Model training method, image classification method, device and electronic equipment

By inversely processing RGB data to RAW data and combining knowledge distillation of teacher network and student network, a target classification model that can directly classify images in the RAW domain is generated, which solves the problem of poor image classification effect in the existing technology and improves the classification Accuracy and efficiency.

CN116051931BActive Publication Date: 2024-12-03AXERA TECH (BEIJING) CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202310117555.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-02-09
Publication Date
2024-12-03
Estimated Expiration
2043-02-09

AI Technical Summary

Technical Problem

The existing image classification method is carried out in the RGB domain, resulting in dynamic range compression and information loss during the conversion process from RAW data to RGB data, especially in low-light or side backlight scenarios, affecting model performance, and the conversion process takes a long time, resulting in classification. Poor results.

Method used

By inversely processing RGB data as RAW data, and combining RGB classification model to train the RAW classification model, a target classification model can be generated that can directly classify images in the RAW domain, reducing the impact of adverse transformation, and improving knowledge distillation of teachers' networks and students' networks Model training effect.

Benefits of technology

It improves the accuracy and efficiency of image classification, is suitable for original image classification in various scenarios, reduces adverse effects during the conversion process, and optimizes the classification effect.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN116051931B_ABST
    Figure CN116051931B_ABST
Patent Text Reader

Abstract

The present application provides a model training method, an image classification method, a device and an electronic device, and relates to the field of image processing technology. The model training method includes: inverse processing the first type of data to obtain the second type of data; training the second classification model based on the first classification model and the second type of data to obtain a target classification model; wherein the first type of data includes RGB data, and the second type of data includes RAW data; the first classification model includes an RGB classification model, and the second classification model includes a RAW classification model. The image classification method includes: obtaining an image to be classified acquired by a target camera; classifying the image to be classified based on the target classification model to obtain a classification result; wherein the image to be classified includes RAW data, and the target classification model is generated according to the model training method. By training the RAW classification model, there is no need to convert the original data, and the image can be directly classified in the RAW domain, thereby optimizing the classification effect of the image.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a model training method, an image classification method, a device and an electronic device. Background Art

[0002] Existing image classification methods are generally performed in the RGB (red, green, and blue color mode) domain. First, a large amount of annotated RGB image data is used to train an RGB image classification model. When applied, the RAW data (original image file) collected by the image sensor is converted into an RGB image through a series of isp-ppl (Image Signal Processing, Perceptual Path Length), and then sent to the trained RGB classification model to obtain the classification result.

[0003] However, since existing image classification methods are generally performed in the RGB domain, and the RAW data collected by the image sensor needs to go through a series of isp-ppl conversion processes to convert it into an RGB image, there is dynamic range compression and information loss in this process. In some scenarios, such as low-light scenes and side-backlit scenes, the converted RGB image quality will be poor, thus affecting the classification performance of the model. In addition, since the entire ppl process has many steps, the classification process takes a long time, resulting in poor image classification results. Summary of the invention

[0004] In view of this, the purpose of the embodiments of the present application is to provide a model training method, an image classification method, a device and an electronic device to improve the problem of poor image classification effect existing in the prior art.

[0005] In order to solve the above problems, in a first aspect, an embodiment of the present application provides a model training method, the method comprising:

[0006] Perform inverse processing on the first type of data to obtain the second type of data;

[0007] Training the second classification model based on the first classification model and the second category of data to obtain a target classification model;

[0008] The first type of data includes RGB data, and the second type of data includes RAW data; the first classification model includes an RGB classification model, and the second classification model includes a RAW classification model.

[0009] In the above implementation process, by inversely processing the first type of data in the RGB domain into the second type of data in the RAW domain, the second type of data can be used as training data for model information training, and the first classification model, that is, the RAW classification model, can be trained in combination with the RGB classification model, thereby obtaining a target classification model that can directly classify RAW data in the RAW domain. The RAW classification model can be trained in combination with the classification method of the RGB classification model, thereby eliminating the need to convert the original RAW image into an RGB image before classification, reducing adverse effects during the conversion process, and optimizing the classification effect of the obtained target classification model.

[0010] Optionally, the training the second classification model based on the first classification model and the second category of data to obtain a target classification model includes:

[0011] Classify the first category of data according to the first classification model to obtain a teacher network;

[0012] Training the second classification model based on the second category of data to obtain a student network;

[0013] The target classification model is obtained by performing knowledge distillation on the student network based on the teacher network.

[0014] In the above implementation process, when training the second classification model, the first classification model can be used as the teacher network and the second classification model can be used as the student network, so that knowledge distillation is performed according to the classification of the corresponding data by the teacher network, the student network, and the two networks, so as to train and obtain the corresponding target classification model. Model training and knowledge distillation can be performed in the form of teacher network and student network, which effectively improves the effect of training the second classification model, thereby improving the effectiveness and accuracy of the obtained target classification model in classifying images.

[0015] Optionally, performing inverse processing on the first type of data to obtain the second type of data includes:

[0016] Performing inverse gamma correction processing on the first type of data to obtain gamma data;

[0017] Performing color correction inverse processing on the gamma data to obtain corrected data;

[0018] Performing dimensionality reduction processing on the correction data to obtain dimensionality reduction data;

[0019] Performing white balance inverse processing on the dimension reduction data to obtain balanced data;

[0020] Noise compensation is performed on the balanced data to obtain the second type of data.

[0021] In the above implementation process, when converting the original RAW data into RGB data, it is necessary to go through a complete ISP processing process. Therefore, when converting the RGB data into RAW data, it is also necessary to go through a complete inverse ISP processing process. By performing various processes such as gamma value inverse conversion, color correction inverse processing, dimensionality reduction processing, white balance inverse processing, noise compensation, etc. on the first type of data, the matrices of multiple color channels are converted accordingly, so that the RGB data is converted into RAW data and synthesized to obtain the estimated second type of data. The complete inverse processing enables the RGB data to be accurately and efficiently converted into RAW data, thereby improving the similarity between the second type of data and the real RAW data corresponding to the first type of data.

[0022] Optionally, performing noise compensation on the balanced data to obtain the second type of data includes:

[0023] Obtaining camera parameters corresponding to the first type of data;

[0024] determining simulated noise data based on the camera parameters;

[0025] Noise compensation is performed on the balanced data based on the simulated noise data to obtain the second type of data.

[0026] In the above implementation process, the corresponding simulated noise data can be determined according to the camera parameters of the shooting camera corresponding to the first type of data, so as to perform corresponding noise compensation on the balanced data according to the simulated noise data, so that the second type of data obtained after the inverse processing is closer to the real RAW data corresponding to the first type of data. The image data can be compensated for noise accordingly according to the actual situation of each camera, which effectively improves the accuracy and effectiveness of the second type of data.

[0027] Optionally, the second classification model is generated by:

[0028] Determining a model head of the first classification model;

[0029] Determine the target encoding based on classification requirements;

[0030] The target code is added to the model header to obtain the second classification model.

[0031] In the above implementation process, in order to obtain the second type of model that can classify RAW data, the existing first type of model can be modified, and the corresponding target code can be added to the model header to obtain the second classification model. There is no need to completely redevelop the model, which can effectively save time and cost in model development and improve the construction efficiency of the second classification model.

[0032] In a second aspect, an embodiment of the present application further provides an image classification method, the method comprising:

[0033] Obtain the image to be classified acquired by the target camera;

[0034] Classify the image to be classified based on the target classification model to obtain a classification result;

[0035] Wherein, the image to be classified includes RAW data, and the target classification model is generated according to any of the model training methods described above.

[0036] In the above implementation process, in the actual application scenario of image classification, the original RAW domain image data to be classified acquired by the target camera can be directly obtained, so as to directly classify the image to be classified based on the target classification model obtained by the above training, and obtain the corresponding image classification result. The image data in the RAW domain can be directly classified without converting the RAW data into RGB data before classification, thereby reducing the adverse effects of the conversion process, and is suitable for classifying original images taken in a variety of different scenarios, which improves the efficiency of image classification and the accuracy and effectiveness of the classification results.

[0037] Optionally, classifying the image to be classified based on the target classification model to obtain a classification result includes:

[0038] Preprocessing the image to be classified to obtain preprocessed data;

[0039] Processing the preprocessed data based on the target classification model to obtain the classification result;

[0040] Wherein, the preprocessing includes black level correction.

[0041] In the above implementation process, before classification, in order to further improve the accuracy of classification, the image to be classified can be preprocessed such as black level correction, and the preprocessed data can be used as the data in the output target classification model for processing, thereby reducing the adverse effects of black level during classification and improving the accuracy of the classification results.

[0042] In a third aspect, an embodiment of the present application further provides a model training device, the device comprising: an inverse processing module and a training module;

[0043] The inverse processing module is used to perform inverse processing on the first type of data to obtain the second type of data;

[0044] The training module is used to train the second classification model based on the first classification model and the second type of data to obtain a target classification model;

[0045] The first type of data includes RGB data, and the second type of data includes RAW data; the first classification model includes an RGB classification model, and the second classification model includes a RAW classification model.

[0046] In the above implementation process, the first type of data in the RGB domain is inversely processed into the second type of data in the RAW domain through the inverse processing module, and the second type of data is used as training data for training model information through the training module. The first classification model, that is, the RAW classification model is trained in combination with the RGB classification model, so as to obtain a target classification model that can directly classify RAW data in the RAW domain.

[0047] In a fourth aspect, an embodiment of the present application further provides an image classification device, the device comprising: an acquisition module and a classification module;

[0048] The acquisition module is used to acquire the image to be classified acquired by the target camera;

[0049] The classification module is used to classify the image to be classified based on the target classification model to obtain a classification result;

[0050] Wherein, the image to be classified includes RAW data, and the target classification model is generated according to any of the model training methods described above.

[0051] In the above implementation process, the acquisition module directly obtains the original RAW domain image data to be classified acquired by the target camera, and the classification module directly classifies the image to be classified based on the target classification model obtained by the above training to obtain the corresponding image classification result.

[0052] In a fifth aspect, an embodiment of the present application further provides an electronic device, comprising a memory and a processor, wherein the memory stores program instructions, and when the processor reads and runs the program instructions, it executes the steps in any implementation of the above-mentioned model training method or image classification method.

[0053] In a sixth aspect, an embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer program instructions. When the computer program instructions are read and executed by a processor, the steps in any implementation of the above-mentioned model training method or image classification method are executed.

[0054] In summary, the embodiments of the present application provide a model training method, an image classification method, an apparatus and an electronic device. By training the RAW classification model, there is no need to convert the original RAW data into RGB data before classification. The image can be classified directly in the RAW domain, reducing the adverse effects of the conversion process. It is suitable for classifying original images taken in a variety of different scenarios, thereby improving the efficiency of image classification and the accuracy and effectiveness of the classification results. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0056] Figure 1 A block diagram of an electronic device provided in an embodiment of the present application;

[0057] Figure 2 A flowchart of a model training method provided in an embodiment of the present application;

[0058] Figure 3 A detailed flowchart of step S300 provided in an embodiment of the present application;

[0059] Figure 4 A detailed flow chart of step S200 provided in an embodiment of the present application;

[0060] Figure 5 A detailed flow chart of step S250 provided in an embodiment of the present application;

[0061] Figure 6 A flowchart of another model training method provided in an embodiment of the present application;

[0062] Figure 7 A flowchart of an image classification method provided in an embodiment of the present application;

[0063] Figure 8 A detailed flowchart of step S600 provided in an embodiment of the present application;

[0064] Fig. 9 A schematic diagram of the structure of a model training device provided in an embodiment of the present application;

[0065] Fig.10 A schematic diagram of the structure of an image classification device provided in an embodiment of the present application.

[0066] Icons: 100-electronic device; 111-memory; 112-storage controller; 113-processor; 114-peripheral interface; 115-input and output unit; 116-display unit; 700-model training device; 710-inverse processing module; 720-training module; 800-image classification device; 810-acquisition module; 820-classification module. DETAILED DESCRIPTION

[0067] The technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments of the embodiments of the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the embodiments of the present application.

[0068] Existing image classification methods are generally performed in the RGB domain. The RAW data (original data) collected by the image sensor needs to be converted into RGB images through a series of isp-ppl, and then sent to the pre-trained RGB classification model to obtain the classification results. However, in the isp-ppl process of converting RAW data to RGB data, there is dynamic range compression and information loss. In some scenes, such as low-light scenes and side-backlit scenes, the converted RGB image quality will be poor, thus affecting the classification performance of the model. In addition, since the entire ppl has many steps, the classification takes a long time, resulting in poor image classification results.

[0069] In order to solve the above problems, the embodiments of the present application provide a model training method and an image classification method, which are applied to electronic devices. The electronic devices can be servers, personal computers (PCs), tablet computers, smart phones, personal digital assistants (PDAs), and other electronic devices with logical computing functions. The RAW classification model that can classify the original RAW data can be trained to directly classify images in the RAW domain, thereby optimizing the image classification effect.

[0070] Optionally, see Figure 1 , Figure 1 The block diagram of an electronic device provided in an embodiment of the present application. The electronic device 100 may include a memory 111, a storage controller 112, a processor 113, a peripheral interface 114, an input and output unit 115, and a display unit 116. A person skilled in the art may understand that Figure 1 The structure shown is only for illustration and does not limit the structure of the electronic device 100. For example, the electronic device 100 may further include Figure 1More or fewer components as shown, or with Figure 1 Different configurations shown.

[0071] The memory 111, storage controller 112, processor 113, peripheral interface 114, input / output unit 115 and display unit 116 are electrically connected to each other directly or indirectly to achieve data transmission or interaction. For example, these components can be electrically connected to each other via one or more communication buses or signal lines. The processor 113 is used to execute the executable module stored in the memory.

[0072] The memory 111 may be, but is not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable read-only memory (EEPROM), etc. The memory 111 is used to store programs, and the processor 113 executes the program after receiving the execution instruction. The method executed by the electronic device 100 defined by the process disclosed in any embodiment of the present application can be applied to the processor 113, or implemented by the processor 113.

[0073] The processor 113 may be an integrated circuit chip with signal processing capability. The processor 113 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application may be implemented or executed. The general-purpose processor may be a microprocessor or any conventional processor.

[0074] The peripheral interface 114 couples various input / output devices to the processor 113 and the memory 111. In some embodiments, the peripheral interface 114, the processor 113 and the memory controller 112 can be implemented in a single chip. In other embodiments, they can be implemented by separate chips.

[0075] The input / output unit 115 is used to provide input data to the user. The input / output unit 115 may be, but is not limited to, a mouse and a keyboard.

[0076] The above-mentioned display unit 116 provides an interactive interface (such as a user operation interface) between the electronic device 100 and the user or is used to display image data for user reference. In this embodiment, the display unit can be a liquid crystal display or a touch display. If it is a touch display, it can be a capacitive touch screen or a resistive touch screen that supports single-point and multi-point touch operations. Supporting single-point and multi-point touch operations means that the touch display can sense touch operations generated simultaneously from one or more positions on the touch display, and the sensed touch operations are handed over to the processor for calculation and processing. In an embodiment of the present application, the display unit 116 can display the first category of data in the RGB domain, the second category of data in the RAW domain, the classified images to be classified, the obtained classification results and other information.

[0077] The electronic device in this embodiment can be used to execute each step in each model training method and image classification method provided in the embodiment of the present application. The implementation process of the model training method and the image classification method is described in detail below through several embodiments.

[0078] See also Figure 2 , Figure 2 A flow chart of a model training method provided in an embodiment of the present application, the method may include steps S200-S300.

[0079] Step S200, performing inverse processing on the first type of data to obtain the second type of data.

[0080] The first type of data includes RGB data in the RGB domain, such as RGB images, and the second type of data includes RAW data in the RAW domain, such as RAW images. Since RAW data is the original data obtained after the camera shoots, and RGB data is the data obtained after a series of ISP conversions of RAW data. It should be noted that since the actual RAW data corresponding to the existing first type of data may be unknown, the existing first type of data can be obtained to perform inverse ISP processing on the first type of data, thereby obtaining the second type of data that is close to the actual RAW data corresponding to the first type of data.

[0081] Step S300: training the second classification model based on the first classification model and the second type of data to obtain a target classification model.

[0082] The first classification model includes an RGB classification model, and the second classification model includes a RAW classification model. The second classification model can be trained to classify the second type of data according to the way the first classification model classifies the first type of data, so as to obtain a corresponding target classification model capable of classifying RAW domain data.

[0083] Optionally, the second classification model may be a model modified based on the first classification model, or may be a separately constructed image classification model.

[0084] exist Figure 2 In the illustrated embodiment, the RAW classification model can be trained in combination with the classification method of the RGB classification model, thereby eliminating the need to convert the original RAW image into an RGB image before classification, thereby reducing adverse effects during the conversion process and optimizing the classification effect of the obtained target classification model.

[0085] Optionally, see Figure 3 , Figure 3 A detailed flowchart of step S300 is provided for an embodiment of the present application. Step S300 may include steps S310-S330.

[0086] Step S310, classify the first category of data according to the first classification model to obtain a teacher network.

[0087] Among them, the first category of data in the RGB domain is classified by the first classification model to serve as the corresponding teacher network, namely, teacher.

[0088] Step S320: training the second classification model based on the second category of data to obtain a student network.

[0089] The second type of data is input into the second classification model to train the classification algorithm of the second classification model to serve as the corresponding student network, namely, student.

[0090] Step S330, performing knowledge distillation on the student network based on the teacher network to obtain a target classification model.

[0091] Among them, knowledge distillation can be performed based on the classification of the teacher network, the student network, and the classification of the corresponding data by the two networks. For example, the training results or labels obtained at each stage when the two models classify the corresponding image data can be compared to adjust the classification algorithm and parameters of the student network accordingly, so that the final classification results of the teacher network and the student network are similar. A corresponding threshold can be set, and when the similarity between the classification results of the student network and the teacher network reaches a higher similarity threshold, the corresponding target classification model is obtained.

[0092] exist Figure 3 In the illustrated embodiment, model training and knowledge distillation can be performed in the form of a teacher network and a student network, which effectively improves the effect of training the second type of classification model, thereby improving the effectiveness and accuracy of the obtained target classification model in classifying images.

[0093] Optionally, see Figure 4 , Figure 4 A detailed flow chart of step S200 is provided for an embodiment of the present application. Step S200 may include steps S210-S250.

[0094] It should be noted that when converting RAW domain data to RGB domain data, it needs to go through a series of ISP processing, such as gamma correction, CCM (Color Correction), dimensionality upscaling, WBC (White Balance) processing, noise reduction, etc. Therefore, when performing inverse processing, it is necessary to perform various inverse operations on the image to convert RGB domain data into RAW domain data.

[0095] Step S210, performing inverse gamma correction processing on the first type of data to obtain gamma data.

[0096] Among them, the first type of data may be subjected to inverse gamma correction first, and the gamma curve or parameters of the image may be adjusted to obtain matrices corresponding to multiple color channels of RGB as corresponding gamma data.

[0097] Step S220, performing color correction inverse processing on the gamma data to obtain corrected data.

[0098] The gamma data may be further subjected to inverse color correction processing to adjust and calculate the colors of matrices of multiple color channels, and obtain matrices corresponding to multiple adjusted color channels as corresponding correction data.

[0099] Step S230, performing dimensionality reduction processing on the corrected data to obtain dimensionality reduced data.

[0100] Wherein, corresponding mosaic processing may be performed on multiple color channel matrices in the correction data to reduce the dimension of the correction data and obtain corresponding reduced-dimensional data.

[0101] Step S240, performing inverse white balance processing on the dimension reduction data to obtain balanced data.

[0102] Among them, the dimensionality reduction data can continue to be subjected to reverse white balance processing to obtain restored balanced data.

[0103] Step S250: noise compensation is performed on the balanced data to obtain second type data.

[0104] Among them, considering that when the camera shoots the original RAW data, the lens has a certain noise effect. In order to make the acquired second type of data closer to the real RAW data corresponding to the first type of data, the balance data can be compensated for noise to improve the accuracy of the second type of data.

[0105] exist Figure 4 In the illustrated embodiment, the RGB data can be accurately and efficiently converted into RAW data through complete inverse processing, thereby improving the similarity between the second type of data and the real RAW data corresponding to the first type of data.

[0106] Optionally, see Figure 5 , Figure 5 A detailed flowchart of step S250 is provided for an embodiment of the present application. Step S250 may include steps S251-S253.

[0107] Step S251, obtaining camera parameters corresponding to the first type of data.

[0108] Among them, since the noise of RAW data is related to the camera used during shooting, the camera parameters of the shooting camera corresponding to the first type of data can be found. The camera parameters may include various related data such as the type, model, and lens parameters of the camera.

[0109] Step S252: Determine simulated noise data based on camera parameters.

[0110] The calculation may be performed according to the corresponding camera parameters, or a query may be performed in a plurality of historical noise data in a database according to similar parameters, so as to calculate or query the simulated noise data of the shooting camera lens.

[0111] Step S253: noise compensation is performed on the balanced data based on the simulated noise data to obtain second type data.

[0112] The balanced data is subjected to corresponding noise compensation according to the simulated noise data, so that the second type of data obtained after the inverse processing is closer to the real RAW data corresponding to the first type of data.

[0113] exist Figure 5 In the illustrated embodiment, the image data can be correspondingly noise compensated according to the actual situation of each camera, thereby effectively improving the accuracy and effectiveness of the second type of data.

[0114] Optionally, see Figure 6 , Figure 6 A flowchart of another model training method provided in an embodiment of the present application, wherein the second classification model can be generated by steps S410-S430.

[0115] Step S410, determining the model head of the first classification model.

[0116] In order to obtain the second type of model capable of classifying RAW data, the existing first type of model may be modified. In order to reduce the cost and difficulty of modification, the model head of the first classification model may be directly obtained as the corresponding modification object.

[0117] Step S420, determining the target code according to the classification requirement.

[0118] Among them, the corresponding target encoding, that is, the encoder, can be determined according to the actual classification requirements, and the size and number of layers of the encoder can be set and modified accordingly according to the classification requirements.

[0119] Step S430, adding the target code to the model header to obtain a second classification model.

[0120] Among them, adding the corresponding target code in the convolutional layer of the model head can simply and efficiently obtain the second classification model that can classify the RAW domain data.

[0121] exist Figure 6 In the illustrated embodiment, there is no need to completely redevelop the model, which can effectively save time and cost in model development and improve the efficiency of building the second classification model.

[0122] See also Figure 7 , Figure 7 A flowchart of an image classification method provided in an embodiment of the present application, the method may include steps S500-S600.

[0123] Step S500: obtaining an image to be classified acquired by a target camera.

[0124] In the actual application scenario of classifying images, the original RAW domain image to be classified acquired by the target camera can be directly obtained through the communication connection with the target camera for processing.

[0125] Step S600: classify the image to be classified based on the target classification model to obtain a classification result.

[0126] The target classification model is generated according to any of the above model training methods. The RAW domain image to be classified can be directly classified according to the target classification model to obtain the corresponding image classification result. Compared with RGB domain data, RAW domain data has a wider dynamic range and can contain richer detail information. Classification can be performed directly on the RAW domain data. It can still be accurately classified in low-light, side backlight and other scenes to optimize the effect of image classification. It can also reduce the delay of ppl during image classification, improve the overall running speed, and improve the efficiency of image classification.

[0127] exist Figure 7 In the illustrated embodiment, image data in the RAW domain can be directly classified without converting the RAW data into RGB data for classification, thereby reducing adverse effects during the conversion process. This is suitable for classifying original images taken in a variety of different scenarios, thereby improving the efficiency of image classification and the accuracy and effectiveness of the classification results.

[0128] Optionally, see Figure 8 , Figure 8 A detailed flow chart of step S600 is provided for an embodiment of the present application. Step S600 may also include steps S610-S620.

[0129] Step S610, preprocessing the image to be classified to obtain preprocessed data.

[0130] Considering that the image to be classified may be affected by the black level, the classified image may be preprocessed first. The preprocessing may include performing black level correction on the image to be classified to obtain corresponding preprocessing data.

[0131] Step S620, processing the preprocessed data based on the target classification model to obtain a classification result.

[0132] The preprocessed data is used as the data in the output target classification model to be processed to obtain a corresponding classification result with less interference. The classification result may include various data such as the type and parameters of the image.

[0133] exist Figure 8 In the illustrated embodiment, the adverse effects of black level and the like during classification can be reduced, thereby improving the accuracy of the classification result.

[0134] See also Fig. 9 , Fig. 9 A schematic diagram of the structure of a model training device provided in an embodiment of the present application, wherein the model training device 700 may include: an inverse processing module 710 and a training module 720;

[0135] The inverse processing module 710 is used to perform inverse processing on the first type of data to obtain the second type of data;

[0136] The training module 720 is used to train the second classification model based on the first classification model and the second type of data to obtain a target classification model;

[0137] The first type of data includes RGB data, and the second type of data includes RAW data; the first classification model includes an RGB classification model, and the second classification model includes a RAW classification model.

[0138] In an optional embodiment, the training module 720 may further include a teacher submodule, a student submodule, and a distillation submodule;

[0139] The teacher submodule is used to classify the first type of data according to the first classification model to obtain a teacher network;

[0140] The student submodule is used to train the second classification model based on the second category data to obtain a student network;

[0141] The distillation submodule is used to perform knowledge distillation on the student network based on the teacher network to obtain the target classification model.

[0142] In an optional implementation, the inverse processing module 710 may further include an inverse conversion submodule, a color submodule, a dimension reduction submodule, a white balance submodule, and a compensation submodule;

[0143] The inverse conversion submodule is used to perform inverse gamma correction processing on the first type of data to obtain gamma data;

[0144] The color submodule is used to perform color correction inverse processing on the gamma data to obtain corrected data;

[0145] The dimension reduction submodule is used to perform dimension reduction processing on the correction data to obtain dimension reduction data;

[0146] The white balance submodule is used to perform white balance inverse processing on the dimension reduction data to obtain balanced data;

[0147] The compensation module is used to perform noise compensation on the balanced data to obtain the second type of data.

[0148] In an optional implementation, the compensation submodule is specifically used to: obtain camera parameters corresponding to the first type of data; determine simulated noise data based on the camera parameters; and perform noise compensation on the balanced data based on the simulated noise data to obtain the second type of data.

[0149] In an optional implementation, the model training device 700 may further include a modification module for determining a model head of the first classification model; determining a target code according to classification requirements; and adding the target code to the model head to obtain a second classification model.

[0150] Since the principle of solving the problem by the model training device 700 in the embodiment of the present application is similar to that of the embodiment of the aforementioned model training method, the implementation of the model training device 700 in this embodiment can refer to the description in the embodiment of the aforementioned model training method, and the repeated parts will not be repeated.

[0151] See also Fig.10 , Fig.10 This is a schematic diagram of the structure of an image classification device provided in an embodiment of the present application. The image classification device 800 may include: an acquisition module 810 and a classification module 820;

[0152] The acquisition module 810 is used to acquire the image to be classified acquired by the target camera;

[0153] The classification module 820 is used to classify the image to be classified based on the target classification model to obtain a classification result;

[0154] The image to be classified includes RAW data, and the target classification model is generated according to any of the above model training methods.

[0155] In an optional implementation, the classification module 820 may include a preprocessing submodule and a classification submodule;

[0156] The preprocessing submodule is used to preprocess the image to be classified to obtain preprocessed data;

[0157] The classification submodule is used to process the preprocessed data based on the target classification model to obtain a classification result; wherein the preprocessing includes black level correction.

[0158] Since the principle of solving the problem by the image classification device 800 in the embodiment of the present application is similar to that in the embodiment of the aforementioned image classification method, the implementation of the image classification device 800 in this embodiment can refer to the description in the embodiment of the aforementioned image classification method, and the repeated parts will not be repeated.

[0159] An embodiment of the present application also provides a computer-readable storage medium, in which computer program instructions are stored. When the computer program instructions are read and executed by a processor, the steps of any one of the model training method or image classification method provided in the present embodiment are executed.

[0160] In several embodiments provided in the present application, it should be understood that the disclosed device can also be implemented in other ways. The device embodiments described above are merely schematic, for example, the block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices according to the multiple embodiments of the present application. In this regard, each box in the block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram, and the combination of the block diagrams, can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0161] In addition, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0162] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0163] The above description is only an embodiment of the present application and is not intended to limit the scope of protection of the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application. It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings.

[0164] The above description is only a specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application.

[0165] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "include..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

Claims

1. A model training method, It is characterized in that The method comprises: Perform inverse processing on the first type of data to obtain the second type of data; Training the second classification model based on the first classification model and the second category of data to obtain a target classification model; The first type of data includes RGB data, and the second type of data includes RAW data; the first classification model includes an RGB classification model, and the second classification model includes a RAW classification model.

2. The method according to claim 1, It is characterized in that The step of training the second classification model based on the first classification model and the second type of data to obtain a target classification model includes: Classify the first category of data according to the first classification model to obtain a teacher network; Training the second classification model based on the second category of data to obtain a student network; The target classification model is obtained by performing knowledge distillation on the student network based on the teacher network.

3. The method according to claim 1, It is characterized in that The inverse processing of the first type of data to obtain the second type of data includes: Performing inverse gamma correction processing on the first type of data to obtain gamma data; Performing color correction inverse processing on the gamma data to obtain corrected data; Performing dimensionality reduction processing on the correction data to obtain dimensionality reduction data; Performing white balance inverse processing on the dimension reduction data to obtain balanced data; Noise compensation is performed on the balanced data to obtain the second type of data.

4. The method according to claim 3, It is characterized in that The performing noise compensation on the balanced data to obtain the second type of data includes: Obtaining camera parameters corresponding to the first type of data; determining simulated noise data based on the camera parameters; Noise compensation is performed on the balanced data based on the simulated noise data to obtain the second type of data.

5. The method according to any one of claims 1 to 4, It is characterized in that in, The second classification model is generated by: Determining a model head of the first classification model; Determine the target encoding based on classification requirements; The target code is added to the model header to obtain the second classification model.

6. A method for image classification, It is characterized in that The method comprises: Obtain the image to be classified acquired by the target camera; Classify the image to be classified based on the target classification model to obtain a classification result; Wherein, the image to be classified includes RAW data, and the target classification model is generated according to the model training method described in any one of claims 1-5.

7. The method according to claim 6, It is characterized in that The classifying the image to be classified based on the target classification model to obtain a classification result includes: Preprocessing the image to be classified to obtain preprocessed data; Processing the preprocessed data based on the target classification model to obtain the classification result; Wherein, the preprocessing includes black level correction.

8. A model training device, It is characterized in that The device comprises: an inverse processing module and a training module; The inverse processing module is used to perform inverse processing on the first type of data to obtain the second type of data; The training module is used to train the second classification model based on the first classification model and the second type of data to obtain a target classification model; The first type of data includes RGB data, and the second type of data includes RAW data; the first classification model includes an RGB classification model, and the second classification model includes a RAW classification model.

9. An image classification device, It is characterized in that The device comprises: an acquisition module and a classification module; The acquisition module is used to acquire the image to be classified acquired by the target camera; The classification module is used to classify the image to be classified based on the target classification model to obtain a classification result; Wherein, the image to be classified includes RAW data, and the target classification model is generated according to the model training method described in any one of claims 1-5.

10. An electronic device, It is characterized in that The electronic device comprises a memory and a processor, wherein the memory stores program instructions, and when the processor runs the program instructions, the steps in the method according to any one of claims 1 to 7 are executed.

11. A computer-readable storage medium, It is characterized in that The readable storage medium stores computer program instructions, and when the computer program instructions are executed by a processor, the steps in the method according to any one of claims 1 to 7 are executed.