Neural network training method, device and electronic equipment
By dividing the degree of degradation according to the environmental parameters of the image and performing targeted processing, a variety of sample sets are generated to train neural networks, which solves the problem of poor processing of image levels for different degradation in the existing technology, and achieves better Image quality improvement.
Patent Information
- Application Number
- CN202110838140.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-07-23
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2041-07-23
AI Technical Summary
In the prior art, deep learning networks have poor results in image processing of different degrees of degradation, resulting in insufficient improvement in image quality during low-illumination scenes and group portrait shooting.
By determining the environmental parameters of the image (such as ISO values) to divide the different degrees of degradation, perform targeted degradation processing, generate sample sets of multiple degrees of degradation, and train neural networks of multiple complexity to adapt to different shooting scenes.
It improves the pertinence and effect of image enhancement processing, is suitable for shooting images under different environmental parameters, and improves the performance of face quality enhancement.
Smart Images

Figure CN113706402B_ABST
Abstract
Description
Technical Field
[0001] The present application belongs to the field of image processing and deep learning, and specifically relates to a neural network training method, device and electronic equipment. Background Art
[0002] With the continuous development of artificial intelligence technology, electronic devices are becoming more and more popular, and users have higher and higher requirements for the shooting quality of electronic devices (such as smart terminals). At present, the solution to improve the shooting quality of electronic devices is mainly to learn the deep network of face quality enhancement through convolutional neural network.
[0003] In the related art, a single input resolution (such as 256, 512) and a single image degradation deep network are trained through a training set to perform image enhancement processing on the captured image. With the above image processing method, in some special scenes, such as low-light scenes, the amount of light entering during shooting is small and the noise has a greater impact. Compared with scenes with normal brightness, the quality of images captured in low-light scenes degrades more seriously (i.e., the image quality is lower). For another example, when shooting group portraits, the degradation degree of each face of different sizes in the lens is also different.
[0004] Therefore, if the deep learning network trained with a single input resolution and a single data degradation in the related technology is still used to process the captured images, the trained deep learning network will not be able to perform targeted enhancement processing on images with different degrees of degradation, making it difficult to achieve an obvious face quality enhancement effect, which in turn leads to poor image processing effects. Summary of the invention
[0005] The purpose of the embodiments of the present application is to provide a neural network training method, device and electronic device, which can solve the technical problem of poor image processing effect of deep learning networks in related technologies.
[0006] In order to solve the above technical problems, this application is implemented as follows:
[0007] In a first aspect, an embodiment of the present application provides a neural network training method, the method comprising: determining N degrees of degradation based on M environmental parameters of M acquired first images, one degree of degradation corresponding to at least one environmental parameter, one degree of degradation corresponding to at least one first image, and M and N being positive integers; based on each degree of degradation, performing degradation processing on the first image corresponding to each degree of degradation to obtain a second image corresponding to each degree of degradation, and each second image corresponding to a first image; based on the first image and the second image corresponding to each degree of degradation, generating a sample set to obtain N sample sets; based on the N sample sets, training Q neural networks respectively, one sample set corresponding to at least one neural network, and Q being a positive integer.
[0008] In a second aspect, an embodiment of the present application provides a neural network training device, which includes: the device includes: a determination module, a processing module, a generation module and a training module, wherein: the determination module is used to determine N degradation degrees according to M environmental parameters of M first images obtained, one degradation degree corresponds to at least one environmental parameter, one degradation degree corresponds to at least one first image, and M and N are both positive integers; the above-mentioned processing module is used to perform degradation processing on the first image corresponding to each degradation degree based on each degradation degree determined by the determination module, so as to obtain a second image corresponding to each degradation degree, and each second image corresponds to a first image; the above-mentioned generation module is used to generate a sample set based on the first image and the second image corresponding to each degradation degree obtained by the processing module, so as to obtain N sample sets; the above-mentioned training module is used to train Q neural networks based on the N sample sets obtained by the generation module, one sample set corresponds to at least one neural network, and Q is a positive integer.
[0009] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the method described in the first aspect.
[0010] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.
[0011] In a fifth aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the method described in the first aspect.
[0012] In a sixth aspect, an embodiment of the present application provides a computer program product, which is stored in a non-volatile storage medium and is executed by at least one processor to implement the method described in the first aspect.
[0013] In the embodiment of the present application, the neural network training device determines N degradation degrees according to the M environmental parameters of the acquired M first images, one degradation degree corresponds to at least one environmental parameter, one degradation degree corresponds to at least one first image, and M and N are both positive integers; and based on each degradation degree, the first image corresponding to each degradation degree is degraded respectively to obtain a second image corresponding to each degradation degree, and each second image corresponds to a first image respectively, and then, based on the first image and the second image corresponding to each degradation degree, a sample set is generated respectively to obtain N sample sets, and finally, based on the N sample sets, Q neural networks are trained respectively, one sample set corresponds to at least one neural network, and Q is a positive integer. Through this method, the neural network training device can degrade the first image under different environmental parameters with corresponding degradation degrees (i.e., with differences) to obtain the degraded image (i.e., the second image) corresponding to the first image after differential degradation, thereby constructing a training sample set (i.e., different degradation degrees) that meets the shooting quality under multiple environmental parameters, and then a neural network suitable for targeted processing of the captured images under different environmental parameters can be obtained, thereby improving the image enhancement processing effect. BRIEF DESCRIPTION OF THE DRAWINGS
[0014] Figure 1 is a flow chart of a neural network training method provided in an embodiment of the present application;
[0015] Figure 2 It is a schematic diagram of a process flow of a multi-strength degradation module provided in an embodiment of the present application;
[0016] Figure 3 This is a schematic diagram of constructing a low-definition to high-definition training set provided in an embodiment of the present application;
[0017] Figure 4 This is a schematic diagram of ISO and face size information processing provided by an embodiment of the present application;
[0018] Figure 5 This is a schematic diagram of a multi-complexity GAN network process provided by an embodiment of the present application;
[0019] Figure 6 is a flow chart of an image processing method provided by an embodiment of the present application;
[0020] Figure 7 is a structural schematic diagram of a neural network training device provided in an embodiment of the present application;
[0021] Figure 8 is a structural schematic diagram of an image processing device provided in an embodiment of the present application;
[0022] Fig. 9This is one of the hardware structure diagrams of an electronic device provided in an embodiment of the present application;
[0023] Fig.10 This is the second schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0024] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0025] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.
[0026] The neural network training method provided in the embodiment of the present application is described in detail below through specific embodiments and their application scenarios in conjunction with the accompanying drawings.
[0027] The present application embodiment provides a neural network training method, which can be applied to electronic devices. Figure 1 FIG. 1 is a flowchart of a neural network training method provided in an embodiment of the present application. Figure 1 As shown, the neural network training method provided in the embodiment of the present application may include the following steps 101 to 104:
[0028] Step 101: Determine N degradation degrees according to M environmental parameters of M acquired first images.
[0029] Among them, one degradation degree corresponds to at least one environmental parameter, one degradation degree corresponds to at least one first image, and M and N are both positive integers.
[0030] Optionally, in an embodiment of the present application, the above-mentioned neural network training method can be a face enhancement GAN network (faceEnhanceGAN) training method. Since the face enhancement GAN network (faceEnhanceGAN) training method is a supervised AI training method, paired image pairs (i.e., low-definition images-high-definition images) are required to constitute a training data set. However, due to factors such as the inability to align pixels, it is impossible to directly capture a one-to-one corresponding low-definition-high-definition image pair. Therefore, a low-definition image can be obtained by performing a variety of degradation processing (such as blurring, adding noise, etc.) on the acquired high-definition image (such as a high-definition image taken by a SLR) to obtain a low-definition image to obtain a low-definition-high-definition image pair, i.e., an LQ-HQ data set.
[0031] In the embodiment of the present application, the first image may be a high-definition picture. For example, the first image may be a high-definition picture taken by a camera of a high-quality imaging device (such as a SLR camera), or the first image may be a high-definition picture obtained by enhancing a low-quality image.
[0032] It should be noted that the clarity and quality of an image can be measured by the degree of blur and noise. The above-mentioned high-definition images refer to images with less blur and noise.
[0033] Exemplarily, when the neural network training device obtains the above-mentioned M first images, it can save each first image and the environmental parameters corresponding to the first image in a database, and the neural network training device can call the M first images from the database when needed.
[0034] Optionally, in the embodiment of the present application, the above-mentioned environmental parameters may include at least one of the following: ISO value, brightness.
[0035] It should be noted that sensitivity refers to the sensitivity to light expressed in numbers. The higher the ISO value, the greater the sensitivity to light, and vice versa.
[0036] Optionally, in the embodiment of the present application, M environmental parameters of the M first images can be obtained by inputting the M first images into the ISO and face size information processing module for processing.
[0037] Exemplarily, the neural network training device can determine the degradation degree of the first image according to the magnitude relationship between the environmental parameter of the first image and the first threshold. Further, the first threshold is an ISO threshold, which can be specifically the size of ISO. For example, when the ISO value of image 1 is greater than the first threshold, the image 1 corresponds to the first degradation degree (i.e., a higher degradation degree).
[0038] Furthermore, the multi-stage ISO threshold (ISO_threshold) can be reasonably set by collecting the user's photo data and analyzing the darkness of the scene.
[0039] It should be noted that the above environmental parameter (ie ISO) can be called prior information, and the above ISO and face size information processing module is a preprocessing module for obtaining the prior information.
[0040] Optionally, in an embodiment of the present application, the N degradation degrees may be one or more of the preset L degradation degrees. Exemplarily, the L degradation degrees may include: a first degradation degree and a second degradation degree. Further, the first degradation degree may be a high degradation degree, and the second degradation degree may be a low degradation degree. It should be noted that the above-mentioned degradation degrees may be flexibly determined according to actual conditions. For example, three or more degradation degrees may be set, and the embodiment of the present application does not impose any limitation on this.
[0041] Optionally, in an embodiment of the present application, the above-mentioned N degradation degrees correspond to N degradation modules respectively, each degradation module is used to perform corresponding degradation processing on the first image, and the degradation algorithm of each degradation module is different, that is, each degradation module corresponds to a different degradation effect.
[0042] Exemplarily, the N degradation modules may include: a high ISO segment degradation module and a low ISO segment degradation module. Further, the high ISO segment degradation module may correspond to the first degradation degree, and the low ISO segment degradation module may correspond to the second degradation degree.
[0043] It should be noted that the above-mentioned N degradation modules can also be refined into degradation modules of more ISO segments according to actual needs, and the embodiment of the present application does not impose any limitation on this.
[0044] It should be noted that the high ISO segment degradation module is used to process the first image with a larger ISO value, and its degradation degree is higher, and the low ISO segment degradation module is used to process the first image with a larger ISO value, and its degradation degree is lower.
[0045] Exemplarily, the neural network training device may determine the degradation degree corresponding to the environmental parameter of each first image in the M first images, so as to determine the degradation degree of each first image. For example, taking an ISO value lower than 50 as a low sensitivity corresponding to a first degradation degree (e.g., a low degradation degree) as an example, if the ISO value of image 1 is 45, the degradation degree corresponding to image 1 is the first degradation degree, that is, image 1 corresponds to a low degradation degree.
[0046] Step 102: Based on each degradation degree, degradation processing is performed on the first image corresponding to each degradation degree to obtain a second image corresponding to each degradation degree.
[0047] Each second image corresponds to a first image.
[0048] In an embodiment of the present application, the neural network training device can use a degradation module corresponding to each degradation degree to perform quality degradation processing on the first image corresponding to each degradation degree to obtain a low-quality image or a low-definition image (i.e., the above-mentioned second image) corresponding to the first image.
[0049] Optionally, in an embodiment of the present application, the neural network training device may determine corresponding data degradation modules of different intensities according to the degree of degradation corresponding to the first image through a multi-intensity data degradation module, and perform corresponding quality degradation processing on the first image.
[0050] Exemplarily, the neural network training device can determine the ISO flag information through the ISO and face size information processing module, and then determine the corresponding degradation module based on the ISO flag information through the multi-intensity data degradation module. Further, the ISO flag is used to characterize the high and low ISO, for example, the ISO flag is a high flag (i.e., highISO_flag), or the ISO flag is a low flag (i.e., lowISO_flag).
[0051] Furthermore, when the ISO flag information is determined by the ISO and face size information processing module, the relationship between the ISO value of the first image and the ISO threshold (i.e., ISO_threshold) can be determined. If the ISO value is greater than ISO_threshold, the high ISO flag (or high ISO sign bit) is set to 1; if the ISO value is less than ISO_threshold, the low ISO flag is set to 1.
[0052] Furthermore, when the corresponding degradation module is determined based on the ISO flag information through the above-mentioned multi-intensity data degradation module, the ISO flag information in the ISO and face size information processing module can be identified, and then the corresponding degradation module is determined based on the identified flag information.
[0053] Example 1: After multiple high-definition images are input into the ISO and face size information processing module and the multi-intensity data degradation module, if the high ISO flag highISO_flag is identified as 1 from the ISO and face size information processing module, it means that the high ISO segment is entered. At this time, the high ISO segment degradation module is entered. Through the degradation module, multiple types of noise with higher intensity can be added to the image (i.e., the image to be degraded) during the degradation process, and multiple model operations with higher intensity can be added, such as the blur degree of the blur functions such as the triangular model, Gaussian model, linear model, motion blur, etc. are all increased. In addition, the input high-definition image is randomly darkened to simulate the brightness of the image under low light.
[0054] Example 2, after multiple high-definition images are input into the ISO and face size information processing module and the multi-intensity data degradation module, if the low ISO flag lowISO_flag is identified as 1 from the ISO and face size information processing module, it means that the low ISO segment is entered. At this time, the low ISO segment degradation module is entered. Through the degradation module, less intensity noise can be added to the image, and fewer types of blur functions can be added to perform lighter blur processing.
[0055] Combining the processes of Example 1 and Example 2 above, Figure 2 Figure 1 is a flow chart of processing using a multi-strength degradation module. Figure 2 As shown, if the low ISO flag lowISO_flag corresponding to the high-definition image is equal to 1, the high-definition image is processed in the low ISO degradation module to obtain the corresponding low-definition image 1; if the high ISO flag highISO_flag corresponding to the high-definition image is equal to 1, the high ISO degradation module is processed in the high-definition image to obtain the corresponding low-definition image 2.
[0056] Exemplarily, the above-mentioned degradation processing includes but is not limited to at least one of the following: noise adding processing, blur processing (defocus blur, motion blur, Gaussian blur, linear blur), reducing / increasing brightness (3D lighting), increasing / reducing shadows (random shadows), and area-perceived degradation.
[0057] Furthermore, when the degradation module is used to process the first image, multiple blur processing models can be added to blur the first image using multiple blur functions, such as defocus model, Gaussian model, linear model, motion model and other blur functions.
[0058] Exemplarily, the degradation degree may include at least one of the following: degradation intensity and degradation mode. Further, the degradation intensity refers to the degree of processing, for example, the amount of noise added, the degree of blur; the degradation mode refers to the processing method, for example, adding noise to image 1 and blurring image 2 are two different processing modes.
[0059] It should be noted that in actual shooting scenarios, the degree of image quality degradation obtained by shooting in different ISO scenes is usually different. The degree of image quality degradation in high ISO scenes is higher than that in low ISO scenes, and the quality of the obtained images is worse. Specifically, through multiple experimental tests, it was found that the worse image quality in high ISO scenes is mainly manifested as: 1) the image is more blurred and the focus is more inaccurate, 2) the noise in the image is larger and the noise is uneven. Therefore, the embodiment of the present application designs a multi-intensity degradation module to perform differential image quality degradation on the first images of different ISOs (i.e., high ISO and low ISO) according to the degradation intensity and degradation mode, thereby constructing a data set to better simulate the image quality of electronic devices shot at different ISO segments. Figure 3 A schematic diagram of constructing a low-definition and high-definition training set is shown. Figure 3 As shown, the image training device performs one or more of the above degradation processes on a number of high-definition images to obtain a number of quality-degraded images (ie, low-definition images), thereby obtaining a high-definition-low-definition training set.
[0060] For example, the M first images are 100 high-definition images. Assuming that the ISO values of 40 images among the 100 high-definition images are greater than the ISO threshold, corresponding to the first degradation degree, and the ISO values of 60 images other than the 40 images are less than the ISO threshold, corresponding to the second degradation degree, the 40 images are input to the high ISO segment degradation module corresponding to the first degradation degree for processing, and the 60 images are input to the low ISO segment degradation module corresponding to the second degradation degree for processing, so as to obtain corresponding low-definition images with different degradation degrees.
[0061] Step 103: Generate a sample set based on the first image and the second image corresponding to each degradation degree, respectively, to obtain N sample sets.
[0062] In the embodiment of the present application, one degradation degree corresponds to one sample set, and N degradation degrees correspond to N sample sets, that is, the above N sample sets correspond to N degradation degrees respectively, and the N sample sets constitute a training data set.
[0063] Optionally, in an embodiment of the present application, a sample set includes at least one first image and at least one second image, that is, each sample set includes at least one low-definition-high-definition image pair, that is, an LQ-HQ data set.
[0064] Optionally, in the embodiment of the present application, one degradation degree corresponds to one training data set. Exemplarily, the neural network training device can associate the first image and the second image corresponding to the first image as training sample images and save them in the corresponding training data set.
[0065] It should be noted that in actual shooting scenes, the larger the ISO value, the darker the shooting scene, and the more serious the quality degradation of the pictures taken in the darker scenes, such as blurriness and noise; the smaller the ISO value, the brighter the shooting scene, and the degree of quality degradation of pictures taken in bright scenes is significantly weaker than that in high ISO scenes. Based on this, when training a neural network to enhance images taken by electronic devices, from an algorithmic perspective, if the image signal processing (i.e., ISP) process of the electronic device and the image quality degradation pattern during the shooting process can be simulated more closely, a high-quality training set can be constructed, and a neural network with better face enhancement effect can be trained.
[0066] Step 104: Based on the N sample sets, train the Q neural networks respectively.
[0067] Among them, one sample set corresponds to at least one neural network, and Q is a positive integer.
[0068] In an embodiment of the present application, the above-mentioned Q neural networks may include a generative adversarial network (GAN network).
[0069] Exemplarily, the Q neural networks mentioned above may be neural networks among a plurality of preset neural networks of different complexity.
[0070] Optionally, in an embodiment of the present application, the neural network training device can train Q neural networks respectively based on the above-mentioned N sample sets in a multi-complexity GAN network module. Exemplarily, the above-mentioned multi-complexity GAN network module includes P neural networks, and the neural network training device can determine Q neural networks from the P neural networks for training. Further, the neural network training device determines Q neural networks from the P neural networks for training according to the degree of degradation corresponding to the N sample sets, where P is greater than Q and P is a positive integer.
[0071] Exemplarily, the neural network training device can input the above N sample sets into Q neural networks to train the neural networks, thereby obtaining Q trained neural networks. Furthermore, the above Q neural networks can be the same, different or partially the same.
[0072] It should be noted that the same neural network refers to neural networks with the same complexity and input resolution.
[0073] It should be noted that, since the degradation degrees of the above-mentioned N sample sets are different, after the neural network is trained based on each sample set in the N sample sets, the parameters (i.e., weights) of the trained neural network are different, that is, the neural networks trained with different sample sets have different image enhancement processing effects.
[0074] In the neural network training method provided in the embodiment of the present application, the neural network training device determines N degradation degrees based on M environmental parameters of the acquired M first images, one degradation degree corresponds to at least one environmental parameter, one degradation degree corresponds to at least one first image, and M and N are both positive integers; and uses each degradation degree to perform degradation processing on the first image corresponding to each degradation degree to obtain a second image corresponding to each degradation degree, and each second image corresponds to a first image respectively. Then, based on the first image and the second image corresponding to each degradation degree, a sample set is generated respectively to obtain N sample sets. Finally, based on the N sample sets, Q neural networks are trained respectively, one sample set corresponds to at least one neural network, and Q is a positive integer. Through this method, the neural network training device can degrade the first image under different environmental parameters to the corresponding degree of degradation (i.e., with differences) to obtain the degraded image (i.e., the second image) corresponding to the first image after the differential degradation, thereby constructing a training sample set (i.e., different degrees of degradation) that meets the shooting quality under multiple environmental parameters, and then obtain a neural network suitable for targeted processing of images shot under different environmental parameters, thereby improving the image enhancement processing effect.
[0075] Optionally, in the embodiment of the present application, the process of step 101 may include the following steps 101a and 101b:
[0076] Step 101a: Determine N target parameter ranges corresponding to the M environmental parameters from the X parameter ranges.
[0077] Each parameter range corresponds to a degradation degree, and a target parameter range corresponds to at least one environmental parameter.
[0078] Step 101b: Determine the degradation degree corresponding to the i-th target parameter range among the N target parameter ranges as the i-th degradation degree.
[0079] Among them, the above-mentioned i-th target parameter range is: the i-th target parameter range after sorting the above-mentioned N target parameter ranges in order of the corresponding degradation degrees from small to large, and i is a positive integer; the environmental parameter corresponding to the above-mentioned i-th target parameter range is less than the environmental parameter corresponding to the i+1-th target parameter range.
[0080] Optionally, the X parameter ranges may be preset parameter ranges, which refer to ISO value ranges. For example, the X parameter ranges may include three parameter ranges: ISO value less than 50, ISO value greater than or equal to 50 and less than 200, and ISO value greater than or equal to 200.
[0081] It should be noted that the larger the environmental parameter corresponding to the image, the darker the shooting scene corresponding to the image, and the greater the degree of degradation of the captured image. Therefore, it is necessary to degrade the image to a greater extent to obtain a training data set that can better simulate the real shooting scene. That is, the larger the environmental parameter (ISO value) in the parameter range of the environmental parameter of the first image, the larger the ISO value of the first image, and the higher the degree of degradation corresponding to the first image.
[0082] Optionally, in an embodiment of the present application, each of the M first images includes at least one facial element.
[0083] Optionally, before the process of step 104, the neural network training method provided in the embodiment of the present application further includes the following steps A1 and B1:
[0084] Step A1: Obtain R region size parameters corresponding to the facial elements in the M first images.
[0085] Among them, each area size parameter is: the size parameter of the area where the facial element in a first image is located, R is a positive integer, and R is greater than or equal to M.
[0086] Step B1: Determine the Q neural networks according to the M environmental parameters and the R area size parameters.
[0087] Among them, one neural network corresponds to: at least one environmental parameter and at least one area size parameter.
[0088] Optionally, a first image may include one or more face elements. For example, if image 2 is a photo of three persons, image 2 includes three face elements, and image 2 corresponds to three region size parameters.
[0089] Optionally, the above region size parameters may include at least one of the following: length, width, and area.
[0090] Exemplarily, the neural network training device can obtain the region size parameters corresponding to at least one facial element in each of the M first images to obtain R region size parameters, wherein each first image can correspond to one or more of the R region size parameters.
[0091] Optionally, the neural network training device can determine the corresponding environmental parameter level and the area size level according to the above-mentioned M environmental parameters and the above-mentioned R area size parameters, respectively. Further, the above-mentioned environmental parameter level can include low ISO scenes and high ISO scenes, and the above-mentioned area size level can include large areas and small areas. Exemplarily, the position information of the face in the first image can be obtained by a face detection algorithm. Furthermore, a rectangular frame can be used to select the facial elements in the first image, and the rectangular frame information can be expanded by a certain proportion to obtain the width (i.e., face_width) and height (i.e., face_height) of the facial area corresponding to the facial elements, so that the area (face_area) of the face size can be calculated as face_area=face_width*face_height.
[0092] Exemplarily, the face area flag information can be determined by the above-mentioned ISO and face size information processing module. Further, the face area flag is used to characterize the size of the area of the face element, for example, the face area flag is a large area flag (ie, bigFace_flag), or the face area flag is a small area flag (ie, smallFace_flag).
[0093] Furthermore, when determining the face area flag information through the above-mentioned ISO and face size information processing module, the size relationship between the area of the face element of the first image and the area threshold (i.e., area_threshold) can be determined. If face_area is greater than area_threshold, the large area flag (or large area symbol) is set to 1, and if face_area is less than area_threshold, the small area flag is set to 1. Furthermore, the above-mentioned area threshold is a multi-stage area threshold set in advance, and the specific area threshold can be determined by collecting user photo data.
[0094] Furthermore, the neural network training device can identify the flag information of the face area in the ISO and face size information processing module, and then determine the level of the face area based on the identified flag information. For example, if bigFace_flag is identified as 1, the face element is determined to be a large-area face element.
[0095] Exemplarily, the neural network training device may determine the ISO flag by comparing the environmental parameter (i.e., ISO value) of the first image with the ISO threshold value. For example, if the ISO value is greater than ISO_threshold, the high ISO flag (or high ISO sign bit) is set to 1, and if the ISO value is less than ISO_threshold, the low ISO flag is set to 1.
[0096] Furthermore, the neural network training device can identify the ISO flag information in the ISO and face size information processing module, and then determine the ISO level based on the identified flag information. For example, if it is identified that highISO_flag is equal to 1, it is determined that the first image corresponds to a low ISO value, that is, a low ISO scene. Figure 4 Shown is a schematic diagram of ISO and face size information processing.
[0097] Exemplarily, the neural network training device can determine the neural network corresponding to each sample set based on the facial area flag information and ISO flag information corresponding to each identified sample set.
[0098] Furthermore, the neural network device can identify N sample sets one by one. When it is identified that the flag information of the face area of the first image in a certain sample set indicates a large-area face element (that is, bigFace_flag is equal to 1), the neural network training device inputs the sample set into a neural network of a first complexity (that is, high complexity), and trains the neural network of the first complexity to obtain a trained neural network; when it is identified that the flag information of the face area of the first image in a certain sample set indicates a small-area face element (that is, smallFace_flag is equal to 1), the sample set is input into a low-complexity neural network for training.
[0099] Further optionally, in the embodiment of the present application, among the Q neural networks, the target parameter corresponding to the neural network with greater complexity is greater than the target parameter corresponding to the neural network with less complexity;
[0100] The above target parameters include: environmental parameters and area size parameters.
[0101] Optionally, the above-mentioned neural network with high complexity refers to a neural network with high time complexity and / or space complexity. Further, the complexity of the neural network can be reflected in the depth and breadth of the neural network.
[0102] For example, for the first image with a larger environmental parameter (i.e., ISO value), that is, the first image in a high ISO scene, since a more complex (i.e., higher degradation degree) degradation processing is performed on it, a set of high-definition-low-definition data sets consisting of images with more severe degradation corresponding to it will be obtained at this time, so a deeper, larger, and more complex neural network can be used, while for the low ISO scene, since a simpler (i.e., lower degradation degree) degradation processing is performed on it, a set of high-definition-low-definition data sets consisting of images with relatively light degradation corresponding to it will be obtained at this time, so a shallower, smaller neural network can be used. That is, a neural network with high complexity can be selected for training for a training set with a high degree of degradation, and a neural network with low complexity can be selected for training for a data set with a low degree of degradation. In this way, while ensuring a better processing effect, the computational complexity can be reduced, the processing time of the algorithm can be accelerated, and the training efficiency can be improved.
[0103] For example, Figure 5 This is a schematic diagram of the multi-complexity GAN network process, such as Figure 5 As shown, assuming that the ISO value of image 1 corresponds to a low ISO scene, it is input into a simple GAN network for training to obtain a trained neural network that has a better processing effect on the image to be processed of the low ISO scene; assuming that the ISO value of image 1 corresponds to a high ISO scene, it is input into a complex GAN network for training to obtain a trained neural network that has a better processing effect on the image to be processed of the high ISO scene.
[0104] Further optionally, in an embodiment of the present application, each sample set corresponds to at least two neural networks.
[0105] Optionally, the process of step B1 above may include the following step C1:
[0106] Step C1: determining a first neural network according to a first environmental parameter and a first area size parameter;
[0107] The first environmental parameter is: an environmental parameter of the first image in any one of the N sample sets; the first region size parameter is: a region size parameter corresponding to the first image in any one of the sample sets.
[0108] The first neural network mentioned above is: the neural network corresponding to any of the above sample sets.
[0109] Optionally, the first image in the above-mentioned sample set corresponds to a degradation degree, and the degradation degree may correspond to a neural network.
[0110] It should be noted that each sample set is a high-definition and low-definition image pair, and the low-definition image in each sample set is an image degraded from the high-definition image.
[0111] Exemplarily, when the regional size parameters (i.e., face area) of the face elements of the first image in a sample set are not the same, for example, when there are first images including large-area face elements and small-area face elements in the sample set, a neural network with a larger input resolution and higher complexity can be selected for training the first image including the large-area face elements, and a neural network with a smaller input resolution and lower complexity can be selected for training the first image including the small-area face elements. In this way, the training efficiency can be improved while achieving better training results.
[0112] An embodiment of the present application provides an image processing method, which can be applied to an electronic device. The electronic device may include Q neural networks trained by the above-mentioned neural network training method, where Q is a positive integer. Figure 6 FIG. 1 is a flow chart showing an image processing method provided by an embodiment of the present application. Figure 6 As shown, the neural network training method provided in the embodiment of the present application may include the following steps 201 and 202:
[0113] Step 201: When displaying a preview image captured by the camera of the electronic device, determine a target neural network from among Q trained neural networks based on target environment parameters of the preview image and target area size parameters corresponding to facial elements in the preview image.
[0114] In the embodiment of the present application, the above-mentioned target environment parameter includes at least one of the following: ISO value, brightness value; the above-mentioned target area size parameter includes at least one of the following: width, height, area.
[0115] Exemplarily, when a preview image is displayed, the current ISO value of the camera can be obtained and determined as the ISO value of the preview image, and the facial area information of the facial elements in the preview image can be obtained.
[0116] Further, the image processing device can determine whether the current shooting scene is a high ISO scene or a low ISO scene based on the ISO value obtained above, and determine whether the facial element to be processed is a large area or a small area based on the acquired facial area information.
[0117] Optionally, in an embodiment of the present application, the target neural network is a trained adversarial neural network that matches the target environment parameters and the target area size parameters. The target neural network is used to perform image enhancement processing on the image captured by the camera.
[0118] Step 202: Input the third image captured by the camera into the target neural network for image processing to obtain a processed image.
[0119] The third image is an image captured by the camera within a predetermined time period, and the predetermined time period is a time period from the moment when the camera captures the preview image to the moment when the camera stops capturing the preview image.
[0120] Optionally, in this embodiment of the present application, the third image may be an image including facial elements.
[0121] Optionally, in an embodiment of the present application, the above-mentioned image processing can be image enhancement processing, for example, enhancing the face area in the image by denoising, removing blur, improving brightness, improving contrast, etc.
[0122] In the image processing method provided in the embodiment of the present application, the image processing device can determine the target neural network from the trained Q neural networks according to the target environment parameters of the preview image and the target area size parameters corresponding to the face elements in the preview image when displaying the preview image captured by the camera of the electronic device. Through this method, the image processing device can intelligently select the neural network that meets the current shooting environment parameters and face size for image enhancement processing based on the environment parameters when the image was shot and the size of the face in the shot image, thereby greatly improving the effect and performance of face enhancement of the shot image.
[0123] Optionally, in the embodiment of the present application, before determining the target neural network from the trained Q neural networks in the above step 201 according to the target environment parameters of the above preview image and the target area size parameters corresponding to the facial elements in the above preview image, the image processing method provided in the embodiment of the present application further includes the following step D1:
[0124] Step D1: when the preview image contains a face element, obtain the target environment parameters and the target area size parameters corresponding to the face element.
[0125] Exemplarily, whether the preview image contains a face element may be detected by acquiring a contour.
[0126] Optionally, in the embodiment of the present application, the process of determining the target neural network from the trained Q neural networks according to the target environment parameters of the preview image and the target area size parameters corresponding to the facial elements in the preview image in the step 201 may include the following step 201a:
[0127] Step 201a: Based on the pre-stored Q corresponding relationships, determine the target neural network corresponding to the target environment parameters and the target area size parameters.
[0128] Each corresponding relationship is a corresponding relationship between at least one environmental parameter, at least one area size parameter and a trained neural network.
[0129] Alternatively, each of the above correspondences is respectively: a correspondence between an environmental parameter level, an area size level and a trained neural network.
[0130] Exemplarily, the above correspondence may be established after training a neural network and obtaining a trained neural network.
[0131] For example, the trained neural network A corresponds to a low ISO scene, and the trained neural network B corresponds to a high ISO scene. Assuming that the above target environment parameters correspond to the low ISO scene, the above target neural network can be neural network A.
[0132] For another example, the trained neural network C corresponds to large-area facial elements, and the trained neural network D corresponds to small-area facial elements. Assuming that the above-mentioned target environment parameters correspond to large-area facial elements, the above-mentioned target neural network can be neural network C.
[0133] It should be noted that the neural network training method provided in the embodiment of the present application can be executed by a neural network training device, or a control module in the neural network training device for executing the neural network training method. In the embodiment of the present application, the neural network training device provided in the embodiment of the present application is described by taking the neural network training method executed by the neural network training device as an example.
[0134] The present application embodiment provides a neural network training device 600, such as Figure 7 As shown, the device includes: a determination module 601, a processing module 602, a generation module 603 and a training module 604, wherein:
[0135] The determination module 601 is used to determine N degradation degrees according to M environmental parameters of the acquired M first images, where one degradation degree corresponds to at least one environmental parameter, and one degradation degree corresponds to at least one first image, and M and N are both positive integers; the processing module 602 is used to perform degradation processing on the first images corresponding to each degradation degree based on each degradation degree determined by the determination module 601, to obtain second images corresponding to each degradation degree, and each second image corresponds to a first image; the generation module 603 is used to generate a sample set based on the first image and the second image corresponding to each degradation degree obtained by the processing module 602, to obtain N sample sets; the training module 604 is used to train Q neural networks based on the N sample sets obtained by the generation module 603, where one sample set corresponds to at least one neural network, and Q is a positive integer.
[0136] Optionally, in an embodiment of the present application, the above-mentioned determination module 601 is specifically used to determine N target parameter ranges corresponding to the above-mentioned M environmental parameters from X parameter ranges, each parameter range corresponds to a degree of degradation, and one target parameter range corresponds to at least one environmental parameter; the above-mentioned determination module 601 is specifically used to determine the degree of degradation corresponding to the i-th target parameter range among the above-mentioned N target parameter ranges as the i-th degree of degradation; wherein the above-mentioned i-th target parameter range is: the i-th target parameter range after sorting the above-mentioned N target parameter ranges in order of the corresponding degree of degradation from small to large, i is a positive integer; the environmental parameter corresponding to the above-mentioned i-th target parameter range is less than the environmental parameter corresponding to the i+1-th target parameter range.
[0137] Optionally, in the embodiment of the present application, each of the M first images includes at least one face element;
[0138] The above device also includes: an acquisition module 605;
[0139] The acquisition module 605 is used to obtain R region size parameters corresponding to the facial elements in the M first images, where each region size parameter is: a size parameter of the region where the facial elements in a first image are located, R is a positive integer, and R is greater than or equal to M; the determination module 601 is also used to determine the Q neural networks based on the M environmental parameters and the R region size parameters obtained by the acquisition module 605; wherein one neural network corresponds to: at least one environmental parameter and at least one region size parameter.
[0140] Optionally, in an embodiment of the present application, among the Q neural networks mentioned above, the target parameters corresponding to the neural networks with greater complexity are greater than the target parameters corresponding to the neural networks with less complexity; wherein the target parameters include: environmental parameters and area size parameters.
[0141] Optionally, in the embodiment of the present application, each sample set corresponds to at least one neural network;
[0142] The determination module 601 is specifically used to determine the first neural network according to the first environmental parameter and the first region size parameter; wherein the first environmental parameter is: the environmental parameter of the first image in any sample set among the N sample sets; the first region size parameter is: the region size parameter corresponding to the first image in any sample set;
[0143] The first neural network mentioned above is: the neural network corresponding to any of the above sample sets.
[0144] In the neural network training device provided in the embodiment of the present application, the neural network training device determines N degradation degrees based on M environmental parameters of the acquired M first images, one degradation degree corresponds to at least one environmental parameter, one degradation degree corresponds to at least one first image, and M and N are both positive integers; and uses each degradation degree to perform degradation processing on the first image corresponding to each degradation degree to obtain a second image corresponding to each degradation degree, and each second image corresponds to a first image respectively. Then, based on the first image and the second image corresponding to each degradation degree, a sample set is generated respectively to obtain N sample sets. Finally, based on the N sample sets, Q neural networks are trained respectively, one sample set corresponds to at least one neural network, and Q is a positive integer. Through this method, the neural network training device can degrade the first image under different environmental parameters to the corresponding degree of degradation (i.e., with differences) to obtain the degraded image (i.e., the second image) corresponding to the first image after the differential degradation, thereby constructing a training sample set (i.e., different degrees of degradation) that meets the shooting quality under multiple environmental parameters, and then obtain a neural network suitable for targeted processing of images shot under different environmental parameters, thereby improving the image enhancement processing effect.
[0145] It should be noted that the image processing method provided in the embodiment of the present application can be executed by an image processing device or a control module in the image processing device for executing the image processing method. In the embodiment of the present application, the image processing device provided in the embodiment of the present application is described by taking the image processing device executing the image processing method as an example.
[0146] The present application embodiment provides an image processing device 700, such as Figure 8As shown, the device includes Q neural networks trained by the above neural network training method, where Q is a positive integer; the device includes: a determination module 701 and a processing module 702, wherein:
[0147] The determination module 701 is used to determine the target neural network from the trained Q neural networks according to the target environment parameters of the preview image and the target area size parameters corresponding to the face elements in the preview image when displaying the preview image captured by the camera of the electronic device;
[0148] The processing module 702 is used to input the third image taken by the camera into the target neural network determined by the determination module 701 for image processing to obtain a processed image, wherein the third image is an image taken by the camera within a predetermined time period, and the predetermined time period is a time period between the moment when the camera captures the preview image and the moment when the camera stops capturing the preview image.
[0149] Optionally, in the embodiment of the present application, the device 700 further includes an acquisition module 703, and the acquisition module 703 is used to acquire the target environment parameters and the target area size parameters corresponding to the facial elements when the preview image contains facial elements.
[0150] Optionally, in an embodiment of the present application, the above-mentioned determination module 701 is specifically used to determine the target neural network corresponding to the above-mentioned target environmental parameters and the above-mentioned target area size parameters based on pre-stored Q corresponding relationships; wherein each corresponding relationship is: a corresponding relationship between at least one environmental parameter, at least one area size parameter and a trained neural network.
[0151] In the image processing device provided in the embodiment of the present application, the image processing device can determine the target neural network from the trained Q neural networks according to the target environment parameters of the preview image and the target area size parameters corresponding to the face elements in the preview image when displaying the preview image captured by the camera of the electronic device. Through this method, the image processing device can intelligently select the neural network that meets the current shooting environment parameters and face size for image enhancement processing based on the environment parameters when the image is shot and the size of the face in the shot image, thereby greatly improving the effect and performance of face enhancement of the shot image.
[0152] The neural network training device and image processing device in the embodiments of the present application can be a device, or a component, integrated circuit, or chip in a terminal. The device can be a mobile electronic device or a non-mobile electronic device. Exemplarily, the mobile electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, an in-vehicle electronic device, a wearable device, an ultra-mobile personal computer (UMPC), a netbook, or a personal digital assistant (PDA), etc. The non-mobile electronic device can be a server, a network attached storage (NAS), a personal computer (PC), a television (TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiments of the present application.
[0153] The neural network training device and the image processing device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.
[0154] The neural network training device provided in the embodiment of the present application can achieve Figures 1 to 5 The various processes implemented by the method embodiment are not described here in detail to avoid repetition. The image processing device provided by the embodiment of the present application can achieve Figure 6 and Figure 7 To avoid repetition, the various processes implemented by the method embodiment are not described here.
[0155] Optional, such as Fig. 9 As shown, the embodiment of the present application also provides an electronic device 800, including a processor 801, a memory 802, and a program or instruction stored in the memory 802 and executable on the processor 801. When the program or instruction is executed by the processor 801, each process of the neural network training method or the image processing method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.
[0156] It should be noted that the electronic devices in the embodiments of the present application include the above-mentioned mobile electronic devices and non-mobile electronic devices.
[0157] Fig.10 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of the present application.
[0158] The electronic device 100 includes but is not limited to components such as a radio frequency unit 101, a network module 102, an audio output unit 103, an input unit 104, a sensor 105, a display unit 106, a user input unit 107, an interface unit 108, a memory 109, and a processor 110.
[0159] Those skilled in the art will appreciate that the electronic device 100 may also include a power source (such as a battery) for supplying power to various components, and the power source may be logically connected to the processor 110 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Fig.10 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.
[0160] Among them, the above-mentioned processor 110 is used to determine N degradation degrees according to M environmental parameters of the acquired M first images, one degradation degree corresponds to at least one environmental parameter, one degradation degree corresponds to at least one first image, and M and N are both positive integers; the above-mentioned processor 110 is used to perform degradation processing on the first image corresponding to each degradation degree based on each degradation degree, so as to obtain the second image corresponding to each degradation degree, and each second image corresponds to a first image; the above-mentioned processor 110 is used to generate a sample set based on the first image and the second image corresponding to each degradation degree, so as to obtain N sample sets; the above-mentioned processor 110 is used to train Q neural networks based on the above-mentioned N sample sets, one sample set corresponds to at least one neural network, and Q is a positive integer.
[0161] Optionally, in an embodiment of the present application, the processor 110 is specifically used to determine N target parameter ranges corresponding to the M environmental parameters from X parameter ranges, each parameter range corresponds to a degree of degradation, and one target parameter range corresponds to at least one environmental parameter; the processor 110 is specifically used to determine the degree of degradation corresponding to the i-th target parameter range among the N target parameter ranges as the i-th degree of degradation; wherein the i-th target parameter range is: the i-th target parameter range after sorting the N target parameter ranges in order of the corresponding degree of degradation from small to large, where i is a positive integer; the environmental parameter corresponding to the i-th target parameter range is less than the environmental parameter corresponding to the i+1-th target parameter range.
[0162] Optionally, in an embodiment of the present application, each of the M first images includes at least one facial element; the processor 110 is used to obtain R region size parameters corresponding to the facial elements in the M first images, each region size parameter being: a size parameter of a region where a facial element in a first image is located, R is a positive integer, and R is greater than or equal to M; the processor 110 is further used to determine the Q neural networks based on the M environmental parameters and the R region size parameters; wherein one neural network corresponds to: at least one environmental parameter and at least one region size parameter.
[0163] Optionally, in an embodiment of the present application, among the Q neural networks mentioned above, the target parameters corresponding to the neural networks with greater complexity are greater than the target parameters corresponding to the neural networks with less complexity; wherein the target parameters include: environmental parameters and area size parameters.
[0164] Optionally, in the embodiment of the present application, each sample set corresponds to at least one neural network;
[0165] The processor 110 is specifically configured to determine the first neural network according to the first environmental parameter and the first region size parameter; wherein the first environmental parameter is: one of the environmental parameters of the first image in any one of the N sample sets; the first region size parameter is: one of the region size parameters corresponding to the first image in any one of the sample sets;
[0166] The first neural network mentioned above is: the neural network corresponding to any of the above sample sets.
[0167] In the electronic device provided in the embodiment of the present application, the electronic device determines N degradation degrees according to M environmental parameters of the acquired M first images, one degradation degree corresponds to at least one environmental parameter, one degradation degree corresponds to at least one first image, and M and N are both positive integers; and each degradation degree is used to perform degradation processing on the first image corresponding to each degradation degree, so as to obtain a second image corresponding to each degradation degree, and each second image corresponds to a first image, and then, based on the first image and the second image corresponding to each degradation degree, a sample set is generated respectively to obtain N sample sets, and finally, based on the N sample sets, Q neural networks are trained respectively, one sample set corresponds to at least one neural network, and Q is a positive integer. Through this method, the electronic device can degrade the first image under different environmental parameters with corresponding degradation degrees (i.e., with differences) to obtain the degraded image (i.e., the second image) corresponding to the first image after differential degradation, so as to construct a training sample set (i.e., different degradation degrees) that meets the shooting quality under multiple environmental parameters, and then a neural network suitable for targeted processing of the captured images under different environmental parameters can be obtained, so as to improve the image enhancement processing effect.
[0168] Alternatively, the processor 110 is used to determine a target neural network from Q trained neural networks based on target environment parameters of the preview image and target area size parameters corresponding to facial elements in the preview image when displaying a preview image captured by a camera of the electronic device; input a third image captured by the camera into the target neural network determined by the determination module 701 for image processing to obtain a processed image, wherein the third image is an image captured by the camera within a predetermined time period, and the predetermined time period is a time period between the moment when the camera captures the preview image and the moment when the camera stops capturing the preview image.
[0169] Optionally, in an embodiment of the present application, the processor 110 is configured to obtain target environment parameters and target area size parameters corresponding to the facial elements when the preview image includes facial elements.
[0170] Optionally, in an embodiment of the present application, the processor 110 is specifically used to determine the target neural network corresponding to the target environment parameters and the target area size parameters based on Q pre-stored correspondences; wherein each correspondence is: a correspondence between at least one environment parameter, at least one area size parameter and a trained neural network.
[0171] In the electronic device provided in the embodiment of the present application, the electronic device can determine the target neural network from the trained Q neural networks according to the target environment parameters of the preview image and the target area size parameters corresponding to the face elements in the preview image when displaying the preview image captured by the camera of the electronic device. Through this method, the electronic device can intelligently select the neural network that meets the current shooting environment parameters and face size for image enhancement processing based on the environment parameters when the image is shot and the size of the face in the shot image, thereby greatly improving the effect and performance of face enhancement of the shot image.
[0172] It should be understood that in the embodiment of the present application, the input unit 104 may include a graphics processor (Graphics Processing Unit, GPU) 1041 and a microphone 1042, and the graphics processor 1041 processes the image data of the static picture or video obtained by the image capture device (such as a camera) in the video capture mode or the image capture mode. The display unit 106 may include a display panel 1061, and the display panel 1061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 107 includes a touch panel 1071 and other input devices 1072. The touch panel 1071 is also called a touch screen. The touch panel 1071 may include two parts: a touch detection device and a touch controller. Other input devices 1072 may include but are not limited to a physical keyboard, a function key (such as a volume control button, a switch button, etc.), a trackball, a mouse, and a joystick, which will not be repeated here. The memory 109 can be used to store software programs and various data, including but not limited to applications and operating systems. The processor 110 can integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, the user interface, and the application program, etc., and the modem processor mainly processes wireless communication. It is understandable that the above-mentioned modem processor may not be integrated into the processor 110.
[0173] The embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the above-mentioned neural network training method embodiment or the various processes of the above-mentioned image processing method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.
[0174] The processor is a processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.
[0175] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned neural network training method embodiment, or the various processes of the above-mentioned image processing method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.
[0176] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.
[0177] An embodiment of the present application provides a computer program product, which is stored in a non-volatile storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned neural network training method embodiment, or the various processes of the above-mentioned image processing method embodiment, and can achieve the same technical effect.
[0178] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.
[0179] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present application.
[0180] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.
Claims
1. A neural network training method, characterized in that: The method comprises: Determine N degradation degrees according to M environmental parameters of the acquired M first images, where one degradation degree corresponds to at least one environmental parameter, one degradation degree corresponds to at least one first image, and M and N are both positive integers; Based on each degradation degree, respectively performing degradation processing on the first images corresponding to each degradation degree to obtain second images corresponding to each degradation degree, each second image respectively corresponding to one first image; Based on the first image and the second image corresponding to each degradation degree, a sample set is generated respectively to obtain N sample sets; Based on the N sample sets, Q neural networks are trained respectively, one sample set corresponds to at least one neural network, and Q is a positive integer; When displaying a preview image captured by a camera of an electronic device, a target neural network is determined from Q trained neural networks based on target environment parameters of the preview image and target area size parameters corresponding to facial elements in the preview image; the image captured by the camera is input into the target neural network for image processing to obtain a processed image.
2. The method according to claim 1, characterized in that: The determining N degradation degrees according to the M environmental parameters of the M first images obtained includes: Determine N target parameter ranges corresponding to the M environmental parameters from the X parameter ranges, each parameter range corresponds to a degradation degree, and one target parameter range corresponds to at least one environmental parameter; The degradation degree corresponding to the i-th target parameter range among the N target parameter ranges is determined as the i-th degradation degree.
3. The method according to claim 1, characterized in that: Each of the M first images includes at least one face element; Before respectively training the Q neural networks based on the N sample sets, the method further includes: Obtain R region size parameters corresponding to the facial elements in the M first images, where each region size parameter is: a size parameter of a region where a facial element in one of the first images is located, R is a positive integer, and R is greater than or equal to M; Determining the Q neural networks according to the M environmental parameters and the R area size parameters; Among them, one neural network corresponds to: at least one environmental parameter and at least one area size parameter.
4. The method according to claim 3, characterized in that Among the Q neural networks, the target parameter corresponding to the neural network with greater complexity is greater than the target parameter corresponding to the neural network with less complexity; Wherein, the target parameters include: environmental parameters and area size parameters.
5. The method according to claim 3, characterized in that: Each sample set corresponds to at least two neural networks; The step of determining the Q neural networks according to the M environmental parameters and the R area size parameters includes: Determining a first neural network according to a first environmental parameter and a first area size parameter; The first environmental parameter is: an environmental parameter of a first image in any sample set among the N sample sets; the first region size parameter is: a region size parameter corresponding to the first image in any sample set; The first neural network is: the neural network corresponding to any one of the sample sets.
6. An image processing method, applied to an electronic device, characterized in that: The electronic device comprises Q neural networks trained by the neural network training method according to any one of claims 1 to 5, where Q is a positive integer; the method comprises: In the case of displaying a preview image captured by a camera of the electronic device, determining a target neural network from among the trained Q neural networks according to target environment parameters of the preview image and target area size parameters corresponding to facial elements in the preview image; The third image taken by the camera is input into the target neural network for image processing to obtain a processed image, wherein the third image is: an image taken by the camera within a predetermined time period, and the predetermined time period is: a time period between the moment when the camera captures the preview image and the moment when the camera stops capturing the preview image.
7. The method according to claim 6, characterized in that Before determining the target neural network from the trained Q neural networks based on the target environment parameters of the preview image and the target area size parameters corresponding to the facial elements in the preview image, the method further includes: In the case where the preview image includes a face element, a target environment parameter and a target area size parameter corresponding to the face element are obtained.
8. The method according to claim 6, characterized in that The step of determining a target neural network from the trained Q neural networks according to the target environment parameters of the preview image and the target area size parameters corresponding to the face elements in the preview image comprises: Based on the pre-stored Q corresponding relationships, determining the target neural network corresponding to the target environment parameter and the target area size parameter; Each corresponding relationship is a corresponding relationship between at least one environmental parameter, at least one area size parameter and a trained neural network.
9. A neural network training device, characterized in that: The device comprises: a determination module, a processing module, a generation module and a training module, wherein: The determination module is used to determine N degradation degrees according to the M environmental parameters of the acquired M first images, where one degradation degree corresponds to at least one environmental parameter, one degradation degree corresponds to at least one first image, and M and N are both positive integers; The processing module is used to perform degradation processing on the first images corresponding to each degradation degree based on each degradation degree determined by the determination module to obtain second images corresponding to each degradation degree, each second image corresponding to one first image; The generating module is used to generate a sample set based on the first image and the second image corresponding to each degradation degree obtained by the processing module, respectively, to obtain N sample sets; The training module is used to train Q neural networks based on the N sample sets obtained by the generation module, one sample set corresponds to at least one neural network, and Q is a positive integer; When displaying a preview image captured by a camera of an electronic device, a target neural network is determined from Q trained neural networks based on target environment parameters of the preview image and target area size parameters corresponding to facial elements in the preview image; the image captured by the camera is input into the target neural network for image processing to obtain a processed image.
10. An image processing device, characterized in that: The device comprises Q neural networks trained by the neural network training method according to any one of claims 1 to 5, where Q is a positive integer; the device comprises: a determination module and a processing module, wherein: The determination module is used to determine the target neural network from the trained Q neural networks according to the target environment parameters of the preview image and the target area size parameters corresponding to the facial elements in the preview image when displaying the preview image captured by the camera of the electronic device; The processing module is used to input the third image taken by the camera into the target neural network determined by the determination module for image processing to obtain a processed image, wherein the third image is: an image taken by the camera within a predetermined time period, and the predetermined time period is: a time period between the moment when the camera captures the preview image and the moment when the camera stops capturing the preview image.
11. An electronic device, characterized in that: The invention comprises a processor, a memory and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the neural network training method according to any one of claims 1 to 5, or the steps of the image processing method according to any one of claims 6 to 8.
12. A readable storage medium, characterized in that: The readable storage medium stores a program or instruction, and when the program or instruction is executed by the processor, the steps of the neural network training method according to any one of claims 1 to 5, or the steps of the image processing method according to any one of claims 6 to 8 are implemented.
