Image processing method, device, electronic device and storage medium

Abstract

The present application discloses an image processing method, device, electronic device and storage medium, which belongs to the field of image processing. The method comprises: inputting the image to be processed into a trained portrait skin recognition model, determining a portrait skin area and N portrait areas, where N is a positive integer; based on the N portrait areas and the portrait skin area, determining a portrait skin sub-area in each portrait area.

Description

Technical Field

[0001] The present application belongs to the field of image processing, and specifically relates to an image processing method, device, electronic device and storage medium. Background Art

[0002] With the continuous development of mobile terminals, users often take photos directly through mobile terminals in their daily lives. In order to achieve better photo effects, mobile terminals often optimize photos when taking photos, especially optimizing the skin area of ​​the portrait in the photo. Only by accurately identifying the skin area of ​​the portrait in the photo can better optimization be performed.

[0003] Therefore, how to identify the skin area of ​​the portrait in the photo has become a problem that needs to be solved urgently in the industry. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide an image processing method, device, electronic device and storage medium, which can solve the problem of identifying the skin area of ​​a portrait in a photo.

[0005] In a first aspect, an embodiment of the present application provides an image processing method, the method comprising:

[0006] Input the image to be processed into the trained portrait skin recognition model to determine the portrait skin area and N portrait areas, where N is a positive integer;

[0007] Based on the N portrait regions and the portrait skin region, a portrait skin sub-region in each portrait region is determined.

[0008] In a second aspect, an embodiment of the present application provides an image processing device, including:

[0009] A determination module, used for inputting the image to be processed into a trained portrait skin recognition model, and determining a portrait skin area and N portrait areas, where N is a positive integer;

[0010] A determination module is used to determine a portrait skin sub-region in each portrait region based on the N portrait regions and the portrait skin region.

[0011] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the method described in the first aspect.

[0012] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0013] In a fifth aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the method described in the first aspect.

[0014] In an embodiment of the present application, a portrait skin recognition model can be used to simultaneously identify the portrait skin area and N portrait areas in the image to be processed, that is, two types of areas can be identified through one model, which can speed up the recognition speed. At the same time, the model can identify the portrait areas of different portraits, and thus can identify the portrait skin sub-areas corresponding to each portrait. By accurately identifying the skin areas of different portraits, skin optimization can be performed more accurately according to the skin characteristics of different portraits, thereby achieving better image optimization effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 A schematic diagram of the image processing method provided in the embodiment of the present application;

[0016] Figure 2 A schematic diagram of multi-task training of the first training sample and the second training sample described in the embodiment of the present application;

[0017] Figure 3 A schematic diagram of the associated operation flow provided in the embodiment of the present application;

[0018] Figure 4 This is a schematic diagram of image processing provided in an embodiment of the present application;

[0019] Figure 5 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application;

[0020] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0021] Figure 7 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of the present application. DETAILED DESCRIPTION

[0022] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0023] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0024] In the following, in conjunction with the accompanying drawings, an image processing method, device, electronic device and storage medium provided by the embodiments of the present application are described in detail through specific embodiments and their application scenarios.

[0025] Figure 1 A schematic diagram of the image processing method provided in the embodiment of the present application is shown in FIG. Figure 1 As shown, including:

[0026] Step 110, inputting the image to be processed into a trained portrait skin recognition model to determine a portrait skin region and N portrait regions, where N is a positive integer;

[0027] Specifically, the image to be processed described in the present application may specifically refer to an image that needs to be optimized and processed, which may be an original image taken by a camera of a mobile terminal, which needs further optimization and processing.

[0028] The trained portrait skin recognition model described in the present application can be installed in the mobile terminal itself or in a server connected to the mobile terminal.

[0029] The mobile terminal described in this application may specifically refer to a mobile phone, a tablet computer, a camera, a drone equipped with a camera, or a gimbal, etc.

[0030] The trained portrait skin recognition model in the present application can be specifically obtained by training a first sample image to be processed carrying M different portrait region identifiers and a second sample image to be processed carrying a portrait skin region identifier.

[0031] The portrait skin recognition model trained in this application can simultaneously recognize the portrait skin area in an image, as well as the portrait areas corresponding to different portraits.

[0032] In the embodiment of the present application, the same model can be used to simultaneously identify the portrait skin area and N portrait areas, which can effectively save the model training time and recognition time, effectively improve the image processing speed, and only one model needs to be maintained subsequently, which is simple and convenient.

[0033] In the present application, each portrait region corresponds to a person instance. For example, when there are two people in the image to be processed, the trained portrait skin recognition model determines the portrait region of user one and the portrait region of user two.

[0034] Step 120: determining a portrait skin sub-region in each portrait region based on the N portrait regions and the portrait skin region.

[0035] In an embodiment of the present application, when taking a group photo, due to differences in skin conditions, such as skin color, of different people, if a unified skin optimization solution is used, it may lead to poor optimization effects. If the skin conditions of different subjects can be optimized, better image optimization effects can be achieved.

[0036] At the same time, in order to distinguish and identify the skin areas of different people, that is, to identify the facial skin area and non-facial skin area of ​​each person, an intuitive method is to directly perform skin instance segmentation on the photo, but the skin area will be divided into multiple parts, and due to different clothing styles, the exposed skin area of ​​each person is different, and it is difficult to identify which person each skin area belongs to.

[0037] Since skin only exists on portraits, the portrait skin area only exists within the portrait area. Therefore, in the embodiment of the present application, the skin areas corresponding to different portraits can be distinguished by determining the portrait skin sub-area within each portrait area.

[0038] Optionally, after identifying the portrait skin sub-region within each portrait area, an image optimization scheme can be generated for each portrait skin sub-region, such as using whitening or skin smoothing treatment for the skin area. In other words, different image optimization schemes are set for different portraits to ensure that each user's optimization scheme is closer to the skin condition of his or her application, and there will be no situation where the optimization effect of some portraits is poor due to the use of a unified image optimization scheme.

[0039] In an embodiment of the present application, a portrait skin recognition model can be used to simultaneously identify the portrait skin area and N portrait areas in the image to be processed, that is, two types of areas can be identified through one model, which can speed up the recognition speed. At the same time, the model can identify the portrait areas of different portraits, and thus can identify the portrait skin sub-areas corresponding to each portrait. By accurately identifying the skin areas of different portraits, skin optimization can be performed more accurately according to the skin characteristics of different portraits, thereby achieving better image optimization effect.

[0040] Optionally, before inputting the image to be processed into the trained portrait skin recognition model, the method further includes:

[0041] Acquire a plurality of first sample images to be processed and a plurality of second sample images to be processed;

[0042] Obtaining M portrait region identifiers corresponding to each of the first sample images to be processed, and obtaining a portrait skin region identifier for each of the second sample images to be processed, where M is a positive integer;

[0043] Taking each first sample image to be processed that carries the M portrait region identifiers as a first training sample, taking each second sample image to be processed that carries the portrait skin region identifier as a second training sample, and obtaining a plurality of first training samples and a plurality of second training samples;

[0044] The preset neural network is trained using the plurality of the first training samples and the plurality of the second training samples.

[0045] Specifically, the first sample image to be processed and the second sample image to be processed described in the present application may be portrait images, and the first sample image to be processed and the second sample image to be processed may be the same image or different images.

[0046] The portrait area identifier described in the embodiment of the present application is an identifier used to mark the portrait area in the sample image to be processed. Since there may be multiple portraits in the sample image to be processed, each of the first sample images to be processed may carry one or more portrait area identifiers. Therefore, the M portrait area identifiers in the embodiment of the present application may refer to one or more portrait area identifiers.

[0047] If a sample image to be processed has multiple portrait region identifiers, each portrait region identifier is different. For example, when there are two portrait region identifiers, they may be portrait region identifier one and portrait region identifier two respectively.

[0048] The portrait skin area identification described in the embodiment of the present application can specifically be an identification used to mark the portrait skin area in the sample image to be processed. The portrait skin area refers to the area with exposed skin in the portrait area, that is, the hair area in the portrait area and the area covered by clothes do not belong to the portrait skin area.

[0049] Optionally, the training sample in the present application may be a first sample image to be processed carrying M portrait region identifiers, or a second sample image to be processed carrying portrait skin region identifiers.

[0050] The training samples in the present application may also be sample images to be processed that carry M portrait region identifiers and portrait skin region identifiers.

[0051] In this application, by diversifying the sample images to be processed and the data annotations, the difficulty of data annotation is reduced, the richness of training samples is improved, and the accuracy of model training can be effectively guaranteed.

[0052] Optionally, the using the plurality of the first training samples and the plurality of the second training samples to train a preset neural network includes:

[0053] For any one of the first training samples, the first training sample is input into the encoding part of the preset neural network, and after the output result of the encoding part is input into the decoding part of the preset neural network, the portrait mask feature information, the portrait convolution kernel information and the portrait classification information corresponding to the first training sample are output, wherein the encoding part is used for convolution downsampling processing, and the decoding part is used for convolution upsampling processing;

[0054] For any one of the second training samples, the second training sample is input into the encoding part and the decoding part, and the skin semantic feature information corresponding to the second training sample is output.

[0055] Specifically, the preset neural network described in the embodiments of the present application may specifically refer to a convolutional neural network (CNN) or other neural network models that can achieve this function, and the embodiments of the present application are not limited to this.

[0056] The encoding part described in this application is performed by convolution downsampling, and the decoding part is performed by convolution upsampling.

[0057] Specifically, in the encoding part, the first training sample is first subjected to convolution down-sampling processing to enhance the feature richness in the training sample image and reduce the size of the training sample image. For example, the first training sample image can be reduced to 1 / 32 of the original size. After convolution down-sampling, the result of the convolution down-sampling is input into the decoding part for convolution up-sampling processing to continuously restore the resolution of the first training sample and obtain the portrait mask feature information, portrait convolution kernel information and portrait classification information corresponding to the first training sample.

[0058] At the same time, during the training process, the second training sample shares the encoding part and decoding part used in the first stage sample training. That is to say, during the training process, the first training sample and the second training sample jointly train the encoding part and decoding part, making the model training less time-consuming.

[0059] In the encoding part, the second training sample is first subjected to convolution downsampling processing to improve the feature richness in the training sample image and reduce the size of the training sample image. Then, in the decoding part, the downsampling result is subjected to convolution upsampling processing to obtain the skin semantic feature information.

[0060] In the present application, the first training sample and the second training sample can be simultaneously input into the model for multi-task training, or the first training sample can be input first and then the second training sample.

[0061] Figure 2 Schematic diagram of multi-task training of the first training sample and the second training sample described in the embodiment of the present application, such as Figure 2 As shown, the first training sample and the second training sample are input into the common encoding part and decoding part for convolution down-sampling and convolution up-sampling processing, and the portrait mask feature information, portrait convolution kernel information and portrait classification information corresponding to the first training sample and the skin semantic feature information corresponding to the second training sample are obtained respectively.

[0062] In an embodiment of the present application, the first training sample and the second training sample share the encoding part and the decoding part of a preset neural network during the training process, which can effectively reduce the model time consumption compared to using two models to train two sets of data separately.

[0063] Optionally, after the step of outputting the skin semantic feature information corresponding to the second training sample, the method further includes:

[0064] Obtaining multiple portrait sample areas based on the portrait mask feature information, the portrait convolution kernel information, and the portrait classification information corresponding to the first training sample;

[0065] Associating the portrait mask feature information with the skin semantic feature information, and performing convolution up-sampling processing on the associated portrait mask feature information and the skin semantic feature information to obtain a portrait skin sample area;

[0066] Calculating a first loss value based on the multiple portrait sample areas and the M portrait area identifiers corresponding to the first training sample, and calculating a second loss value based on the portrait skin sample area and the portrait skin area identifier corresponding to the second training sample;

[0067] When the first loss value is less than a first preset threshold and the second loss value is less than a second preset threshold, the training of the preset neural network is stopped to obtain a trained portrait skin recognition model.

[0068] Specifically, since portrait skin only exists on portraits, portrait skin must also exist in the portrait area. Therefore, when identifying the portrait skin area, only the portrait area needs to be focused on. Therefore, when determining the portrait skin sample area, the portrait mask feature information and the skin semantic feature information can be associated to constrain and guide the determination of the portrait skin area. Figure 3 A schematic diagram of the associated operation flow provided in the embodiment of the present application, such as Figure 3 As shown, the portrait mask feature information and the skin semantic feature information are associated, and the associated portrait mask feature information and the skin semantic feature information are convolutionally upsampled to obtain a portrait skin sample area.

[0069] Since the portrait mask feature information and the skin semantic feature information are both already generated data, in the process of generating the portrait skin sample area, it is only necessary to perform concat operations on the portrait mask feature information and the skin semantic feature information, which improves the segmentation progress of the portrait skin sample area without adding additional overhead.

[0070] Optionally, the association operation described in the embodiments of the present application may be performed after a preset number of first training samples are trained, or the association may be started when the first training samples and the second training samples are just started to be trained.

[0071] Specifically, the portrait classification information described in the embodiments of the present application is used to determine whether the area belongs to the background category or the portrait category.

[0072] In an embodiment of the present application, the portrait mask feature information and the portrait convolution kernel information are dynamically convolved, and then combined with the portrait classification information, multiple portrait instance masks in the training samples, that is, multiple portrait sample areas, can be obtained.

[0073] Based on the comparison between the multiple portrait sample areas output by the model and the M portrait area identifications in the first training sample, a first loss value can be calculated by a preset first loss function. Based on the comparison between the portrait skin sample areas output by the model and the portrait skin area identifications, a second loss value can be calculated by a preset second loss function.

[0074] Through the first loss function value and the second loss function value, it can be determined whether the preset neural network needs to continue training. If the first loss value is greater than the first preset threshold, or the second loss value is greater than the second preset threshold, then continue to obtain new first training samples and new second training samples, and continue to input them into the preset neural network for training until the first loss value is less than the first preset threshold and the second loss value is less than the second preset threshold. Then, stop training the preset neural network to obtain a trained portrait skin recognition model.

[0075] In an embodiment of the present application, during the process of portrait skin area recognition, the portrait mask feature information and the skin semantic feature information can be associated with each other, and the portrait skin recognition area can be limited by the portrait mask feature to improve the recognition progress and avoid misrecognition caused by skin-like background outside the portrait. Moreover, through model training, a trained portrait skin recognition model can be obtained, so that only one test picture needs to be input to obtain the effects of the portrait skin area and N portrait areas at the same time, which can effectively ensure the accurate recognition of subsequent portrait skin areas.

[0076] Optionally, the determining, based on the N portrait regions and the portrait skin region, a portrait skin subregion in each portrait region includes:

[0077] Determine an overlapping area between each portrait area and the portrait skin area;

[0078] Based on the overlapping area, a portrait skin sub-region in each portrait region is determined.

[0079] Specifically, each portrait area described in the embodiments of the present application refers to an area corresponding to a different portrait subject. For example, there are three portrait areas, which correspond to the portrait areas of user A, user B, and user C respectively.

[0080] The portrait skin region described in this application refers to a set of regions belonging to the exposed skin of each portrait subject in the image to be processed.

[0081] Since there may be a portrait skin area in each portrait area, the portrait skin sub-area in each portrait area can be determined by determining the overlapping area between each portrait area and the portrait skin area, that is, the portrait skin sub-area corresponding to each portrait subject can be determined.

[0082] In the embodiment of the present application, by determining the overlapping area between each portrait area and the portrait skin area, the portrait skin sub-area corresponding to each portrait subject can be effectively determined, which is conducive to image optimization according to the skin conditions corresponding to different portrait subjects.

[0083] Optionally, Figure 4 This is a schematic diagram of image processing provided in the embodiment of the present application, such as Figure 4 As shown, after the image to be processed is input into the trained portrait skin recognition model, the model will output a portrait skin region 41, a first portrait region 42, and a second portrait region 43, and then the portrait skin region and the two portrait regions are fused to determine a portrait skin sub-region 44 in the first portrait region 42 and a portrait skin sub-region 45 in the second portrait region 43. Since an individual portrait may not contain a skin region, after fusion, there is no portrait fusion sub-region in the portrait region.

[0084] It should be noted that the image processing method provided in the embodiment of the present application can be executed by an image processing device or a control module in the image processing device for executing the image processing method. In the embodiment of the present application, the image processing device provided in the embodiment of the present application is described by taking the image processing method executed by the image processing device as an example.

[0085] Figure 5 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application is shown in FIG. Figure 5 As shown, it includes a determination module 510 and a determination module 520; wherein the determination module 510 is used to input the image to be processed into a trained portrait skin recognition model to determine the portrait skin area and N portrait areas, wherein N is a positive integer; wherein the determination module 520 is used to determine the portrait skin sub-area in each portrait area based on the N portrait areas and the portrait skin area.

[0086] Optionally, the device further comprises:

[0087] A first acquisition module, used for acquiring a plurality of first sample images to be processed and a plurality of second sample images to be processed;

[0088] A second acquisition module, used to acquire M portrait region identifiers corresponding to each of the first sample images to be processed, and acquire a portrait skin region identifier for each of the second sample images to be processed, where M is a positive integer;

[0089] A first processing module is used to take each first sample image to be processed carrying M portrait region identifiers as a first training sample, and take each second sample image to be processed carrying portrait skin region identifiers as a second training sample, to obtain a plurality of first training samples and a plurality of second training samples;

[0090] The training module is used to train a preset neural network using a plurality of the first training samples and a plurality of the second training samples.

[0091] Optionally, the training module is specifically used to: for any first training sample, input the first training sample into the encoding part of the preset neural network, input the output result of the encoding part into the decoding part of the preset neural network, and then output the portrait mask feature information, portrait convolution kernel information and portrait classification information corresponding to the first training sample; wherein the encoding part is performed by convolution downsampling, and the decoding part is performed by convolution upsampling;

[0092] For any one of the second training samples, the second training sample is input into the encoding part and the decoding part, and the skin semantic feature information corresponding to the second training sample is output.

[0093] Optionally, the training module is further configured to obtain a plurality of portrait sample regions based on portrait mask feature information, portrait convolution kernel information and portrait classification information corresponding to the first training sample;

[0094] Associating the portrait mask feature information with the skin semantic feature information, and performing convolution up-sampling processing on the associated portrait mask feature information and the skin semantic feature information to obtain a portrait skin sample area;

[0095] Calculating a first loss value based on the multiple portrait sample areas and the M portrait area identifiers corresponding to the first training sample, and calculating a second loss value based on the portrait skin sample area and the portrait skin area identifier corresponding to the second training sample;

[0096] When the first loss value is less than a first preset threshold and the second loss value is less than a second preset threshold, the training of the preset neural network is stopped to obtain a trained portrait skin recognition model.

[0097] The image processing device in the embodiment of the present application can be a device, or a component, integrated circuit, or chip in a terminal. The device can be a mobile electronic device or a non-mobile electronic device. Exemplarily, the mobile electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, an in-vehicle electronic device, a wearable device, an ultra-mobile personal computer (UMPC), a netbook, or a personal digital assistant (PDA), etc. The non-mobile electronic device can be a server, a network attached storage (NAS), a personal computer (PC), a television (TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.

[0098] The image processing device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0099] The image processing device provided in the embodiment of the present application can achieve Figures 1 to 4 To avoid repetition, the various processes implemented by the method embodiment are not described here.

[0100] Optionally, Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application, such as Figure 6 As shown, an embodiment of the present application also provides an electronic device 600, including a processor 601, a memory 602, and a program or instruction stored in the memory 602 and executable on the processor 601. When the program or instruction is executed by the processor 601, each process of the above-mentioned image processing method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0101] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0102] Figure 7 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of the present application.

[0103] The electronic device 700 includes but is not limited to: a radio frequency unit 701, a network module 702, an audio output unit 703, an input unit 704, a sensor 705, a display unit 706, a user input unit 707, an interface unit 708, a memory 709, and a processor 710.

[0104] Those skilled in the art will appreciate that the electronic device 700 may also include a power source (such as a battery) for supplying power to each component, and the power source may be logically connected to the processor 710 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Figure 7 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0105] The processor 710 is used to input the image to be processed into the trained portrait skin recognition model to determine the portrait skin area and N portrait areas, where N is a positive integer;

[0106] The processor 710 is used to determine a portrait skin sub-region in each portrait region based on the N portrait regions and the portrait skin region.

[0107] Optionally, the processor 710 is used to obtain a plurality of first sample images to be processed and a plurality of second sample images to be processed;

[0108] Obtaining M portrait region identifiers corresponding to each of the first sample images to be processed, and obtaining a portrait skin region identifier for each of the second sample images to be processed, where M is a positive integer;

[0109] Taking each first sample image to be processed that carries the M portrait region identifiers as a first training sample, taking each second sample image to be processed that carries the portrait skin region identifier as a second training sample, and obtaining a plurality of first training samples and a plurality of second training samples;

[0110] The preset neural network is trained using the plurality of the first training samples and the plurality of the second training samples.

[0111] Optionally, the processor 710 is configured to, for any one of the first training samples, input the first training sample into an encoding part of the preset neural network, input an output result of the encoding part into a decoding part of the preset neural network, and then output portrait mask feature information, portrait convolution kernel information, and portrait classification information corresponding to the first training sample, wherein the encoding part is performed by convolution downsampling, and the decoding part is performed by convolution upsampling;

[0112] For any one of the second training samples, the second training sample is input into the encoding part and the decoding part, and the skin semantic feature information corresponding to the second training sample is output.

[0113] Optionally, the processor 710 is configured to obtain a plurality of portrait sample regions based on portrait mask feature information, portrait convolution kernel information, and portrait classification information corresponding to the first training sample;

[0114] An association operation is performed based on the portrait mask feature information and the skin semantic feature information, and a convolution up-sampling process is performed on the associated portrait mask feature information and the skin semantic feature information to obtain a portrait skin sample area;

[0115] Calculating a first loss value based on the multiple portrait sample areas and the M portrait area identifiers corresponding to the first training sample, and calculating a second loss value based on the portrait skin sample area and the portrait skin area identifier corresponding to the second training sample;

[0116] When the first loss value is less than a first preset threshold and the second loss value is less than a second preset threshold, the training of the preset neural network is stopped to obtain a trained portrait skin recognition model.

[0117] In an embodiment of the present application, a portrait skin recognition model can be used to simultaneously identify the portrait skin area and N portrait areas in the image to be processed, that is, two types of areas can be identified through one model, which can speed up the recognition speed. At the same time, the model can identify the portrait areas of different portraits, and thus can identify the portrait skin sub-areas corresponding to each portrait. By accurately identifying the skin areas of different portraits, skin optimization can be performed more accurately according to the skin characteristics of different portraits, thereby achieving better image optimization effect.

[0118] It should be understood that in the embodiment of the present application, the input unit 704 may include a graphics processor (Graphics Processing Unit, GPU) 7041 and a microphone 7042, and the graphics processor 7041 processes the image data of the static picture or video obtained by the image capture device (such as a camera) in the video capture mode or the image capture mode. The display unit 706 may include a display panel 7061, and the display panel 7061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 707 includes a touch panel 7071 and other input devices 7072. The touch panel 7071 is also called a touch screen. The touch panel 7071 may include two parts: a touch detection device and a touch controller. Other input devices 7072 may include but are not limited to a physical keyboard, a function key (such as a volume control button, a switch button, etc.), a trackball, a mouse, and a joystick, which will not be repeated here. The memory 709 can be used to store software programs and various data, including but not limited to applications and operating systems. The processor 710 can integrate an application processor and a modem processor, wherein the application processor mainly processes an operating system, a user interface, and applications, etc., and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 710.

[0119] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the above-mentioned image processing method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0120] The processor is a processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0121] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned image processing method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0122] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0123] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0124] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0125] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. An image processing method, characterized in that: include: Input the image to be processed into the trained portrait skin recognition model to determine the portrait skin area and N portrait areas, where N is a positive integer; Based on the N portrait regions and the portrait skin region, determining a portrait skin sub-region within each portrait region; Before inputting the image to be processed into the trained portrait skin recognition model, the method further includes: Acquire a plurality of first sample images to be processed and a plurality of second sample images to be processed; Obtaining M portrait region identifiers corresponding to each of the first sample images to be processed, and obtaining a portrait skin region identifier for each of the second sample images to be processed, where M is a positive integer; Taking each first sample image to be processed that carries the M portrait region identifiers as a first training sample, taking each second sample image to be processed that carries the portrait skin region identifier as a second training sample, and obtaining a plurality of first training samples and a plurality of second training samples; Training a preset neural network using the plurality of the first training samples and the plurality of the second training samples; The step of training a preset neural network using a plurality of the first training samples and a plurality of the second training samples includes: For any one of the first training samples, the first training sample is input into the encoding part of the preset neural network, and after the output result of the encoding part is input into the decoding part of the preset neural network, the portrait mask feature information, the portrait convolution kernel information and the portrait classification information corresponding to the first training sample are output, wherein the encoding part is performed by convolution downsampling, and the decoding part is performed by convolution upsampling; For any one of the second training samples, input the second training sample into the encoding part and the decoding part, and output the skin semantic feature information corresponding to the second training sample; Wherein, after the step of outputting the skin semantic feature information corresponding to the second training sample, the method further includes: Obtaining multiple portrait sample areas based on the portrait mask feature information, the portrait convolution kernel information, and the portrait classification information corresponding to the first training sample; Associating the portrait mask feature information with the skin semantic feature information, and performing convolution up-sampling processing on the associated portrait mask feature information and the skin semantic feature information to obtain a portrait skin sample area; Calculating a first loss value based on the multiple portrait sample areas and the M portrait area identifiers corresponding to the first training sample, and calculating a second loss value based on the portrait skin sample area and the portrait skin area identifier corresponding to the second training sample; When the first loss value is less than a first preset threshold and the second loss value is less than a second preset threshold, the training of the preset neural network is stopped to obtain a trained portrait skin recognition model.

2. The image processing method according to claim 1, characterized in that: The determining of the portrait skin sub-region in each portrait region based on the N portrait regions and the portrait skin region includes: Determine an overlapping area between each portrait area and the portrait skin area; Based on the overlapping area, a portrait skin sub-region in each portrait region is determined.

3. An image processing device, characterized in that: include: A determination module, used for inputting the image to be processed into a trained portrait skin recognition model, and determining a portrait skin area and N portrait areas, where N is a positive integer; A determination module, configured to determine a portrait skin sub-region in each portrait region based on the N portrait regions and the portrait skin region; Wherein, the device further comprises: A first acquisition module, used for acquiring a plurality of first sample images to be processed and a plurality of second sample images to be processed; A second acquisition module, configured to acquire M portrait region identifiers corresponding to each of the first sample images to be processed, and acquire a portrait skin region identifier for each of the second sample images to be processed, wherein M is a positive integer; A first processing module is used to take each first sample image to be processed carrying M portrait region identifiers as a first training sample, and take each second sample image to be processed carrying portrait skin region identifiers as a second training sample, to obtain a plurality of first training samples and a plurality of second training samples; A training module, used for training a preset neural network using a plurality of the first training samples and a plurality of the second training samples; The training module is specifically used to: for any one of the first training samples, input the first training sample into the encoding part of the preset neural network, input the output result of the encoding part into the decoding part of the preset neural network, and then output the portrait mask feature information, portrait convolution kernel information and portrait classification information corresponding to the first training sample; wherein the encoding part is performed by convolution downsampling, and the decoding part is performed by convolution upsampling; For any one of the second training samples, input the second training sample into the encoding part and the decoding part, and output the skin semantic feature information corresponding to the second training sample; Wherein, the device is also used for: Obtaining multiple portrait sample areas based on the portrait mask feature information, the portrait convolution kernel information, and the portrait classification information corresponding to the first training sample; Associating the portrait mask feature information with the skin semantic feature information, and performing convolution up-sampling processing on the associated portrait mask feature information and the skin semantic feature information to obtain a portrait skin sample area; Calculating a first loss value based on the multiple portrait sample areas and the M portrait area identifiers corresponding to the first training sample, and calculating a second loss value based on the portrait skin sample area and the portrait skin area identifier corresponding to the second training sample; When the first loss value is less than a first preset threshold and the second loss value is less than a second preset threshold, the training of the preset neural network is stopped to obtain a trained portrait skin recognition model.

4. An electronic device, characterized in that: The method comprises a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the image processing method according to any one of claims 1 to 2.

5. A readable storage medium, characterized in that: The readable storage medium stores a program or instruction, and when the program or instruction is executed by a processor, the steps of the image processing method according to any one of claims 1 to 2 are implemented.

Images