Image processing method and device

Abstract

The invention discloses an image processing method and device, and belongs to the technical field of image processing. The method comprises the following steps: acquiring a first image, wherein the first image comprises at least one shooting object; identifying each shooting object in the at least one shooting object; according to the area image corresponding to each shooting object, light supplementing intensity information of each shooting object is determined, and the light supplementing intensity information of each shooting object is intensity information for carrying out light supplementing on the shooting object; and according to the light supplement intensity information, performing light supplement on the corresponding shooting object in the at least one shooting object to obtain a second image.

Description

Technical Field

[0001] The present application belongs to the field of image processing technology, and specifically relates to an image processing method and a device thereof. Background Art

[0002] With the increasing popularity of electronic devices, users have an increasing demand for imaging technology in electronic devices, especially portrait mode. Users expect to take portrait photos that are more atmospheric and closer to SLR effects.

[0003] Light and shadow are key evaluation dimensions for the beauty of portrait photos. When taking images, natural light is limited and affected by many factors and is difficult to strictly control. Therefore, professional photographers will use special light source equipment to light the photos according to the natural light conditions on site to obtain portrait photos with better light and shadow effects. However, the above method increases the hardware cost. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide an image processing method and device thereof, which can solve the problem of excessively high hardware costs when obtaining portrait photos with better portrait lighting effects.

[0005] In a first aspect, an embodiment of the present application provides an image processing method, the method comprising:

[0006] Acquire a first image, wherein the first image includes at least one photographed object;

[0007] identifying each of the at least one subject;

[0008] Determine fill light intensity information of each of the photographed objects according to the area image corresponding to each of the photographed objects, wherein the fill light intensity information of each of the photographed objects is intensity information of fill light for the photographed objects;

[0009] Fill light is respectively performed on corresponding photographic objects among the at least one photographic object according to the fill light intensity information to obtain a second image.

[0010] In a second aspect, an embodiment of the present application provides an image processing device, the device comprising:

[0011] A first acquisition module, used to acquire a first image, wherein the first image includes at least one photographed object;

[0012] A first determination module, configured to identify each of the at least one photographic object;

[0013] A second determining module is used to determine fill light intensity information of each of the photographed objects according to the area image corresponding to each of the photographed objects, wherein the fill light intensity information of each of the photographed objects is intensity information of fill light for the photographed objects;

[0014] The third determining module is used to perform fill light on corresponding photographic objects among the at least one photographic object according to the fill light intensity information to obtain a second image.

[0015] In a third aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0016] In a fourth aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the method described in the first aspect.

[0017] In a fifth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the method described in the first aspect.

[0018] In an embodiment of the present application, by identifying each photographic object in a first image including at least one photographic object, a regional image corresponding to each photographic object can be obtained. For each photographic object, fill light intensity information for fill light on the photographic object can be determined based on the regional image corresponding to the photographic object. For each photographic object, fill light is performed on the photographic object based on the fill light intensity information for fill light on the photographic object, and a second image after fill light for each photographic object can be obtained. In this way, through the image processing method provided in an embodiment of the present application, there is no need to add other fill light equipment to fill light the photographic object to obtain a portrait photo with better portrait light and shadow effects. Not only does it reduce the hardware cost, but it also solves the problem of poor portrait light and shadow effects due to insufficient lighting conditions or hardware limitations of electronic equipment, thereby achieving the effect of high-quality portrait photos at a low cost, while improving the filming rate and image aesthetics of the electronic device. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 is a flowchart of an image processing method provided by some embodiments of the present application;

[0020] Figure 2 is a schematic diagram of determining fill light intensity information for fill light of a target photographed object provided by some embodiments of the present application;

[0021] Figure 3is a schematic diagram of collecting a first training sample provided by some embodiments of the present application;

[0022] Figure 4 is a schematic diagram of a flow chart of data processing of a first training sample provided by some embodiments of the present application;

[0023] Figure 5 is a flowchart of an image processing method provided by some embodiments of the present application;

[0024] Figure 6 is a schematic diagram of the structure of an image processing device shown in some embodiments of the present application;

[0025] Figure 7 is a schematic diagram of the structure of an electronic device shown in some embodiments of the present application;

[0026] Figure 8 It is a schematic diagram of the hardware structure of an electronic device shown in some embodiments of the present application. DETAILED DESCRIPTION

[0027] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0028] The terms "first", "second", etc. in the specification and claims of this application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or N. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0029] As described in the background technology section, in the existing scheme, the hardware cost is relatively high to obtain a portrait photo with good portrait lighting and shadow effects. In order to solve the above problems, the embodiment of the present application provides an image processing method and a device thereof. By identifying each photographed object in a first image including at least one photographed object, a regional image corresponding to each photographed object can be obtained. For each photographed object, fill light intensity information for fill light on the photographed object can be determined according to the regional image corresponding to the photographed object. For each photographed object, fill light is performed on the photographed object according to the fill light intensity information for fill light on the photographed object, and a second image after fill light is performed on each photographed object can be obtained. In this way, through the image processing method provided by the embodiment of the present application, there is no need to add other fill light equipment to fill light the photographed object to obtain a portrait photo with good portrait lighting and shadow effects. Not only does it reduce the hardware cost, but it also solves the problem of poor portrait lighting and shadow effects caused by insufficient lighting conditions or hardware limitations of electronic equipment, thereby achieving the effect of high-quality portrait photos at a low cost, while improving the film formation rate and image aesthetics of the electronic device.

[0030] The technical solution of the embodiment of the present application can be applied to the scene of processing portrait photos with poor lighting and shadow effects to obtain portrait photos with better lighting and shadow effects.

[0031] The image processing method provided in the embodiment of the present application is described in detail below through specific embodiments and their application scenarios in conjunction with the accompanying drawings.

[0032] Figure 1 It is a flow chart of an image processing method provided in an embodiment of the present application. The executor of the image processing method may be an electronic device, which may be but is not limited to a personal computer (PC), a smart phone, a tablet computer or a personal digital assistant (PDA).

[0033] like Figure 1 As shown, the image processing method provided in the embodiment of the present application may include S110-S140.

[0034] S110: Acquire a first image.

[0035] The first image may be an image captured by an electronic device having an image capture function, or may be an image stored locally or downloaded from the Internet via a network, and the first image may include at least one photographed object. The photographed object here may be a photographed object, such as a person.

[0036] S120: Identify each photographed object in at least one photographed object.

[0037] In some embodiments of the present application, each photographed object may be identified from the first image, and a regional image corresponding to each photographed object may be obtained.

[0038] In some embodiments of the present application, in order to accurately obtain the regional image corresponding to each photographed object, S120 may specifically include:

[0039] Respectively obtain position information of each photographed object in the mask image of the first image;

[0040] According to the position information of each photographed object, each photographed object is extracted from the mask image of the first image to obtain a regional image corresponding to each photographed object.

[0041] The mask image of the first image may be an image obtained by extracting the foreground image from the first image.

[0042] In some embodiments of the present application, the position information of each photographed object in the mask image of the first image can be obtained respectively, and then each photographed object can be cut out from the mask image of the first image according to the position information of each photographed object, so as to obtain the regional image corresponding to each photographed object.

[0043] In some embodiments of the present application, the first image can be processed by image processing technology to obtain a mask image of the first image. For example, the first image can be converted into a binary image by threshold segmentation, and the foreground and background can be distinguished by setting a threshold to obtain the mask image of the first image. It can also be through edge detection, such as identifying the edges in the first image by an edge detection algorithm (such as the Canny edge detection algorithm, etc.), and then using the edge information to generate a mask to obtain the mask image of the first image. It can also be obtained by regional growing. The specific method of obtaining the mask image of the first image can be set according to user needs and is not limited in the embodiments of the present application.

[0044] In an embodiment of the present application, by respectively acquiring the position information of each photographed object in the mask image of the first image, and then respectively extracting each photographed object from the mask image of the first image according to the position information of each photographed object, the regional image corresponding to each photographed object can be accurately obtained.

[0045] S130: Determine fill light intensity information of each photographed object according to the area image corresponding to each photographed object.

[0046] The fill light intensity information of each photographed object may be the intensity information of the fill light for the photographed object.

[0047] In some embodiments of the present application, to improve the efficiency of determining fill light intensity information of each photographed object, S130 may specifically include:

[0048] Select a target subject from each subject;

[0049] The fill light intensity information of each object is determined according to the area image corresponding to the target object.

[0050] The target photographing object may be a photographing object selected from at least one photograph of the first image.

[0051] In some embodiments of the present application, one photographic object may be selected from at least one photographic object as a target photographic object, and then fill light intensity information for fill light to each photographic object may be determined based on a regional image corresponding to the target photographic object.

[0052] In an embodiment of the present application, by selecting a photographic object from at least one photographic object as the target photographic object, and then using the regional image of the target photographic object to respectively determine the fill light intensity information for each photographic object, there is no need to process the regional image of each photographic object, thereby improving the efficiency of determining the fill light intensity information for each photographic object.

[0053] In some embodiments of the present application, in order to improve the flexibility of selecting the target photographing object, the selecting the target photographing object from each photographing object may specifically include:

[0054] When the number of photographed objects in the first image is 1, determining that the photographed object is a target photographed object;

[0055] When the number of photographing subjects in the first image is greater than 1, the target photographing subject is determined according to the size information of the face area of ​​each photographing subject.

[0056] In some embodiments of the present application, different methods of selecting a target subject may be provided according to the number of subjects in the first image. Specifically, when the number of subjects in the first image is 1, the subject may be determined as the target subject. When the number of subjects in the first image is greater than 1, the target subject may be determined from multiple subjects according to the size information of the face area of ​​each subject.

[0057] In the embodiments of the present application, different methods are selected to determine the target photographic object according to the number of photographic objects in the first image, thereby improving the flexibility of selecting the target photographic object.

[0058] In some embodiments of the present application, when the number of photographed objects in the first image is greater than 1, in order to prevent the light and shadow effects between the photographed objects in the image after fill light from changing too much, determining the target photographed object according to the size information of the face area of ​​each photographed object may specifically include:

[0059] Selecting maximum size information from the size information of the face area of ​​each photographed subject;

[0060] The photographic object corresponding to the maximum size information is used as the target photographic object.

[0061] The maximum size information may be the largest size information among the size information of the face area of ​​each photographed object.

[0062] In some embodiments of the present application, maximum size information can be selected from the size information of the face area of ​​each photographed object, and then the photographed object corresponding to the maximum size information is used as the target photographed object. This is because the target photographed object occupies the largest space in the first image, and the target photographed object can be used as a reference to determine the fill light intensity information for fill light of other photographed objects based on the fill light intensity information for fill light used for it, so that the light and shadow effects between the photographed objects in the image after fill light are obtained will not be too abrupt.

[0063] In an embodiment of the present application, the subject corresponding to the maximum size information in the size information of the facial area of ​​each subject is used as the target subject, so that the light and shadow effects between the subjects in the image after fill-in will not be too abrupt, thereby obtaining a second image with better light and shadow effects.

[0064] In some embodiments of the present application, in order to accurately determine the fill light intensity information of each photographed object, determining the fill light intensity information of each photographed object according to the regional image corresponding to the target photographed object may specifically include:

[0065] Gray-scale the area image corresponding to the target photographed object to obtain a gray-scaled area image;

[0066] Sort the brightness values ​​of each pixel in the grayscale area image in ascending order to obtain a brightness value sequence;

[0067] Obtaining a first normalized brightness value sequence according to the position sequence of each brightness value in the brightness value sequence;

[0068] According to each brightness value in the brightness value sequence and the position sequence of each brightness value in the brightness value sequence, a second normalized brightness value sequence is obtained;

[0069] Determining fill light intensity information of the target photographed object according to the first normalized brightness value sequence and the second normalized brightness value sequence;

[0070] The fill light intensity information of each photographed object is determined according to the fill light intensity information of the target photographed object.

[0071] The brightness value sequence may be a sequence obtained by sorting the brightness values ​​of each pixel in the grayscale area image in ascending order.

[0072] The first normalized brightness value sequence may be a sequence consisting of brightness values ​​after each brightness value is normalized, obtained according to the bit sequence of each brightness value in the brightness value sequence.

[0073] The second normalized brightness value sequence may be a sequence consisting of brightness values ​​after each brightness value is normalized, obtained according to each brightness value in the brightness value sequence and the position sequence of each brightness value in the brightness value sequence.

[0074] In some embodiments of the present application, the concept of the gini coefficient may be used to calculate the light ratio of the target object, and the light ratio may be used as fill light intensity information for fill light to the target object.

[0075] Specifically, the region image corresponding to the target object is grayscaled to obtain the grayscaled region image, that is, the three-channel region image corresponding to the target object is converted into a one-channel region image. Since the region image corresponding to the target object is an RGB image, which is a three-channel image, if it is to be converted into a one-channel region image, it can be grayscaled. For example, the region image corresponding to the target object is an image of (512, 512, 3), and after grayscale, the obtained image is (512, 512).

[0076] Then, the brightness values ​​of each pixel in the grayscale area image are sorted in ascending order to obtain a brightness value sequence, that is, the brightness value sequence is a 1-row and 1-column sequence formed by the brightness values ​​of each pixel, in which each brightness value is arranged in ascending order. For example, there are 512×512=262144 pixels in the grayscale area image (512,512), and then the brightness values ​​of these 262144 pixels are sorted in ascending order to form a brightness value sequence w(j)={a1,a2,a3,…a 262144}.

[0077] Then, according to the position order of each brightness value in the brightness value sequence, the first normalized brightness value sequence x can be obtained according to the following formula (1): i :

[0078]

[0079] In the above formula (1), i is the position sequence of each brightness value in the brightness value sequence, i∈(1,2,…262144), and the initial value of i is 1.

[0080] That is to say, first let i = 1, and according to formula (1), we get the value of x1, then let i = 2, and according to formula (1), we get the value of x2, then let i = 3, and so on, until we let i = 262144, and according to formula (1), we get x 262144 Then, x1, x2, ..., x 262144 A sequence is formed to obtain a first normalized brightness value sequence, wherein each value in the first normalized brightness value sequence is between (0, 1).

[0081] According to each brightness value in the brightness value sequence and the position order of each brightness value in the brightness value sequence, the second normalized brightness value sequence y can be obtained according to the following formula (2): i :

[0082]

[0083] According to the above formula (2), we can sum the first i brightness values ​​and then find the quotient with the sum of each brightness value in the brightness value sequence to get y i .

[0084] That is to say, let i = 1, and according to formula (1), we get Then get Then let i = 1, according to formula (1), we get Then get Then let i = 3, and so on, until i = 262144, according to formula (2), we get y 262144 Then, y1, y2, ..., y 262144 A sequence is formed to obtain a second normalized brightness value sequence, and each value in the second normalized brightness value sequence is between (0, 1).

[0085] Then, fill light intensity information for fill light to the target object can be obtained according to the first normalized brightness value sequence and the second normalized brightness value sequence. Specifically, each value in the first normalized brightness value sequence can be matched with each value in the second normalized brightness value sequence in order, that is, x1 is matched with y1, x2 is matched with y2, and so on. Then, as Figure 2 As shown, each value in the first normalized brightness value sequence is used as the horizontal coordinate, and each value in the second normalized brightness value sequence is used as the vertical coordinate. Figure 2In the figure, curve 21 is a curve fitted by the second normalized brightness value sequence, and the area between curve 21 and the horizontal axis is S B , since each value in the first normalized brightness value sequence and each value in the second normalized brightness value sequence are between (0,1), the area of ​​the triangle enclosed by the first normalized brightness value sequence and the second normalized brightness value sequence can be calculated, and then according to the area and S B , we can find the area S A , and then according to the following formula (3), the fill light intensity information G of the target object can be obtained:

[0086]

[0087] Then, the fill light intensity information of each photographic object may be determined according to the fill light intensity information of the target photographic object.

[0088] In an embodiment of the present application, the fill light intensity information for fill light for each photographic object is determined by a software method, which saves hardware costs. In addition, the fill light intensity information for fill light for the target object can be obtained by calculating the light ratio of the target object, and then the fill light intensity information for fill light for each object can be obtained respectively. In this way, the fill light intensity information for fill light for each object can be accurately determined, thereby improving the accuracy of determining the fill light intensity information for fill light for each object.

[0089] In some embodiments of the present application, when the number of photographed objects in the first image is greater than 1, in order to prevent the light and shadow effects between the photographed objects in the image after fill light from changing too much and to improve the efficiency of determining the fill light intensity information of each photographed object, the method of determining the fill light intensity information of each photographed object according to the fill light intensity information of the target photographed object may specifically include:

[0090] The fill light intensity information of the target photographed object is determined as the fill light intensity information corresponding to each photographed object.

[0091] In some embodiments of the present application, after obtaining the fill light intensity information for fill light of the target subject, the fill light intensity information for fill light of the target subject can be determined as the fill light intensity information for fill light of each subject respectively. This ensures that the portrait of each subject in the same image has the same fill light intensity information and does not change too much.

[0092] In the embodiment of the present application, the fill light intensity information for fill light of the target photographic object is determined as the fill light intensity information for fill light of each photographic object, so that the light and shadow effects between the photographic objects in the image after fill light are not too jumpy. And the fill light intensity information for fill light of the target photographic object is directly determined as the fill light intensity information for fill light of each photographic object, so that there is no need to repeatedly calculate the fill light intensity information of other photographic objects, thereby improving the efficiency of determining the fill light intensity information of each photographic object.

[0093] It should be noted that, in order to further improve the light and shadow effect of the second image, the fill light intensity information of each photographed object can also be determined by the above-mentioned method for determining the fill light intensity information of the target photographed object, so that the light and shadow effect of the second image can be further improved. Specifically, the fill light intensity information of each photographed object is calculated and determined in turn, or the fill light intensity information of the target photographed object is used as the fill light intensity information of other photographed objects. It can be selected according to user needs and is not limited in the embodiments of the present application.

[0094] S140. Perform fill light on corresponding objects in at least one photographed object according to the fill light intensity information to obtain a second image.

[0095] The second image may be an image with better light and shadow effects obtained after fill-in lighting is performed on each photographed object in the first image.

[0096] In some embodiments of the present application, in order to improve the fill light efficiency for each photographed object, S140 may specifically include:

[0097] Determining a fill light intensity information image of the first image according to the fill light intensity information of each photographed object;

[0098] splicing image data of the first image, image data of the mask image of the first image, image data of a skin image of at least one photographed object in the first image, and image data of the fill light intensity information image to obtain a first image data vector;

[0099] Inputting the first image data vector into a fill light model, and performing fill light on each photographed object based on the fill light model to obtain a second image data vector;

[0100] Based on the second image data vector, a second image is determined.

[0101] The fill light intensity information image may be an image of fill light intensity information obtained according to the fill light intensity information of each photographed object.

[0102] The skin image of at least one photographed subject may be an image of the skin area of ​​each photographed subject. For example, in a first image, there are the upper body area of ​​subject 1 and the upper body area of ​​subject 2, subject 1 is wearing short sleeves, and subject 2 is wearing long sleeves and wearing sunglasses. Then, in the first image, the facial skin of subject 1 and the skin of the arms exposed outside the short sleeves can be seen, and the facial skin of subject 2 except the skin covered by the sunglasses can be seen. Then, the image composed of the facial skin of subject 1, the skin of the arms of subject 1 exposed outside the short sleeves, and the facial skin of subject 2 except the skin covered by the sunglasses is the skin image of at least one photographed subject.

[0103] The first image data vector may be an image data vector formed by splicing the image data of the first image, the image data of the mask image of the first image, the image data of the skin image of at least one object in the first image, and the image data of the fill light intensity information image. For example, the first image data vector may be a data vector of [1,5,1024,1024].

[0104] The second image data vector may be an image data vector output based on the fill light model after the first image data vector is input into the fill light model. The second image data vector may be a data vector of [1, 3, 1024, 1024].

[0105] The fill light model may be a pre-trained model for performing fill light on each photographed object to obtain a second image data vector. The fill light model may be, but is not limited to, a neural network model, a support vector machine model, a decision tree model, or the like based on deep learning.

[0106] In some embodiments of the present application, a fill light intensity information image of the first image can be obtained based on the fill light intensity information of each photographed object, and then the image data of the first image, the image data of the mask image of the first image, the image data of the skin image of at least one photographed object in the first image, and the image data of the fill light intensity information image can be spliced ​​to obtain a first image data vector, and the first image data vector is input into a fill light model to obtain a second image data vector, and then the second image can be determined based on the second image data vector.

[0107] It should be noted that before splicing the image data of the first image, the image data of the mask image of the first image, the image data of the skin image of at least one subject in the first image, and the image data of the fill light intensity information image, the weights of the first image, the mask image of the first image, the skin image, and the fill light intensity information image can be set respectively, and then the image data of the first image, the image data of the mask image of the first image, the image data of the skin image of at least one subject in the first image, and the image data of the fill light intensity information image can be spliced ​​according to the weights of the first image, the mask image of the first image, the skin image, and the fill light intensity information image respectively.

[0108] The weights of the specific first image, the mask image of the first image, the skin image and the fill light intensity information image can be set based on prior experience or other reference factors. The values ​​of the weight values ​​corresponding to the specific first image, the mask image of the first image, the skin image and the fill light intensity information image can be set according to user needs and are not limited in the embodiments of the present application.

[0109] In the embodiments of the present application, fill light is provided to each photographed object through a fill light model, thereby improving the fill light efficiency of each photographed object.

[0110] It should be noted that since the fill light model processes image data, that is, the fill light model processes graphics, the format of the data to be processed should be consistent with the data format that can be processed by the graphics processor, that is, the above-mentioned first image data and second image data vectors can both be the first vector format corresponding to the graphics processor, that is to say, the first vector format can be a vector format that can be processed by the graphics processor, such as a tensor format.

[0111] In some embodiments of the present application, in order to avoid the user having to adjust the obtained image in such a way that the format and size of the obtained image with better light and shadow effects do not meet the requirements, the determining of the second image according to the second image data vector may specifically include:

[0112] Converting the format of the second image data vector to obtain a second image data vector in a second vector format;

[0113] The second image data vector in the second vector format is upsampled to obtain a second image.

[0114] The second vector format may be a vector format corresponding to the processor, such as a numpy format.

[0115] In some embodiments of the present application, since the number of channels and size information of the second image data vector output by the fill light model are different from those of the first image, it is necessary to adjust the number of channels and size information of the second image data vector to be the same as those of the first image. However, this process is performed in a processor. Since the vector format of the data that the processor can process is different from the vector format of the data processed by the graphics processor, it is necessary to first convert the format of the second image data vector into a second image data vector in a second vector format that the processor can process, and then upsample the second image data vector in the second vector format to obtain a second image that is consistent with the size information and number of channels of the first image.

[0116] In an embodiment of the present application, by converting the format of the second image data vector to obtain a second image data vector in a second vector format, and then upsampling the second image data vector in the second vector format, a second image that is consistent with the size information and number of channels of the first image can be obtained. This ensures that the obtained second image with better light and shadow effects is consistent with the size information of the first image, and the user does not need to adjust the obtained image because the format and size of the obtained image with better light and shadow effects do not meet the requirements, thereby simplifying user operations.

[0117] In some embodiments of the present application, in order to improve the training efficiency of the preset fill light model, before inputting the first image data vector into the fill light model, and performing fill light on each photographed object based on the fill light model to obtain the second image data vector, the above method may further include:

[0118] Obtaining a plurality of first training samples;

[0119] Determine a sample fill light intensity information image of the first sample image according to the brightness value of each pixel point in the area image corresponding to each sample photographed object in the first sample image;

[0120] splicing the image data of the first sample image, the image data of the mask image of the first sample image, the image data of the skin image of each sample photographed object in the first sample image, and the image data of the sample fill light intensity information image to obtain a first sample image data vector;

[0121] Down-sampling the image data of the second sample image to obtain a second sample image data vector;

[0122] Obtain a second training sample according to the first sample image data vector and the second sample image data vector;

[0123] The preset fill light model is trained based on the multiple second training samples to obtain a fill light model.

[0124] The first training sample may be an initially acquired sample used to train a preset fill light model.

[0125] Each of the above-mentioned first training samples may include a first sample image and a second sample image. The first sample image may include at least one sample photographed object. The sample photographed object may be a photographed object used to obtain a sample for training a preset fill light model, such as a person. The sample photographed object may be the same as or different from the photographed object in the first image in the above-mentioned S110, which is not limited in the embodiment of the present application.

[0126] The first sample image mentioned above may be an image captured before fill-in lighting is applied to the sample photographed object, and the second sample image may be an image captured after fill-in lighting is applied to the sample photographed object, that is, the second sample image may be an image with perfect light and shadow effects.

[0127] In one example, reference Figure 3 Before using the fill light 31 to fill in the light of the sample photographing object 32, the sample photographing object 32 can be photographed by using the image acquisition device 33 to obtain a first sample image, and then the sample photographing object 32 can be kept still, the fill light 31 can be turned on, and the sample photographing object 32 can be filled in the light of the fill light 31, and then the sample photographing object 32 can be photographed by using the image acquisition device 33 to obtain a second sample image.

[0128] Referring to the above process, different shooting objects may be changed to collect several pairs of first sampling images and second sampling images to obtain multiple training samples.

[0129] It should be noted that in Figure 3 In the figure, the number of fill lights 31, the number of sample photographing objects 32 and the number of image acquisition devices 33 are only examples and not limitations, that is, the number of fill lights 31, the number of sample photographing objects 32 and the number of image acquisition devices 33 can be set according to user needs and are not limited in the embodiments of the present application.

[0130] It should be noted that before using the fill light and image acquisition equipment, the fill light intensity of the fill light has been adjusted based on prior experience, and the placement angle, color temperature, camera parameters, etc. of the image acquisition equipment have all been adjusted.

[0131] For each first training sample, the sample fill light intensity information image of the first sample image therein may be an image of fill light intensity information for fill light of each sample photographed object obtained according to the brightness value of each pixel point in the regional image corresponding to each sample photographed object in the first sample image. The specific determination of the sample fill light intensity information image is consistent with the determination method of the fill light intensity information image in the above embodiment, and will not be repeated here.

[0132] The first sample image data vector may be a data vector obtained by splicing the image data of the first sample image, the image data of the mask image of the first sample image, the image data of the skin image of each sample photographed object in the first sample image, and the image data of the sample fill light intensity information image. The method for determining the first sample image data vector may refer to the method for determining the first image data vector in the above embodiment, and will not be described in detail here.

[0133] The second sample image data vector may be an image data vector obtained by downsampling the image data of the second sample image, and the dimension of the second sample image data vector is the same as the dimension of the first sample image data vector. Specifically, the dimension of the second sample image data vector and the dimension of the first sample image data vector are consistent with the dimension of the second image data vector and the dimension of the first image data vector in the above embodiment.

[0134] The second training sample may be a training sample obtained after a series of processing on the first training sample and finally used to train the preset fill light model. The second training sample may be a training sample obtained based on the first sample image data vector and the second sample image data vector.

[0135] The preset fill light model may be a preset fill light model that has not been trained yet. After the preset fill light model is trained, the above-mentioned fill light model may be obtained.

[0136] In some embodiments of the present application, for each first training sample, a corresponding second training sample can be obtained through the above processing method, and then the preset fill light model is trained by the second training sample corresponding to each first training sample to obtain the above fill light model.

[0137] In an embodiment of the present application, by processing each first training sample initially acquired, a plurality of second training samples of image data vectors based on the image can be obtained, and then the preset fill light model can be trained based on the plurality of second training samples to obtain the fill light model. In this way, the preset fill light model is trained by the image data vector rather than directly using the image to train the preset fill light model, thereby avoiding an image that is too large and increasing the processing burden of the preset fill light model. Training the preset fill light model by the image data vector reduces the computational power of the preset fill light model and improves the training efficiency of the preset fill light model.

[0138] In some embodiments of the present application, in order to improve the training effect of the preset fill light model, after obtaining the plurality of first training samples, the above method may further include:

[0139] Preprocessing the plurality of first training samples to obtain the plurality of preprocessed first training samples;

[0140] Determining the sample fill light intensity information image of the first sample image according to the brightness value of each pixel in the regional image corresponding to each sample photographed object in the first sample image may specifically include:

[0141] For each of the preprocessed first training samples, a sample fill light intensity information image of the first sample image is determined according to the brightness value of each pixel in the area image corresponding to each sample photographed object in the first sample image in the first training sample.

[0142] In some embodiments of the present application, after obtaining multiple first training samples, the quality of the first training samples may be poor, and they need to be preprocessed. The specific preprocessing may include but is not limited to eliminating first training samples with a large deviation between the first sample image and the second sample image, and adjusting first training samples with a small deviation between the first sample image and the second sample image.

[0143] In some embodiments of the present application, among multiple first training samples, there may be some training samples whose first sample images and second sample images have large deviations, such as movement of the photographed object, movement of the image acquisition device, untimely signal transmission, failure to turn on the fill light in time, incorrect setting of image acquisition device parameters, incorrect setting of fill light parameters, etc., which lead to large deviations between the first sample image and the second sample image. Therefore, these training samples need to be eliminated or processed to obtain better training samples.

[0144] Specifically, the first training samples with large deviations between the first sample image and the second sample image among the multiple first training samples can be removed. For example, for a certain training sample, after the first sample image is captured, the movement of the photographed object or the movement of the image capture device causes the position of the photographed object in the captured second sample image to have a large deviation relative to the first sample image, so that the position and brightness value of each pixel point of the first sample image have a large deviation from the position and brightness value of each pixel point of the second sample image, which leads to a mismatch between the first sample image data vector and the second sample image data vector in the second training sample obtained based on the first training sample, thereby affecting the training effect of the preset fill light model, so the first training sample needs to be removed.

[0145] After removing the first training samples with larger deviations between the first sample image and the second sample image from the plurality of first training samples, the remaining first training samples can be processed as follows: Figure 4 The data processing flow shown in the figure is used for processing. Figure 4 As shown, the data processing flow for performing data processing on the remaining first training samples includes S410 - S440 .

[0146] S410, determining whether the first training sample meets the preset training condition, if so, executing S440, if not, executing S420.

[0147] The first training sample in S410 refers to the first training sample remaining after removing the first training sample with a larger deviation between the first sample image and the second sample image from the acquired multiple first training samples.

[0148] The above-mentioned preset training conditions may be conditions for whether the pre-set first training sample can be directly used to train the preset fill light model. The preset training conditions may include but are not limited to whether the quality of the first sample image and the second sample image in the remaining first training samples meets the requirements, such as whether each pixel in the first sample image and each pixel in the second sample image in the first training sample are aligned, because when the first sample image and the second sample image in a certain first training sample are collected, the sample shooting object remains stationary, and only the fill light is added. Therefore, for the first sample image and the second sample image in a certain first training sample, the pixels in the two images should be alignable.

[0149] S420: Register and align the first sample image and the second sample image.

[0150] In the remaining first training samples, if the deviation between the first sample image and the second sample image in a certain first training sample is small, the deviation between the first sample image and the second sample image can be adjusted by the registration and alignment algorithm, without eliminating the first training sample. For example, for a certain first training sample, after collecting the first sample image, when collecting the second sample image, the sample subject blinked, that is, the eyes of the sample subject in the second sample image are closed, but the eyes of the sample subject in the first sample image are open, and the deviation between the first sample image and the second sample image is small. In this way, the deviation between the first sample image and the second sample image can be corrected by the registration and alignment algorithm, without eliminating the first training sample.

[0151] S430: Remove the black border region in the first sample image and the black border region in the second sample image after the registration and alignment.

[0152] The image after the registration and alignment algorithm may have black edges, so the black edges of the image can be cropped through a cropping algorithm.

[0153] S440: Calculate a mask image of the first sample image and a mask image of the second sample image.

[0154] After preprocessing the first training sample, a mask image of the first sample image and a mask image of the second sample image may be calculated, and the mask image of the first sample image and the mask image of the second sample image are used for subsequent training of a preset fill light model.

[0155] In an embodiment of the present application, a problematic first training sample may be preprocessed to obtain a preprocessed first training sample, which may improve the training accuracy of the preset fill light model.

[0156] In some embodiments of the present application, if the subject is wearing glasses, then when the image is captured, there will be a hard shadow cast by the glasses on the face of the subject in the image. This hard shadow cannot be resolved by fill light. Therefore, when training the preset fill light model, parameters for shadow degradation of the data can be added to improve the generalization ability of the preset fill light model.

[0157] In addition, due to the parameter setting problem of the preset fill light model, there may be a problem of image blurring corresponding to the image data vector obtained after processing by the preset fill light model. Therefore, when training the preset fill light model, parameters for blurring and degrading the data can be added to improve the generalization ability of the preset fill light model.

[0158] It should be noted that when setting the parameters for shadow degradation of data and the parameters for blur degradation of data, the parameters for shadow degradation of data and the parameters for blur degradation of data can be set according to a certain ratio. The specific ratio can be set according to user experience and is not limited in the embodiments of the present application.

[0159] In some embodiments of the present application, since model training requires a large number of training samples for different situations, for example, if the photographed object in the first image is not upright but rotated at a certain angle, if the preset fill light model is not trained using the training samples of this situation, the fill light model cannot process the first image, or the processing effect of the first image is not good. Therefore, in order to increase the robustness of the fill light model, data augmentation can be performed on the pre-processed multiple first training samples to obtain a larger number of training samples containing more situations. Specific data augmentation can be, for example, rotating, mirroring, color adjusting, etc. the first sample image and the second sample image in the first training sample.

[0160] It should be noted that when performing data augmentation on a first training sample, the first sample image and the second sample image in the first training sample need to be subjected to the same data augmentation. For example, if the first sample image in a first training sample is rotated 60° to the left, the second sample image in the first training sample also needs to be rotated 60° to the left.

[0161] It should be noted that, for the multiple first training samples after preprocessing, different proportions of the first training samples can be selected for data augmentation in different ways. For example, there are 10 first training samples in total, and 30% of the first training samples can be selected for rotation data augmentation, 30% of the first training samples can be selected for mirroring data augmentation, and 40% of the first training samples can be selected for color adjustment data augmentation. The specific proportions of the first training samples selected for different augmentation methods can be set according to user needs and are not limited in the embodiments of the present application.

[0162] In some embodiments of the present application, in order to improve the efficiency of fill light for images shot under poor lighting conditions, the preset fill light model is trained based on multiple groups of second training samples to obtain the fill light model, which may specifically include:

[0163] For each second training sample, input the second training sample into a preset fill light model to obtain a predicted image data vector;

[0164] Determining a loss function value of a preset fill light model according to the predicted image data vector and the second sample image data vector;

[0165] When the loss function value does not meet the preset conditions, the model parameters of the preset fill light model are adjusted, and the preset fill light model with adjusted model parameters is trained using multiple second training samples until the loss function value meets the preset conditions to obtain the fill light model.

[0166] The predicted image data vector may be an image data vector after fill lighting is performed on the image represented by the first sample image data vector after the second training sample is input into a preset fill lighting model and the first sample image data vector is processed based on the preset fill lighting model.

[0167] The preset condition may be a preset condition that the loss function value must satisfy in order to stop training the preset fill light model. For example, the preset condition may be that the loss function value is less than a certain threshold.

[0168] In some embodiments of the present application, for each second training sample, the second training sample can be input into a preset fill light model to output a predicted image data vector, and then the loss function value of the preset fill light model can be determined based on the predicted image data vector and the second sample image data vector. When the loss function value does not meet the preset conditions, the model parameters of the preset fill light model can be adjusted, and then the preset fill light model with adjusted parameters can be continued to be trained using multiple second training samples until the loss function value meets the preset conditions. Then, the training of the preset fill light model can be stopped to obtain the fill light model.

[0169] In an embodiment of the present application, a fill light model can be obtained by continuously training a preset fill light model using multiple second training samples, so that the first image can be subsequently processed based on the fill light model to obtain a second image, thereby improving the efficiency of fill light for images taken due to poor lighting conditions.

[0170] In some embodiments of the present application, in order to improve the robustness of the preset fill light model, after obtaining the plurality of first training samples, the above method may further include:

[0171] For each first training sample, a sample shadow mask image and a sample specular reflection image are obtained respectively according to a first sample image and a second sample image in the first training sample;

[0172] The obtaining of the second training sample according to the first sample image data vector and the second sample image data vector may specifically include:

[0173] A second training sample is obtained according to the first sample image data vector, the second sample image data vector, the sample shadow mask image, the sample specular reflection image, the second sample image and the first sample image.

[0174] The sample shadow mask image may be a mask image obtained based on the first sample image and the second sample image and used to characterize the shadow area of ​​the first sample image relative to the second sample image.

[0175] The sample specular reflection image may be an image obtained based on the first sample image and the second sample image and used to characterize the specular reflection condition of the portrait of the first sample image relative to the second sample image.

[0176] In some embodiments of the present application, for each first training sample, a sample shadow mask image and a sample mirror reflection image can be obtained respectively according to the first sample image and the second sample image in the first training sample, and the sample shadow mask image and the sample mirror reflection image can roughly reflect the approximate distribution of the shadow on the portrait.

[0177] Specifically, for each first training sample, a sample shadow mask image S for characterizing the shadow area of ​​the first sample image relative to the second sample image can be obtained according to the first sample image and the second sample image according to the following formula (4):

[0178] S=max(min(1-I in gray / I gt ),0) (4)

[0179] In the above formula (4), I in gray is the grayscale image of the first sample image, I gt is the grayscale image of the second sample image.

[0180] For each first training sample, a sample specular reflection image D for characterizing the specular reflection of the portrait of the first sample image relative to the second sample image can be obtained according to the first sample image and the second sample image according to the following formula (5):

[0181] D=max(min(1-I gt / I in gray ),0) (5)

[0182] In the above formula (5), I in gray is the grayscale image of the first sample image, I gt is the grayscale image of the second sample image.

[0183] The sample shadow mask image and sample mirror reflection image obtained above can be used to monitor the shadow area or the area with low brightness value in the portrait image (the first sample image and the second sample image). For the preset fill light model, the ability to identify the details and structural information of the dark area or the strong shadow area in the portrait image is relatively weak. Therefore, the sample shadow mask image obtained by the above formula (4) for characterizing the shadow area of ​​the first sample image relative to the second sample image, and the sample mirror reflection image obtained by the above formula (5) for characterizing the portrait mirror reflection of the first sample image relative to the second sample image, together with the first sample image data vector and the second sample image data vector, can be used as the second training sample for training the preset fill light model, so as to enhance the preset fill light model's recognition of the details and structural information of the dark area or the strong shadow area in the portrait image.

[0184] In an embodiment of the present application, for each first training sample, a sample shadow mask image and a sample mirror reflection image are respectively obtained according to the first sample image and the second sample image in the first training sample, and then the sample shadow mask image and the sample mirror reflection image are also used as second training samples for training the preset fill light model, which can enhance the preset fill light model's details and structural information of dark areas or strong shadow areas in portrait images, so as to improve the robustness of the preset fill light model.

[0185] In some embodiments of the present application, in order to improve the generalization ability and robustness of the preset fill light model, the step of inputting the second training sample into the preset fill light model to obtain the predicted image data vector may specifically include:

[0186] Inputting the second training sample into the preset fill light model to obtain a predicted image data vector and a predicted image;

[0187] The step of determining the loss function value of the preset fill light model according to the predicted image data vector and the second sample image data vector includes:

[0188] Determine a first loss function value of a preset fill light model according to the predicted image, the sample shadow mask image, the sample specular reflection image, and the second sample image;

[0189] Determine a second loss function value of a preset fill light model according to the predicted image data vector and the second sample image data vector;

[0190] A loss function value of a preset fill light model is determined according to the first loss function value and the second loss function value.

[0191] The predicted image may be an image after fill-lighting the first sample image, which is output after the second training sample is input into a preset fill-light model and the first sample image is processed based on the preset fill-light model.

[0192] The first loss function value may be a loss function value of a preset fill light model obtained based on the predicted image, the sample shadow mask image, the sample specular reflection image, the second sample image, and the first sample image.

[0193] The second loss function value may be a loss function value of a preset fill light model obtained based on the predicted image data vector and the second sample image data vector.

[0194] In some embodiments of the present application, a first loss function value of a preset fill light model can be obtained based on a predicted image, a sample shadow mask image, a sample mirror reflection image, a second sample image and a first sample image, and a second loss function value of the preset fill light model can be obtained by predicting an image data vector and a second sample image data vector. Then, the loss function value of the preset fill light model is determined by combining the first loss function value and the second loss function value, so as to comprehensively evaluate the training of the preset fill light model.

[0195] In an embodiment of the present application, the preset fill light model's ability to recognize details and structural information of dark areas or strong shadow areas in a portrait image can be enhanced based on the predicted image, the sample shadow mask image, the sample mirror reflection image, the second sample image, and the first sample image. The ability of the preset fill light model to fill light the portrait image can be enhanced by predicting the image data vector and the second sample image data vector. The generalization ability and robustness of the preset fill light model can be improved by utilizing a variety of different loss function values.

[0196] In some embodiments of the present application, the above-mentioned first loss function value may include a first sub-loss function value and a second sub-loss function value.

[0197] The above-mentioned first sub-loss function value can be the loss function value of the preset fill light model obtained based on the predicted image, the second sample image and the sample shadow mask image. The first sub-loss function value can be used to characterize the difference between the pixel points of the predicted image and the pixel points of the second sample image.

[0198] The second sub-loss function value may be a loss function value of a preset fill light model obtained based on the predicted image, the second sample image and the sample specular reflection image. The second sub-loss function value may be used to characterize the difference between feature information of the predicted image and feature information of the second sample image.

[0199] In order to improve the generalization ability and robustness of the preset fill light model, determining the first loss function value of the preset fill light model according to the predicted image, the sample shadow mask image, the sample specular reflection image and the second sample image may specifically include:

[0200] Determine a first sub-loss function value of a preset fill light model according to the predicted image, the second sample image and the sample shadow mask image;

[0201] A second sub-loss function value of a preset fill light model is determined according to the predicted image, the second sample image and the sample specular reflection image.

[0202] In some embodiments of the present application, the first sub-loss function L for characterizing the difference between each pixel in the predicted image and each pixel in the second sample image can be calculated according to the predicted image, the second sample image and the sample shadow mask image according to the following formula (6): pixel :

[0203]

[0204] In the above formula (6), x is the second sample image, is the predicted image, and S is the sample shadow mask image obtained by the above formula (4).

[0205] The second sub-loss function value L for characterizing the difference between the feature information of the predicted image and the feature information of the second sample image can be calculated according to the following formula (7) based on the predicted image, the second sample image and the sample mirror reflection image: vgg :

[0206]

[0207] In the above formula (7), D is the sample mirror reflection image obtained by the above formula (5), and VGG(·) is the feature information captured based on the VGG network model.

[0208] In an embodiment of the present application, the first sub-loss function value of the preset fill light model is determined by the predicted image, the second sample image and the sample shadow mask image, and the second sub-loss function value of the preset fill light model can be determined based on the predicted image, the second sample image and the sample mirror reflection image. In this way, the generalization ability and robustness of the preset fill light model can be improved by utilizing a variety of different loss function values.

[0209] In some embodiments of the present application, the above-mentioned second loss function value may include a third sub-loss function value and a fourth sub-loss function value.

[0210] The third sub-loss function value mentioned above may be a loss function value of a preset fill light model obtained according to the predicted image data vector and the second sample image data vector. The third sub-loss function value may be used to characterize the authenticity of the predicted image data vector.

[0211] The fourth sub-loss function value may be a loss function value of a preset fill light model obtained based on the predicted image data vector and the second sample image data vector. The fourth sub-loss function value may be used to characterize the similarity between the predicted image data vector and the second sample image data vector.

[0212] In order to improve the generalization ability and robustness of the preset fill light model, determining the first loss function value of the preset fill light model according to the predicted image data vector and the second sample image data vector may specifically include:

[0213] According to the predicted image data vector and the second sample image data vector, a third sub-loss function value and a fourth sub-loss function value of the preset fill light model are determined.

[0214] In some embodiments of the present application, a Generative Adversarial Network (GAN) may be added to the preset fill light model to improve the authenticity of the predicted image data vector. Specifically, the third sub-loss function value L used to characterize the authenticity of the predicted image data vector may be obtained according to the predicted image data vector and the second sample image data vector according to the following formula (8): adv :

[0215]

[0216] In formula (8), GAN(·) is an algorithm used to determine the authenticity of the generated image based on the GAN network.

[0217] In some embodiments of the present application, a model for calculating structural similarity (SSIM) may be added to the preset fill light model to improve the authenticity of the output preset image. Specifically, the fourth sub-loss function value L for characterizing the similarity between the predicted image data vector and the second sample image data vector may be obtained according to the following formula (9): ssim :

[0218]

[0219] In formula (8), SSIM(·) is an algorithm for calculating structural similarity.

[0220] In an embodiment of the present application, by predicting the image data vector and the second sample image data vector, a third sub-loss function value for characterizing the authenticity of the predicted image data vector in the preset fill light model and a fourth sub-loss function value for characterizing the similarity between the predicted image data vector and the second sample image data vector can be obtained. In this way, by utilizing a variety of different loss function values, the generalization ability and robustness of the preset fill light model can be improved.

[0221] After obtaining the first sub-loss function value, the second sub-loss function value, the third sub-loss function value and the fourth sub-loss function value, the first sub-loss function value, the second sub-loss function value, the third sub-loss function value and the fourth sub-loss function value can be weighted and summed to obtain the loss function value of the preset fill light model. The specific weights corresponding to the first sub-loss function value, the second sub-loss function value, the third sub-loss function value and the fourth sub-loss function value can be set according to user needs and are not limited in the embodiments of the present application.

[0222] In order to better understand the image processing method of the embodiment of the present application, the embodiment of the present application describes in detail the image processing method provided by the embodiment of the present application with a specific scenario.

[0223] like Figure 5 As shown, the image processing method provided in the embodiment of the present application may include S510-S570.

[0224] S510: Acquire a first image.

[0225] S520: Calculate a mask image of the first image.

[0226] S530: Determine whether the number of photographed objects in the acquired first image is 1, if so, execute S540, if not, execute S550.

[0227] S540: Determine fill light intensity information of each photographed object according to the mask image of the first image.

[0228] In S540 , the specific calculation of the fill light intensity information of each photographed object may refer to the method for determining the fill light intensity information of the target photographed object in the above embodiment, which will not be described in detail herein.

[0229] S550: Obtain the photographed object corresponding to the maximum size information in the size information of the face region of each photographed object.

[0230] S560: Use the fill-light model to fill-light the photographed object in the first image to obtain a second image data vector.

[0231] In S560, the first image data vector can be calculated according to the determined fill light intensity information for fill light on the photographed object and the calculation method of the first image data vector in the above embodiment, and then the first image data vector is input into the fill light model, and the photographed object in the first image is fill lighted based on the fill light model to obtain the second image data vector.

[0232] S570: Post-process the second image data vector to obtain a second image.

[0233] In S570, the second image data vector may be processed with reference to the above embodiment to obtain the second image, which will not be described in detail here.

[0234] The image processing method provided in the embodiment of the present application can be executed by an image processing device. In the embodiment of the present application, an image processing device executing the image processing method is taken as an example to illustrate the image processing device provided in the embodiment of the present application.

[0235] Figure 6 FIG. 1 is a schematic diagram showing the structure of an image processing device according to an exemplary embodiment. Figure 6 As shown, the image processing device 600 may include:

[0236] A first acquisition module 610 is used to acquire a first image, wherein the first image includes at least one photographed object;

[0237] A first determination module 620, configured to identify each of the at least one photographic object;

[0238] A second determining module 630 is used to determine fill light intensity information of each of the photographed objects according to the regional image corresponding to each of the photographed objects, wherein the fill light intensity information of each of the photographed objects is intensity information of fill light for the photographed objects;

[0239] The third determining module 640 is configured to perform fill light on corresponding objects in the at least one photographic object according to the fill light intensity information to obtain a second image.

[0240] In an embodiment of the present application, by identifying each photographic object in a first image including at least one photographic object, a regional image corresponding to each photographic object can be obtained. For each photographic object, fill light intensity information for fill light on the photographic object can be determined based on the regional image corresponding to the photographic object. For each photographic object, fill light is performed on the photographic object based on the fill light intensity information for fill light on the photographic object, and a second image after fill light for each photographic object can be obtained. In this way, through the image processing method provided in an embodiment of the present application, there is no need to add other fill light equipment to fill light the photographic object to obtain a portrait photo with better portrait light and shadow effects. Not only does it reduce the hardware cost, but it also solves the problem of poor portrait light and shadow effects due to insufficient lighting conditions or hardware limitations of electronic equipment, thereby achieving the effect of high-quality portrait photos at a low cost, while improving the filming rate and image aesthetics of the electronic device.

[0241] In some embodiments of the present application, the second determining module 630 may include:

[0242] A first selection submodule is used to select a target photographing object from each photographing object;

[0243] The first determination submodule is used to determine fill light intensity information of each of the photographed objects according to the area image corresponding to the target photographed object.

[0244] In some embodiments of the present application, the first selection submodule may include:

[0245] A first selection unit, configured to determine that the photographed object is a target photographed object when the number of photographed objects in the first image is 1;

[0246] The second selection unit is used to determine the target photographing object according to the size information of the face area of ​​each photographing object when the number of photographing objects in the first image is greater than 1.

[0247] In some embodiments of the present application, the first determining submodule may include:

[0248] A first determining unit is used to grayscale the area image corresponding to the target photographed object to obtain a grayscaled area image;

[0249] A second determining unit is used to sort the brightness values ​​of each pixel in the grayscale regional image in ascending order to obtain a brightness value sequence;

[0250] A third determining unit, configured to obtain a first normalized brightness value sequence according to a position sequence of each brightness value in the brightness value sequence in the brightness value sequence;

[0251] a fourth determining unit, configured to obtain a second normalized brightness value sequence according to each brightness value in the brightness value sequence and a bit sequence of each brightness value in the brightness value sequence;

[0252] a fifth determining unit, configured to determine fill light intensity information for the target photographic object according to the first normalized brightness value sequence and the second normalized brightness value sequence;

[0253] The sixth determining unit is used to determine the fill light intensity information of each photographic object according to the fill light intensity information of the target photographic object.

[0254] In some embodiments of the present application, the third determination module 640 may include:

[0255] A second determining submodule, configured to determine a fill light intensity information image of the first image according to fill light intensity information of each photographed object;

[0256] A third determination submodule is configured to splice the image data of the first image, the image data of the mask image of the first image, the image data of the skin image of the at least one photographed object in the first image, and the image data of the fill light intensity information image to obtain a first image data vector;

[0257] a fourth determination submodule, configured to input the first image data vector into a fill light model, and perform fill light on each photographed object based on the fill light model to obtain a second image data vector;

[0258] The fifth determination submodule is used to determine the second image according to the second image data vector.

[0259] The image processing device in the embodiment of the present application can be an electronic device, or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or it can be other devices other than a terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmentedreality, AR) / virtual reality (virtualreality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc., and can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.

[0260] The image processing device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0261] The image processing device provided in the embodiment of the present application can achieve Figure 1 To avoid repetition, the various processes implemented by the method embodiment are not described here.

[0262] Alternatively, if Figure 7 As shown, an embodiment of the present application further provides an electronic device 700, including a processor 701 and a memory 702, wherein the memory 702 stores programs or instructions that can be executed on the processor 701, and when the program or instructions are executed by the processor 701, the various steps of the above-mentioned image processing method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, they are not described here.

[0263] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0264] Figure 8 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of the present application.

[0265] The electronic device 800 includes but is not limited to: a radio frequency unit 801, a network module 802, an audio output unit 803, an input unit 804, a sensor 805, a display unit 806, a user input unit 807, an interface unit 808, a memory 809, and a processor 810 and other components.

[0266] Those skilled in the art will appreciate that the electronic device 800 may also include a power source (such as a battery) for supplying power to each component, and the power source may be logically connected to the processor 810 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Figure 8 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0267] Among them, the processor 810 is used to obtain a first image, which includes at least one photographed object; identify each photographed object in the at least one photographed object; determine the fill light intensity information of each photographed object based on the regional image corresponding to each of the photographed objects, and the fill light intensity information of each of the photographed objects is the intensity information of fill light for the photographed object; fill light is performed on the corresponding photographed objects in the at least one photographed object according to the fill light intensity information to obtain a second image.

[0268] In this way, by identifying each photographed object in a first image including at least one photographed object, a regional image corresponding to each photographed object can be obtained. For each photographed object, fill light intensity information for fill light on the photographed object can be determined based on the regional image corresponding to the photographed object. For each photographed object, fill light is performed on the photographed object based on the fill light intensity information for fill light on the photographed object, and a second image after fill light for each photographed object can be obtained. In this way, through the image processing method provided in the embodiment of the present application, there is no need to add other fill light equipment to fill light the photographed object to obtain a portrait photo with better portrait light and shadow effects. Not only does it reduce the hardware cost, but it also solves the problem of poor portrait light and shadow effects due to insufficient lighting conditions or hardware limitations of electronic equipment, thereby achieving the effect of high-quality portrait photos at a low cost, while improving the filming rate and image aesthetics of the electronic device.

[0269] Optionally, the processor 810 is further configured to select a target photographic object from each photographic object; and determine fill light intensity information of each photographic object according to a regional image corresponding to the target photographic object.

[0270] In this way, by selecting a photographic object from at least one photographic object as the target photographic object, and then determining the fill light intensity information for each photographic object respectively through the regional image of the target photographic object, there is no need to process the regional image of each photographic object, thereby improving the efficiency of determining the fill light intensity information for each photographic object.

[0271] Optionally, the processor 810 is further used to determine the photographed object as the target photographed object when the number of photographed objects in the first image is 1; and to determine the target photographed object based on size information of the facial area of ​​each photographed object when the number of photographed objects in the first image is greater than 1.

[0272] In this way, different methods are selected according to the number of photographed objects in the first image to determine the target photographed object, thereby improving the flexibility of selecting the target photographed object.

[0273] Optionally, the processor 810 is further used to grayscale the regional image corresponding to the target photographic object to obtain a grayscale regional image; sort the brightness values ​​of each pixel in the grayscale regional image in ascending order to obtain a brightness value sequence; obtain a first normalized brightness value sequence according to the bit sequence of each brightness value in the brightness value sequence in the brightness value sequence; obtain a second normalized brightness value sequence according to each brightness value in the brightness value sequence and the bit sequence of each brightness value in the brightness value sequence; determine the fill light intensity information of the target photographic object according to the first normalized brightness value sequence and the second normalized brightness value sequence; determine the fill light intensity information of each photographic object according to the fill light intensity information of the target photographic object.

[0274] In this way, the fill light intensity information for fill light for each photographic object is determined by software method, which saves hardware costs. In addition, the fill light intensity information for fill light for the target object can be obtained by calculating the light ratio of the target object, and then the fill light intensity information for fill light for each object can be obtained respectively. In this way, the fill light intensity information for fill light for each object can be accurately determined, thereby improving the accuracy of determining the fill light intensity information for fill light for each object.

[0275] Optionally, the processor 810 is further used to determine the fill light intensity information image of the first image based on the fill light intensity information of each photographed object; splice the image data of the first image, the image data of the mask image of the first image, the image data of the skin image of at least one photographed object in the first image, and the image data of the fill light intensity information image to obtain a first image data vector; input the first image data vector into a fill light model, and perform fill light on each photographed object based on the fill light model to obtain a second image data vector; and determine the second image based on the second image data vector.

[0276] In this way, each photographed object is filled with light by using the fill light model, thereby improving the fill light efficiency of each photographed object.

[0277] It should be understood that in the embodiment of the present application, the input unit 804 may include a graphics processor (Graphics Processing Unit, GPU) 8041 and a microphone 8042, and the graphics processor 8041 processes the image data of the static picture or video obtained by the image capture device (such as a color camera) in the video capture mode or the image capture mode. The display unit 806 may include a display panel 8061, and the display panel 8061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 807 includes a touch panel 8071 and at least one of other input devices 8072. The touch panel 8071 is also called a touch screen. The touch panel 8071 may include two parts: a touch detection device and a touch controller. Other input devices 8072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0278] The memory 809 can be used to store software programs and various data. The memory 809 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 809 may include a volatile memory or a non-volatile memory, or the memory 809 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 809 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0279] The processor 810 may include one or more processing units; optionally, the processor 810 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 810.

[0280] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the above-mentioned image processing method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0281] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0282] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned image processing method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0283] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0284] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned image processing method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0285] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0286] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0287] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. An image processing method, characterized in that: The method comprises: Acquire a first image, wherein the first image includes at least one photographed object; identifying each of the at least one subject; Determine fill light intensity information of each of the photographed objects according to the area image corresponding to each of the photographed objects, wherein the fill light intensity information of each of the photographed objects is intensity information of fill light for the photographed objects; Fill light is respectively performed on corresponding photographic objects among the at least one photographic object according to the fill light intensity information to obtain a second image.

2. The method according to claim 1, characterized in that The step of determining fill light intensity information of each photographed object according to the area image corresponding to each photographed object includes: Select a target subject from each subject; The fill light intensity information of each of the target photographed objects is determined according to the area image corresponding to the target photographed object.

3. The method according to claim 2, characterized in that The step of selecting a target photographing object from each photographing object includes: When the number of photographed objects in the first image is 1, determining that the photographed object is a target photographed object; When the number of photographed subjects in the first image is greater than 1, the target photographed subject is determined according to the size information of the face area of ​​each photographed subject.

4. The method according to claim 2, characterized in that: The step of determining fill light intensity information of each of the photographed objects according to the area image corresponding to the target photographed object includes: Gray-scaling the area image corresponding to the target photographed object to obtain a gray-scaling area image; Sorting the brightness values ​​of the pixels in the grayscale area image in ascending order to obtain a brightness value sequence; Obtaining a first normalized brightness value sequence according to the position sequence of each brightness value in the brightness value sequence; Obtaining a second normalized brightness value sequence according to each brightness value in the brightness value sequence and the position sequence of each brightness value in the brightness value sequence; Determining fill light intensity information of the target photographed object according to the first normalized brightness value sequence and the second normalized brightness value sequence; The fill light intensity information of each of the photographic objects is determined according to the fill light intensity information of the target photographic object.

5. The method according to any one of claims 1 to 4, characterized in that: The step of performing fill light on corresponding objects among the at least one photographic object according to the fill light intensity information to obtain a second image includes: Determining a fill light intensity information image of the first image according to the fill light intensity information of each photographed object; splicing the image data of the first image, the image data of the mask image of the first image, the image data of the skin image of the at least one photographed object in the first image, and the image data of the fill light intensity information image to obtain a first image data vector; Inputting the first image data vector into a fill light model, and performing fill light on each photographed object based on the fill light model to obtain a second image data vector; Based on the second image data vector, a second image is determined.

6. An image processing device, characterized in that: The device comprises: A first acquisition module, used to acquire a first image, wherein the first image includes at least one photographed object; A first determination module, configured to identify each of the at least one photographic object; A second determining module is used to determine fill light intensity information of each of the photographed objects according to the area image corresponding to each of the photographed objects, wherein the fill light intensity information of each of the photographed objects is intensity information of fill light for the photographed objects; The third determining module is used to perform fill light on corresponding photographic objects among the at least one photographic object according to the fill light intensity information to obtain a second image.

7. The device according to claim 6, characterized in that The second determining module includes: A first selection submodule is used to select a target photographing object from each photographing object; The first determination submodule is used to determine fill light intensity information of each of the photographed objects according to the area image corresponding to the target photographed object.

8. The device according to claim 7, characterized in that The first selection submodule includes: A first selection unit, configured to determine that the photographed object is a target photographed object when the number of photographed objects in the first image is 1; The second selection unit is used to determine the target photographing object according to the size information of the face area of ​​each photographing object when the number of photographing objects in the first image is greater than 1.

9. The device according to claim 7, characterized in that The first determining submodule includes: A first determining unit is used to grayscale the area image corresponding to the target photographed object to obtain a grayscaled area image; A second determining unit is used to sort the brightness values ​​of each pixel in the grayscale regional image in ascending order to obtain a brightness value sequence; A third determining unit, configured to obtain a first normalized brightness value sequence according to a position sequence of each brightness value in the brightness value sequence in the brightness value sequence; a fourth determining unit, configured to obtain a second normalized brightness value sequence according to each brightness value in the brightness value sequence and a bit sequence of each brightness value in the brightness value sequence; a fifth determining unit, configured to determine fill light intensity information of the target photographic object according to the first normalized brightness value sequence and the second normalized brightness value sequence; A sixth determining unit is used to determine the fill light intensity information of each of the photographic objects according to the fill light intensity information of the target photographic object.

10. The device according to any one of claims 6 to 9, characterized in that: The third determination module comprises: A second determining submodule, configured to determine a fill light intensity information image of the first image according to fill light intensity information of each photographed object; A third determination submodule is configured to concatenate the image data of the first image, the image data of the mask image of the first image, the image data of the skin image of the at least one photographed object in the first image, and the image data of the fill light intensity information image to obtain a first image data vector; a fourth determination submodule, configured to input the first image data vector into a fill light model, and perform fill light on each photographed object based on the fill light model to obtain a second image data vector; The fifth determination submodule is used to determine the second image according to the second image data vector.

Images