A method and system for generating recommended photographing postures according to photographing scenes

Abstract

The present invention discloses a method and system for generating recommended photographing postures according to a photographing scene, relating to the field of deep learning. By means of an end-to-end deep convolutional neural network, a direct mapping from a scene picture to a posture action is found, so as to recommend a posture suitable for the current photographing scene to a user, and solve the pain points that the user cannot think of a suitable posture and cannot pose.

Description

Technical Field

[0001] The present invention relates to the field of deep learning, and particularly to a method and system for generating recommended photographing poses from a photographing scene. Background Art

[0002] At present, the photographing function of smart phones is developing rapidly, and its camera pixels, imaging quality, etc. are getting higher and higher. The photographing functions of many smart phones can even rival those of professional cameras. At the same time, the booming development of the mobile Internet and social networks has also promoted people's willingness to take and share photos. Against this background, coupled with the convenience of mobile phones, people are using smart phones to take photos more and more frequently. Whether professional or amateur, digital photography is interesting to many people. It is estimated that more than 1 billion photos are taken by smart phones every year. People on social networks often share their photos with friends. The continuously enhanced computing power of smart phones and the ability to connect to more powerful computing platforms via the network have made them creative assistants for amateur photographers, and major smart phone manufacturers have also started to introduce photo enhancement functions on their devices.

[0003] However, taking satisfactory photos usually requires the professional knowledge and experience of professional photographers. Like other visual arts, it is difficult to impart photographic knowledge because there is no common representation similar to symbols or mathematical equations. Many people have single and rigid photographing poses and cannot make poses that are suitable for the current scene and aesthetically pleasing. It is also difficult to quickly find suitable photographing poses for the current scene on the Internet in real life, resulting in a situation where, although the pixels are high, the quality of portrait photos is not very satisfactory. Nevertheless, many people are still interested in photos of professional quality, and they hope to create photos of similar quality for the scenes or events they are interested in.

[0004] At present, there are already some technologies for recommending posing postures in China. For example, in the patent "A Method and System for Intelligently Recommending Photographing Expressions and Posing Postures during Photographing" (Patent No.: CN107911601A), photographing expressions and posing postures are recommended by identifying expression information, inputting mood information, and obtaining information such as the relationship between the photographer and the holder of the mobile terminal; in the patent "Picture Output Method and Device" (Patent No.: CN103220466B), the posture pictures in the posture library are recommended by parameter matching through judging information such as the number of human faces and the positions of human faces in the picture; in the patent "Shooting Template Recommendation Method, Device and Shooting Equipment" (Patent No.: CN107018333A), posture templates are recommended through the number of human faces and the identity information corresponding to the human faces. In the patent "Photographing Method, Intelligent Terminal and Cloud Server" (Patent No.: CN104284092A), by analyzing the environmental information (location information and environmental picture information) around the photographer, pictures with outlined photographing postures that are consistent with the current environment are found through matching, and this picture material is recommended to the user as a recommended posture. The essence of its posture recommendation is a picture matching process and cannot generate recommended postures according to the specific environment. It can be seen that the existing technologies have not realized the method of directly generating photographing postures according to the scene. Summary of the Invention

[0005] To solve the above problems, the present invention proposes a method and system for generating recommended photographing postures according to the photographing scene, and finds the direct mapping from the scene picture to the posture action through an end-to-end deep convolutional neural network, so as to recommend postures suitable for the current photographing scene to the user and solve the pain points that the user cannot think of suitable postures and cannot pose.

[0006] The technical solution adopted by the present invention is as follows:

[0007] A method for generating recommended photographing postures according to the photographing scene includes the following steps:

[0008] Collect pictures of human postures that have been screened and have clear backgrounds to obtain an initial data set;

[0009] Obtain the human posture skeleton points in each picture in the initial data set to obtain a posture data set;

[0010] Deduct the human body in each picture in the initial data set and complete the background filling for the deducted part to obtain pictures with only background scenes and form a scene data set;

[0011] Make a training data set according to the posture data set and the scene data set;

[0012] Input the images in the training data set into the deep convolutional neural network model, and this deep convolutional neural network model learns the mapping relationship from the scene picture to the human posture skeleton points through iterative training;

[0013] When the user takes a photo, first take a scene picture and input it into the trained deep convolutional neural network model, which outputs the recommended photo-taking position and pose to the user.

[0014] Furthermore, use the OpenPose algorithm (see Zhe Cao, Gines Hidalgo, Tomas Simon, Shih-En Wei, Yaser Sheikh, OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields, Computer Vision and Pattern Recognition 2018.) to obtain the human pose skeleton points in the picture.

[0015] Furthermore, use the Deep Image Matting human portrait segmentation algorithm (see Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, Thomas S. Huang Generative Image Inpainting with Contextual Attention, Computer Vision and Pattern Recognition 2018.) to subtract the human body in the picture.

[0016] Furthermore, use the Generative Image Inpainting image blank filling algorithm (see Ning Xu, Brian Price, Scott Cohen, Thomas Huang Deep Image Matting, Computer Vision and Pattern Recognition 2017.) to complete the background of the subtracted part in the picture.

[0017] Furthermore, the deep convolutional neural network model is based on ResNet50, and the intermediate structure consists of 50 layers of residual network convolutional layers. Two parallel pooling layers are connected at the last layer. Two shared multi-dimensional fully connected layers are used after the pooling layer, and the ReLU activation function is used after each fully connected layer; the deep convolutional neural network model has two output branches, one for outputting the two-dimensional coordinates of the human pose skeleton points, and the other for outputting a one-dimensional vector composed of the photo-taking position and the face position coordinates.

[0018] Further, the method for creating a training dataset based on the pose dataset and the scene dataset is as follows: Obtain the L2-normalized human pose skeleton point coordinates from the pose dataset; obtain the face center coordinates from the pose dataset and the coordinates of the four directional boundary points, namely the leftmost point, the rightmost point, the uppermost point, and the lowermost point, among the un-normalized human pose skeleton points, and flatten the face center point coordinates and the four directional boundary point coordinates into one-dimensional vector position point coordinates; construct the training dataset from the images in the scene dataset, the above-mentioned L2-normalized human pose skeleton point coordinates, and the above-mentioned position point coordinates.

[0019] Further, the method for iteratively training the deep convolutional neural network model is as follows: In one round of iterative training, first perform forward computation based on the input image and simultaneously calculate the loss function; then calculate the gradient value of the network in the reverse direction according to the loss function, propagate the gradient value in the reverse direction, and update the parameters of the deep convolutional neural network; perform multiple rounds of iterative training until the preset number of iterations or convergence condition is reached.

[0020] A system for generating recommended photographing poses according to the photographing scene includes a memory and a processor. A computer program is stored on the memory, and when the processor executes the program, the steps of the above method are implemented.

[0021] The beneficial effects achieved by the present invention are as follows:

[0022] The network model used in the present invention adopts a structure with dual-branch output, and a fully connected layer with shared weights is adopted after the branch pooling layer. Such a model structure can not only train and output the normalized pose skeleton points to eliminate the influence of the original image character size on the pose, but also train and output the position of the original image character as an auxiliary reference for the pose. The end-to-end deep learning network can also learn and represent the complex mapping relationship from the scene to the pose, and the recommended results will not be limited to the poses that have appeared in the dataset, making the recommendation more flexible and diverse. Description of the Drawings

[0023] Figure 1 is a flowchart of a method for generating recommended photographing poses according to the photographing scene in an embodiment of the present invention.

[0024] Figure 2 is a structure diagram of the deep convolutional neural network model.

[0025] Figure 3 is a sample diagram of pose skeleton points Detailed Embodiment

[0026] To make the above features and advantages of the present invention more obvious and understandable, specific embodiments are hereinafter given and described in detail in conjunction with the accompanying drawings as follows.

[0027] An embodiment of the present invention provides a method for generating recommended photographing postures according to a photographing scene. By constructing an end-to-end deep convolutional neural network, a direct mapping from a scene picture to a posture and a posture position is found. As Figure 1 shown, this method consists of two parts: training the model and model prediction. First, an initial dataset is used to generate a posture dataset through human posture skeleton point detection; a scene dataset is generated by a portrait matting and blank filling algorithm, and the input labels and output labels of a deep convolutional neural network model ( Figure 1 shown as an end-to-end model) are made respectively, and then sent to the model framework for training. In addition, the model prediction process can start from an input scene photo, be sent into the trained model, predict and output a recommended human posture skeleton map, and give the recommended human position and face position. The technical solution of this method is specifically described as follows:

[0028] 1.1 Training data preparation

[0029] S1: Collect N single-person portrait photos, travel tourist photos, and life photos with clear backgrounds (without background blurring processing) and beautiful postures as the initial dataset D.

[0030] S2: Apply the OpenPose algorithm to obtain the human key points, that is, the human posture skeleton points, of each picture in the initial dataset and store them as the posture dataset G.

[0031] S3: Apply the Deep Image Matting portrait segmentation (matting) algorithm to identify the portrait edges of each photo in the initial dataset, subtract the portrait, and use the Generative Image Inpainting image blank filling algorithm to fill in the original portrait position to obtain N scene datasets S with only background scenes.

[0032] 1.2 Model construction and training

[0033] The present invention constructs a deep convolutional neural network model with a dual-output branch to learn the mapping relationship from a scene picture to a human posture skeleton point. Due to the need for user recommendation, this model will also output the recommended photographing position and face position in the original scene photo. Finally, this model will achieve that given a scene photo, the model automatically outputs the recommended photographing position, face position, and shows the recommended posture skeleton map at the recommended photographing position.

[0034] As Figure 2As shown, the deep convolutional neural network applied in the present invention is a modification based on ResNet50. The intermediate structure of the model consists of 50 layers of residual network convolutional layers. At the last layer, two parallel pooling layers (1×1, 2×2) are connected. After the pooling layers, two shared fully connected layers with 1000 dimensions are used, and the ReLU activation function is used after each fully connected layer.

[0035] The two output branches of the present invention are respectively the two-dimensional pose skeleton point coordinates and the one-dimensional vector composed of the photographing position and the face position coordinates. Therefore, the training dataset of (X, Y1, Y2) needs to be made before training. Among them, X = [x 1 , x 2 , x 3 , …, x N is the image in the scene dataset. is the human pose skeleton point coordinates after L2 normalization in the pose dataset; among them, represents the set of human pose skeleton point coordinates corresponding to the first photo among N photos. Similarly for others, such as represents the set of human pose skeleton point coordinates corresponding to the Nth photo among N photos (the superscript N is the total number of photos). Each is composed of 25 coordinate points. For example, is composed of 25 coordinate points ; and represent the abscissa and ordinate of the first coordinate point of the first image after L2 normalization (that is, in , p represents the abscissa, i represents the i-th photo, the subscript 1 represents the first coordinate point, and the subscript L2 represents after L2 normalization; , q represents the ordinate, i represents the i-th photo, the subscript 1 represents the first coordinate point, and the subscript L2 represents after L2 normalization; similarly for others);

[0036]

[0037] are the coordinates of the face center point (such as point 0 in Figure 3 ) and the leftmost point (such as point 4 in Figure 3 ), the rightmost point (such as point 7 in Figure 3 ), the uppermost point (such as point 16 in Figure 3 ) and the lowermost point (such as point 22 in Figure 3 ) of the human pose skeleton points in the pose dataset that have not been L2-normalized, and the position point coordinates flattened into a one-dimensional vector (for example, represents the set of position point coordinates corresponding to the first photo among N photos, which is composed of the above 5 coordinate points. and (representing the horizontal and vertical coordinates of the center point of the face in the first picture).

[0038] The loss function is the mean squared error function L2loss (MSE), written as:

[0039]

[0040]

[0041] Loss = Loss1 + Loss2 (3)

[0042] In the formula, N represents the number of photos in the training dataset, represents the set of coordinates of the human body pose skeleton points corresponding to the i-th photo, represents the set of coordinates of the corresponding position points of the i-th photo, and the result of f(·) is the training output corresponding to the model.

[0043] Based on the above descriptions and definitions of data, model, and loss function, the present invention trains the above-defined deep recurrent neural network model according to the following steps:

[0044] S1: Produce the training dataset (X, Y1, Y2) according to the method described above,

[0045] S2: Each time during iteration, send 100 samples as the same batch into the above-defined neural network model;

[0046] S3: The above-defined neural network model performs forward operations according to the input image, and at the same time calculates the loss of this iteration according to formula (3);

[0047] S4: Calculate the gradient value of the network in reverse according to the loss function, perform backpropagation of the gradient value, and update the neural network parameters;

[0048] S5: Return to step S2, enter the next round of iteration, until the preset number of iterations or convergence condition is reached.

[0049] Although the present invention has been disclosed above with embodiments, it is not intended to limit the present invention. Any appropriate modifications or equivalent replacements made by those of ordinary skill in the art to the technical solutions of the present invention shall be covered within the protection scope of the present invention. The protection scope of the present invention shall be subject to that defined by the claims.

Claims

1. A method for generating recommended photographing postures according to a photographing scene, characterized in that, It includes the following steps: Collect pictures of human postures that are screened and have clear backgrounds to obtain an initial dataset; Obtain the human posture skeleton points in each picture in the initial dataset to obtain a posture dataset; Deduct the human body in each picture in the initial dataset and complete the background filling for the deducted part to obtain pictures with only background scenes and form a scene dataset; Produce a training dataset according to the posture dataset and the scene dataset; Input the images in the training dataset into a deep convolutional neural network model. This deep convolutional neural network model learns the mapping relationship from scene pictures to human posture skeleton points through iterative training; the deep convolutional neural network model is based on ResNet50, and the intermediate structure consists of 50 layers of residual network convolutional layers. Two parallel pooling layers are connected at the last layer, and two shared multi-dimensional fully connected layers are used after the pooling layer. The ReLU activation function is used after each fully connected layer; the deep convolutional neural network model contains two output branches, one for outputting the two-dimensional coordinates of human posture skeleton points, and the other for outputting a one-dimensional vector composed of the photographing position and the face position coordinates; When the user takes a photo, first take a scene picture and input it into the trained deep convolutional neural network model, and this model outputs the recommended photographing position and photographing posture to the user.

2. The method according to claim 1, wherein Use the OpenPose algorithm to obtain the human posture skeleton points in the picture.

3. The method according to claim 1, wherein Use the Deep Image Matting human portrait segmentation algorithm to deduct the human body in the picture.

4. The method according to claim 1, wherein Use the Generative Image Inpainting image blank filling algorithm to complete the background filling for the deducted part in the picture.

5. The method according to claim 1, wherein The method for producing a training dataset according to the posture dataset and the scene dataset is: obtain the L2-normalized human posture skeleton point coordinates from the posture dataset; Obtain the face center coordinates and the coordinates of the four direction boundary points, namely the leftmost point, the rightmost point, the uppermost point, and the lowermost point, among the un-L2-normalized human posture skeleton points from the posture dataset, and flatten the face center point coordinates and the four direction boundary point coordinates into the position point coordinates of a one-dimensional vector; The training dataset is composed of the images in the scene dataset, the above-mentioned L2-normalized human posture skeleton point coordinates, and the above-mentioned position point coordinates.

6. The method according to claim 1, wherein The method for iterative training of the deep convolutional neural network model is: in one round of iterative training, first perform forward calculation according to the input image and calculate the loss function at the same time; then calculate the gradient value of the network in reverse according to the loss function, propagate the gradient value in reverse, and update the parameters of the deep convolutional neural network; stop after multiple rounds of iterative training until the preset number of iterations or convergence condition is reached.

7. The method according to claim 6, characterized in that, The loss function is the mean square error function Loss = Loss1 + Loss2, where: Where N represents the number of photos in the training dataset, represents the set of coordinates of the human body pose skeleton points corresponding to the i-th photo, represents the set of coordinates of the corresponding position points. The result of f(·) is the training output corresponding to the deep convolutional neural network model.

8. A system for generating recommended photographing postures according to a photographing scene, characterized in that, It includes a memory and a processor. A computer program is stored on this memory, and when the processor executes this program, it implements the steps of the method described in any one of claims 1-7.

Images