A photo pose recommendation method integrating image content and feature extraction

Abstract

The present invention discloses a method for recommending photographing postures by fusing image content and geometric feature extraction, and the steps include: extracting scene semantic features, scene features, and viewing angle features from an image input by a user; fusing the extracted scene semantic features, scene features, and viewing angle features according to set rules; calculating the similarity between the fused features and the features in a feature database, and returning posture pictures corresponding to the m features with the highest similarity to the user. The present invention can meet the posture preferences of users of different body shapes, skin colors, genders, and ages. Introducing scene category information, texture information, and viewing angle information in the feature extraction step can more accurately retrieve posture photos with similar scenes in a posture picture library. Introducing semantic information and a strong tag system can more intelligently identify objects in the screen with which the user can interact and take photos, and can meet the user's demand for immediacy of posture recommendations.

Description

Technical Field

[0001] The present invention belongs to the field of deep learning and image retrieval, and in particular relates to a method for recommending photographing postures by integrating image content and geometric feature extraction. Background Art

[0002] At present, the camera function of smartphones is developing rapidly, and its camera pixels and image quality are getting higher and higher. The camera function of many smartphones can even rival that of professional cameras. In this context, coupled with the convenience of mobile phones, people are more willing to use mobile phones to complete personal photography. Whether professionals or amateurs, digital photography is interesting to many people. It is estimated that more than 1 billion photos are taken by smartphones every year. People on social networks often share their photos with friends. The increasing computing power of smartphones and the ability to connect to more powerful computing platforms through the network make them creative assistants for amateur photographers, and major smartphone manufacturers have also begun to introduce photo enhancement functions on devices.

[0003] However, taking satisfactory photos usually requires the expertise and experience of professional photographers. As with other visual arts, the lack of universal representations such as symbols or mathematical equations makes it difficult to impart knowledge of photography. Many people's photo poses are monotonous and rigid, and they cannot make poses that are suitable for the current scene and beautiful. In real life, it is also difficult to quickly find photo poses that are suitable for the current scene on the Internet, resulting in high-pixel but unsatisfactory portrait photos. Despite this, many people are still interested in professional-quality photos, and they hope to be able to create photos of similar quality for scenes or events that interest them.

[0004] In order to solve such a problem, the present invention proposes a technical solution and a system for intelligently recommending photographic poses based on photographic scenes, and performs retrieval-based pose image recommendation by utilizing features of different levels extracted by deep neural networks.

[0005] At present, there are some practices of recommending photo poses in China, such as recommending photo expressions and photo poses by recognizing facial expressions, inputting mood information, and obtaining information such as the relationship between the photographer and the mobile terminal holder; matching the pose images in the recommended pose library by parameters based on the number of faces and face positions in the picture; and recommending pose templates based on the number of faces and the identity information corresponding to the faces. The environmental information around the photographer (GPS positioning information and compass information) is analyzed and poses are recommended. In addition, there are some patents on building a photo template database. Other existing pose recommendation technologies include pose retrieval recommendation based on photography rules and pose retrieval recommendation based on aesthetic scores.

[0006] None of the above technologies have mined the semantic information, scene category information, and texture information in the scene. Using this rich information to search the pose image library can make the results more diverse and meet the pose preferences of users of different body shapes, skin colors, genders, and ages. In addition, introducing semantic information can more intelligently identify objects in the picture that users can interact with to take photos, such as park benches, stairs, etc. Summary of the invention

[0007] In view of the problems existing in the prior art, the purpose of the present invention is to provide a method for recommending photographic postures by integrating image content and geometric feature extraction. The present invention includes obtaining the initial photographed picture and the coordinates of the user's focus point; extracting the scene category, semantic texture features, perspective features and user focus category information of the picture based on the picture; calculating the similarity between the features and the features in the own database; and finally returning several optimal photographic postures suitable for the current scene after retrieval. The present invention intelligently and quickly recommends photographic postures suitable for the current scene and shooting perspective of the user to the user, so as to meet the user's needs for postures during the shooting process. Providing different posture prompts and guidance according to different scenes also enriches the photographing function and improves the photographing experience. In addition, the present invention pays attention to the objects and scenes that the user is interested in. When the object scene that the user focuses on and hopes to interact with appears, the user can click on the objects and scenes of interest, and the present invention will make recommendations based on the user's interests.

[0008] The technical solution of the present invention is:

[0009] A method for recommending photographing postures by integrating image content and geometric feature extraction, the steps of which include:

[0010] Extract scene semantic features, scene features, and perspective features from the image input by the user;

[0011] The extracted scene semantic features, scene features, and view features are integrated according to the set rules;

[0012] The similarity between the fused features and the features in the feature database is calculated, and the posture images corresponding to the m features with the highest similarity are returned to the user.

[0013] Furthermore, the method for extracting the scene semantic features is: The image is divided into left sub-image, right sub-image, upper sub-image and lower sub-image; the semantic segmentation model is used to extract the two-dimensional feature map G of each sub-image, and each two-dimensional feature map G is compressed into a one-dimensional vector as the scene semantic information of the corresponding sub-image, denoted as f s_left ,f s_right ,f s_up ,f s_down; Then the scene semantic information of each sub-image is weightedly spliced ​​to obtain the scene semantic feature f s .

[0014] Furthermore, when the user clicks on a target in the image as a focused scene or an item that the user wishes to interact with, the coordinates of the target clicked by the user (p x ,p y ) will be used as the input of the semantic segmentation model, and the corresponding two-dimensional coordinates in the output two-dimensional feature map G The corresponding category P is used as a strong label.

[0015] Furthermore, the cosine distance formula is used to calculate the fused feature f input With each feature in the feature library Similarity between them; the confidence of the image category P in the posture image database is greater than the set threshold, and the features of the image are consistent with the fusion feature f input The m posture images with the highest similarity are returned to the user.

[0016] Furthermore, a residual network is used to extract the perspective feature and the scene feature; the residual network model uses a ResNet-50 model as a basic network structure, and the last convolution module of the ResNet-50 model is respectively connected to the first average pooling layer and the second average pooling layer for scene feature extraction, and is respectively connected to the third average pooling layer and the fourth average pooling layer; the third average pooling layer is connected to the first fully connected layer for outputting the scene category, and the fourth average pooling layer is connected to the second fully connected layer for outputting the perspective category; wherein the method for training the residual network is: firstly, a picture data set of different scenes is collected, and then for each picture in the picture data set, a vanishing point V (v x ,v y ), and through The image is classified by perspective, and the scene category of the image is marked according to the visual classification result View of the image, where L input is the length of the image, W input is the width of the image; then the training data set X=[x1,x2,x3,…,x N ] and the corresponding scene category labels and perspective category labels, train the residual network, and the loss function used in training the residual network includes Among them, View i represents the view category label of the i-th image, y i represents the scene category label of the i-th image, f view (x i) is the i-th photo x i is the viewing angle category output by the model as input, f scene (x i ) is the i-th photo x i is the scene category output by the model when it is input.

[0017] Furthermore, the last convolution module output of the trained ResNet-50 model is used as the scene feature f c_1×1 ; and changing the size parameter of the average pooling layer in the last convolutional module of the trained ResNet-50 model to 2×2, and outputting the feature map f of the convolutional module c_2×2 Compressed into a one-dimensional vector f c_2×2 ; Then the scene feature f c_1×1 , one-dimensional vector f c_2×2 As the scene features

[0018] Furthermore, the scene semantic features, scene features, and viewing angle features are respectively normalized by L2 norm and then weighted fused to obtain fused features.

[0019] Furthermore, feature extraction is performed on each photo in the posture image database D, and the extracted scene semantic features, scene features, and perspective features are fused according to the set rules to obtain the corresponding feature library D f .

[0020] A server, characterized in that it comprises a memory and a processor, wherein the memory stores a computer program, the computer program is configured to be executed by the processor, and the computer program comprises instructions for executing each step in the above method.

[0021] A computer-readable storage medium stores a computer program, wherein the computer program implements the steps of the above method when executed by a processor.

[0022] The advantages of the present invention are as follows:

[0023] The system logic proposed in the technical solution of the present invention can return a variety of posture photos to meet the posture preferences of users of different body shapes, skin colors, genders, and ages. Introducing scene category information, texture information, and perspective information in the feature extraction step can more accurately retrieve posture photos with similar scenes in the posture image library. Introducing semantic information and a strong label system can more intelligently identify objects in the picture that users can interact with to take photos, such as park benches, stairs, etc. The method of the present invention will have strong interpretability, modifiability, and robustness. Experiments have proved that the system provided by the present technical solution can respond to new scenes more quickly and can meet users' immediate needs for posture recommendations. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 The present invention is a flow chart of the method.

[0025] Figure 2 Example graph for segmenting detail sub-graphs for semantic segmentation;

[0026] (a) Up and down division, (b) left and right division.

[0027] Figure 3 This is an example diagram of scene classification and perspective features. DETAILED DESCRIPTION

[0028] The present invention is further described in detail below in conjunction with the accompanying drawings. The examples given are only used to explain the present invention but not to limit the scope of the present invention.

[0029] This technical solution hopes to find the nonlinear mapping relationship between scene photos and postures by combining image feature similarity comparison and image database retrieval. Figure 1 As shown, the technical solution is divided into four steps as a whole. It starts with the input picture. First, the image features are extracted, and the three sets of features are fused according to certain rules. Next, the obtained feature values ​​are compared with the features in the feature database, and finally the posture picture database is searched. If the user clicks on an object or scene of interest, the coordinates of the user's click will be used as another input to enter the semantic segmentation network, and then the strong label will be identified, and targeted recommendations will be made including scenes that the user is interested in or objects that the user wants to interact with. In addition, in order to construct the feature database of the posture picture library, it is also necessary to perform the same steps of feature extraction on the pictures in the posture picture library.

[0030] Preparation of pose image library and feature library

[0031] Collect N photos of tourists, select the better ones (usually taken by photographers or people with certain photography experience who guide the subjects) and build a pose image database D. The requirements for the photos include: clear background, diverse scenes, and beautiful poses.

[0032] The steps of the present invention include:

[0033] Step 1: Scene semantic feature extraction and identification of strong labels. This technical solution uses the semantic segmentation model Pyramid Scene Parsing Network (PSPNet) and the 50-layer deep residual network (ResNet50) as the basic model structure, with the aim of mining image scene category information, semantic information, picture texture information, and photo perspective information. The last convolution module of the ResNet-50 model is connected to the first average pooling layer and the second average pooling layer for scene feature extraction, as well as the third average pooling layer and the fourth average pooling layer; the third average pooling layer is connected to the first fully connected layer for outputting scene categories, and the fourth average pooling layer is connected to the second fully connected layer for outputting perspective categories.

[0034] First, a semantic segmentation model is constructed to extract scene semantic feature representation. Figure 1 As shown, this technical solution uses PSPNet as the basic network structure of the model, compressing the two-dimensional feature map G output by PSPNet into a one-dimensional vector f with 150 components s_all , where f s_all Each component represents the number of pixels classified into a set category. The PSPNet semantic segmentation model uses the open source ADE20K as a training data set, which consists of 20,000 images containing 150 types of semantic information. In order to obtain semantic information related to spatial location information, this technical solution adopts the method of extracting feature values ​​by grid, that is, Figure 2 As shown, according to the original area Extract the scene semantic information of the four sub-images of the image, namely the left sub-image, the right sub-image, the upper sub-image and the lower sub-image, denoted as f s_left ,f s_right ,f s_up ,f s_down And add weights and perform eigenvalue splicing (the splicing of eigenvalues ​​is done with operator), usually with weight w left ,w right ,w up ,w down Set to w all of As shown below:

[0035]

[0036] Get 750-dimensional scene semantic features f s =[x1,x2,x3,…,x 750 ].

[0037] In addition, if the user clicks on the scene in the viewfinder that is of focus or an object that the user wishes to interact with (such as a chair, table, railing, staircase, grass, etc.), the coordinates of the user's click (px ,p y ) will also be used as the input of the semantic segmentation model, and this coordinate will be brought into the two-dimensional feature map G output by the PSPNet model to obtain the two-dimensional coordinates The corresponding category is P. From this we can see that the strong tag that the user is concerned about is category P.

[0038] Step 2: Construct a residual network model to extract scene feature representation and view feature representation. Figure 1 As shown in the figure, this technical solution uses ResNet-50 as the basic network structure of the model. In order to extract scene information and texture information, the output of the last convolution module of the ResNet-50 model is used as the feature map f of the image scene information. c_1×1 =[y1,y2,y3,…,y 2048 ],like Figure 3 In order to obtain scene category information related to spatial position, this technical solution also provides another output as an option, that is, the size parameter of the average pooling layer in the last convolution module of the ResNet-50 model is changed from 1×1 to 2×2, which is recorded as feature map f c_2×2 , to more conveniently obtain spatial information, such as Figure 3 As shown. For the convenience of processing, the feature map f c_2×2 Compressed into a one-dimensional vector f c_2×2 =[z1,z2,z3,…,z 8192 ]. In addition, for the extraction of perspective features, this solution adopts the residual network multi-task training method for training. The training details are as follows (Note: The size parameter of the average pooling layer in the last convolution module of the ResNet-50 model is changed to 2×2 for the purpose of facilitating feature extraction. During the model training process, the average pooling layer of size 1×1 is still added with two fully connected layers of different sizes for dual-output multi-task training. After the training is completed, the scene category features under the two average pooling layers are generated in the prediction stage. For details, see Figure 3 ):

[0039] First, the vanishing point V (v) in the scene is detected by the method of [Rother 2002] (see Rother CA new approach to vanishing point detection in architectural environments [J]. Image and Vision Computing, 2002, 20 (9-10): 647-655). x ,v y ), and classify the viewing angles in the following way (the default image length and width are L input and W input ):

[0040]

[0041] From the above formula, we can see that the perspective classification is divided into 4 categories, that is, the original image is divided into four equal parts. Categories 0, 1, 2, and 3 represent the four categories where the scene vanishing points are located in the upper right part, lower right part, upper left part, and lower left part of the original image respectively.

[0042] First, we collect image datasets of different scenes (such as the Places365 public dataset), annotate different scene categories (such as grass, beach, bedroom, cafe, etc.), and use the above [Rother 2002] method combined with formula (2) to annotate the perspective category, and obtain a training dataset X containing N photos, X = [x1, x2, x3, ..., x N ] and their corresponding scene category labels and view category labels. During model training, the model outputs scene categories and view categories. When extracting features after training, since this solution changes the output of the last convolutional module of ResNet50, the model output is no longer the scene category. The model will be more aware of the scene texture information of the photo, and with the output of the view category, the model will also be aware of the spatial direction information.

[0043] Two loss functions for scene features and view features in multi-task training (Loss scene ,Loss view ) is the mean square error function L2loss (MSE), written as:

[0044]

[0045]

[0046] Where N is the number of photos in the training dataset, View i Indicates the view category label corresponding to the i-th photo (since there are 4 view categories, View i There will be 4 dimensions, corresponding to the label category value of 1, and the rest of the values ​​​​are 0), y i represents the scene category label corresponding to the i-th photo (e.g., for the Places365 dataset with 365 categories, y i There will be 365 dimensions, corresponding to the label category value of 1, and the rest of the values ​​0), f view The result of (·) is the i-th photo x i is the viewing angle category output by the model as input, f scene The result of (·) is the i-th photo x i is the scene category output by the model when input. Thus, the view feature f can be obtained through the residual network according to the input picture. view=[v1,v2,v3,v4].

[0047] Step 3: Feature fusion. According to the feature vector f obtained in step 1 s , the feature f obtained in step 2 c_1×1 ,f c_2×2 , f view Normalize them each using the L2 norm:

[0048]

[0049]

[0050]

[0051]

[0052] And with weight q s ,q c_1×1 ,q c_2×2 ,q v Add (usually q c_1×1 and q c_2×2 The weight values ​​are set to the same, and are all q s Weight value q v Weight and q s same):

[0053] f input =f s_l2 ·q s +f c_1×1_l2 ·q c_1×1 +f c_2×2_l2 ·q c_2×2 +f view_l2 ·q v (9)

[0054] Step 4: Create a feature library. Perform the feature extraction process described above on each photo in the posture image database D and store it in the feature library D. f , where the eigenvalue corresponding to the i-th picture is recorded as

[0055]

[0056] Step 5: Feature library comparison. Use the cosine distance formula to calculate f input With each feature library Similarity between:

[0057]

[0058] Step 6: Retrieve the pose image library. Sort by the similarity obtained in step 3, return the first m results, and return the m photos corresponding to these m features in the image library, which are the m poses recommended by this technical solution in the current scene. If there is a strong label, filter the photos classified as category P with a confidence level of more than 80%, sort by similarity, and return the recommended results.

[0059] Although the specific embodiments of the present invention are disclosed for the purpose of illustration, the purpose is to help understand the content of the present invention and implement it accordingly, those skilled in the art will understand that various substitutions, changes and modifications are possible without departing from the spirit and scope of the present invention and the appended claims. Therefore, the present invention should not be limited to the content disclosed in the best embodiment, and the scope of the present invention is subject to the scope defined in the claims.

Claims

1. A method for recommending photographing postures by integrating image content and geometric feature extraction, the steps of which include: Extract scene semantic features, scene features, and view features from the image input by the user ； The extracted scene semantic features, scene features, and view features are integrated according to the set rules; Calculate the similarity between the fused features and the features in the feature database, and return the posture images corresponding to the m features with the highest similarity to the user; The method for extracting the scene semantic features is as follows: The image is divided into left sub-image, right sub-image, upper sub-image and lower sub-image; the semantic segmentation model is used to extract the two-dimensional feature map G of each sub-image, and each two-dimensional feature map G is compressed into a one-dimensional vector as the scene semantic information of the corresponding sub-image, denoted as f s_left ,f s_right ,f s_up ,f s_down ; Then the scene semantic information of each sub-image is weighted and spliced ​​to obtain the scene semantic feature f s ; Wherein, a residual network is used to extract the viewing angle feature and the scene feature; the residual network model uses a ResNet-50 model as a basic network structure, and the last convolution module of the ResNet-50 model is respectively connected to a first average pooling layer and a second average pooling layer for scene feature extraction, and is respectively connected to a third average pooling layer and a fourth average pooling layer; The third average pooling layer is connected to the first fully connected layer to output the scene category, and the fourth average pooling layer is connected to the second fully connected layer to output the perspective category. The method for training the residual network is: first, collect image data sets of different scenes, and then for each image in the image data set, detect the vanishing point V (v x ,v y ), and through The image is classified by perspective, and the scene category of the image is marked according to the visual classification result View of the image, where L input is the length of the image, W input is the width of the image; then the training data set X=[x1,x2,x3,…,x N ] and the corresponding scene category labels and perspective category labels, train the residual network, and the loss function used in training the residual network includes Among them, View i represents the view category label of the i-th image, y i represents the scene category label of the i-th image, f view (x i ) is the i-th photo x i is the viewing angle category output by the model as input, f scene (x i ) is the i-th photo x i is the scene category output by the model when it is input.

2. The method according to claim 1, characterized in that When the user clicks on an object in the image as a focused scene or an object that the user wishes to interact with, the coordinates of the object clicked by the user (p x ,p y ) will be used as the input of the semantic segmentation model, and the corresponding two-dimensional coordinates in the output two-dimensional feature map G The corresponding category P is used as a strong label.

3. The method according to claim 2, characterized in that Use the cosine distance formula to calculate the fused feature f input With each feature in the feature library Similarity between them; the confidence of the image category P in the posture image database is greater than the set threshold, and the features of the image are consistent with the fusion feature f input The m posture images with the highest similarity are returned to the user.

4. The method according to claim 1, characterized in that: The last convolutional module output of the trained ResNet-50 model is used as the scene feature f c_1×1 ; and changing the size parameter of the average pooling layer in the last convolutional module of the trained ResNet-50 model to 2×2, and outputting the feature map f of the convolutional module c_2×2 Compressed into a one-dimensional vector f c_2×2 ; Then the scene feature f c_1×1 , one-dimensional vector f c_2×2 As the scene feature.

5. The method according to claim 1, characterized in that The scene semantic features, scene features, and viewing angle features are respectively normalized by L2 norm and then weighted fused to obtain fused features.

6. The method according to claim 1, characterized in that Extract features from each photo in the posture image database D, and fuse the extracted scene semantic features, scene features, and view features according to the set rules to obtain the corresponding feature library D f .

7. A server, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, the computer program is configured to be executed by the processor, and the computer program comprises instructions for executing each step in the method according to any one of claims 1 to 6.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images