A method, system, camera device and storage medium for taking a composition
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SHENZHEN LINKRIC TECH CO LTD
- Filing Date
- 2024-03-15
- Publication Date
- 2026-05-29
Smart Images
Figure CN118250554B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image capture technology, and more specifically, to a method, system, camera device, and storage medium for composing images. Background Technology
[0002] Existing camera devices typically require manual control of the camera movement to compose a shot. This high degree of human intervention makes it difficult for ordinary users lacking photography knowledge to produce high-quality photographs. Summary of the Invention
[0003] In view of this, the present invention discloses a method, system, camera device and storage medium for shooting composition, which predicts the optimal composition frame and aesthetic score of the image to be processed through an aesthetic evaluation model, and automatically captures the target composition area with a high aesthetic score. Therefore, it can produce high-quality photographic works and saves users from manually controlling the movement of the camera device to shoot composition.
[0004] A method of composing a photograph includes:
[0005] The video sequence to be processed is obtained from the captured video stream at preset time intervals;
[0006] Obtain the image to be processed from the video sequence to be processed;
[0007] The image to be processed is input into a pre-trained aesthetic evaluation model to predict the optimal composition box of the image to be processed under a preset aspect ratio, and the aesthetic score of the target composition area captured by the optimal composition box.
[0008] When the aesthetic score is greater than the aesthetic score threshold, the target composition area is captured.
[0009] Optionally, the aesthetic evaluation model includes: a feature extraction backbone network, an aesthetic composition branch, and an aesthetic scoring branch;
[0010] The step of inputting the image to be processed into a pre-trained aesthetic evaluation model to predict the optimal composition box of the image to be processed under a preset aspect ratio, and the aesthetic score of the target composition region cropped by the optimal composition box, includes:
[0011] Based on the feature extraction backbone network, image features of the image to be processed are extracted;
[0012] The image features are input into the aesthetic composition branch to predict the optimal composition box of the image to be processed under the preset aspect ratio.
[0013] The image features and the optimal composition box are input into the aesthetic scoring branch to predict the aesthetic score of the target composition region.
[0014] Optionally, the step of inputting the image features into the aesthetic composition branch to predict the optimal composition box of the image to be processed under the preset aspect ratio includes:
[0015] The image features are input into the aesthetic composition branch to predict the bounding boxes of each position of the image to be processed under the preset aspect ratio.
[0016] Based on the aesthetic composition branch, the global weighted weights of the image to be processed at each of the aforementioned locations are predicted;
[0017] Based on the bounding box corresponding to each predicted position and the global weighting weight, the global weighted average of the bounding box prediction results is calculated channel by channel to obtain the average box under the preset aspect ratio.
[0018] The average frame is determined as the optimal composition frame.
[0019] Optionally, the step of inputting the image features and the optimal composition box into the aesthetic scoring branch to predict the aesthetic score of the target composition region includes:
[0020] Based on the aesthetic scoring feature mapping layer of the aesthetic scoring branch, the image features are mapped to the aesthetic scoring feature space to obtain aesthetic scoring features;
[0021] The aesthetic score features and the optimal composition box are input into the aesthetic score prediction layer of the aesthetic score branch for region of interest pooling and multilayer perceptron processing to predict the aesthetic score of the target composition region.
[0022] Optionally, after obtaining the video sequence to be processed from the captured video stream at preset time intervals, the method further includes:
[0023] The video sequence to be processed is input into the highlight behavior recognition model, and the features of each single frame image are extracted based on the backbone network of the highlight behavior recognition model. The highlight behavior recognition model is trained using a video sequence dataset labeled with highlight behavior categories and trained with classification loss as supervision.
[0024] Behavioral features are extracted from the features of each single-frame image, and the information from multiple frames of images is integrated to obtain video sequence features;
[0025] Based on the video sequence features, the behavior recognition module in the highlight behavior recognition model is used to determine all behaviors and their corresponding probability values in the video sequence to be processed.
[0026] The highest probability value among all the stated behavior values is selected as the highlight moment score.
[0027] When the highlight moment score is greater than the highlight moment score threshold, the behavior corresponding to the highlight moment score is determined as a highlight behavior;
[0028] Capture the target video sequence containing the highlight behavior from the video sequence to be processed;
[0029] The aesthetic evaluation model is used to predict the optimal composition box for all single-frame images in the target video sequence and the aesthetic score corresponding to the target composition region captured by the optimal composition box;
[0030] From the multiple predicted optimal composition boxes, select the optimal composition box with an aesthetic score greater than the aesthetic score threshold as the target composition box;
[0031] The image of the area where the target frame is located is captured as the highlight moment image.
[0032] Optionally, the training process of the aesthetic evaluation model includes:
[0033] Obtain an aesthetic evaluation training dataset, wherein each training image in the aesthetic evaluation training dataset is labeled with an optimal composition box and a corresponding aesthetic score;
[0034] The aesthetic evaluation training dataset is input into the initial aesthetic evaluation model to obtain the predicted bounding box and predicted aesthetic score for each training image.
[0035] Based on the optimal bounding box and the predicted bounding box of each of the training images, the aesthetic composition loss function value is obtained, and based on the aesthetic score and the predicted aesthetic score of each of the training images, the aesthetic score loss function value is obtained.
[0036] The final loss function value is obtained based on the aesthetic composition loss function value and the aesthetic scoring loss function value;
[0037] Using the loss function value, the parameters of the initial aesthetic evaluation model are optimized through the gradient descent algorithm to obtain the final aesthetic evaluation model.
[0038] Optionally, obtaining the final loss function value based on the aesthetic composition loss function value and the aesthetic scoring loss function value includes:
[0039] Determine whether the main subject of the photograph has framing information;
[0040] If not, the sum of the aesthetic composition loss function value and the aesthetic scoring loss function value is determined as the loss function value;
[0041] If so, then define the main objective integrity loss function;
[0042] Based on the main target integrity loss function, the negative logarithm of the ratio of the area of the target region contained within the composition frame to the total area of the target region is calculated, and the negative logarithm is determined as the value of the main target integrity loss function;
[0043] The sum of the aesthetic composition loss function value, the aesthetic scoring loss function value, and the subject target integrity loss function value is determined as the loss function value.
[0044] A system for composing a photograph, comprising:
[0045] The video sequence acquisition unit is used to acquire video sequences to be processed from the captured video stream at preset time intervals.
[0046] An image acquisition unit is used to acquire images to be processed from the video sequence to be processed;
[0047] The prediction unit is used to input the image to be processed into a pre-trained aesthetic evaluation model to predict the optimal composition box of the image to be processed under a preset aspect ratio, and the aesthetic score of the target composition area captured by the optimal composition box.
[0048] The snapshot unit is used to capture the target composition area when the aesthetic score is greater than the aesthetic score threshold.
[0049] A camera device includes: a lens, a gimbal, and a control module, wherein the lens is mounted on the gimbal, and the gimbal is connected to the control module;
[0050] The control module includes a memory and a processor;
[0051] The memory is used to store at least one instruction;
[0052] The processor is used to execute the at least one instruction to implement the above-described method for composing a photograph.
[0053] A computer-readable storage medium storing at least one instruction that, when executed by a processor, implements the above-described method for composing a photograph.
[0054] As can be seen from the above technical solution, this invention discloses a method, system, camera device, and storage medium for shooting composition. It acquires a video sequence to be processed from a captured video stream at preset time intervals, acquires an image to be processed from the video sequence, and inputs the image to be processed into a pre-trained aesthetic evaluation model. The model predicts the optimal composition frame of the image to be processed at a preset aspect ratio, and the aesthetic score of the target composition area captured by the optimal composition frame. When the aesthetic score is greater than an aesthetic score threshold, the target composition area is captured. This invention predicts the optimal composition frame and aesthetic score of the image to be processed through an aesthetic evaluation model, and automatically captures the target composition area with a high aesthetic score. Therefore, it can produce high-quality photographic works and eliminates the need for the user to manually control the camera device to move and compose the shot. Attached Figure Description
[0055] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the published drawings without creative effort.
[0056] Figure 1 This is a flowchart of a method for composing a photograph, as disclosed in an embodiment of the present invention.
[0057] Figure 2 This invention discloses an embodiment of an aesthetic evaluation model that inputs an image to be processed into a pre-trained model.
[0058] Figure 3 This is a structural diagram of an aesthetic evaluation model disclosed in an embodiment of the present invention;
[0059] Figure 4 This is a flowchart of another method for composing a photograph disclosed in an embodiment of the present invention;
[0060] Figure 5 This is a flowchart illustrating the training and highlight behavior recognition process of a highlight behavior recognition model disclosed in an embodiment of the present invention.
[0061] Figure 6 This is a flowchart of a training method for an aesthetic evaluation model disclosed in an embodiment of the present invention;
[0062] Figure 7 This is a schematic diagram illustrating the training of an aesthetic evaluation model disclosed in an embodiment of the present invention;
[0063] Figure 8 This is a schematic diagram of the structure of a shooting composition system disclosed in an embodiment of the present invention;
[0064] Figure 9This is a schematic diagram of the structure of a control module in a camera device disclosed in an embodiment of the present invention. Detailed Implementation
[0065] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0066] This invention discloses a method, system, camera device, and storage medium for shooting composition. It involves acquiring a video sequence to be processed from a captured video stream at preset time intervals, acquiring an image to be processed from the video sequence, and inputting the image to be processed into a pre-trained aesthetic evaluation model. The model predicts the optimal composition frame of the image to be processed at a preset aspect ratio, as well as the aesthetic score of the target composition area captured by the optimal composition frame. When the aesthetic score is greater than an aesthetic score threshold, the target composition area is captured. This invention predicts the optimal composition frame and aesthetic score of the image to be processed through an aesthetic evaluation model, and automatically captures target composition areas with high aesthetic scores. Therefore, it can produce high-quality photographic works and eliminates the need for users to manually control the camera device to move and compose the shot.
[0067] See Figure 1 The present invention discloses a flowchart of a method for composing a photograph, the method comprising:
[0068] Step S101: Obtain the video sequence to be processed from the captured video stream at preset time intervals;
[0069] In this embodiment, the video stream is the video stream captured by the camera device.
[0070] The video sequence to be processed consists of video segments obtained from a video stream.
[0071] The value of the preset time interval depends on the actual needs, such as 5 seconds, but this invention does not limit it.
[0072] Step S102: Obtain the image to be processed from the video sequence to be processed;
[0073] The video sequence to be processed consists of many frames of images. In this embodiment, the images to be processed are obtained from the video sequence to be processed.
[0074] Step S103: Input the image to be processed into a pre-trained aesthetic evaluation model to predict the optimal composition box of the image to be processed under a preset aspect ratio, and the aesthetic score of the target composition area captured by the optimal composition box.
[0075] The aesthetic evaluation model in this embodiment uses a deep neural network to extract features and can predict the target composition area and its aesthetic score under the current situation.
[0076] Among them, the higher the aesthetic score of the composition area, the more it conforms to photographic aesthetics.
[0077] In this embodiment, the target composition area is the image area framed by the optimal composition frame in the image to be processed.
[0078] The preset aspect ratio determines the size of the composition frame. The value of the preset aspect ratio depends on the actual needs, such as a 9:16 frame, a 16:9 frame, and an unlimited frame.
[0079] A 9:16 box indicates that the aspect ratio of the prediction box is fixed at 9:16.
[0080] A 16:9 box indicates that the aspect ratio of the prediction box is fixed at 16:9.
[0081] An unrestricted box can represent a prediction box with any aspect ratio.
[0082] Step S104: When the aesthetic score is greater than the aesthetic score threshold, the target composition area is captured.
[0083] The value of the aesthetic rating threshold is determined according to actual needs, and this invention does not limit it.
[0084] In summary, this invention discloses a method for shooting composition. It involves acquiring a video sequence to be processed from a captured video stream at preset time intervals, acquiring an image to be processed from the video sequence, and inputting the image to be processed into a pre-trained aesthetic evaluation model. This model predicts the optimal composition frame of the image to be processed at a preset aspect ratio, as well as the aesthetic score of the target composition area captured by the optimal composition frame. When the aesthetic score is greater than an aesthetic score threshold, the target composition area is captured. This invention predicts the optimal composition frame and aesthetic score of the image to be processed through an aesthetic evaluation model, and automatically captures target composition areas with high aesthetic scores. Therefore, it can produce high-quality photographic works and eliminates the need for users to manually control the movement of the camera device for shooting composition.
[0085] It should be noted that the aesthetic evaluation model in this invention includes: a feature extraction backbone network, an aesthetic composition branch, and an aesthetic scoring branch.
[0086] To further optimize the above embodiments, see [link to relevant documentation]. Figure 2This invention discloses a method flowchart for inputting an image to be processed into a pre-trained aesthetic evaluation model to predict the optimal composition box of the image under a preset aspect ratio, and the aesthetic score of the target composition region captured by the optimal composition box. The method includes:
[0087] Step S201: Extract image features of the image to be processed based on the feature extraction backbone network.
[0088] Image features include color features, texture features, and spatial relationship features.
[0089] Step S202: Input the image features into the aesthetic composition branch to predict the optimal composition box of the image to be processed under the preset aspect ratio.
[0090] Specifically, (1) the image features are input into the aesthetic composition branch to predict the bounding boxes of each position of the image to be processed under the preset aspect ratio.
[0091] See details Figure 3 The aesthetic evaluation model structure diagram shown assumes that the bounding boxes under the preset aspect ratio include three types: 9:16 box, 16:9 box, and unrestricted box.
[0092] For 9:16 and 16:9 bounding boxes, only the distances from the anchor point to the left (l), top (t), and right (r) sides of the bounding box are predicted when any point on the image to be processed is used as the anchor point.
[0093] See details Figure 3 The distances of the three sides predicted for a 9:16 box include: l1 t1 r1, and the distances of the three sides predicted for a 16:9 box include: l2 t2 r2.
[0094] For unbounded boxes, predict the distances from the anchor point to the left (l), top (t), right (r), and bottom (b) edges of the bounding box. See details. Figure 3 The predicted distances for the four edges are: l0 t0 r0b0.
[0095] (2) Based on the aesthetic composition branch, the global weighted weight of the image to be processed at each position is predicted.
[0096] The global weighting weight can be represented by w. The global weighting weight represents the confidence level of the prediction of each bounding box boundary at each location in the image to be processed. The higher the weighting weight of a certain channel at a certain location, the more the aesthetic evaluation model trusts the prediction result of the corresponding bounding box boundary at that location.
[0097] In this embodiment, the global weighting is the same as the number of channels and the spatial size of the bounding box. See details. Figure 3As shown, the global weighted weights for the 9:16 bounding box are: w1 w2 w3. The global weighted weights for the 16:9 bounding box are: w4 w5 w6. The global weighted weights for the unrestricted bounding box are: w7 w8 w9 w 10 .
[0098] (3) Based on the bounding boxes corresponding to each predicted position and the global weighted weight, calculate the global weighted average of the bounding box prediction results for each channel to obtain the average box under the preset aspect ratio.
[0099] The aesthetic composition branch calculates a global weighted average of the predicted bounding box results channel by channel based on the predicted bounding box and the weights of each position, to obtain the average box under the preset aspect ratio.
[0100] See details Figure 3 The average frame corresponding to frame 9:16 is
[0101] The average frame corresponding to the 16:9 frame is:
[0102] The average box corresponding to an unrestricted box is:
[0103] Among them, for the 16:9 average frame and the 9:16 average frame with a fixed aspect ratio, the position of the lower boundary needs to be calculated based on the left, top, and right boundaries. and To ensure that the aspect ratio meets the requirements, the unrestricted average frame is obtained directly through global weighted averaging.
[0104] (4) Determine the average frame as the optimal frame.
[0105] Step S203: Input the image features and the optimal composition box into the aesthetic scoring branch to predict the aesthetic score of the target composition area.
[0106] Assuming the optimal frame composition consists of three types: a 9:16 average frame, a 16:9 average frame, and an unrestricted average frame, see details below. Figure 3 As shown, the aesthetic score of the composition area corresponding to the predicted 9:16 average frame is s1.
[0107] The predicted aesthetic score for the composition area corresponding to the 16:9 average frame is s2.
[0108] The predicted aesthetic score for the composition area corresponding to the unrestricted average frame is s0.
[0109] Specifically, (1) the aesthetic rating feature mapping layer based on the aesthetic rating branch maps the image features to the aesthetic rating feature space to obtain the aesthetic rating features;
[0110] (2) Input the aesthetic score features and the optimal composition box into the aesthetic score prediction layer of the aesthetic score branch for region of interest pooling and multilayer perceptron processing to predict the aesthetic score of the target composition region.
[0111] Among them, the aesthetic score can be used as a basis for identifying highlight moments and automatically capturing images with high aesthetic scores.
[0112] It should be noted that, in addition to automatic composition and snapshot capture, this invention can also identify highlights in videos. The highlight recognition model is trained for specific application scenarios, using specific behaviors or events as conditions for high scores, such as the moment a goal is scored in a basketball game.
[0113] Therefore, to further optimize the above embodiments, after obtaining the video sequence to be processed from the captured video stream at preset time intervals, in Figure 1 Based on the illustrated embodiment, it is also possible to identify target video sequences containing highlight behavior and capture them.
[0114] For details, see Figure 4 The present invention discloses another method for composing a photograph, the method comprising:
[0115] Step S301: Obtain the video sequence to be processed from the captured video stream at preset time intervals;
[0116] Step S302: Input the video sequence to be processed into the specular behavior recognition model, and extract the features of each single frame image based on the backbone network of the specular behavior recognition model;
[0117] See details Figure 5 The flowchart shown is for the training and recognition of the highlight behavior recognition model. The highlight behavior recognition model is trained using a video sequence dataset labeled with highlight behavior categories and trained with classification loss as supervision.
[0118] Specifically, the training and highlight behavior recognition process of the highlight behavior recognition model includes:
[0119] See details Figure 5 Using a video sequence dataset labeled with highlight behavior categories as training data, a backbone network is employed to extract behavioral features from the training data center and perform behavior recognition on these features. The recognized behaviors include various highlight behaviors, such as standing (S1), rolling (S2), ..., jumping (S...). n etc.
[0120] When training the highlight behavior recognition model, the cross-entropy classification loss function is used as supervision. Based on the classification prediction results (1, 2, ..., n) and the corresponding training data labels (such as standing S1), the cross-entropy classification loss function value is calculated, and the parameters of the highlight behavior recognition model are optimized by the gradient descent algorithm.
[0121] After training, the highlight behavior is identified based on the highlight behavior recognition model.
[0122] It should be noted that the categories of highlight behaviors are determined based on actual needs. For example, the categories of highlight behaviors for pets include: standing, rolling, sticking out tongue, jumping, etc.
[0123] In training the highlight behavior recognition model, datasets from different specific scenarios can be used to expand the application scope of the highlight moment recognition model.
[0124] Step S303: Extract behavioral features from the features of each single frame image and integrate the information from multiple frames to obtain video sequence features;
[0125] Step S304: Based on the video sequence features, use the behavior recognition module in the highlight behavior recognition model to determine all behaviors and their corresponding probability values in the video sequence to be processed.
[0126] Step S305: Select the behavior with the highest probability from all behavior probability values as the highlight moment score;
[0127] See details Figure 5 From standing S1, rolling S2, ..., jumping S n The one with the highest probability is selected as the highlight moment score.
[0128] Step S306: When the highlight moment score is greater than the highlight moment score threshold, the behavior corresponding to the highlight moment score is determined as the highlight behavior;
[0129] Step S307: Capture the target video sequence containing highlight behavior in the video sequence to be processed;
[0130] Step S308: Use an aesthetic evaluation model to predict the optimal composition box for all single-frame images in the target video sequence and the aesthetic score corresponding to the target composition area captured by the optimal composition box.
[0131] Step S309: From the multiple predicted optimal composition boxes, select the optimal composition box with an aesthetic score greater than the aesthetic score threshold as the target composition box;
[0132] Step S310: Capture the image of the area where the target frame is located as the highlight moment image.
[0133] In summary, this invention identifies highlight behavior in a video sequence by inputting it into a highlight behavior recognition model, thereby capturing the target video sequence containing highlight behavior. This results in high-quality photographic works and eliminates the need for users to manually control the camera movement for shooting and composition.
[0134] To prevent the recording of too many repetitive highlight moments in a single day, this invention proposes a hierarchical scoring system. This system scores and sorts the highlight moments recorded throughout the day, selecting a certain number of videos and images from each category for saving, while other materials are automatically deleted. Specifically, the hierarchical scoring system is first divided into an image scoring system and a video scoring system. The image scoring system sorts based on image aesthetic scores, retaining images with higher aesthetic scores. The video scoring system sorts according to highlight moment categories. Taking pet highlight behaviors as an example, video clips categorized into standing, rolling, sticking out their tongues, jumping, etc., are sorted by highlight moment score within their respective categories, and then a certain number of high-scoring videos are selected and saved from each category.
[0135] To further optimize the above embodiments, see [link to relevant documentation]. Figure 6 The present invention discloses a flowchart of a training method for an aesthetic evaluation model, the method comprising:
[0136] Step S401: Obtain the aesthetic evaluation training dataset.
[0137] In the aesthetic evaluation training dataset, each training image is labeled with its optimal bounding box and corresponding aesthetic score. Therefore, see details... Figure 7 The diagram shown illustrates the training of the aesthetic evaluation model. The aesthetic evaluation training dataset includes: an aesthetic composition dataset and an aesthetic rating dataset.
[0138] Step S402: Input the aesthetic evaluation training dataset into the initial aesthetic evaluation model to obtain the predicted bounding box and predicted aesthetic score for each training image.
[0139] Specifically, the initial aesthetic evaluation model includes: a feature extraction backbone network, an aesthetic composition branch, and an aesthetic scoring branch.
[0140] The aesthetic evaluation training dataset is input into the initial aesthetic evaluation model. The feature extraction backbone network, aesthetic composition branch, and aesthetic scoring branch are used to obtain the predicted bounding boxes and predicted aesthetic scores for each training image. For example, bounding boxes with aspect ratios of 9:16, 16:9, and unlimited aspect ratios are obtained, as well as the aesthetic scores for the training images.
[0141] Step S403: Based on the optimal bounding box and predicted bounding box of each training image, obtain the aesthetic composition loss function value, and based on the aesthetic score and predicted aesthetic score of each training image, obtain the aesthetic score loss function value.
[0142] The aesthetic composition loss function is the sum of the distances between the predicted composition box and the four sides of the labeled optimal composition box, which is the L1 norm loss.
[0143] The aesthetic rating loss function is the root mean square of the difference between the predicted aesthetic rating and the labeled aesthetic rating, i.e., the MSE loss.
[0144] Step S404: Obtain the final loss function value based on the aesthetic composition loss function value and the aesthetic scoring loss function value;
[0145] Step S405: Using the loss function value, optimize the parameters of the initial aesthetic evaluation model through the gradient descent algorithm to obtain the final aesthetic evaluation model.
[0146] It should be noted that the gradient descent algorithm used in this embodiment is also known as... Figure 7 The propagation process is shown by the dashed arrows indicating backpropagation in the illustrated embodiment.
[0147] To further optimize the above embodiments, step S404 may specifically include:
[0148] Determine whether the main subject of the photograph has framing information;
[0149] If not, the sum of the aesthetic composition loss function value and the aesthetic scoring loss function value is determined as the loss function value;
[0150] If so, then define the main objective integrity loss function;
[0151] Based on the main target integrity loss function, the negative logarithm of the ratio of the area of the target region contained within the composition frame to the total area of the target region is calculated, and the negative logarithm is determined as the value of the main target integrity loss function.
[0152] The sum of the aesthetic composition loss function value, the aesthetic scoring loss function value, and the subject object integrity loss function value is determined as the loss function value.
[0153] Before determining the final loss function value, this embodiment needs to determine whether the captured subject has frame information. If the subject has frame information, a subject integrity loss function is defined, causing the aesthetic evaluation model to tend to include the complete subject when predicting the optimal frame. Specifically, the subject integrity loss function value is the negative logarithm of the ratio of the area of the target region contained within the frame to the total area of the target region. At this point, the sum of the aesthetic composition loss function value, the aesthetic scoring loss function value, and the subject integrity loss function value is determined as the final loss function value.
[0154] Corresponding to the above method embodiments, the present invention also discloses a system for composing a photograph.
[0155] See Figure 8 The present invention discloses a structural schematic diagram of a system for shooting composition, the system comprising:
[0156] The video sequence acquisition unit 501 is used to acquire a video sequence to be processed from the captured video stream at preset time intervals.
[0157] In this embodiment, the video stream is the video stream captured by the camera device.
[0158] The video sequence to be processed consists of video segments obtained from a video stream.
[0159] The value of the preset time interval depends on the actual needs, such as 5 seconds, but this invention does not limit it.
[0160] Image acquisition unit 502 is used to acquire images to be processed from the video sequence to be processed;
[0161] The video sequence to be processed consists of many frames of images. In this embodiment, the images to be processed are obtained from the video sequence to be processed.
[0162] The prediction unit 503 is used to input the image to be processed into a pre-trained aesthetic evaluation model to predict the optimal composition box of the image to be processed under a preset aspect ratio, and the aesthetic score of the target composition area captured by the optimal composition box.
[0163] The aesthetic evaluation model in this embodiment uses a deep neural network to extract features and can predict the target composition area and its aesthetic score under the current situation.
[0164] Among them, the higher the aesthetic score of the composition area, the more it conforms to photographic aesthetics.
[0165] In this embodiment, the target composition area is the image area framed by the optimal composition frame in the image to be processed.
[0166] The preset aspect ratio determines the size of the composition frame. The value of the preset aspect ratio depends on the actual needs, such as a 9:16 frame, a 16:9 frame, and an unlimited frame.
[0167] A 9:16 box indicates that the aspect ratio of the prediction box is fixed at 9:16.
[0168] A 16:9 box indicates that the aspect ratio of the prediction box is fixed at 16:9.
[0169] An unrestricted box can represent a prediction box with any aspect ratio.
[0170] The snapshot unit 504 is used to capture the target composition area when the aesthetic score is greater than the aesthetic score threshold.
[0171] The value of the aesthetic rating threshold is determined according to actual needs, and this invention does not limit it.
[0172] In summary, this invention discloses a system for shooting composition. It acquires a video sequence to be processed from a captured video stream at preset time intervals, obtains an image to be processed from the video sequence, and inputs the image to be processed into a pre-trained aesthetic evaluation model. The model predicts the optimal composition frame of the image to be processed at a preset aspect ratio, as well as the aesthetic score of the target composition area captured by the optimal composition frame. When the aesthetic score is greater than an aesthetic score threshold, the target composition area is captured. This invention predicts the optimal composition frame and aesthetic score of the image to be processed through an aesthetic evaluation model, and automatically captures the target composition area with a high aesthetic score. Therefore, it can produce high-quality photographic works and eliminates the need for users to manually control the movement of the camera device for shooting composition.
[0173] The aesthetic evaluation model includes: a feature extraction backbone network, an aesthetic composition branch, and an aesthetic scoring branch;
[0174] Prediction unit 503 includes:
[0175] The feature extraction subunit is used to extract image features of the image to be processed based on the feature extraction backbone network.
[0176] The optimal composition box prediction subunit is used to input the image features into the aesthetic composition branch and predict the optimal composition box of the image to be processed under the preset aspect ratio.
[0177] The aesthetic score prediction subunit is used to input the image features and the optimal composition box into the aesthetic score branch to predict the aesthetic score of the target composition region.
[0178] The optimal frame prediction sub-unit is specifically used for:
[0179] The image features are input into the aesthetic composition branch to predict the bounding boxes of each position of the image to be processed under the preset aspect ratio.
[0180] Based on the aesthetic composition branch, the global weighted weights of the image to be processed at each of the aforementioned locations are predicted;
[0181] Based on the bounding box corresponding to each predicted position and the global weighting weight, the global weighted average of the bounding box prediction results is calculated channel by channel to obtain the average box under the preset aspect ratio.
[0182] The average frame is determined as the optimal composition frame.
[0183] The aesthetic rating prediction subunit is specifically used for:
[0184] Based on the aesthetic scoring feature mapping layer of the aesthetic scoring branch, the image features are mapped to the aesthetic scoring feature space to obtain aesthetic scoring features;
[0185] The aesthetic score features and the optimal composition box are input into the aesthetic score prediction layer of the aesthetic score branch for region of interest pooling and multilayer perceptron processing to predict the aesthetic score of the target composition region.
[0186] The system for composing a shot may also include:
[0187] The image feature extraction unit is used to input the video sequence to be processed into the highlight behavior recognition model, and extract the image features of each single frame based on the backbone network of the highlight behavior recognition model. The highlight behavior recognition model is trained using a video sequence dataset labeled with highlight behavior categories and trained with classification loss.
[0188] The video sequence feature determination unit is used to extract behavioral features from the features of each single frame image and integrate the information of multiple frames to obtain video sequence features;
[0189] The behavior probability value determination unit is used to determine all behaviors and their corresponding behavior probability values in the video sequence to be processed based on the video sequence features and using the behavior recognition module in the highlight behavior recognition model.
[0190] The highlight moment scoring determination unit is used to select the maximum behavior probability from all the behavior probability values as the highlight moment score;
[0191] The highlight behavior determination unit is used to determine the behavior corresponding to the highlight moment score as a highlight behavior when the highlight moment score is greater than the highlight moment score threshold.
[0192] The video capture unit is used to capture the target video sequence containing the highlight behavior in the video sequence to be processed;
[0193] The aesthetic score determination unit is used to predict the optimal composition box and the aesthetic score corresponding to the target composition region captured by the optimal composition box for all single-frame images in the target video sequence using the aesthetic evaluation model.
[0194] The target composition box determination unit is used to select the optimal composition box with an aesthetic score greater than the aesthetic score threshold from the multiple predicted optimal composition boxes as the target composition box;
[0195] The highlight image capture unit is used to capture the image of the area where the target composition frame is located as the highlight moment image.
[0196] The system for shooting composition may also include: an aesthetic evaluation model training unit.
[0197] The aesthetic evaluation model training unit is specifically used for:
[0198] Obtain an aesthetic evaluation training dataset, wherein each training image in the aesthetic evaluation training dataset is labeled with an optimal composition box and a corresponding aesthetic score;
[0199] The aesthetic evaluation training dataset is input into the initial aesthetic evaluation model to obtain the predicted bounding box and predicted aesthetic score for each training image.
[0200] Based on the optimal bounding box and the predicted bounding box of each of the training images, the aesthetic composition loss function value is obtained, and based on the aesthetic score and the predicted aesthetic score of each of the training images, the aesthetic score loss function value is obtained.
[0201] The final loss function value is obtained based on the aesthetic composition loss function value and the aesthetic scoring loss function value;
[0202] Using the loss function value, the parameters of the initial aesthetic evaluation model are optimized through the gradient descent algorithm to obtain the final aesthetic evaluation model.
[0203] The aesthetic evaluation model training unit is also specifically used for:
[0204] Determine whether the main subject of the photograph has framing information;
[0205] If not, the sum of the aesthetic composition loss function value and the aesthetic scoring loss function value is determined as the loss function value;
[0206] If so, then define the main objective integrity loss function;
[0207] Based on the main target integrity loss function, the negative logarithm of the ratio of the area of the target region contained within the composition frame to the total area of the target region is calculated, and the negative logarithm is determined as the value of the main target integrity loss function;
[0208] The sum of the aesthetic composition loss function value, the aesthetic scoring loss function value, and the subject target integrity loss function value is determined as the loss function value.
[0209] It should be noted that for the specific working principles of each component in the device embodiment, please refer to the corresponding section of the method embodiment, which will not be repeated here.
[0210] Corresponding to the above embodiments, see [link to relevant documentation]. Figure 9 The present invention also discloses a schematic diagram of the structure of a control module in a camera device. The camera device includes a lens, a gimbal, and a control module. The lens is mounted on the gimbal, and the gimbal is connected to the control module.
[0211] The control module may include: processor 1 and memory 2;
[0212] The processor 1 and memory 2 communicate with each other via communication bus 3.
[0213] Processor 1, for executing at least one instruction;
[0214] Memory 2 is used to store at least one instruction;
[0215] Processor 1 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention.
[0216] Memory 2 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.
[0217] The processor executes at least one instruction to implement the steps shown in the embodiment of the method for composing a photograph.
[0218] Corresponding to the above embodiments, the present invention also discloses a computer-readable storage medium that stores at least one instruction, which, when executed by a processor, implements the steps shown in the embodiments of the method for composing a photograph.
[0219] In summary, this invention discloses an electronic device and a computer-readable storage medium. It acquires a video sequence to be processed from a captured video stream at preset time intervals, acquires an image to be processed from the video sequence, and inputs the image to be processed into a pre-trained aesthetic evaluation model. The model predicts the optimal composition frame of the image to be processed at a preset aspect ratio, and the aesthetic score of the target composition area captured by the optimal composition frame. When the aesthetic score is greater than an aesthetic score threshold, the target composition area is captured. This invention predicts the optimal composition frame and aesthetic score of the image to be processed through an aesthetic evaluation model, and automatically captures target composition areas with high aesthetic scores. Therefore, it can produce high-quality photographic works and eliminates the need for users to manually control the movement of the camera device for shooting and composition.
[0220] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0221] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.
[0222] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A method for composing a photograph, characterized in that, include: The video sequence to be processed is obtained from the captured video stream at preset time intervals; Obtain the image to be processed from the video sequence to be processed; The image to be processed is input into a pre-trained aesthetic evaluation model to predict the optimal composition box of the image to be processed under a preset aspect ratio, and the aesthetic score of the target composition area captured by the optimal composition box. When the aesthetic score is greater than the aesthetic score threshold, the target composition area is captured. The process of inputting the image to be processed into a pre-trained aesthetic evaluation model to predict the optimal frame of the image under a preset aspect ratio includes: The aesthetic evaluation model predicts the bounding box of each position in the image to be processed under a preset aspect ratio. The bounding box under the preset aspect ratio includes a fixed aspect ratio box and an unrestricted box. The fixed aspect ratio includes a 9:16 box and a 16:9 box. For the fixed aspect ratio bounding box, only the distances from the anchor point to the left, top, and right sides of the bounding box are predicted when any point on the image to be processed is used as the anchor point. For the unbounded bounding box, predict the distances from the anchor point to the left, top, right, and bottom four sides of the bounding box; Based on the bounding boxes corresponding to each predicted position and the global weighted weights, the global weighted average of the bounding box prediction results is calculated channel by channel to obtain the average box under the preset aspect ratio. When the average frame is an average frame with a fixed aspect ratio, the position of the lower boundary is calculated based on the left, top, and right boundaries of the average frame, and the average frame after calculating the lower boundary is determined as the optimal composition frame; When the average frame is an unrestricted average frame, the average frame containing four boundaries is determined as the optimal frame.
2. The method according to claim 1, characterized in that, The aesthetic evaluation model includes: a feature extraction backbone network, an aesthetic composition branch, and an aesthetic scoring branch; The step of inputting the image to be processed into a pre-trained aesthetic evaluation model to predict the optimal composition box of the image to be processed under a preset aspect ratio, and the aesthetic score of the target composition region cropped by the optimal composition box, includes: Based on the feature extraction backbone network, image features of the image to be processed are extracted; The image features are input into the aesthetic composition branch to predict the optimal composition box of the image to be processed under the preset aspect ratio. The image features and the optimal composition box are input into the aesthetic scoring branch to predict the aesthetic score of the target composition region.
3. The method according to claim 2, characterized in that, The step of inputting the image features and the optimal composition box into the aesthetic scoring branch to predict the aesthetic score of the target composition region includes: Based on the aesthetic scoring feature mapping layer of the aesthetic scoring branch, the image features are mapped to the aesthetic scoring feature space to obtain aesthetic scoring features; The aesthetic score features and the optimal composition box are input into the aesthetic score prediction layer of the aesthetic score branch for region of interest pooling and multilayer perceptron processing to predict the aesthetic score of the target composition region.
4. The method according to claim 1, characterized in that, After obtaining the video sequence to be processed from the captured video stream at preset time intervals, the method further includes: The video sequence to be processed is input into the highlight behavior recognition model, and the features of each single frame image are extracted based on the backbone network of the highlight behavior recognition model. The highlight behavior recognition model is trained using a video sequence dataset labeled with highlight behavior categories and trained with classification loss as supervision. Behavioral features are extracted from the features of each single-frame image, and the information from multiple frames of images is integrated to obtain video sequence features; Based on the video sequence features, the behavior recognition module in the highlight behavior recognition model is used to determine all behaviors and their corresponding probability values in the video sequence to be processed. The highest probability value among all the stated behavior values is selected as the highlight moment score. When the highlight moment score is greater than the highlight moment score threshold, the behavior corresponding to the highlight moment score is determined as a highlight behavior; Capture the target video sequence containing the highlight behavior from the video sequence to be processed; The aesthetic evaluation model is used to predict the optimal composition box for all single-frame images in the target video sequence and the aesthetic score corresponding to the target composition region captured by the optimal composition box; From the multiple predicted optimal composition boxes, select the optimal composition box with an aesthetic score greater than the aesthetic score threshold as the target composition box; The image of the area where the target frame is located is captured as the highlight moment image.
5. The method according to any one of claims 1 to 4, characterized in that, The training process of the aesthetic evaluation model includes: Obtain an aesthetic evaluation training dataset, wherein each training image in the aesthetic evaluation training dataset is labeled with an optimal composition box and a corresponding aesthetic score; The aesthetic evaluation training dataset is input into the initial aesthetic evaluation model to obtain the predicted bounding box and predicted aesthetic score for each training image. Based on the optimal bounding box and the predicted bounding box of each of the training images, the aesthetic composition loss function value is obtained, and based on the aesthetic score and the predicted aesthetic score of each of the training images, the aesthetic score loss function value is obtained. The final loss function value is obtained based on the aesthetic composition loss function value and the aesthetic scoring loss function value; Using the loss function value, the parameters of the initial aesthetic evaluation model are optimized through the gradient descent algorithm to obtain the final aesthetic evaluation model.
6. The method according to claim 5, characterized in that, The process of obtaining the final loss function value based on the aesthetic composition loss function value and the aesthetic scoring loss function value includes: Determine whether the main subject of the photograph has framing information; If not, the sum of the aesthetic composition loss function value and the aesthetic scoring loss function value is determined as the loss function value; If so, then define the main objective integrity loss function; Based on the main target integrity loss function, the negative logarithm of the ratio of the area of the target region contained within the composition frame to the total area of the target region is calculated, and the negative logarithm is determined as the value of the main target integrity loss function; The sum of the aesthetic composition loss function value, the aesthetic scoring loss function value, and the subject target integrity loss function value is determined as the loss function value.
7. A system for composing a photograph, characterized in that, include: The video sequence acquisition unit is used to acquire video sequences to be processed from the captured video stream at preset time intervals. An image acquisition unit is used to acquire images to be processed from the video sequence to be processed; The prediction unit is used to input the image to be processed into a pre-trained aesthetic evaluation model to predict the optimal composition box of the image to be processed under a preset aspect ratio, and the aesthetic score of the target composition area captured by the optimal composition box. The snapshot unit is used to capture the target composition area when the aesthetic score is greater than the aesthetic score threshold. The process by which the prediction unit inputs the image to be processed into a pre-trained aesthetic evaluation model and predicts the optimal frame of the image to be processed under a preset aspect ratio includes: the aesthetic evaluation model predicts the bounding box of each position of the image to be processed under the preset aspect ratio. The bounding box under the preset aspect ratio includes a fixed aspect ratio box and an unrestricted box. The fixed aspect ratio includes a 9:16 box and a 16:9 box. For the fixed aspect ratio bounding box, only the distances from the anchor point to the left, top, and right sides of the bounding box are predicted when any point on the image to be processed is used as the anchor point. For the unbounded bounding box, predict the distances from the anchor point to the left, top, right, and bottom four sides of the bounding box; Based on the bounding boxes corresponding to each predicted position and the global weighted weights, the global weighted average of the bounding box prediction results is calculated channel by channel to obtain the average box under the preset aspect ratio. When the average frame is an average frame with a fixed aspect ratio, the position of the lower boundary is calculated based on the left, top, and right boundaries of the average frame, and the average frame after calculating the lower boundary is determined as the optimal composition frame; When the average frame is an unrestricted average frame, the average frame containing four boundaries is determined as the optimal frame.
8. A camera device, characterized in that, include: The lens, the gimbal, and the control module are provided, wherein the lens is mounted on the gimbal and the gimbal is connected to the control module. The control module includes a memory and a processor; The memory is used to store at least one instruction; The processor is used to execute the at least one instruction to implement the shooting composition method as described in any one of claims 1 to 6.
9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one instruction, which, when executed by a processor, implements the method of composing a photograph as described in any one of claims 1 to 6.