Auxiliary shooting method and device, electronic equipment and storage medium

CN119012001APending Publication Date: 2024-11-22SAMSUNG ELECTRONICS CHINA R&D CENT +1
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202411077385.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-07
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

When ordinary users use mobile camera equipment to shoot, it is difficult for them to take satisfactory photos or videos because they lack professional photography knowledge, resulting in poor shooting results.

Method used

By identifying the user's shooting intention, a three-dimensional model of the shooting area is built, and based on this, the shooting suggestions are generated, including shooting position, angle and parameters, to help the user improve the shooting quality.

Benefits of technology

It effectively improves users' shooting confidence and experience, enables ordinary users to take professional-level photos and videos, solving the problem of poor shooting.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119012001A_ABST
    Figure CN119012001A_ABST
Patent Text Reader

Abstract

The invention provides an auxiliary shooting method and device, electronic equipment and a storage medium, and relates to the technical field of computer vision algorithms, in particular to the technical field of camera shooting. According to the specific implementation scheme, the method comprises the following steps: constructing a spatial three-dimensional model of an area shown by a pre-shot image according to the pre-shot image of mobile camera equipment; determining a shooting intention corresponding to the pre-shot image; and outputting shooting suggestion information for the area based on the spatial three-dimensional model and the shooting intention. On the basis of identifying the shooting intention of the user, the shooting suggestion information for assisting the user in shooting is accurately generated.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the technical field of computer vision algorithms, in particular to the technical field of camera technology, and in particular to an auxiliary shooting method, device, electronic device and storage medium. Background Art

[0002] As an important product of modern technology, mobile camera equipment is widely used in many fields such as monitoring, photography, and live broadcast.

[0003] However, since users do not have professional photography knowledge, it is often difficult for them to take satisfactory photos or videos when using mobile camera equipment to shoot subjects such as scenery, people, buildings, and crafts. Summary of the invention

[0004] The present disclosure provides an auxiliary shooting method, device, electronic device and storage medium. The present disclosure accurately generates shooting suggestion information for assisting the user in shooting based on identifying the user's shooting intention.

[0005] According to a first aspect of the present disclosure, an auxiliary shooting method is provided, including: constructing a spatial three-dimensional model of the area shown in the pre-shot image based on a pre-shot image of a mobile camera device; determining the shooting intention corresponding to the pre-shot image; and outputting shooting suggestion information for the area based on the spatial three-dimensional model and the shooting intention.

[0006] In some embodiments, the above-mentioned construction of a spatial three-dimensional model of the area shown in the pre-captured image based on the pre-captured image of the mobile camera device includes: determining the local appearance information of objects in the area and the relative positions between objects based on the pre-captured image; constructing a three-dimensional model of the object based on the local appearance information; and generating a spatial three-dimensional model based on the three-dimensional model and the relative positions between the objects.

[0007] In some embodiments, the above-mentioned construction of a three-dimensional model of an object based on local appearance information includes: inputting the local appearance information into the object appearance model to determine the appearance characteristics of the object; building a three-dimensional model of the object based on the appearance characteristics; wherein the object appearance model is used to characterize the correspondence between the local appearance information of the object and the global appearance characteristics.

[0008] In some embodiments, the above-mentioned determination of the shooting intention corresponding to the pre-shot image includes: extracting features of the pre-shot image to obtain shooting features; inputting the shooting features into an intention recognition model to determine the shooting intention, wherein the intention recognition model is a pre-trained large language model.

[0009] In some embodiments, the above-mentioned output of shooting suggestion information for an area based on the spatial three-dimensional model and shooting intention includes: generating multiple initial image frames based on the spatial three-dimensional model and multiple sets of preset shooting information, wherein the shooting information includes shooting position, shooting angle and shooting parameters; screening the multiple initial image frames based on the shooting intention to obtain multiple candidate image frames; and outputting shooting suggestion information for the area based on the multiple candidate image frames.

[0010] In some embodiments, the above-mentioned output of shooting suggestion information for a region based on multiple candidate image frames includes: in response to the user's shooting mode taking a photo of a scene, inputting multiple candidate image frames into a photo model to determine aesthetic scores of the multiple candidate image frames, wherein the photo model is a multimodal collaborative self-attention neural network model; based on the aesthetic scores, selecting a target image frame from the multiple candidate image frames; outputting shooting suggestion information for the region, wherein the shooting suggestion information includes the target image frame and its shooting information.

[0011] In some embodiments, the above-mentioned output of shooting suggestion information for an area based on multiple candidate image frames includes: in response to a user shooting mode to take a photo of a person, determining multiple alternative image frames based on multiple pre-selected virtual objects and multiple candidate image frames, wherein the virtual objects are person postures or matching schemes; inputting the multiple alternative image frames into the photo model respectively to determine the aesthetic scores of the multiple alternative image frames; selecting a target image frame from the multiple candidate image frames based on the aesthetic scores of the multiple alternative image frames; outputting shooting suggestion information for the area, wherein the shooting suggestion information includes the target image frame and the virtual object information and shooting information corresponding to the target image frame.

[0012] In some embodiments, the above-mentioned output of shooting suggestion information for a region based on multiple candidate image frames includes: in response to the user's shooting mode being video recording, generating multiple image frame combinations based on multiple candidate image frames; inputting the multiple image frame combinations into a video model respectively to determine aesthetic scores of the multiple image frame combinations; based on the aesthetic scores, selecting a target image frame combination from the multiple image frame combinations; generating a target video based on the target image frame combination; outputting shooting suggestion information for the region, wherein the shooting suggestion information includes shooting information of each frame image in the target video as well as shot switching information and shooting parameter adjustment information between adjacent image frames.

[0013] In some embodiments, the construction process of the above-mentioned video model includes: inputting the video sample into a deep learning model based on the self-attention mechanism to determine the video description corresponding to the video sample; based on the video description, determining the main subject in the video sample; inputting the video frame containing the main subject in the video sample into the photo model to determine the aesthetic score of the video frame; based on the aesthetic score of the video frame, determining the key frame corresponding to the video sample; and training the video model using the key frames and aesthetic scores corresponding to the video sample.

[0014] According to a second aspect of the present disclosure, an auxiliary shooting device is provided, including: a construction module, configured to construct a spatial three-dimensional model of the area shown in the pre-shot image based on the pre-shot image of a mobile camera device; a determination module, configured to determine the shooting intention corresponding to the pre-shot image; and an output module, configured to output shooting suggestion information for the area based on the spatial three-dimensional model and the shooting intention.

[0015] According to a third aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the auxiliary shooting method provided by the first aspect.

[0016] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the auxiliary shooting method provided by the first aspect.

[0017] According to a fifth aspect of the present disclosure, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the auxiliary shooting method provided by the first aspect is implemented.

[0018] The auxiliary shooting method, device, electronic device and storage medium provided by the present disclosure construct a spatial three-dimensional model of the area shown in the pre-shot image based on the pre-shot image of the mobile camera device; determine the shooting intention corresponding to the pre-shot image; and output shooting suggestion information for the area based on the spatial three-dimensional model and the shooting intention. Based on the understanding of the user's shooting intention, appropriate shooting suggestion information is recommended to the user based on the regional three-dimensional model of the scene that the user wants to shoot, so as to avoid users without shooting skills from not being able to take good photos and videos, and effectively improve the user's shooting confidence and shooting experience.

[0019] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0021] Figure 1 is an exemplary system architecture diagram in which the present disclosure may be applied;

[0022] Figure 2 is a flow chart of an embodiment of an auxiliary shooting method according to the present disclosure;

[0023] Figure 3 is a flow chart of another embodiment of the auxiliary shooting method according to the present disclosure;

[0024] Figure 4 is a flow chart of another embodiment of the auxiliary shooting method according to the present disclosure;

[0025] Figure 5 is a structural schematic diagram of an embodiment of the auxiliary shooting device disclosed in the present invention;

[0026] Figure 6 is a block diagram of an electronic device for implementing the auxiliary shooting method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0027] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0028] Figure 1 An exemplary system architecture 100 is shown to which embodiments of the auxiliary shooting method and auxiliary shooting device of the present disclosure can be applied.

[0029] like Figure 1 As shown, the system architecture 100 may include mobile camera devices 101, 102, 103, a network 104, and a server 105. The network 104 is used to provide a medium for communication links between the mobile camera devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.

[0030] Users can use mobile camera devices 101, 102, 103 to interact with server 105 via network 104, and can also interact with each other via server 105. Mobile camera devices 101, 102, 103 can be various electronic devices with camera functions, including but not limited to cameras on smartphones, portable mobile cameras, head-mounted cameras, etc.

[0031] The server 105 can provide various services. For example, the server 105 obtains pre-shot images from the mobile camera devices 101, 102, and 103, analyzes and processes the pre-shot images, and obtains shooting suggestion information for the area shown in the pre-shot images. The server 105 recommends the shooting suggestion information to the mobile camera devices 101, 102, and 103 to assist the users of the mobile camera devices 101, 102, and 103 to shoot.

[0032] It should be noted that the server 105 can be hardware or software. When the server 105 is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or it can be implemented as a single server. When the server 105 is software, it can be implemented as multiple software or software modules (for example, for providing distributed services), or it can be implemented as a single software or software module. No specific limitation is made here.

[0033] It should be noted that the auxiliary shooting method provided in the embodiment of the present disclosure is generally executed by the server 105 , and accordingly, the auxiliary shooting device is generally disposed in the server 105 .

[0034] It should be understood that Figure 1 The number of mobile devices, networks and servers in the embodiment is only for illustration. Any number of mobile devices, networks and servers may be provided as required.

[0035] Continue to refer Figure 2 , which shows a process 200 of an embodiment of an auxiliary shooting method according to the present disclosure. The method comprises the following steps:

[0036] Step 201: construct a spatial three-dimensional model of the area shown in the pre-shot image based on the pre-shot image of the mobile camera device.

[0037] In this embodiment, the execution subject of the auxiliary shooting method (for example Figure 1 The server 105 shown in the figure may first obtain a mobile camera device (eg Figure 1 The pre-shot images uploaded by the mobile camera devices 101, 102, 103 shown in the figure are used to construct a spatial three-dimensional model of the area shown in the pre-shot images.

[0038] In some optional implementations of this embodiment, the mobile camera device may include one camera or multiple cameras, and the types of cameras include but are not limited to RGB cameras, infrared cameras, and depth-of-field cameras.

[0039] The above-mentioned pre-captured image, also called preview image or viewfinder image, refers to the image that has been pre-stored in the internal buffer of the mobile camera device before the shutter is officially pressed. As an example, the above-mentioned pre-captured image may be one, such as the imaging picture of a single camera at a specific shooting angle and shooting position; or it may be multiple, such as the pictures presented by a single or multiple cameras at multiple shooting positions and / or shooting angles. The area shown in the above-mentioned pre-captured image refers to the area covered by all the pre-captured images. Compared with one pre-captured image, multiple pre-captured images will be more helpful in constructing a spatial three-dimensional model in the area.

[0040] In some optional implementations of this embodiment, the step of constructing a spatial three-dimensional model of the area shown in the pre-shot image according to the pre-shot image of the mobile camera device includes:

[0041] Step 1: Determine the object image and relative position of the objects in the area shown by the pre-shot image.

[0042] As an example, the above-mentioned determination of the object images and the relative positions of objects in the area shown by the pre-captured image includes: in response to there being only one pre-captured image, capturing the object images of all objects therein from the pre-captured image, determining each object image as the object image of the corresponding object in the area shown by the pre-captured image, and determining the relative positions of objects in the pre-captured image as the relative positions of objects in the area shown by the pre-captured image.

[0043] Among them, capturing the object image of all objects in the pre-captured image from the pre-captured image includes: inputting the pre-captured image into a visual language pre-training model to determine the category label of each object in the pre-captured image; inputting the category label of each object into an open world target detection model to determine the detection frame of each object in the pre-captured image; inputting the detection frame of each object into an image segmentation model to determine the object image of each object in the pre-captured image.

[0044] The above-mentioned visual language pre-training model is an image labeling model, such as the RAM model (RecognizeAnything Model) and the Tag2Text model (visual-language model based on image labeling), etc. It recognizes objects in the input image and adds category labels. For example, an image describing a cat sleeping on the sofa is input into the visual language pre-training model, and the visual language pre-training model recognizes the cat and the sofa and adds corresponding labels of cat and sofa.

[0045] The above-mentioned open-world object detection model, such as detecting arbitrary objects in images by text prompts (Grounding-DINO), can detect the task object through the input text (such as category name, referential expression, etc.) and obtain the corresponding detection box.

[0046] The above-mentioned image segmentation models, such as the SAM model (Segment Anything Model), can segment the edges of the object to be segmented through the input segmentation hints to obtain the corresponding object image of the object to be segmented.

[0047] Furthermore, the process of determining the relative positions of objects in the pre-shot image includes: determining the object position of each object in the pre-shot image based on the detection frame of each object in the pre-shot image; and determining the relative positions of objects in the pre-shot image based on the object position of each object.

[0048] As another example, the above-mentioned determination of the object images of objects and the relative positions between objects in the area shown by the pre-captured image includes: in response to the presence of multiple pre-captured images, capturing the object images of all objects therein from each pre-captured image; for each object, determining the object images corresponding to the object in all pre-captured images as the object image of the object in the area shown by the pre-captured image; and determining the fusion calculation result of the relative positions between objects in each pre-captured image as the relative positions between objects in the area shown by the pre-captured image.

[0049] Step 2: Based on the object images and the relative positions between the objects, a spatial three-dimensional model of the area shown in the pre-shot image is constructed.

[0050] As an example, the above-mentioned construction of a spatial three-dimensional model of the area shown in the pre-shot image based on the object image and the relative position between the objects includes: constructing a three-dimensional model of the object based on the object image; constructing a spatial three-dimensional model of the area shown in the pre-shot image based on the three-dimensional model of the object and the relative position between the objects.

[0051] The above-mentioned construction of the three-dimensional model of the object according to the object image may be: in response to there being only one image of the object and it being taken by a depth-of-field camera, the three-dimensional model of the object is constructed based on the object image and the depth information contained therein.

[0052] The above-mentioned construction of a three-dimensional model of an object based on the object image can also be: in response to there being only one image of the object, obtaining depth information corresponding to the object image based on a sensor (such as a radar sensor) installed on a mobile camera device, the depth information including the angular distance relationship between the sensor and the object corresponding to the object image, and constructing a three-dimensional model of the object based on the object image and its corresponding depth information.

[0053] The above-mentioned constructing the three-dimensional model of the object according to the object image may also be: in response to there being multiple object images, constructing the three-dimensional model of the object using the multiple object images.

[0054] Step 202: Determine the shooting intention corresponding to the pre-shot image.

[0055] In this embodiment, after acquiring the pre-shot image, the execution subject may determine the shooting intention corresponding to the pre-shot image.

[0056] In some optional implementations of this embodiment, the above-mentioned shooting intention mainly refers to the main content that the photographer wants to shoot, such as the description of the objects, positions, and distance relationships within the viewfinder.

[0057] As an example, the above-mentioned execution subject can receive the shooting intention input by the user through voice, text, touch, etc.

[0058] As another example, the above-mentioned execution subject can extract features from the pre-shot image to obtain shooting features; input the shooting features into the intention recognition model to determine the shooting intention.

[0059] In response to there being only one pre-shot image, feature extraction is performed on the pre-shot image to obtain a shooting feature. In response to there being multiple pre-shot images, feature extraction is performed on each pre-shot image respectively, and the extracted results are fused to obtain a shooting feature.

[0060] The above-mentioned intention recognition model is a large language model pre-selected and trained using the first training sample set. The large language model performs semantic analysis on the input image and obtains the main content that the photographer wants to shoot. The samples in the first training sample set include image features and their corresponding shooting intentions.

[0061] As another example, the execution subject may obtain a labeled image obtained by processing a pre-shot image using a visual language pre-training model; and input the labeled image into a first intention recognition model to determine the shooting intention.

[0062] In response to there being only one pre-shot image, the labeled image is an image obtained by inputting the pre-shot image into the visual language pre-training model. In response to there being multiple pre-shot images, the labeled image is a fused image of each first image obtained by inputting each pre-shot image into the visual language pre-training model.

[0063] The first intention recognition model is a large language model pre-selected and trained using a second training sample set, where samples in the second training sample set include labeled images and their corresponding shooting intentions.

[0064] Step 203: Based on the spatial three-dimensional model and the shooting intention, output shooting suggestion information for the area shown in the pre-shot image.

[0065] In this embodiment, after constructing a spatial three-dimensional model of the area shown in the pre-shot image and determining the shooting intention, the above-mentioned execution entity can output shooting suggestion information for the area shown in the pre-shot image based on the spatial three-dimensional model and the shooting intention.

[0066] In some optional implementations of this embodiment, the above-mentioned shooting suggestion information refers to information that can assist users in shooting and improve the user's shooting effect, including but not limited to shooting position, shooting composition, shooting angle and shooting parameters.

[0067] The above shooting parameters include but are not limited to hue, saturation, brightness, complementary colors, dual tones, high dynamic range, image grain, white background light, soft focus and vanishing point.

[0068] The above-mentioned output of shooting suggestion information for the area shown in the pre-shot image based on the spatial three-dimensional model and shooting intention can be: based on the spatial three-dimensional model, determine multiple two-dimensional images with different shooting positions and different shooting angles; filter out two-dimensional images that do not match the shooting intention from the multiple two-dimensional images; and output shooting suggestion information for the area based on the aesthetic effect of the multiple two-dimensional images after filtering.

[0069] The auxiliary shooting method provided by the embodiment of the present disclosure first constructs a three-dimensional regional model of the scene to be shot by the user based on the pre-shot image, and this regional three-dimensional model can provide a preview of the shooting images at multiple shooting positions and multiple shooting angles; secondly, the pre-shot image is semantically understood to understand the user's shooting intention; finally, appropriate shooting suggestion information is recommended to the user based on the user's intention and the regional three-dimensional model, which solves the pain point that ordinary users cannot perform professional-level shooting and effectively improves the user's shooting confidence and shooting experience.

[0070] Continue to refer Figure 3 , which shows a process 300 of another embodiment of the auxiliary shooting method according to the present disclosure. The method includes the following steps:

[0071] Step 301: Determine local appearance information of objects in the area shown in the pre-shot image and relative positions between objects.

[0072] In this embodiment, the execution subject of the auxiliary shooting method (for example Figure 1 The server 105 shown in the figure may first obtain a mobile camera device (eg Figure 1 The method further comprises the step of: obtaining a pre-shot image uploaded by the mobile camera devices 101, 102, 103 as shown in the figure, and then determining the local appearance information of the objects in the area shown in the pre-shot image and the relative positions between the objects.

[0073] In some optional implementations of this embodiment, the above-mentioned appearance information refers to external visual information of the object, such as shape, color, size, etc.

[0074] The above-mentioned pre-shot image and the above-mentioned specific operation of determining the relative positions of the objects in the area shown in the pre-shot image have been Figure 2 Step 201 in the illustrated embodiment has been described in detail and will not be repeated here.

[0075] The above-mentioned determining the local appearance information of the object in the area shown by the pre-shot image includes: determining the object image of the object in the area shown by the pre-shot image; and determining the local appearance information of the object according to the object image.

[0076] The specific operation of determining the object image of the object in the area shown in the pre-captured image has been described in Figure 2 Step 201 in the illustrated embodiment is described in detail and will not be repeated here. The above-mentioned determining the local appearance information of the object based on the object image includes: in response to there being only one object image, extracting the appearance information of the object from the object image, and recording it as the local appearance information of the object. In response to there being multiple object images, extracting the appearance information of the object from each object image, and fusing the extracted appearance information, and recording the fusion result as the local appearance information of the object.

[0077] Step 302: construct a three-dimensional model of the object based on the local appearance information.

[0078] In this embodiment, after determining the local appearance information of the object, the execution subject may construct a three-dimensional model of the object based on the local appearance information of the object.

[0079] In some optional implementations of this embodiment, the above-mentioned constructing a three-dimensional model of the object based on local appearance information includes: determining the appearance features of the object based on the local appearance information; and constructing the three-dimensional model of the object based on the appearance features of the object.

[0080] The main purpose of determining the appearance features of an object based on local appearance information is to infer the global appearance features of the object based on the local appearance information of the object (the photographed area), and the global appearance features can characterize the appearance information of the unphotographed area of ​​the object.

[0081] As an example, the above-mentioned determination of the appearance features of an object based on local appearance information can be implemented as follows: matching the local appearance information with the appearance information in the object appearance library; and using the appearance features corresponding to the matched appearance information as the appearance features of the object.

[0082] Among them, the above-mentioned matching of local appearance information with the appearance information in the object appearance library means that the similarity between the local appearance information and the appearance information in the object appearance library is greater than a preset threshold; the above-mentioned object appearance library records the global appearance information and global appearance features of each object.

[0083] As another example, the above-mentioned determination of the appearance features of an object based on local appearance information can be implemented as follows: the local appearance information is input into the object appearance model to determine the appearance features of the object; wherein the object appearance model is used to characterize the correspondence between the local appearance information of the object and the global appearance features.

[0084] As another example, the above-mentioned determination of the appearance features of an object based on local appearance information can be implemented as follows: in response to the absence of appearance information matching the local appearance information in the object appearance library or the object appearance model having difficulty in learning the appearance features of the object based on the local appearance information of the object, the appearance features of the object are acquired from an external network based on the category labeling of the object.

[0085] The above-mentioned construction of a three-dimensional model of an object based on the appearance features of the object can be implemented as follows: constructing a three-dimensional structure of the object according to the appearance features, performing skin rendering on the three-dimensional structure, and obtaining a three-dimensional model of the object.

[0086] Step 303: Generate a spatial three-dimensional model of the area shown in the pre-shot image according to the relative position between the three-dimensional model and the object.

[0087] In this embodiment, after determining the three-dimensional model of the object, the execution subject may generate a spatial three-dimensional model of the area shown in the pre-shot image according to the relative position between the three-dimensional model and the object.

[0088] In some optional implementations of this embodiment, generating a spatial three-dimensional model of the area shown in the pre-shot image according to the relative position between the three-dimensional model and the object includes:

[0089] The three-dimensional models of the objects are combined according to the relative positional relationship between the objects to obtain a spatial three-dimensional model of the area shown in the pre-shot image.

[0090] Step 304: Determine the shooting intention corresponding to the pre-shot image.

[0091] Step 305: Based on the spatial three-dimensional model and the shooting intention, output shooting suggestion information for the area shown in the pre-shot image.

[0092] The specific operations of step 304 and step 305 are already described in Figure 2 In the illustrated embodiment, steps 202 and 203 are described in detail and will not be repeated here.

[0093] The auxiliary shooting method provided by the embodiment of the present disclosure infers the global appearance information of the object through the appearance information of the photographed area of ​​the object, and then uses the global appearance information of the object to construct a three-dimensional model of the object, which can improve the accuracy of the construction.

[0094] Continue to refer Figure 4 , which shows a process 400 of another embodiment of the auxiliary shooting method according to the present disclosure. The method includes the following steps:

[0095] Step 401: construct a spatial three-dimensional model of the area shown in the pre-shot image based on the pre-shot image of the mobile camera device.

[0096] Step 402: Determine the shooting intention corresponding to the pre-shot image.

[0097] The specific operations of step 401 and step 402 are already in Figure 2 In the illustrated embodiment, steps 201 and 202 are described in detail and will not be repeated here.

[0098] Step 403: Generate multiple initial image frames based on the spatial three-dimensional model and multiple sets of preset shooting information, wherein the shooting information includes shooting position, shooting angle and shooting parameters;

[0099] In this embodiment, after acquiring the spatial three-dimensional model, the execution subject may generate a plurality of initial image frames according to the spatial three-dimensional model and a plurality of sets of preset shooting information.

[0100] In some optional implementations of the present embodiment, the above-mentioned execution entity can determine the two-dimensional picture corresponding to each shooting position and each shooting angle through the above-mentioned three-dimensional spatial model, and then determine the effect picture when the mobile camera device shoots the two-dimensional picture corresponding to each shooting position and each shooting angle with each set of shooting parameters, and the effect picture is the initial image frame.

[0101] Step 404: based on the shooting intention, multiple initial image frames are screened to obtain candidate image frames;

[0102] In this embodiment, after determining a plurality of initial image frames, the execution subject may filter the plurality of initial image frames based on the shooting intention to obtain candidate image frames.

[0103] In some optional implementations of the present embodiment, the execution subject may respectively determine the shooting intentions of a plurality of initial image frames; and use the initial image frames whose shooting intentions match the above shooting intentions among the plurality of initial image frames as candidate image frames.

[0104] The execution subject may also determine the main subject based on the shooting intention; and use the initial image frame containing the main subject among the multiple initial image frames as the candidate image frame.

[0105] Step 405: Based on the multiple candidate image frames, output shooting suggestion information for the area shown in the pre-shot image.

[0106] In this embodiment, after determining a plurality of candidate image frames, the execution subject may output shooting suggestion information for the area shown in the pre-shot image according to the plurality of candidate image frames.

[0107] In some optional implementations of this embodiment, the above-mentioned execution subject may first determine the user shooting mode, and then output shooting suggestion information for the area shown in the pre-shot image according to the user shooting mode and multiple candidate image frames.

[0108] The above user shooting modes are classified according to different shooting objects, including but not limited to scene shooting, person shooting and video shooting.

[0109] As an example, the above-mentioned execution entity outputs shooting suggestion information for the area shown in the pre-shot image based on multiple candidate image frames. It can be as follows: in response to the user's shooting mode to take a photo of the scene, the multiple candidate image frames are respectively input into the photo model to determine the aesthetic scores of the multiple candidate image frames; based on the aesthetic scores, a target image frame is selected from the multiple candidate image frames; based on the target image frame, shooting suggestion information for the area shown in the pre-shot image is output.

[0110] Among them, the above-mentioned photo model is a multimodal collaborative self-attention neural network model, which can learn the attributes of the input image and predict the overall aesthetic level of the image based on the attributes.

[0111] The above attributes include but are not limited to photographic parameters and composition parameters. The above composition parameters include but are not limited to the distance between the identified objects, the distance between the identified objects and the photographer, the shooting angle, the light and dark contrast and shadows formed by the lighting, etc.

[0112] The above-mentioned selecting a target image frame from multiple candidate image frames based on aesthetic scores includes: sorting the multiple candidate image frames in descending order of aesthetic scores; and selecting the first candidate image frame or the first preset number of candidate image frames in the front sorting as the target image frame.

[0113] The outputting of shooting suggestion information for the area shown in the pre-shot image based on the target image frame includes: using the target image frame and its shooting information as the shooting suggestion information.

[0114] In this example, the above-mentioned execution entity selects one or more image frames that meet the user's shooting intention and are ranked at the top in aesthetic level as target image frames, and outputs the target image frames with their corresponding shooting information. The mobile camera device displays the scene shooting effects and shooting information of the target image frames on its display screen to guide the user to compose the shot.

[0115] As another example, the above-mentioned execution entity outputs shooting suggestion information for the area shown in the pre-captured image based on multiple candidate image frames, which can be: in response to the user's shooting mode to take a photo of a person, multiple candidate image frames are adjusted based on multiple pre-selected person postures to obtain multiple alternative image frames; the multiple alternative image frames are respectively input into the photo model to determine the aesthetic scores of the multiple alternative image frames; based on the aesthetic scores of the multiple alternative image frames, a target image frame is selected from the multiple candidate image frames; based on the target image frame, the shooting suggestion information for the area is output.

[0116] The above-mentioned multiple pre-selected character postures are the character postures to be recommended selected by the above-mentioned execution subject after analyzing the character features and scene features in the pre-shot image. The above-mentioned character features refer to the character's expression, body shape, clothing, etc.

[0117] The above-mentioned selecting a target image frame from multiple candidate image frames based on aesthetic scores includes: sorting the multiple candidate image frames in descending order of aesthetic scores; and selecting the first candidate image frame or a second preset number of candidate image frames in the front order as the target image frame.

[0118] The above-mentioned output of shooting suggestion information for the area shown in the pre-shot image based on the target image frame may include: in response to the virtual object being a human posture, using the target image frame, the shooting information of the target image frame and the human posture in the target image frame as shooting suggestion information.

[0119] In this example, the above-mentioned execution entity combines the virtual character posture with the candidate image frames determined based on the three-dimensional model to generate alternative image frames, and then selects one or more of the candidate image frames with the highest aesthetic level as the target image frames, and outputs the target image frames with their corresponding shooting information. The mobile camera device displays the character posture, shooting effect and shooting information of the target image frame on its display screen to guide the user to perform posture placement and shooting composition.

[0120] As another example, the above-mentioned execution entity outputs shooting suggestion information for the area shown in the pre-captured image based on multiple candidate image frames, which can be: in response to the user's shooting mode to take a photo of a person, multiple candidate image frames are determined based on multiple pre-selected matching schemes and multiple candidate image frames; the multiple candidate image frames are respectively input into the photo model to determine the aesthetic scores of the multiple candidate image frames; based on the aesthetic scores of the multiple candidate image frames, a target image frame is selected from the multiple candidate image frames; based on the target image frame, the shooting suggestion information for the area is output.

[0121] The above-mentioned determining multiple candidate image frames based on multiple pre-selected matching schemes and multiple candidate image frames includes: determining a virtual matching object in each pre-selected matching scheme and its corresponding matching position; for each pre-selected matching scheme and each candidate image frame, rendering the virtual matching object into the candidate image frame according to its matching position to obtain a corresponding candidate image frame.

[0122] The pre-selected multiple matching schemes may be matching schemes recommended by the execution subject after analyzing factors such as the shooting object and the environment, or may be matching schemes selected by the user based on the prop library. The prop library provides virtual matching objects, such as decorations, clothing, etc.

[0123] The above-mentioned selecting a target image frame from multiple candidate image frames based on aesthetic scores includes: sorting the multiple candidate image frames in descending order of aesthetic scores; and selecting the first candidate image frame or the third preset number of candidate image frames in the front sort as the target image frame.

[0124] The outputting of shooting suggestion information for the area shown in the pre-shot image based on the target image frame includes: taking the target image frame, shooting information of the target image frame and a matching scheme corresponding to the target image frame as the shooting suggestion information.

[0125] In this example, the execution subject renders the virtual collocation object into the candidate image frame determined based on the three-dimensional model to generate the candidate image frame, then selects one or more of the candidate image frames with the highest aesthetic level as the target image frame, and outputs the target image frame with its corresponding shooting information, and the mobile camera device displays the virtual collocation object, matching effect and shooting information of the target image frame on its display screen to guide the user to shoot matching and shooting composition. Accordingly, when shooting, the virtual collocation object is placed in the shooting preview box for the user to directly observe the shooting effect and integrate into the real shooting environment, so that the virtual collocation object is more realistic and integrated with the environment.

[0126] It should be noted that when the user's shooting mode is to take pictures of people or scenery, the shooting information of the target image frame can be displayed by: marking the shooting information of the target image frame with text on the display screen, or providing a play button on the display screen, and when the user clicks the play button, the shooting information of the target image frame is displayed in a voice broadcast form; the dynamic operation guide corresponding to the shooting information of the target image frame can also be displayed on the display screen.

[0127] As another example, the above-mentioned execution entity outputs shooting suggestion information for the area shown in the pre-captured image based on multiple candidate image frames, which can be: in response to the user's shooting mode being video recording, generating multiple image frame combinations based on multiple candidate image frames; inputting the multiple image frame combinations into a video model respectively, and determining aesthetic scores of the multiple image frame combinations; selecting a target image frame combination from the multiple image frame combinations based on the aesthetic scores; generating a target video based on the target image frame combination; and outputting shooting suggestion information for the area, wherein the shooting suggestion information includes shooting information of each frame image in the target video and shot switching information and shooting parameter adjustment information between adjacent image frames.

[0128] The construction process of the above-mentioned video model includes: inputting the video sample into a deep learning model (Transformer model) based on the self-attention mechanism to determine the video description corresponding to the video sample; based on the video description, determining the main subject in the video sample; inputting the video frame containing the main subject in the video sample into the photo model to determine the aesthetic score of the video frame; based on the aesthetic score of the video frame, determining the key frame corresponding to the video sample; and using the key frame and aesthetic score corresponding to the video sample to train the video model.

[0129] Among them, the deep learning model (Transformer model) based on the self-attention mechanism is used to understand the main content of the video within a certain period of time (i.e., the video description), and it can be trained using large-scale video-description texts. The above-mentioned determination of the key frames corresponding to the video sample based on the aesthetic score of the video frame includes: arranging the video frames containing the main objects in the video sample in descending order according to the aesthetic score, and taking the fourth preset number of video frames in the sorting as the key frames corresponding to the video sample. The above-mentioned training of the video model using the key frames and aesthetic scores corresponding to the video sample includes: inputting the key frames corresponding to the video sample into the initial network model, predicting the predicted aesthetic score of the video sample, and optimizing the initial network model using the predicted aesthetic score and the aesthetic score corresponding to the video sample; repeating the above operation until the preset conditions are met, and taking the initial network model at this time as the video model. The video content and aesthetic effect are associated through the video model, so that a combination of image frames with a higher aesthetic level can be recommended when the user records the video.

[0130] The target video generated based on the target image frame combination includes: using the image frames in the target image frame combination as key frames to generate the target video; the target video directly reflects the best camera movement and lens switching method based on the key frames.

[0131] In this example, the execution subject determines the key frames of the target video based on the video model, generates the target video from these key frames, and provides the target video's camera movement, angle switching, shooting parameter adjustment, etc., to provide the user with real-time dynamic shooting route guidance. Furthermore, the shooting parameters can be output to smart devices such as smart pan / tilts, drones, mobile robots, etc., to complete the shooting autonomously or semi-autonomously.

[0132] The auxiliary shooting method provided by the embodiment of the present disclosure fully understands the user's shooting intention, recommends different shooting plans for different shooting objects in combination with the spatial three-dimensional model, and can enable ordinary users to shoot photos that give the feeling of professional blockbusters.

[0133] The auxiliary shooting method provided by the embodiment of the present disclosure can also be applied to professional photography to provide references for professional photography and give practitioners creative inspiration. Alternatively, the shooting plan can be transmitted to intelligent shooting devices such as intelligent gimbals, drones, and movable shooting robots to complete automated shooting work.

[0134] In addition, the present disclosure also provides other application methods of the photo model, for example: in real-time capture scenarios, some wonderful moments are difficult to grasp, resulting in frequent misses, or it takes a long time to focus on the object to be photographed in order to capture the wonderful moment. The present disclosure can focus on tracking the object that the user wants to shoot, and determine the shooting intention of each frame of the image based on the intention recognition model, and select meaningful images with high aesthetic scores based on the photo model and the shooting intention and temporarily save them for users to choose. In addition, the user's selection results can also be learned to improve the accuracy of the image saved next time. At the same time, it supports some user shortcut commands (such as voice, gestures, etc.) for shooting.

[0135] For another example: based on the pre-shot image, the object that the user wants to shoot can be identified, and the shooting parameters of the objects can be optimized based on the characteristics of the objects in the spatial three-dimensional model. For example, the sky can be bluer, the grass can be greener, and the water can be clearer, instead of the traditional shooting parameter adjustment of the entire picture, thus avoiding the situation of losing sight of one thing while focusing on another.

[0136] Continue to refer Figure 5 , Figure 5 A structural diagram 500 of an embodiment of the auxiliary shooting device disclosed in the present invention is illustrated; the device embodiment is similar to Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0137] like Figure 5As shown, the auxiliary shooting device of this embodiment may include a construction module 501, a determination module 502 and an output module 503. The construction module 501 is configured to construct a spatial three-dimensional model of the area shown in the pre-shot image according to the pre-shot image of the mobile camera device; the determination module 502 is configured to determine the shooting intention corresponding to the pre-shot image; and the output module 503 is configured to output shooting suggestion information for the area shown in the pre-shot image based on the spatial three-dimensional model and the shooting intention.

[0138] In this embodiment, the specific processing of the construction module 501, the determination module 502 and the output module 503 and the technical effects thereof can be referred to in Figure 2 The relevant descriptions of steps 201 - 203 in the corresponding embodiment are not repeated here.

[0139] In some optional implementations of this embodiment, the construction module 504 includes a first determination submodule, a construction submodule and a first generation submodule, wherein the first determination submodule is configured to determine the local appearance information of the objects in the area shown in the pre-captured image and the relative positions between the objects based on the pre-captured image; the construction submodule is configured to construct a three-dimensional model of the object based on the local appearance information; and the first generation submodule is configured to generate a spatial three-dimensional model based on the three-dimensional model and the relative positions between the objects.

[0140] In some optional implementations of this embodiment, the above-mentioned construction submodule includes a determination unit and a construction unit, wherein the determination unit is configured to input local appearance information into the object appearance model to determine the appearance features of the object; the construction unit is configured to construct a three-dimensional model of the object based on the appearance features; wherein the object appearance model is used to characterize the correspondence between the local appearance information of the object and the global appearance features.

[0141] In some optional implementations of the present embodiment, the determination module 502 includes an extraction submodule and a second determination submodule, wherein the extraction submodule is configured to perform feature extraction on the pre-shot image to obtain shooting features; and the second determination submodule is configured to input the shooting features into an intention recognition model to determine the shooting intention, wherein the intention recognition model is a pre-trained large language model.

[0142] In some optional implementations of the present embodiment, the output module 503 includes a first generation submodule, a screening submodule and an output submodule, wherein the first generation submodule is configured to generate multiple initial image frames based on a spatial three-dimensional model and multiple sets of preset shooting information, wherein the shooting information includes a shooting position, a shooting angle and shooting parameters; the screening submodule is configured to screen the multiple initial image frames based on the shooting intention to obtain multiple candidate image frames; the output submodule is configured to output shooting suggestion information for the area shown in the pre-shot image based on the multiple candidate image frames.

[0143] In some optional implementations of the present embodiment, the output submodule includes a first determination subunit, a first selection subunit and a first output subunit, wherein the first determination subunit is configured to take a picture of the scene in response to a user shooting mode, input multiple candidate image frames into a photo model respectively, and determine the aesthetic scores of the multiple candidate image frames, wherein the photo model is a multimodal collaborative self-attention neural network model; the first selection subunit is configured to select a target image frame from the multiple candidate image frames based on the aesthetic score; the first output subunit is configured to output shooting suggestion information for the area shown in the pre-shot image, wherein the shooting suggestion information includes the target image frame and its shooting information.

[0144] In some optional implementations of the present embodiment, the output submodule further includes a second determination subunit, a third determination subunit, a second selection subunit and a second output subunit, wherein the second determination subunit is configured to take a photo of a person in response to a user shooting mode, and determine a plurality of candidate image frames based on a plurality of preselected virtual objects and a plurality of candidate image frames, wherein the virtual objects are person postures or matching schemes; the third determination subunit is configured to input the plurality of candidate image frames into the photo model respectively, and determine the aesthetic scores of the plurality of candidate image frames; the second selection subunit is configured to select a target image frame from a plurality of candidate image frames based on the aesthetic scores of the plurality of candidate image frames; the second output subunit is configured to output shooting suggestion information for the area shown in the pre-shot image, wherein the shooting suggestion information includes the target image frame and the virtual object information and shooting information corresponding to the target image frame.

[0145] In some optional implementations of the present embodiment, the output submodule also includes a first generating subunit, a fourth determining subunit, a third selecting subunit, a second generating subunit and a third output subunit, wherein the first generating subunit is configured to generate a plurality of image frame combinations based on a plurality of candidate image frames in response to the user shooting mode being video recording; the fourth determining subunit is configured to input the plurality of image frame combinations into a video model respectively to determine the aesthetic scores of the plurality of image frame combinations; the third selecting subunit is configured to select a target image frame combination from the plurality of image frame combinations based on the aesthetic scores; the second generating subunit is configured to generate a target video based on the target image frame combination; the third output subunit is configured to output shooting suggestion information for the area shown in the pre-shot image, wherein the shooting suggestion information includes shooting information of each frame image in the target video as well as shot switching information and shooting parameter adjustment information between adjacent image frames.

[0146] In some optional implementations of the present embodiment, the above-mentioned device also includes a video model construction module, and the video model construction module includes a third determination submodule, a fourth determination submodule, a fifth determination submodule, a sixth determination submodule and a training submodule, wherein the third determination submodule is configured to input the video sample into a deep learning model based on the self-attention mechanism to determine the video description corresponding to the video sample; the fourth determination submodule is configured to determine the main subject in the video sample based on the video description; the fifth determination submodule is configured to input the video frame containing the main subject in the video sample into the photo model to determine the aesthetic score of the video frame; the sixth determination submodule is configured to determine the key frame corresponding to the video sample based on the aesthetic score of the video frame; the training submodule is configured to train the video model using the key frames and aesthetic scores corresponding to the video samples.

[0147] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0148] Figure 6 A schematic block diagram of an example electronic device 600 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0149] like Figure 6 As shown, the device 600 includes a computing unit 601, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the device 600 can also be stored. The computing unit 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0150] A number of components in the device 600 are connected to the I / O interface 605, including: an input unit 606, such as a keyboard, a mouse, etc.; an output unit 607, such as various types of displays, speakers, etc.; a storage unit 608, such as a disk, an optical disk, etc.; and a communication unit 609, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 609 allows the device 600 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0151] The computing unit 601 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 601 performs the various methods and processes described above, such as the auxiliary shooting method. For example, in some embodiments, the auxiliary shooting method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 600 via the ROM 602 and / or the communication unit 609. When the computer program is loaded into the RAM 603 and executed by the computing unit 601, one or more steps of the auxiliary shooting method described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to perform the auxiliary shooting method in any other appropriate manner (e.g., by means of firmware).

[0152] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0153] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0154] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0155] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0156] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0157] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0158] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0159] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0160] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. An auxiliary shooting method, include: Constructing a spatial three-dimensional model of the area shown in the pre-shot image according to the pre-shot image of the mobile camera device; Determining a shooting intention corresponding to the pre-shot image; Based on the spatial three-dimensional model and the shooting intention, shooting suggestion information for the area is output.

2. The method according to claim 1, in, The step of constructing a spatial three-dimensional model of the area shown in the pre-shot image according to the pre-shot image of the mobile camera device includes: Determining local appearance information of objects in the area and relative positions between objects based on the pre-captured image; Based on the local appearance information, construct a three-dimensional model of the object; The spatial three-dimensional model is generated according to the relative position between the three-dimensional model and the object.

3. The method according to claim 2, in, The constructing of the three-dimensional model of the object based on the local appearance information includes: Inputting the local appearance information into an object appearance model to determine the appearance features of the object; Based on the appearance features, construct a three-dimensional model of the object; The object appearance model is used to characterize the correspondence between the local appearance information and the global appearance features of the object.

4. The method according to claim 1, in, The determining the shooting intention corresponding to the pre-shot image includes: Extracting features from the pre-shot image to obtain shooting features; The shooting features are input into an intention recognition model to determine the shooting intention.

5. The method according to claim 1, in, The outputting shooting suggestion information for the area based on the spatial three-dimensional model and the shooting intention includes: Generate a plurality of initial image frames based on the spatial three-dimensional model and a plurality of sets of preset shooting information, wherein the shooting information includes a shooting position, a shooting angle, and shooting parameters; Based on the shooting intention, the multiple initial image frames are screened to obtain multiple candidate image frames; Based on the multiple candidate image frames, shooting suggestion information for the area is output.

6. The method according to claim 5, in, The outputting shooting suggestion information for the area based on the multiple candidate image frames includes: In response to the user taking a picture of a scene in shooting mode, the plurality of candidate image frames are respectively input into a photo model to determine aesthetic scores of the plurality of candidate image frames, wherein the photo model is a multimodal collaborative self-attention neural network model; Selecting a target image frame from the plurality of candidate image frames based on the aesthetic score; Output shooting suggestion information for the area, wherein the shooting suggestion information includes the target image frame and its shooting information.

7. The method according to claim 5, in, The outputting shooting suggestion information for the area based on the multiple candidate image frames includes: In response to the user taking a photo of a person in shooting mode, a plurality of candidate image frames are determined based on a plurality of pre-selected virtual objects and the plurality of candidate image frames, wherein the virtual objects are postures or matching schemes of the person; Inputting the multiple candidate image frames into a photo model respectively to determine aesthetic scores of the multiple candidate image frames, wherein the photo model is a multimodal collaborative self-attention neural network model; Selecting a target image frame from the plurality of candidate image frames based on the aesthetic scores of the plurality of candidate image frames; Output shooting suggestion information for the area, wherein the shooting suggestion information includes the target image frame and virtual object information and shooting information corresponding to the target image frame.

8. The method according to claim 5, in, The outputting shooting suggestion information for the area based on the multiple candidate image frames includes: In response to the user shooting mode being video recording, generating a plurality of image frame combinations based on the plurality of candidate image frames; Inputting the plurality of image frame combinations into a video model respectively, and determining aesthetic scores of the plurality of image frame combinations; Based on the aesthetic score, selecting a target image frame combination from the plurality of image frame combinations; Based on the target image frame combination, generating a target video; Output shooting suggestion information for the area, wherein the shooting suggestion information includes shooting information of each frame image in the target video and shot switching information and shooting parameter adjustment information between adjacent image frames.

9. The method according to claim 8, in, The video model construction process includes: Inputting the video sample into a deep learning model based on a self-attention mechanism to determine a video description corresponding to the video sample; Based on the video description, determining a main subject in the video sample; Inputting the video frame containing the main subject in the video sample into a photo model to determine an aesthetic score of the video frame; Determining a key frame corresponding to the video sample based on the aesthetic score of the video frame; The video model is trained using the key frames and aesthetic scores corresponding to the video samples.

10. An auxiliary shooting device, include: A construction module, configured to construct a spatial three-dimensional model of the area shown in the pre-shot image according to the pre-shot image of the mobile camera device; A determination module, configured to determine a shooting intention corresponding to the pre-shot image; The output module is configured to output shooting suggestion information for the area based on the spatial three-dimensional model and the shooting intention.

11. An electronic device, include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 10.

12. A non-transitory computer-readable storage medium storing computer instructions, in, The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 10.

13. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the method according to any one of claims 1 to 10.

Citation Information

Cited By

  • Control method and device, storage medium and electronic equipment

    CN119863810A