Action recommendation and photographing control method and device, and terminal equipment

Abstract

The invention provides an action recommendation method and device, a photographing control method and device and terminal equipment, and relates to the technical field of image processing, in particular to the technical fields of artificial intelligence, big data and the like. The method comprises the following steps: acquiring a preview image acquired by a camera assembly in real time, and acquiring a reference image; performing posture recognition on the target objects of the preview image and the reference image to obtain a first posture of each part of the target object in the preview image and a second posture of each part of the target object in the reference image; calling a preset posture guidance model, comparing the first posture with the second posture, and obtaining a target action adjustment statement of each part from the first posture to the second posture; and based on the target action adjustment statement of each part from the first posture to the second posture, referring to the second posture of each part of the target object in the picture, and carrying out action recommendation and photographing control. The picture composition and action can be guided in real time according to a specific scene, and the photographing efficiency and the user photographing convenience are improved.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to technical fields such as big data and artificial intelligence. Background Art

[0002] In the related art, in a complex shooting environment, the relationship between the shooting background and the target object, the changes in natural light or lighting over time and weather, and the model's posture will all affect the efficiency and effect of the shooting.

[0003] How to guide composition and action in real time according to specific scenes, assist in action recommendations during shooting, and improve shooting efficiency and user convenience has become one of the important research directions. Summary of the invention

[0004] The present invention provides an action recommendation and photo taking control method, apparatus, and terminal device.

[0005] According to one aspect of the present disclosure, there is provided an action recommendation and photo taking control method, comprising:

[0006] Obtain the preview image collected by the camera component in real time and obtain the reference image;

[0007] Performing posture recognition on the target object in the preview image and the reference image respectively to obtain a first posture of each part of the target object in the preview image and a second posture of each part of the target object in the reference image;

[0008] Calling a preset posture guidance model, comparing the first posture with the second posture, and obtaining a target action adjustment statement for each part from the first posture to the second posture;

[0009] Based on the target action adjustment statements from the first posture to the second posture of each part and the second posture of each part of the target object in the reference picture, action recommendation and photo control are performed.

[0010] In the embodiment of the present application, posture recognition is performed on the target object of the preview image and the reference image respectively, and the first posture of each part of the target object in the preview image and the second posture of each part of the target object in the reference image are obtained. Different parts of the target object can be divided for real-time guidance prompts; the preset posture guidance model is called to compare the first posture and the second posture, and the target action adjustment statement of each part from the first posture to the second posture is obtained. The adjustment instruction of the specific action can be generated at the same time. Based on the target action adjustment statement from the first posture to the second posture of each part and the second posture of each part of the target object in the reference image, action recommendation and photo control are performed, which can guide the composition and action in real time, assist in action recommendation during shooting, and improve the shooting efficiency and user convenience.

[0011] According to another aspect of the present disclosure, there is provided an action recommendation and photo taking control device, comprising:

[0012] An acquisition module is used to acquire a preview image collected by the camera component in real time and to acquire a reference image;

[0013] A recognition module, used to perform posture recognition on the target object in the preview image and the reference image respectively, to obtain a first posture of each part of the target object in the preview image and a second posture of each part of the target object in the reference image;

[0014] A comparison module is used to call a preset posture guidance model, compare the first posture with the second posture, and obtain a target action adjustment statement for each part from the first posture to the second posture;

[0015] The first processing module is used to perform action recommendation and photo taking control based on the target action adjustment statement of each part from the first posture to the second posture and the second posture of each part of the target object in the reference picture.

[0016] According to another aspect of the present disclosure, there is provided an electronic device, including at least one processor, and

[0017] a memory communicatively connected to at least one processor; wherein,

[0018] The memory stores instructions that can be executed by at least one processor, and the instructions are executed by at least one processor so that the at least one processor can execute the action recommendation and photo-taking control method of the embodiment of the first aspect of the present disclosure.

[0019] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to execute the action recommendation and photo-taking control method according to the embodiment of the first aspect of the present disclosure.

[0020] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, which, when executed by a processor, implements the steps of the action recommendation and photo-taking control method of the first aspect of the present disclosure.

[0021] According to another aspect of the present disclosure, a terminal device is provided, including an action recommendation and photo taking control device according to the embodiment of the second aspect of the present disclosure as described above.

[0022] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0024] Figure 1 is a flow chart of an action recommendation and photo taking control method according to an embodiment of the present disclosure;

[0025] Figure 2 is a schematic diagram of gesture recognition according to an embodiment of the present disclosure;

[0026] Figure 3 is a schematic diagram showing target action adjustment statements for various parts according to an embodiment of the present disclosure;

[0027] Figure 4 is a flow chart of an action recommendation and photo taking control method according to an embodiment of the present disclosure;

[0028] Figure 5 is a schematic diagram of target detection according to an embodiment of the present disclosure;

[0029] Figure 6 is a flow chart of an action recommendation and photo taking control method according to an embodiment of the present disclosure;

[0030] Figure 7 is a schematic diagram of navigation based on aircraft position navigation information according to an embodiment of the present disclosure;

[0031] Figure 8 is a flow chart of an action recommendation and photo taking control method according to an embodiment of the present disclosure;

[0032] Fig. 9 is a schematic diagram of image description according to an embodiment of the present disclosure;

[0033] Fig.10 is a flow chart of an action recommendation and photo taking control method according to an embodiment of the present disclosure;

[0034] Fig.11 is a structural diagram of an action recommendation and photo taking control device according to an embodiment of the present disclosure;

[0035] Fig.12 The block diagram is a block diagram of an electronic device for implementing the method of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0036] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0037] The embodiments of the present disclosure relate to technical fields such as data processing and artificial intelligence.

[0038] Image processing: Image processing is a technique that uses computers to analyze images to achieve the desired results. It is also called image processing. Image processing generally refers to digital image processing. A digital image is a large two-dimensional array obtained by shooting with industrial cameras, video cameras, scanners and other equipment. The elements of the array are called pixels, and their values ​​are called grayscale values.

[0039] Artificial Intelligence (AI) is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence.

[0040] Big data, or massive data, refers to the amount of data involved that is so large that it cannot be captured, managed, processed, and organized into information that helps companies make more proactive business decisions within a reasonable time through mainstream software tools. "Big data" requires new processing models to have stronger decision-making power, insight discovery, and process optimization capabilities to adapt to massive, high-growth, and diverse information assets.

[0041] The following are explanations of the terms used in this disclosure:

[0042] The Vision-Language Model (VLM) is a multimodal model that combines visual and language information. The VLM model processes image and text data simultaneously, and is capable of performing tasks such as image description generation and visual question answering. It enhances the understanding and association between vision and language through contrastive learning or other methods.

[0043] Data flywheel can quickly iterate large models through data feedback such as prompt words. Online learning of data flywheel is a training method for models in deep learning. It can update the model online through online streaming data and adjust the model according to one or a batch of data samples obtained in real time. It can capture data changes in a timely manner and effectively increase the update frequency of the model.

[0044] The following describes the action recommendation and photo taking control method, apparatus, and terminal device disclosed in the present invention with reference to the accompanying drawings.

[0045] Figure 1 is a flow chart of an action recommendation and photo taking control method according to an embodiment of the present disclosure. Figure 1 As shown, the method comprises the following steps:

[0046] S101, obtaining a preview image captured by a camera component in real time, and obtaining a reference image.

[0047] In some implementations, taking a terminal device as an example, a real-time image frame of a camera component of the terminal device may be obtained as a preview image.

[0048] In some implementations, a designated image may be directly used as a reference image; in some implementations, multiple candidate images may be obtained from a preset database, and then the candidate images may be screened based on background information of the preview image to determine a reference image from the candidate images.

[0049] S102 , performing posture recognition on the target object in the preview image and the reference image respectively, to obtain a first posture of each part of the target object in the preview image and a second posture of each part of the target object in the reference image.

[0050] For example, the target object may be a person, and the preview image may include one target object or multiple target objects. Similarly, the reference image may include one target object or multiple target objects.

[0051] In some embodiments, Figure 2 As shown, key point detection can also be performed on the target objects of the preview image and the reference image respectively to realize posture recognition of the target objects of the preview image and the reference image. That is to say, the first posture of each part of the target object in the preview image and the second posture of each part of the target object in the reference image are obtained through key point-based action recognition.

[0052] Perform posture detection on the target object in the preview image and the reference image respectively, and obtain the first posture of each part of the target object in the preview image and the second posture of each part of the target object in the reference image, wherein the various parts of the target object may include, for example, legs, head, arms, feet, waist, etc.

[0053] S103, calling a preset posture guidance model, comparing the first posture and the second posture, and obtaining a target action adjustment statement for each part from the first posture to the second posture.

[0054] In some embodiments, the posture guidance model can be a pre-deployed large language model, and in other implementations, it can also be other multimodal models, which is not limited by the present disclosure. Optionally, in order to improve the accuracy of the data, the large language model can be fine-tuned and trained to obtain the posture guidance model.

[0055] In some embodiments, a preset posture guidance model is called to compare and process each part from a first posture and a second posture, and a target action adjustment statement for each part from the first posture to the second posture is obtained. The part of the target object is an arm for illustration. If the first posture of the arm of the target object in the preview image is the arm hanging down, and the second posture of the arm of the target object in the reference image is the arm raised high, the preset posture guidance model is called, and after comparing and processing the arm from the first posture and the second posture, the target action adjustment statement for the arm obtained may be "raise the arm."

[0056] S104, based on the target action adjustment statement of each part from the first posture to the second posture and the second posture of each part of the target object in the reference picture, action recommendation and photo control are performed.

[0057] like Figure 3 As shown, in the embodiment of the present disclosure, the position of each part in the preview image can be determined based on target detection, and based on the position of each part in the preview image, the target action adjustment statement of each part is displayed to assist in action recommendation.

[0058] In some embodiments, the preset photo control is detected to be triggered, and the camera assembly is controlled to take a photo. Optionally, the change of the first posture of each part in the preview image can also be detected in real time, and when the posture of each part is consistent with the second posture, the camera assembly is automatically controlled to take a photo.

[0059] In the embodiment of the present application, posture recognition is performed on the target object of the preview image and the reference image respectively, and the first posture of each part of the target object in the preview image and the second posture of each part of the target object in the reference image are obtained. Different parts of the target object can be divided for real-time guidance prompts; the preset posture guidance model is called to compare the first posture and the second posture, and the target action adjustment statement of each part from the first posture to the second posture is obtained. The adjustment instruction of the specific action can be generated at the same time. Based on the target action adjustment statement from the first posture to the second posture of each part and the second posture of each part of the target object in the reference image, action recommendation and photo control are performed, which can guide the composition and action in real time, assist in action recommendation during shooting, and improve the shooting efficiency and user convenience.

[0060] Figure 4 is a flow chart of an action recommendation and photo taking control method according to an embodiment of the present disclosure. Figure 4 As shown, the method comprises the following steps:

[0061] S401, obtaining a preview image captured by a camera component in real time.

[0062] For the introduction of step S401, please refer to the content of the above embodiment, which will not be repeated here.

[0063] S402, obtaining multiple candidate images from a preset database.

[0064] In some implementations, images in a preset database are directly obtained as candidate images.

[0065] In some implementations, image data and corresponding text information are crawled from an open source database to obtain candidate images and their associated captions.

[0066] S403, obtaining background information of the preview image and background information of each candidate image.

[0067] like Figure 5 As shown, in some implementations, target detection is performed on the preview image and each candidate image respectively to identify the first reference object and the first target object in the preview image, and the second reference object and the second target object in each candidate image.

[0068] In some embodiments, such as in a tourism scenario, the reference object can be a building, such as a tower or a bridge. In some embodiments, such as in a conference room scenario, the reference object can be various types of equipment or objects, such as a table or a cabinet. In other implementations, the reference object can also be other types of reference objects, such as sculptures or paintings. The present disclosure does not limit this.

[0069] Continue to perform depth prediction on the preview image and each candidate image to obtain the depth and orientation information of the first reference object, the first target object, the second reference object and the second target object, respectively; use the depth and orientation information of the first reference object and the first target object in the preview image as the background information of the preview image; use the depth and orientation information of the second reference object and the second target object in each candidate image as the background information of the candidate image.

[0070] Taking a building as an example, in the disclosed embodiment, depth prediction can be used to obtain the depth and orientation of the building, thereby obtaining more accurate location information, so as to facilitate the subsequent determination of a reference image with a background similar to the preview image from the candidate images.

[0071] S404, calling a preset multimodal model, comparing the background information of the preview image with the background information of each candidate image, and determining a reference image from the candidate images.

[0072] In some embodiments, the multimodal model is called to compare the preview image and its background information with each candidate image and its background information to determine N target images from the candidate images, where N is a positive integer. Optionally, the multimodal model in the present disclosure may be a visual language model VLM.

[0073] In some implementations, N takes a value of 1, and a multimodal large model is used to retrieve multiple candidate images to find similar scenes, give a target image, and use the target image as the best reference image.

[0074] In some embodiments, if two or more target images are determined from the candidate images, the reference image is further determined from the target images. For example, each target image is displayed, and the triggered target image is determined to be the reference image. For another example, a preset evaluation model is called to score the preview image and each target image, obtain the first similarity between the preview image and each target image, and determine the target image with the highest first similarity as the reference image. For another example, based on the evaluation information associated with each target image, an evaluation value corresponding to each target image is obtained, and the target image with the highest evaluation value is determined from the target images as the reference image; taking the evaluation information as the number of likes of the picture as an example, the number of likes of each target image can be obtained as the initial evaluation value corresponding to each target image, and the evaluation value and the first similarity are weighted to obtain the target evaluation value, and the target image with the highest target evaluation value is used as the reference image.

[0075] In some embodiments, changes in time and weather may also affect the photographing effect. In order to further improve the accuracy, time and weather factors need to be considered. In the embodiment of the present disclosure, the time information and weather information of the candidate image and each target image are obtained respectively. For example, the description text of the candidate image and each target image can be obtained respectively, and keyword detection and information extraction are performed on the description text to obtain the time information and weather information of the candidate image and each target image.

[0076] In some embodiments, the time information and / or weather information is not present in the captions of certain images, and the time information and / or weather information of the image cannot be extracted from the captions, and the missing information needs to be supplemented. In the embodiment of the present disclosure, the image that is missing the time information and / or weather information in the candidate image and the target image is used as the third image, and the multimodal model is called to predict the missing time information and / or weather information based on preset prompts, the third image and its captions, so as to fill in the data of the missing time information and / or weather information of the third image.

[0077] In the embodiment of the present disclosure, images whose weather is inconsistent with that of the reference image or whose shooting time is greatly different are deleted from the target image. Optionally, based on the time information of the candidate image and each target image, the time difference corresponding to each target image and the candidate image is obtained, the second similarity of the weather information of each target image and the candidate image is obtained, the first image whose time difference is within a preset time range is determined from the target image, and the second image whose second similarity is less than a preset similarity threshold is determined, and the first image and the second image are removed from the target image.

[0078] S405 , performing posture recognition on the target object in the preview image and the reference image respectively, to obtain a first posture of each part of the target object in the preview image and a second posture of each part of the target object in the reference image.

[0079] S406, calling a preset posture guidance model, comparing the first posture and the second posture, and obtaining a target action adjustment statement for each part from the first posture to the second posture.

[0080] S407, based on the target action adjustment statement of each part from the first posture to the second posture and the second posture of each part of the target object in the reference picture, action recommendation and photo control are performed.

[0081] For the introduction of steps S405 to S407, reference may be made to the relevant contents in the above embodiment, which will not be repeated here.

[0082] In an embodiment of the present disclosure, background information of the preview image and background information of each candidate image are obtained, a preset multimodal model is called, the background information of the preview image and the background information of each candidate image are compared, and a reference image is determined from the candidate images. The present disclosure can provide composition guidance and action guidance in real time according to specific scenarios through comprehensive calculation of factors such as environment, reference objects, time, and weather, thereby improving photo-taking efficiency and user convenience.

[0083] Figure 6 is a flow chart of a training posture guidance model according to an embodiment of the present disclosure. Figure 6 As shown, the training process of the posture guidance model includes the following steps:

[0084] S601, obtaining a standard image sample containing a target object and a plurality of initial image samples.

[0085] In some implementations, image data is crawled from an open source database to obtain a standard image sample and a plurality of initial image samples.

[0086] S602, performing posture estimation on the target object in the standard image sample and each initial image sample to obtain a third posture of each part of the target object in each initial image sample and a fourth posture of each part of the target object in the standard image.

[0087] Optionally, two-dimensional 2D posture estimation is used to detect the whole body posture (including various parts such as body, hands, face, etc.) of the target object in the image, and a third posture of each part of the target object in each initial image sample and a fourth posture of each part of the target object in the standard image are obtained.

[0088] S603: for each initial image sample, annotate the initial image sample with action prompt text to obtain reference action adjustment statements for each part from the third posture to the fourth posture.

[0089] A plurality of groups of images may be acquired based on the standard image samples and the initial image samples, and each group of images includes the standard image samples and any initial image sample.

[0090] Each set of images is labeled with “prompt text”, for example: need to lift left leg, raise arms, smile, etc.

[0091] S604, training a preset language model based on reference action adjustment sentences of each part of the target object from the third posture to the fourth posture in each initial image sample to obtain a trained posture guidance model.

[0092] The large language model is used as the basic model, and the data set constructed in the above steps S601 to S603 and the preset low-rank adaptive (LORA) model are used to train the large language model to obtain the posture guidance model.

[0093] First, the disassembly module of the large language model (such as the tokenizer) is expanded into a text tokenizer and an action pose tokenizer, so as to align the data information of the pose to the text space. This can be achieved by annotating single pose data, that is, performing pose captioning in analogy with image captioning, describing the 2D pose as text, and obtaining the pose description information. Optionally, the 2D pose can also be generated based on the pose description information in the text to reversely verify the accuracy of the pose description information.

[0094] Fine-tune the large language model by constructing instructions through prompt engineering.

[0095] In the disclosed embodiment, for each initial image sample, the language model is called to describe the third posture of each part of the target object in the initial image sample to obtain the first posture description information, and the fourth posture of each part of the target object in the standard image sample is described to obtain the second posture description information. The language model is called to generate a predicted action adjustment sentence from the first posture description information to the second posture description information based on the constructed context prompt, construct an action guidance prompt word to fine-tune the language model, and reversely adjust the language model based on the reference action adjustment sentence and the predicted action adjustment sentence.

[0096] In some implementations, after the preset language model is trained, the model can be continuously trained based on the next set of sample data, and the model can be continuously optimized until a preset number of iterations is reached to obtain a trained posture guidance model.

[0097] In some implementations, a data flywheel can also be used to obtain high-quality data to fine-tune the model, thereby obtaining a model that better suits the application scenario, improving the accuracy of the prediction, and optimizing the user experience.

[0098] The disclosed embodiment can disassemble the image, improve the accuracy of the data through model training, and obtain more accurate action adjustment statements.

[0099] In some embodiments, based on the multimodal model, position recognition is performed on the preview image and the reference image, and first relative position information of the first reference object and the first target object in the preview image, the shooting height and orientation corresponding to the preview image, the second relative position information of the third reference object and the third target object in the reference image, and the shooting height and orientation corresponding to the reference image are obtained. Based on the first relative position information and the second relative position information, position adjustment information of the first reference object or the first target object is generated, for example, Figure 7 As shown, the first relative position information is the position information of the first target object when the first reference object is the origin, and the second relative position information is the position information of the second target object when the second reference object is the origin. Therefore, the position adjustment information of the first target object can be obtained based on the difference between the second relative position information and the first relative position information. For another example, the first relative position information is the position information of the first reference object when the first target object is the origin, and the second relative position information is the position information of the second reference object when the second target object is the origin. Therefore, the position adjustment information of the first reference object can be obtained based on the difference between the second relative position information and the first relative position information.

[0100] Further, based on the camera height and orientation corresponding to the preview image and the camera height and orientation corresponding to the reference image, the height and orientation adjustment information of the camera assembly is obtained, such as the height and orientation adjustment information includes the height adjustment information and the orientation adjustment information, the height adjustment information of the camera assembly is obtained based on the difference between the camera heights corresponding to the reference image and the preview image, the orientation adjustment information of the camera assembly is obtained based on the difference between the camera orientations corresponding to the reference image and the preview image, and the position navigation information is generated and displayed based on the position adjustment information of the first reference object or the first target object and the height and orientation adjustment information of the camera assembly, such as Figure 7As shown, based on the camera navigation information, a preset map application can be called for navigation. Among them, the position adjustment information of the first reference object or the first target object can guide the position adjustment of the first reference object or the first target object, and the height and azimuth adjustment information of the camera assembly can guide the position adjustment of the camera assembly, that is, the terminal device. The present disclosure can calculate the camera position and composition guidance to perform precise navigation, and accurately guide the spatial relationship between the camera assembly and the target object and the reference object through depth recognition.

[0101] The multimodal models in the present disclosure may be different multimodal models or different capabilities of the same multimodal model, which will not be elaborated in the embodiments of the present disclosure.

[0102] In some embodiments, in order to provide refined information, provide shooting suggestions to users before taking pictures, and further meet shooting needs, the reference image is segmented based on a preset object segmentation model (SAM2 model), multiple segmented images are obtained, and at least one commodity in each segmented image is determined, and the commodity can be, for example, clothing, accessories, etc. An image description is performed on each segmented image to obtain commodity information of the commodity. For each segmented image, the target application is called to perform information retrieval based on the segmented image and the commodity information of the commodity in the segmented image, obtain commodity links of the same or similar styles of commodities, display the reference image, and display the commodity link at the corresponding position of the reference image, wherein the commodity link is used to jump to the commodity recommendation page of the target application.

[0103] Figure 8 is a flow chart of an action recommendation and photo taking control method according to an embodiment of the present disclosure. Figure 8 As shown, the method comprises the following steps:

[0104] S801, obtaining a preview image captured in real time by a camera component, and obtaining a reference image.

[0105] For the content of step S801, please refer to the relevant introduction in the above embodiment, which will not be repeated here.

[0106] S802, obtaining the description text of the reference image.

[0107] In some embodiments, the reference image has associated captions;

[0108] In some implementations, the reference image is identified to obtain the caption in the reference image.

[0109] S803: extract information from the description text to obtain color information and reference camera parameters corresponding to the reference image.

[0110] like Fig. 9As shown, if the color information corresponding to the reference image is not extracted, an image description is performed on the reference image to obtain the color information of the background object and the foreground target object in the reference image.

[0111] If the reference camera parameters of the reference image are not extracted, the multimodal model is called to perform prediction based on the preset prompt, the reference image and its explanatory text to obtain the reference camera parameters of the reference image.

[0112] In some implementations, if the description text of the reference image is not obtained, it is determined that the color information and reference camera parameters corresponding to the reference image are not extracted, and the color information and reference camera parameters corresponding to the reference image are obtained according to the above steps.

[0113] S804, displaying the color information of the reference image and the reference camera parameters.

[0114] In some implementations, the color information of the reference image and the reference camera parameters are displayed on the interface of the terminal device to provide shooting suggestions to the user before taking the photo, thereby further meeting the shooting needs.

[0115] S805 , in response to monitoring that the reference camera parameter is triggered, adjusting the camera parameter of the camera component based on the reference camera parameter.

[0116] In some embodiments, the reference camera parameters are detected to be triggered, and the reference camera parameters are automatically applied to the camera assembly.

[0117] S806 , performing posture recognition on the target object in the preview image and the reference image respectively, to obtain a first posture of each part of the target object in the preview image and a second posture of each part of the target object in the reference image.

[0118] S807, calling a preset posture guidance model, comparing the first posture and the second posture, and obtaining a target action adjustment statement for each part from the first posture to the second posture.

[0119] S808, based on the target action adjustment statement of each part from the first posture to the second posture and the second posture of each part of the target object in the reference picture, perform action recommendation and photo taking control.

[0120] For the contents of step S806 to step S808, reference may be made to the relevant introduction in the above embodiment, which will not be repeated here.

[0121] The disclosed embodiments can provide real-time guidance on composition color and camera parameters according to the specific scene before shooting, further provide data support and suggestions for the shooting process, improve shooting efficiency and user shooting convenience, and in response to monitoring that the reference camera parameters are triggered, adjust the camera parameters of the camera component based on the reference camera parameters, thereby improving the flexibility of action recommendations and shooting control.

[0122] Fig.10 is a flow chart of an action recommendation and photo taking control method according to an embodiment of the present disclosure. Fig.10 As shown, the method comprises the following steps:

[0123] S1001, obtaining a preview image captured by a camera component in real time, and obtaining a reference image.

[0124] S1002 , performing posture recognition on the target object in the preview image and the reference image respectively, to obtain a first posture of each part of the target object in the preview image and a second posture of each part of the target object in the reference image.

[0125] For the contents of step S1001 to step S1002, reference may be made to the relevant introduction in the above embodiment, which will not be repeated here.

[0126] S1003, performing facial recognition on the target object in the preview image and the reference image respectively to obtain a first facial expression of the target object in the preview image and a second facial expression of the target object in the reference image.

[0127] Optionally, in the embodiment of the present disclosure, various parts of the target object also include face.

[0128] S1004: Using the first facial expression as a first gesture of the target object's face in the preview image, and using the second facial expression as a second gesture of the target object's face in the reference image.

[0129] According to the first facial expression of the target object's face in the preview image, the first posture of each part of the target object in the preview image is updated, and according to the second facial expression of the target object's face in the reference image, the second posture of each part of the target object in the reference image is updated. Therefore, when the target action adjustment statement of each part from the first posture to the second posture is subsequently obtained, the guidance of the facial expression is also included.

[0130] S1005, calling a preset posture guidance model, comparing the first posture and the second posture, and obtaining a target action adjustment statement for each part from the first posture to the second posture.

[0131] S1006, displaying target action adjustment statements for each part from the first posture to the second posture.

[0132] The position of each part of the target object is obtained, and based on the position of each part, a target action adjustment statement of each part from a first posture to a second posture is displayed.

[0133] S1007: Update the preview image and the first posture of each part of the target object in the preview image.

[0134] Since the posture of the target object may change in real time, the preview image and the first posture of each part of the target object in the preview image are continuously updated.

[0135] S1008 , for each part of the target object in the preview image, compare the updated first posture and the updated second posture of the part to determine whether the updated first posture and the updated second posture of the part are the same.

[0136] Optionally, in some embodiments, if the first posture and the second posture of the part after update are different, timing is started, and when the timing duration reaches a preset time threshold, step S1005 is re-executed to update the part target action adjustment statement.

[0137] S1009 , in response to the first posture and the second posture of the updated part being the same, stop displaying the target action adjustment statement of the part.

[0138] In response to the first and second postures of the part being the same after the update, it is determined that the action recommendation guidance for the part is completed, and the display of the target action adjustment statement for the part can be stopped.

[0139] S1010, in response to the updated first posture and the second posture of all parts being the same, controlling the camera component to take a photo, or, in response to monitoring that a preset photo control is triggered, controlling the camera component to take a photo.

[0140] In some embodiments, the updated first posture and the second posture of all parts are the same, and the recommended action guidance for completing the current shooting is determined, and the camera assembly is automatically controlled to take a picture;

[0141] In some implementations, the camera assembly may be controlled to take photos based on preset photo controls.

[0142] In the disclosed embodiments, shooting action recommendations and photo control can be implemented, composition and action can be guided in real time, and action recommendations can be assisted during shooting. In response to the first posture and the second posture after all parts are updated being the same, the camera component is controlled to take a photo, or, in response to monitoring that a preset photo control is triggered, the camera component is controlled to take a photo. This can improve the flexibility of action recommendations and shooting control, and enhance shooting efficiency and user convenience.

[0143] Fig.11 is a structural diagram of an action recommendation and photo taking control device according to an embodiment of the present disclosure, such as Fig.11 As shown, the action recommendation and photo taking control device 1100 includes:

[0144] An acquisition module 1110 is used to acquire a preview image collected by the camera component in real time and to acquire a reference image;

[0145] The recognition module 1120 is used to perform posture recognition on the target object in the preview image and the reference image respectively, and obtain a first posture of each part of the target object in the preview image and a second posture of each part of the target object in the reference image;

[0146] A comparison module 1130 is used to call a preset posture guidance model, compare the first posture with the second posture, and obtain a target action adjustment statement for each part from the first posture to the second posture;

[0147] The first processing module 1140 is used to perform action recommendation and photo taking control based on the target action adjustment statement of each part from the first posture to the second posture and the second posture of each part of the target object in the reference picture.

[0148] In some implementations, the acquisition module 1110 is further configured to:

[0149] Obtain multiple candidate images from a preset database;

[0150] Obtaining background information of the preview image and background information of each candidate image;

[0151] The preset multimodal model is called to compare the background information of the preview image with the background information of each candidate image, and a reference image is determined from the candidate images.

[0152] In some implementations, the acquisition module 1110 is further configured to:

[0153] Performing target detection on the preview image and each candidate image respectively, identifying a first reference object and a first target object in the preview image, and a second reference object and a second target object in each candidate image;

[0154] Continue to perform depth prediction on the preview image and each candidate image to obtain depth and orientation information of the first reference object, the first target object, the second reference object, and the second target object;

[0155] The depth of field and orientation information of the first reference object and the first target object in the preview image are used as the background information of the preview image, and the depth of field and orientation information of the second reference object and the second target object in each candidate image are used as the background information of the candidate image.

[0156] In some implementations, the acquisition module 1110 is further configured to:

[0157] Call the multimodal model to compare the preview image and its background information with each candidate image and its background information, respectively, to determine N target images from the candidate images, where N is a positive integer;

[0158] Determine a reference image from N target images.

[0159] In some implementations, N is greater than 1, and the acquisition module 1110 is further configured to:

[0160] Display each target image and determine that the triggered target image is a reference image; or,

[0161] Obtaining a first similarity between the preview image and each target image, and determining the target image with the highest first similarity as the reference image; or,

[0162] An evaluation value corresponding to each target image is obtained based on the evaluation information associated with each target image, and a target image with the highest evaluation value is determined from the target images as a reference image.

[0163] In some implementations, the acquisition module 1110 is further configured to:

[0164] Obtaining time information and weather information of the candidate image and each target image respectively;

[0165] Based on the time information of the candidate image and each target image, a time difference value corresponding to each target image and the candidate image is obtained;

[0166] Obtaining a second similarity between the weather information of each target image and the candidate image;

[0167] Determine, from the target image, a first image whose time difference is within a preset time range, and determine a second image whose second similarity is less than a preset similarity threshold;

[0168] The first image and the second image are cleared from the target image.

[0169] In some implementations, the acquisition module 1110 is further configured to:

[0170] Obtain the description texts of the candidate images and each target image respectively;

[0171] Extract information from the captions to obtain the time information and weather information of the candidate images and each target image;

[0172] If there is a third image that lacks time information and / or weather information among the candidate images and the target image, the multimodal model is called to perform prediction based on the preset prompt, the third image and its explanatory text to obtain the missing time information and / or weather information of the third image.

[0173] In some embodiments, the apparatus further comprises a second processing module 1150, configured to:

[0174] Get the caption of the reference image;

[0175] Extract information from the caption to obtain color information and reference camera parameters corresponding to the reference image;

[0176] The color information of the reference image and the reference camera parameters are displayed.

[0177] In some implementations, the second processing module 1150 is further configured to:

[0178] If the color information corresponding to the reference image is not extracted, performing image description on the reference image to obtain the color information of the background object and the foreground target object in the reference image; and / or

[0179] If the reference camera parameters of the reference image are not extracted, the multimodal model is called to perform prediction based on the preset prompt, the reference image and its explanatory text to obtain the reference camera parameters of the reference image.

[0180] In some implementations, the second processing module 1150 is further configured to:

[0181] In response to monitoring that the reference camera parameter is triggered, the camera parameter of the camera assembly is adjusted based on the reference camera parameter.

[0182] In some embodiments, the apparatus further includes a model training module 1160 for:

[0183] Acquire a standard image sample containing a target object and a plurality of initial image samples;

[0184] Performing posture estimation on the target object in the standard image sample and each initial image sample to obtain a third posture of each part of the target object in each initial image sample and a fourth posture of each part of the target object in the standard image;

[0185] For each initial image sample, annotate the initial image sample with action prompt text to obtain a reference action adjustment statement of each part from the third posture to the fourth posture;

[0186] Based on the reference action adjustment sentences of each part of the target object from the third posture to the fourth posture in each initial image sample, the preset language model is trained to obtain a trained posture guidance model.

[0187] In some implementations, the model training module 1160 is further configured to:

[0188] For each initial image sample, calling the language model, performing a posture description on the third posture of each part of the target object in the initial image sample to obtain first posture description information, and performing a posture description on the fourth posture of each part of the target object in the standard image sample to obtain second posture description information;

[0189] Calling the language model to generate a predicted action adjustment sentence from the first posture description information to the second posture description information based on the constructed context prompt;

[0190] Construct action guidance prompt words to fine-tune the language model, and reversely adjust the language model based on reference action adjustment sentences and predicted action adjustment sentences.

[0191] In some embodiments, the various parts include a face, and the recognition module 1120 is further configured to:

[0192] Performing facial recognition on the target object in the preview image and the reference image respectively to obtain a first facial expression on the face of the target object in the preview image and a second facial expression on the face of the target object in the reference image;

[0193] The first facial expression is used as a first posture of the target object's face in the preview image, and the second facial expression is used as a second posture of the target object's face in the reference image.

[0194] In some embodiments, the comparison module 1130 is further configured to:

[0195] Displaying target action adjustment statements for each part from the first posture to the second posture;

[0196] updating the preview image and the first posture of each part of the target object in the preview image;

[0197] For each part of the target object in the preview image, comparing the first posture and the second posture of the part after the update, and determining whether the first posture and the second posture of the part after the update are the same;

[0198] In response to the first posture and the second posture after the part is updated being the same, stopping displaying the target action adjustment statement for the part;

[0199] In response to the updated first posture and second posture of all parts being the same, the camera component is controlled to take a photo; or, in response to monitoring that a preset photo control is triggered, the camera component is controlled to take a photo.

[0200] In some embodiments, the apparatus further includes an aircraft position navigation module 1170, which is configured to:

[0201] Based on the multimodal model, position recognition is performed on the preview image and the reference image to obtain first relative position information of the first reference object and the first target object in the preview image, the shooting height and orientation corresponding to the preview image, second relative position information of the third reference object and the third target object in the reference image, and the shooting height and orientation corresponding to the reference image;

[0202] generating position adjustment information of a first reference object or a first target object based on the first relative position information and the second relative position information;

[0203] Acquire height and orientation adjustment information of the camera assembly based on the camera height and orientation corresponding to the preview image and the camera height and orientation corresponding to the reference image;

[0204] Based on the position adjustment information of the first reference object or the first target object and the height and orientation adjustment information of the camera assembly, the camera position navigation information is generated and displayed.

[0205] In some embodiments, the device further includes a product recommendation module 1180, which is used to:

[0206] Segment the reference image based on a preset object segmentation model, obtain a plurality of segmented images, and determine at least one commodity in each segmented image;

[0207] Perform image description on each segmented image to obtain product information of the product;

[0208] For each segmented image, call the target application to perform information retrieval based on the segmented image and the product information of the product in the segmented image to obtain a product link, and the product link is used to jump to the product recommendation page of the target application;

[0209] Display reference images and product links.

[0210] In the embodiment of the present application, posture recognition is performed on the target object of the preview image and the reference image respectively, and the first posture of each part of the target object in the preview image and the second posture of each part of the target object in the reference image are obtained. Different parts of the target object can be divided for real-time guidance prompts; the preset posture guidance model is called to compare the first posture and the second posture, and the target action adjustment statement of each part from the first posture to the second posture is obtained. The adjustment instruction of the specific action can be generated at the same time. Based on the target action adjustment statement from the first posture to the second posture of each part and the second posture of each part of the target object in the reference image, action recommendation and photo control are performed, which can guide the composition and action in real time, assist in action recommendation during shooting, and improve the shooting efficiency and user convenience.

[0211] Based on the same concept, the present disclosure also provides a terminal device, including an action recommendation and photo taking control device according to the above-mentioned embodiment of the present disclosure.

[0212] The acquisition, transmission, storage, use, and processing of data in the technical solution disclosed in this disclosure are in compliance with the relevant provisions of national laws and regulations and do not violate public order and good morals. It should be noted that in the embodiments of this disclosure, some existing solutions in the industry such as certain software, components, and models may be mentioned, which should be considered as exemplary, and their purpose is only to illustrate the feasibility of implementing the technical solution disclosed in this disclosure, but it does not mean that the person who discloses the solution has or will necessarily use it.

[0213] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0214] Fig.12 is a block diagram of an electronic device for implementing an embodiment of the present disclosure. The electronic device can implement the action recommendation and photo control method of the embodiment of the present disclosure, and the electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0215] like Fig.12 As shown, the device 1200 includes a computing unit 1201, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1202 or a computer program loaded from a storage unit 1208 into a random access memory (RAM) 1203. In the RAM 1203, various programs and data required for the operation of the device 1200 can also be stored. The computing unit 1201, the ROM 1202, and the RAM 1203 are connected to each other via a bus 1204. An input / output (I / O) interface 1205 is also connected to the bus 1204.

[0216] A number of components in the device 1200 are connected to the I / O interface 1205, including: an input unit 1206, such as a keyboard, a mouse, etc.; an output unit 1207, such as various types of displays, speakers, etc.; a storage unit 1208, such as a disk, an optical disk, etc.; and a communication unit 1209, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1209 allows the device 1200 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0217] The computing unit 1201 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1201 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1201 performs the various methods and processes described above, such as the action recommendation and photo control method. For example, in some embodiments, the action recommendation and photo control method may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 1208. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 1200 via the ROM 1202 and / or the communication unit 1209. When the computer program is loaded into the RAM 1203 and executed by the computing unit 1201, one or more steps of the action recommendation and photo control method described above may be performed. Alternatively, in other embodiments, the computing unit 1201 may be configured to perform the action recommendation and photo taking control method in any other appropriate manner (eg, by means of firmware).

[0218] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0219] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0220] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0221] To provide interaction with a target object, the systems and techniques described herein may be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the target object; and a keyboard and pointing device (e.g., a mouse or trackball) through which the target object can provide input to the computer. Other types of devices may also be used to provide interaction with the target object; for example, the feedback provided to the target object may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the target object may be received in any form (including acoustic input, voice input, or tactile input).

[0222] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a target object computer with a graphical target object interface or a web browser through which the target object can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0223] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0224] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0225] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for action recommendation and photo taking control, wherein: include: Obtain the preview image collected by the camera component in real time and obtain the reference image; Performing posture recognition on the target object in the preview image and the reference image respectively to obtain a first posture of each part of the target object in the preview image and a second posture of each part of the target object in the reference image; Calling a preset posture guidance model, comparing the first posture with the second posture, and obtaining a target action adjustment statement for each part from the first posture to the second posture; Based on the target action adjustment statement of each part from the first posture to the second posture and the second posture of each part of the target object in the reference picture, action recommendation and photo taking control are performed.

2. The method according to claim 1, wherein: The obtaining of a reference image comprises: Obtain multiple candidate images from a preset database; Acquire background information of the preview image and background information of each of the candidate images; A preset multimodal model is called to compare the background information of the preview image with the background information of each of the candidate images, and the reference image is determined from the candidate images.

3. The method according to claim 2, wherein: The obtaining of the background information of the preview image and the background information of each of the candidate images includes: Performing target detection on the preview image and each of the candidate images respectively, identifying a first reference object and a first target object in the preview image, and a second reference object and a second target object in each of the candidate images; Continue to perform depth prediction on the preview image and each of the candidate images to obtain depth of field and orientation information of the first reference object, the first target object, the second reference object, and the second target object; The depth of field and orientation information of the first reference object and the first target object in the preview image are respectively used as the background information of the preview image, and the depth of field and orientation information of the second reference object and the second target object in each of the candidate images are respectively used as the background information of the candidate image.

4. The method according to claim 2 or 3, wherein: The calling of the preset multimodal model, comparing the background information of the preview image with the background information of each of the candidate images, and determining the reference image from the candidate images includes: Calling the multimodal model to compare the preview image and its background information with each of the candidate images and their background information, respectively, to determine N target images from the candidate images, where N is a positive integer; The reference image is determined from the N target images.

5. The method according to claim 4, wherein: The value of N is greater than 1, and determining the reference image from the N target images includes: Display each of the target images and determine that the triggered target image is the reference image; or, Obtaining a first similarity between the preview image and each of the target images, and determining the target image with the highest first similarity as the reference image; or, An evaluation value corresponding to each of the target images is obtained based on the evaluation information associated with each of the target images, so as to determine the target image with the highest evaluation value from the target images as the reference image.

6. The method according to claim 4, wherein: After calling the multimodal model and comparing each of the candidate images and their background information based on the preview image and its background information to determine a plurality of target images from the candidate images, the method further includes: Respectively obtaining time information and weather information of the candidate image and each of the target images; Based on the time information of the candidate image and each of the target images, obtaining a time difference value corresponding to each of the target images and the candidate image; Obtaining a second similarity between the weather information of each of the target images and the candidate images; Determine, from the target image, a first image whose time difference is within a preset time range, and determine a second image whose second similarity is less than a preset similarity threshold; The first image and the second image are cleared from the target image.

7. The method according to claim 6, wherein: The step of respectively acquiring the time information and weather information of the candidate image and each of the target images comprises: Respectively obtain the description text of the candidate image and each of the target images; Extracting information from the caption text to obtain the time information and the weather information of the candidate image and each of the target images; If there is a third image between the candidate image and the target image that lacks the time information and / or the weather information, the multimodal model is called to perform prediction based on a preset prompt, the third image and its explanatory text to obtain the time information and / or the weather information that is missing from the third image.

8. The method according to any one of claims 1 to 3, wherein: After obtaining the reference image, the method further comprises: Obtaining the caption of the reference image; Extracting information from the caption text to obtain color information and reference camera parameters corresponding to the reference image; The color information of the reference image and the reference camera parameters are displayed.

9. The method according to claim 8, wherein: After extracting information from the description text, the method further includes: If the color information corresponding to the reference image is not extracted, performing image description on the reference image to obtain the color information of the background object and the foreground target object in the reference image; and / or If the reference camera parameters of the reference image are not extracted, the multimodal model is called to perform prediction based on a preset prompt, the reference image and its explanatory text to obtain the reference camera parameters of the reference image.

10. The method according to claim 8, wherein: After displaying the color information of the reference image and the reference camera parameters, the method further includes: In response to monitoring that the reference camera parameter is triggered, adjusting the camera parameter of the camera assembly based on the reference camera parameter.

11. The method according to any one of claims 1 to 3, wherein: The training process of the posture guidance model includes: Acquire a standard image sample containing a target object and a plurality of initial image samples; Performing posture estimation on the target object in the standard image sample and each of the initial image samples to obtain a third posture of each part of the target object in each of the initial image samples and a fourth posture of each part of the target object in the standard image; For each of the initial image samples, annotating the initial image sample with an action prompt text to obtain a reference action adjustment statement for each part from the third posture to the fourth posture; Based on the reference action adjustment sentences of each part of the target object from the third posture to the fourth posture in each of the initial image samples, a preset language model is trained to obtain the trained posture guidance model.

12. The method according to claim 11, wherein: The step of training a preset language model based on the reference action adjustment sentence of each part of the target object from the third posture to the fourth posture in each of the initial image samples comprises: For each of the initial image samples, calling the language model, performing posture description on the third posture of each part of the target object in the initial image sample to obtain first posture description information, and performing posture description on the fourth posture of each part of the target object in the standard image sample to obtain second posture description information; Calling the language model to generate a predicted action adjustment sentence from the first posture description information to the second posture description information based on the constructed context prompt; The action guidance prompt words are constructed to fine-tune the language model, and the language model is reversely adjusted based on the reference action adjustment sentence and the predicted action adjustment sentence.

13. The method according to any one of claims 1 to 3, wherein: The various parts include a face, and before calling the preset posture guidance model and comparing the first posture with the second posture, the method further includes: Performing facial recognition on the target object in the preview image and the reference image respectively to obtain a first facial expression on the face of the target object in the preview image and a second facial expression on the face of the target object in the reference image; The first facial expression is used as a first posture of the target object's face in the preview image, and the second facial expression is used as a second posture of the target object's face in the reference image.

14. The method according to any one of claims 1 to 3, wherein: The step of performing action recommendation and photo taking control based on the target action adjustment statement of each part from the first posture to the second posture and the reference picture includes: Displaying the target action adjustment statement for each part from the first posture to the second posture; updating the preview image and the first posture of each part of the target object in the preview image; For each part of the target object in the preview image, comparing the first posture and the second posture of the part after being updated, and determining whether the first posture and the second posture of the part after being updated are the same; In response to the first posture and the second posture of the part being the same after the update, stopping displaying the target action adjustment statement of the part; In response to the first posture and the second posture being the same after all parts are updated, the camera component is controlled to take a photo; or, in response to monitoring that a preset photo control is triggered, the camera component is controlled to take a photo.

15. The method according to any one of claims 1 to 3, wherein: After obtaining the reference image, the method further includes: Based on the multimodal model, position recognition is performed on the preview image and the reference image to obtain first relative position information of the first reference object and the first target object in the preview image, a photographing height and an orientation corresponding to the preview image, second relative position information of a third reference object and a third target object in the reference image, and a photographing height and an orientation corresponding to the reference image; generating position adjustment information of the first reference object or the first target object based on the first relative position information and the second relative position information; Acquire height and orientation adjustment information of the camera assembly based on the camera height and orientation corresponding to the preview image and the camera height and orientation corresponding to the reference image; Based on the position adjustment information of the first reference object or the first target object and the height and orientation adjustment information of the camera assembly, the camera position navigation information is generated and displayed.

16. The method according to any one of claims 1 to 3, wherein: After obtaining the reference image, the method further includes: Segmenting the reference image based on a preset object segmentation model, obtaining a plurality of segmented images, and determining at least one commodity in each of the segmented images; Performing image description on each of the segmented images to obtain product information of the product; For each of the segmented images, calling a target application to perform information retrieval based on the segmented image and the product information of the product in the segmented image to obtain a product link, wherein the product link is used to jump to a product recommendation page of the target application; The reference image and the product link are displayed.

17. An action recommendation and photo taking control device, wherein: include: An acquisition module is used to acquire a preview image collected by the camera component in real time and to acquire a reference image; A recognition module, used to perform posture recognition on the target object in the preview image and the reference image respectively, to obtain a first posture of each part of the target object in the preview image and a second posture of each part of the target object in the reference image; A comparison module, used for calling a preset posture guidance model, comparing the first posture and the second posture, and obtaining a target action adjustment statement for each part from the first posture to the second posture; The first processing module is used to perform action recommendation and photo control based on the target action adjustment statement of each part from the first posture to the second posture and the second posture of each part of the target object in the reference picture.

18. The device according to claim 17, wherein: The acquisition module is further used for: Obtain multiple candidate images from a preset database; Acquire background information of the preview image and background information of each of the candidate images; A preset multimodal model is called to compare the background information of the preview image with the background information of each of the candidate images, and the reference image is determined from the candidate images.

19. The device according to claim 18, wherein: The acquisition module is further used for: Performing target detection on the preview image and each of the candidate images respectively, identifying a first reference object and a first target object in the preview image, and a second reference object and a second target object in each of the candidate images; Continue to perform depth prediction on the preview image and each of the candidate images to obtain depth of field and orientation information of the first reference object, the first target object, the second reference object, and the second target object; The depth of field and orientation information of the first reference object and the first target object in the preview image are respectively used as the background information of the preview image, and the depth of field and orientation information of the second reference object and the second target object in each of the candidate images are respectively used as the background information of the candidate image.

20. The device according to claim 18 or 19, wherein: The acquisition module is further used for: Calling the multimodal model to compare the preview image and its background information with each of the candidate images and their background information, respectively, to determine N target images from the candidate images, where N is a positive integer; The reference image is determined from the N target images.

21. The device according to claim 20, wherein: The value of N is greater than 1, and the acquisition module is further used to: Display each of the target images and determine that the triggered target image is the reference image; or, Acquire a first similarity between the preview image and each of the target images, and determine the target image with the highest first similarity as the reference image; or, An evaluation value corresponding to each of the target images is obtained based on the evaluation information associated with each of the target images, so as to determine the target image with the highest evaluation value from the target images as the reference image.

22. The device according to claim 20, wherein: The acquisition module is further used for: Respectively obtaining time information and weather information of the candidate image and each of the target images; Based on the time information of the candidate image and each of the target images, obtaining a time difference value corresponding to each of the target images and the candidate image; Obtaining a second similarity between the weather information of each of the target images and the candidate images; Determine, from the target image, a first image whose time difference is within a preset time range, and determine a second image whose second similarity is less than a preset similarity threshold; The first image and the second image are cleared from the target image.

23. The device according to claim 22, wherein: The acquisition module is further used for: Respectively obtain the description text of the candidate image and each of the target images; Extracting information from the caption text to obtain the time information and the weather information of the candidate image and each of the target images; If there is a third image between the candidate image and the target image that lacks the time information and / or the weather information, the multimodal model is called to perform prediction based on a preset prompt, the third image and its explanatory text to obtain the time information and / or the weather information that is missing from the third image.

24. The device according to any one of claims 17 to 19, wherein: The device also includes a second processing module, configured to: Obtaining the caption of the reference image; Extracting information from the caption text to obtain color information and reference camera parameters corresponding to the reference image; The color information of the reference image and the reference camera parameters are displayed.

25. The device according to claim 24, wherein: The second processing module is further used for: If the color information corresponding to the reference image is not extracted, performing image description on the reference image to obtain the color information of the background object and the foreground target object in the reference image; and / or If the reference camera parameters of the reference image are not extracted, the multimodal model is called to perform prediction based on a preset prompt, the reference image and its explanatory text to obtain the reference camera parameters of the reference image.

26. The device according to claim 24, wherein: The second processing module is further used for: In response to monitoring that the reference camera parameter is triggered, adjusting the camera parameter of the camera assembly based on the reference camera parameter.

27. The device according to any one of claims 17 to 19, wherein: The device also includes a model training module, which is used to: Acquire a standard image sample containing a target object and a plurality of initial image samples; Performing posture estimation on the target object in the standard image sample and each of the initial image samples to obtain a third posture of each part of the target object in each of the initial image samples and a fourth posture of each part of the target object in the standard image; For each of the initial image samples, annotating the initial image sample with an action prompt text to obtain a reference action adjustment statement for each part from the third posture to the fourth posture; Based on the reference action adjustment sentences of each part of the target object from the third posture to the fourth posture in each of the initial image samples, a preset language model is trained to obtain the trained posture guidance model.

28. The device according to claim 27, wherein The model training module is also used for: For each of the initial image samples, calling the language model, performing posture description on the third posture of each part of the target object in the initial image sample to obtain first posture description information, and performing posture description on the fourth posture of each part of the target object in the standard image sample to obtain second posture description information; Calling the language model to generate a predicted action adjustment sentence from the first posture description information to the second posture description information based on the constructed context prompt; The action guidance prompt words are constructed to fine-tune the language model, and the language model is reversely adjusted based on the reference action adjustment sentence and the predicted action adjustment sentence.

29. The device according to any one of claims 17 to 19, wherein: The various parts include a face, and the recognition module is further used for: Performing facial recognition on the target object in the preview image and the reference image respectively to obtain a first facial expression on the face of the target object in the preview image and a second facial expression on the face of the target object in the reference image; The first facial expression is used as a first posture of the target object's face in the preview image, and the second facial expression is used as a second posture of the target object's face in the reference image.

30. The device according to any one of claims 17 to 19, wherein: The comparison module is further used for: Displaying the target action adjustment statement for each part from the first posture to the second posture; updating the preview image and the first posture of each part of the target object in the preview image; For each part of the target object in the preview image, comparing the first posture and the second posture of the part after being updated, and determining whether the first posture and the second posture of the part after being updated are the same; In response to the first posture and the second posture of the part being the same after the update, stopping displaying the target action adjustment statement of the part; In response to the first posture and the second posture being the same after all parts are updated, the camera component is controlled to take a photo; or, in response to monitoring that a preset photo control is triggered, the camera component is controlled to take a photo.

31. The device according to any one of claims 17 to 19, wherein: The device also includes an aircraft position navigation module, which is used for: Based on the multimodal model, position recognition is performed on the preview image and the reference image to obtain first relative position information of the first reference object and the first target object in the preview image, a photographing height and an orientation corresponding to the preview image, second relative position information of a third reference object and a third target object in the reference image, and a photographing height and an orientation corresponding to the reference image; generating position adjustment information of the first reference object or the first target object based on the first relative position information and the second relative position information; Acquire height and orientation adjustment information of the camera assembly based on the camera height and orientation corresponding to the preview image and the camera height and orientation corresponding to the reference image; Based on the position adjustment information of the first reference object or the first target object and the height and orientation adjustment information of the camera assembly, the camera position navigation information is generated and displayed.

32. The device according to any one of claims 17 to 19, wherein: The device also includes a commodity recommendation module, which is used to: Segmenting the reference image based on a preset object segmentation model, obtaining a plurality of segmented images, and determining at least one commodity in each of the segmented images; Performing image description on each of the segmented images to obtain product information of the product; For each of the segmented images, calling a target application to perform information retrieval based on the segmented image and the product information of the product in the segmented image to obtain a product link, wherein the product link is used to jump to a product recommendation page of the target application; The reference image and the product link are displayed.

33. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method according to any one of claims 1 to 16.

34. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-16.

35. A computer program product comprising a computer program, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 16.

36. A terminal device, comprising the apparatus according to any one of claims 17-32.

Images