Image processing method, image processing device, electronic device, and storage medium

Abstract

An image processing method, an image processing device, an electronic device, and a storage medium. The image processing method includes: obtaining an original image captured during the framing process; determining whether the original image meets a preset shooting quality; in response to the original image not meeting the preset shooting quality: providing a shooting guidance type corresponding to the original image; in response to a first guidance type in the shooting guidance type, reframing based on a guidance interaction given by the first guidance type to obtain a first intermediate framing image; in response to a second guidance type in the shooting guidance type, displaying a second intermediate framing image after the shooting parameters are adjusted; and obtaining a shooting result image from the first intermediate framing image or the second intermediate framing image. The auxiliary shooting process provided by the image processing method does not require scene recognition, and has wider adaptability and richer auxiliary shooting capabilities.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to an image processing method, an image processing apparatus, an electronic device, and a non-transitory computer-readable storage medium. Background Art

[0002] With the continuous development of artificial intelligence, more and more artificial intelligence applications are being applied in the field of images. For example, when users use mobile devices, cameras, and camcorders to take pictures, they can use artificial intelligence to perform image enhancement, image understanding, image style transformation, etc., and can automatically remove noise, beautify, remove reflections, and automatically correct perspective on the original pictures taken by users. Summary of the invention

[0003] At least one embodiment of the present disclosure provides an image processing method, comprising: acquiring an original image obtained during a framing process; determining whether the original image meets a preset shooting quality; in response to the original image not meeting the preset shooting quality: providing a shooting guide type corresponding to the original image, wherein the shooting guide type includes at least one of a first guide type and a second guide type, the first guide type being used to indicate a composition guide relative to the original image during the framing process, and the second guide type being used to indicate a shooting parameter adjustment relative to the original image during the framing process; in response to the first guide type in the shooting guide types, reframing is performed based on a guide interaction given by the first guide type to obtain a first intermediate framing image; in response to the second guide type in the shooting guide types, displaying a second intermediate framing image after the shooting parameters are adjusted; and obtaining a shooting result image from the first intermediate framing image or the second intermediate framing image.

[0004] For example, in at least one embodiment of the present disclosure, a method for processing an image is provided, in which determining whether the original image meets a preset shooting quality includes: determining whether the original image meets the preset shooting quality through scoring.

[0005] For example, in at least one embodiment of the present disclosure, there is provided an image processing method, in which, based on processing of the original image, a judgment result of whether the original image meets a preset shooting quality is obtained, including: performing image processing on the original image to obtain a score corresponding to the original image; in response to the score being less than a first threshold, determining that the original image does not meet the requirements; in response to the score being greater than the first threshold, determining that the original image meets the requirements.

[0006] For example, in at least one embodiment of the present disclosure, an image processing method is provided in which image processing is performed on the original image to obtain a score corresponding to the original image, including: inputting the original image into a first judgment model, processing the original image using the first judgment model, and obtaining a score corresponding to the original image.

[0007] For example, in at least one embodiment of the present disclosure, there is provided an image processing method, in which, based on processing of the original image, a judgment result of whether the original image meets a preset shooting quality is obtained, including: performing feature extraction on the original image using a neural network, and outputting a category of the original image, wherein the category includes a first category or a second category, the first category indicating that the original image meets the preset shooting quality, and the second category indicating that the original image does not meet the preset shooting quality.

[0008] For example, in at least one embodiment of the present disclosure, there is provided an image processing method, in which a shooting guidance type corresponding to the original image is provided, including: inputting the original image into a second judgment model, processing the original image using the second judgment model, and obtaining a first result value corresponding to the first guidance type and a second result value corresponding to the second guidance type; in response to the first result value being greater than a second threshold, determining that the shooting guidance type corresponding to the original image includes the first guidance type; in response to the second result value being greater than a third threshold, determining that the shooting guidance type corresponding to the original image includes the second guidance type.

[0009] For example, in at least one embodiment of the present disclosure, a method for processing an image is provided, in which a judgment result of whether the original image meets a preset shooting quality is obtained through a neural network, and the second judgment model and the neural network share a network, and the shared network is configured to simultaneously output the judgment result of whether the original image meets the preset shooting quality, as well as the first result value and the second result value.

[0010] For example, in at least one embodiment of the present disclosure, there is provided an image processing method, in response to the first guidance type among the shooting guidance types, reframing is performed based on the guidance interaction given by the first guidance type to obtain a first intermediate framed image, including: in response to the first guidance type among the shooting guidance types, displaying the guidance interaction interface of the first guidance type, performing guidance interaction reframing to obtain the first intermediate framed image.

[0011] In response to the first guidance type in the shooting guidance types, in response to the first guidance type in the shooting guidance types, a guidance interaction interface of the first guidance type is displayed, and a guidance interaction re-frame is performed to obtain a first intermediate framed image, including: displaying the position of a guidance reference object in the original image and a target movement trajectory of the guidance reference object in the guidance interaction interface, prompting the operator to move the guidance reference object according to the target movement trajectory, and guiding the operator to re-frame to obtain the first intermediate framed image, wherein the guidance reference object includes a target point or a target frame in the original image.

[0012] For example, at least one embodiment of the present disclosure provides an image processing method, in which the position of the guiding reference object in the original image and the target moving trajectory of the guiding reference object are displayed in the guiding interaction interface, the operator is prompted to move the guiding reference object according to the target moving trajectory, and the operator is guided to re-frame to obtain the first intermediate framing image, including: during the movement of the guiding reference object, in response to detecting that the image obtained by framing is in a stable focus state, acquiring a framing image that is currently in a stable focus state; based on processing the framing image, acquiring a judgment result of whether the framing image meets a preset shooting quality requirement; in response to the framing image meeting the preset shooting quality requirement, the operator is prompted to shoot to obtain the first intermediate framing image; in response to the framing image not meeting the preset shooting quality requirement, the target moving trajectory of the guiding reference object continues to be displayed, guiding the operator to re-frame until the acquired framing image meets the preset shooting quality, and the operator is prompted to shoot to obtain the first intermediate framing image.

[0013] For example, in at least one embodiment of the present disclosure, there is provided an image processing method, in which the position of the guiding reference object in the original image and the target movement trajectory are obtained through a composition prediction model; the composition prediction model is configured to receive the original image, perform target detection on the original image, so as to obtain the position of the guiding reference object, the target movement distance and the target movement direction, wherein the target movement trajectory of the guiding reference object includes the target movement distance and the target movement direction.

[0014] For example, in at least one embodiment of the present disclosure, there is provided an image processing method, in which, before providing a shooting guidance type corresponding to the original image, the image processing method further includes: displaying a guidance mode prompt interface to prompt the operator to select whether to enter the auxiliary shooting process; and performing subsequent processing in response to selecting to enter the auxiliary shooting process.

[0015] For example, in at least one embodiment of the present disclosure, there is provided an image processing method, in response to the second guidance type in the shooting guidance type, displaying a second intermediate framing image after shooting parameters are adjusted, including: in response to the shooting guidance type including the first guidance type, using the adjustment model to adjust the shooting parameters of the first intermediate framing image to obtain the second intermediate framing image; in response to the shooting guidance type not including the first guidance type, using the adjustment model to adjust the shooting parameters of the original image to obtain the second intermediate framing image; wherein the shooting parameter adjustment includes adjusting one or more of the brightness, hue, saturation, sharpness, and hue of the image.

[0016] For example, in at least one embodiment of the present disclosure, there is provided an image processing method, in which a shooting result image is obtained from the first intermediate framing image or the second intermediate framing image, including: in response to the shooting guidance type including the first guidance type but not including the second guidance type, using the first intermediate framing image as the shooting result image; in response to the shooting guidance type including the second guidance type, using the second intermediate framing image as the shooting result image.

[0017] For example, in at least one embodiment of the present disclosure, there is provided an image processing method, in which the shooting guidance type corresponding to the original image and the judgment result of whether the original image meets the preset shooting quality are determined by a common neural network, and the shooting parameter adjustment in response to the second guidance type is completed by an adjustment model, and the image processing method also includes: using the adjustment model as a generator, using the common neural network as a discriminator, forming a generative adversarial network with the adjustment model and the common neural network, and performing alternating iterative training on the generative adversarial network to obtain the trained common neural network and the adjustment model.

[0018] For example, in at least one embodiment of the present disclosure, there is provided an image processing method, in which the generative adversarial network is alternately iteratively trained to obtain the trained neural network and the adjustment model, including: obtaining a first training set and a second training set, wherein the first training set includes a plurality of first training pictures and a plurality of first standard pictures corresponding one-to-one to the plurality of first training pictures, each first training picture is obtained by changing the composition structure of a first standard picture corresponding to the first training picture, and the second training set includes a plurality of second training pictures and a plurality of second standard pictures corresponding one-to-one to the plurality of second training pictures, each second training picture is obtained by adjusting shooting parameters of the second standard picture corresponding to the second training picture; in the alternating iterative training process, the adjustment model is trained using the second training set, and the common neural network is trained using the output of the adjustment model and the first training set.

[0019] For example, in at least one embodiment of the present disclosure, a method for image processing is provided, in which the position of the guidance reference object in the original image displayed in the guidance interaction interface and the target movement trajectory of the guidance reference object are obtained through a composition prediction model, and the image processing method further includes: training the composition prediction model using the first training set.

[0020] At least one embodiment of the present disclosure provides an image processing device, comprising: an image acquisition unit, configured to acquire an original image obtained during a framing process; a shooting quality judgment unit, configured to judge whether the original image meets a preset shooting quality; an image processing unit, configured to: in response to the original image not meeting the preset shooting quality: provide a shooting guide type corresponding to the original image, wherein the shooting guide type includes at least one of a first guide type and a second guide type, the first guide type is used to indicate a composition guide relative to the original image during the framing process, and the second guide type is used to indicate a shooting parameter adjustment relative to the original image during the framing process; in response to the first guide type in the shooting guide type, re-frame based on the guidance interaction given by the first guide type to obtain a first intermediate framing image; in response to the second guide type in the shooting guide type, display a second intermediate framing image after the shooting parameters are adjusted; and obtain a shooting result image from the first intermediate framing image or the second intermediate framing image.

[0021] At least one embodiment of the present disclosure provides an electronic device, comprising: a memory, which non-transiently stores computer-executable instructions; and a processor, configured to execute the computer-executable instructions, wherein the computer-executable instructions, when executed by the processor, implement the image processing method according to any embodiment of the present disclosure.

[0022] At least one embodiment of the present disclosure provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, the image processing method according to any embodiment of the present disclosure is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings of the embodiments will be briefly introduced below. Obviously, the drawings in the following description only relate to some embodiments of the present disclosure, but are not intended to limit the present disclosure.

[0024] Figure 1 A flowchart of an image processing method provided by an embodiment of the present disclosure;

[0025] Figure 2A flowchart of auxiliary shooting provided by at least one embodiment of the present disclosure;

[0026] Figure 3 A training flow chart of a network model provided for at least one embodiment of the present disclosure;

[0027] Figure 4 A schematic block diagram of an image processing device provided by at least one embodiment of the present disclosure;

[0028] Figure 5 A schematic block diagram of an electronic device provided in accordance with an embodiment of the present disclosure;

[0029] Figure 6 A schematic diagram of a non-transitory computer-readable storage medium provided for at least one embodiment of the present disclosure;

[0030] Figure 7 A schematic diagram of a hardware environment provided for at least one embodiment of the present disclosure. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution of the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings of the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the described embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0032] Unless otherwise defined, the technical terms or scientific terms used in the present disclosure should be understood by people with ordinary skills in the field to which the present disclosure belongs. The "first", "second" and similar words used in the present disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing in front of the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly. In order to keep the following description of the embodiments of the present disclosure clear and concise, the present disclosure omits the detailed description of some known functions and known components.

[0033] At present, most of the auxiliary photography functions provided by shooting devices are to optimize parameters such as image color, usually using scene recognition, and adjusting according to different scenes according to the preset adjustment parameters of the scene. For example, when the user is framing, the shooting device automatically identifies shooting scenes with distinctive characteristics, such as scenery, animals, food, low light, backlight, etc. The algorithm makes a series of adjustments to the shooting parameters of the image according to the preset adjustment parameters of the scene. At the moment the user presses the shutter, the shooting is completed according to the adjusted shooting parameters (such as brightness, saturation, color, etc.).

[0034] In the process of photography, composition is also an important part. Composition refers to the selection and layout of all elements contained in the lens within the framing range of the lens. It plays a role in highlighting the theme, explaining the background, portraying the emotions of the characters, and ensuring the beauty of the image. Composition directly affects the aesthetics of the photo.

[0035] Some current composition assistance methods provide some reference templates based on the scene. The reference template sets the outline position of the main composition elements according to different scenes. When shooting, the user adjusts the direction and distance of the camera according to the outline position of the main composition elements in the reference template to achieve a certain degree of overlap with the reference template. The scene recognition and search matching algorithm are used at the algorithm layer. For example, according to different scenes, the reference templates include mountain peaks, portraits, etc. When it is recognized that the user is shooting a mountain, the reference template of the mountain peak is selected. According to the reference template of the mountain peak, the user is guided to place the mountain in the lens at the position specified by the reference template. This process calculates the matching degree between the reference template and the user's framing image. When the matching degree exceeds the preset threshold, it is considered that the user has placed the mountain in the framing at the specified position and the shooting is completed.

[0036] The above-mentioned composition assistance and shooting parameter assistance functions are of limited help to users. They are restricted by scene types and fixed composition styles, have poor flexibility, and cannot adapt to a wide range of shooting scenes. In addition, the above-mentioned methods have weak modeling capabilities for the shooting experience of professional photographers.

[0037] At least one embodiment of the present disclosure provides an image processing method, an image processing device, an electronic device, and a non-transient storage medium. The image processing method includes: obtaining an original image taken by a user during a framing process; based on processing the original image, obtaining a judgment result of whether the original image meets a preset shooting quality; in response to the original image not meeting the preset shooting quality: providing a shooting guidance type corresponding to the original image, wherein the shooting guidance type includes at least one of a first guidance type and a second guidance type, the first guidance type is used to indicate that a composition guidance needs to be performed relative to the original image during the framing process, and the second guidance type is used to indicate that a shooting parameter needs to be adjusted relative to the original image during the framing process; in response to the first guidance type in the shooting guidance type, displaying a guidance interaction interface of the first guidance type to guide the user to re-frame to obtain a first intermediate framing image; in response to the second guidance type in the shooting guidance type, displaying a second intermediate framing image after the shooting parameters are adjusted; and obtaining a shooting result image from the first intermediate framing image or the second intermediate framing image.

[0038] The image processing method provided by at least one embodiment of the present disclosure first evaluates the professionalism of the original image taken by the user to determine whether the original image meets the preset shooting quality requirements. If the original image does not meet the preset shooting quality requirements, the operator is assisted in shooting. During the operator's framing and shooting process, the shooting skills of professionals (such as professional photographers) learned by the artificial intelligence algorithm can be output in the form of suggestions and guidance to assist the operator in completing high-level professional-level shooting. The auxiliary shooting process does not require scene recognition, is not constrained by scene types and composition rules, has strong scalability, and has wider adaptability and richer auxiliary shooting capabilities.

[0039] The image processing method provided in at least one embodiment of the present disclosure may be applied to the image processing device provided in the embodiment of the present disclosure, and the image processing device may be configured on an electronic device. The electronic device may be a personal computer, a mobile terminal, etc., and the mobile terminal may be a hardware device such as a mobile phone, a tablet computer, a laptop computer, XR glasses, etc.

[0040] The embodiments of the present disclosure are described in detail below with reference to the accompanying drawings, but the present disclosure is not limited to these specific embodiments.

[0041] Figure 1 The present invention provides a flowchart of an image processing method according to an embodiment of the present invention.

[0042] like Figure 1 As shown, the image processing method provided by at least one embodiment of the present disclosure includes steps S10 to S60.

[0043] Step S10, acquiring an original image obtained during the framing process.

[0044] For example, an operator (e.g., a user) starts a camera, selects an area to be photographed within the camera's framing range, and photographs the original image. For example, the original image may be a framing image in a stable focus state, and the operator does not need to press the shutter at this time, and the framing image (i.e., the image within the current lens framing range) presented in the image preview display area of ​​the shooting device is used as the original image. For example, the original image may also be an image obtained after the operator presses the shutter during the framing process.

[0045] For example, the original image may be captured by a video camera or a still camera, or the original image may be captured by a camera (or lens) in a mobile terminal. The present disclosure does not specifically limit the device for acquiring the original image.

[0046] For example, the original image may be various types of images, such as landscape images, character images, object images, etc. For example, landscape images may include landscape objects such as mountains, rivers, plants, animals, and the sky. Character images are images including people (e.g., faces, etc.). For example, character images may include face areas. Object images may include vehicles, houses, and other object objects. Of course, in addition to face areas, character images may also include areas corresponding to landscape objects, object objects, and the like. For example, in some embodiments, the original image may be a character image, such as a close-up image of a person. For example, in other embodiments, the original image may also be a character image with landscape objects or object objects. It should be noted that the present disclosure does not specifically limit the type of the original image or the scene in the original image. The original image may be an image obtained by the operator by photographing the area that he wants to photograph.

[0047] For example, although the above description describes the operation of the operator starting the camera, this is not a necessary operation step. Other operations to start the framing process, such as other applications calling the camera program to start the framing process, or by instructions (such as timed shooting instructions), or by trigger conditions, etc., all include the embodiments of the present disclosure. Regardless of who starts the framing process, it can be logically regarded as the operator.

[0048] Step S20, determining whether the original image meets the preset shooting quality.

[0049] For example, step S20 may include, based on processing the original image, obtaining a judgment result of whether the original image meets a preset shooting quality.

[0050] For example, the judgment result of whether the original image meets the preset shooting quality may include two results, that is, the original image meets the preset shooting quality and the original image does not meet the preset shooting quality.

[0051] For example, if the original image meets the preset shooting quality, it means that the original image shot by the operator meets the shooting level of professional photographers and is an image recognized by professionals. For example, the operator can be prompted to shoot immediately to obtain the final image, or the operator can be prompted to save the shot image. Of course, the operator can also choose to enter the auxiliary shooting process at this time.

[0052] For example, if the original image does not meet the preset shooting quality, it means that the original image taken by the operator does not meet the shooting level of professional photographers and is not recognized by professionals, and shooting guidance is required. At this time, a guidance mode prompt interface can be displayed to prompt the operator to choose whether to enter the auxiliary shooting process; if the operator chooses to enter the auxiliary shooting process, subsequent processing is performed.

[0053] For example, the network model can be trained through machine learning methods so that the network model can receive and process the original image to obtain a judgment result of whether the original image meets the preset shooting quality.

[0054] For example, in some embodiments, step S20 may include: determining whether the original image meets a preset shooting quality through scoring.

[0055] For example, judging whether the original image meets the preset shooting quality through scoring may include: performing image processing on the original image to obtain a score corresponding to the original image; in response to the score being less than a first threshold, determining that the original image does not meet the preset shooting quality; in response to the score being greater than the first threshold, determining that the original image meets the preset shooting quality.

[0056] For example, performing image processing on the original image to obtain a score corresponding to the original image may include: inputting the original image into a first judgment model, and processing the original image using the first judgment model to obtain a score corresponding to the original image.

[0057] For example, a large number of training images can be acquired in advance, the training images can be scored, and the scores can be used as standard values ​​to train the first judgment model, thereby obtaining a trained first judgment model. During use, the original image is input into the first judgment model, and the score corresponding to the original image can be output, and the score is compared with the first threshold to determine whether the original image meets the preset shooting quality.

[0058] For example, the first threshold can be set according to actual usage needs, and the present disclosure does not limit this.

[0059] Since the judgment results include two types, the neural network can also be used to obtain a binary classification result of the original image, one classification is that the original image meets the preset shooting quality, and the other classification is that the original image does not meet the preset shooting quality.

[0060] For example, in some other embodiments, step S20 may include: using a neural network to extract features from the original image, and outputting a category of the original image, wherein the category includes a first category or a second category, the first category indicates that the original image meets a preset shooting quality, and the second category indicates that the original image does not meet the preset shooting quality.

[0061] Similarly, a large number of training images can be acquired in advance, and the training images can be classified to determine whether each training image belongs to the first category or the second category. The category of the training image is used as a standard value to train the neural network, thereby obtaining a trained neural network. During use, the original image is input into the neural network, and the category of the original image can be output. According to the category, it can be determined whether the original image meets the preset shooting quality.

[0062] The above two methods are essentially the same. In the neural network, it can also be understood that the score is determined internally by the model. If the score is greater than the first threshold, the original image is determined to be in the first category. If the score is less than the first threshold, the original image is determined to be in the second category.

[0063] In the above two processes of obtaining the judgment results of whether the original image meets the preset shooting quality, the training images used may include a large number of images recognized by professionals (such as photographers) (regardless of whether such images are taken by photographers or ordinary operators). Such images can be regarded as meeting the preset shooting quality, and they can be marked with a high score, or marked as the first category. The training images may also include a large number of images not recognized by professionals (such as photographers) (regardless of whether such images are taken by photographers or ordinary operators). Such images can be regarded as not meeting the preset shooting quality, and they can be marked with a low score, or marked as the second category. Thus, the neural network or the first judgment model can learn the features recognized by professionals, and learn this experience through machine learning methods, so that the shooting quality of the original images taken by the operator can be automatically evaluated from a professional perspective. When it is evaluated that auxiliary shooting is required, the auxiliary shooting process can be entered to perform subsequent processing.

[0064] In some embodiments of the present disclosure, the description of the preset shooting quality is not limited to a fixed concept. As more and more images are evaluated by professionals, the features recognized by professionals are dynamically changing, and the machine learning method used can be iteratively updated or adjusted. The preset shooting quality generated can be continuously adjusted or updated and pushed to the operator terminal (for example, by pushing the camera module update through the cloud or packaging it in the system update).

[0065] In some embodiments of the present disclosure, it can be implemented through a variety of neural networks, such as CNN (or various implementations derived from CNN, such as AlexNet, VGG, NIN, GoogLeNet, ResNet, FCN, etc.), and classification and scoring can be achieved through softmax, svm, lr, etc.

[0066] For example, the subsequent processing in the auxiliary shooting process may include the following steps S30-S60.

[0067] Step S30, in response to the original image not satisfying a preset shooting quality: providing a shooting guidance type corresponding to the original image, wherein the shooting guidance type includes at least one of a first guidance type and a second guidance type, the first guidance type being used to indicate a composition guidance relative to the original image during the framing process (that is, composition guidance needs to be performed relative to the original image during the framing process), and the second guidance type being used to indicate a shooting parameter adjustment relative to the original image during the framing process (that is, shooting parameters need to be adjusted relative to the original image during the framing process).

[0068] Step S40 , in response to a first guidance type in the shooting guidance types, reframing is performed based on a guidance interaction given by the first guidance type to obtain a first intermediate framing image.

[0069] Step S50 : In response to a second guide type in the shooting guide type, displaying a second intermediate framing image after the shooting parameters are adjusted.

[0070] Step S60: obtaining a shooting result image from the first intermediate framing image or the second intermediate framing image.

[0071] Through the above steps S30-S60, the shooting result image is obtained. For example, the shooting result image can be automatically stored as the final image, or the shooting result image can be displayed in the image preview display area of ​​the shooting device to prompt the operator whether to confirm the use. For example, if the operator confirms the use, the shooting result image is saved, or if the operator chooses to reshoot, the next round of shooting is continued to step S10.

[0072] For example, in the above auxiliary shooting process, an exit button can be set in the display interface, and the operator can actively exit the auxiliary shooting mode at any time during the auxiliary shooting process.

[0073] Through the above method, when the original image does not meet the preset shooting quality, the original image is analyzed to confirm whether it needs composition guidance or shooting parameter adjustment, and the corresponding guidance method is adaptively selected for auxiliary shooting guidance based on the analysis result. Shooting suggestions are provided in the form of text, animation, optional modes, etc. to assist the operator to complete high-level professional-level shooting. The shooting suggestions are not restricted by scenes or fixed styles and are highly flexible.

[0074] The specific execution process of steps S30 - S60 in the auxiliary shooting process is described in detail below.

[0075] Step S30 may include: inputting the original image into a second judgment model, processing the original image using the second judgment model, and obtaining a first result value corresponding to the first guidance type and a second result value corresponding to the second guidance type; in response to the first result value being greater than a second threshold, determining that the shooting guidance type corresponding to the original image includes the first guidance type; in response to the second result value being greater than a third threshold, determining that the shooting guidance type corresponding to the original image includes the second guidance type.

[0076] For example, the second threshold may be equal to the third threshold, or may not be equal to the third threshold. The second threshold and the third threshold may be set as needed, and the present disclosure does not impose any specific limitation on this.

[0077] For example, the second judgment model can process the original image to confirm whether the original image has a composition problem and / or a shooting parameter problem. For example, if it is determined that the original image has a composition problem, the first result value corresponding to the first guidance type output is larger, for example, larger than the second threshold, and it is determined that the shooting guidance type corresponding to the original image includes the first guidance type; for example, if it is determined that the original image has a shooting parameter problem, the second result value corresponding to the second guidance type output is larger, for example, larger than the third threshold, and it is determined that the shooting guidance type corresponding to the original image includes the second guidance type.

[0078] The shooting guide type corresponding to the original image may include only the first guide type or the second guide type, and of course, may also include both the first guide type and the second guide type.

[0079] For example, if a neural network (such as a classification network) is used to obtain a judgment result of whether the original image meets the preset shooting quality, the second judgment model can share a network with the neural network, that is, both use the same network to reduce the number of parameters, especially when configured on a mobile terminal, which can greatly reduce resource overhead and improve operating efficiency. At this time, the shared network is configured to simultaneously output the judgment result of whether the original image meets the preset shooting quality, as well as the first result value and the second result value. In the following, the shared network is referred to as a shared neural network.

[0080] For example, when the shooting guidance type corresponding to the original image includes the first guidance type, step S40 may include: in response to the first guidance type in the shooting guidance type, displaying the guidance interaction interface of the first guidance type, performing guidance interaction reframing to obtain a first intermediate framing image.

[0081] For example, in response to a first guidance type in the shooting guidance types, a guidance interaction interface of the first guidance type is displayed, and a guidance interaction re-frame is performed to obtain a first intermediate framed image. This may include: displaying the position of a guidance reference object in the original image and a target movement trajectory of the guidance reference object in the guidance interaction interface, prompting the operator to move the guidance reference object according to the target movement trajectory, and guiding the operator to re-frame to obtain the first intermediate framed image, wherein the guidance reference object includes a target point or a target frame in the original image.

[0082] For example, displaying the position of the guiding reference object in the original image and the target moving trajectory of the guiding reference object in the guiding interaction interface, prompting the operator to move the guiding reference object according to the target moving trajectory, and guiding the operator to re-frame to obtain a first intermediate framing image, may include: during the movement of the guiding reference object, in response to detecting that the image obtained by framing is in a stable focus state, acquiring a framing image that is currently in a stable focus state; based on processing the framing image, acquiring a judgment result of whether the framing image meets a preset shooting quality; in response to the framing image meeting the preset shooting quality, prompting the user to shoot to obtain the first intermediate framing image; in response to the framing image not meeting the preset shooting quality, continuing to display the target moving trajectory of the guiding reference object, guiding the operator to re-frame, until the acquired framing image meets the preset shooting quality, and prompting the operator to shoot to obtain the first intermediate framing image.

[0083] For example, the position of the guiding reference object in the original image and the target movement trajectory are obtained through a composition prediction model; the composition prediction model is configured to receive the original image, perform target detection on the original image, so as to obtain the position of the guiding reference object, the target movement distance and the target movement direction, wherein the target movement trajectory of the guiding reference object includes the target movement distance and the target movement direction.

[0084] For example, the guide reference object can be a composition element that plays a key role in the composition of the original image obtained by the composition prediction model. The guide reference object can be in the form of a target frame (rectangular frame, elliptical frame, etc., which is not specifically limited in the present disclosure), or in the form of a target point, that is, a key feature point in the original image. The recognition of the guide reference object can be achieved by training the composition prediction model. For the specific process, refer to the relevant description of the subsequent training process.

[0085] For example, taking the example of an operator taking a landscape image with a mountain, the guiding reference object that can be obtained through the composition prediction model is, for example, a rectangular frame (target frame) at the top of the mountain, or a point (target point) at the top of the mountain, etc.

[0086] For example, the target moving distance and the target moving direction may be the distance and direction in a two-dimensional plane, such as the moving distance and the moving direction in the two-dimensional plane coordinate system where the image is located. For example, in some embodiments, the target moving distance and the target moving direction may be the distance and direction in a three-dimensional plane, and in addition to the horizontal direction and the vertical direction in the two-dimensional plane, the moving distance in the depth direction perpendicular to the two-dimensional plane where the image is located may also be provided, for example, the moving distance in the depth direction may indicate zooming in and out of the image, thereby providing more dimensional shooting assistance and providing richer auxiliary shooting capabilities.

[0087] For example, in the guidance interaction interface, the position of the guidance reference object and the target movement trajectory can be annotated in a static or dynamic manner. For example, the target movement trajectory can be a visual line to guide the operator to move along the visual line; or the target movement trajectory can provide a movement start point (that is, the current position of the guidance reference object) and a movement end point, prompting the operator to move the guidance reference object to the position marked by the movement end point in the form of animation or the like.

[0088] During the interaction, the operator moves the guiding reference object according to the moving trajectory. During the movement of the guiding reference object, the lens will reframe accordingly. When it is detected that the image obtained by framing is in a stable focus state, a framing image that is currently in a stable focus state is obtained. The framing image can be an image in a stable focus state obtained by the current camera within the framing range.

[0089] Afterwards, the framing image is evaluated to see whether it meets the preset shooting quality in the manner described in the aforementioned step S20, and a judgment result of whether the framing image meets the preset shooting quality is obtained. For example, the framing image is input into a first judgment model or a neural network, and a judgment result of whether the framing image meets the preset shooting quality is obtained based on the output scoring or classification result.

[0090] For example, if it is determined that the framing image meets the preset shooting quality, the framing image is used as the first intermediate framing image; for example, if it is determined that the framing image does not meet the preset shooting quality, the movement trajectory of the guiding reference object continues to be displayed to guide the operator to continue adjusting the framing until a framing image in a stable focus state is obtained again, and the framing image is evaluated to see whether it meets the preset shooting quality. If the framing image meets the preset shooting quality, the framing image is used as the first intermediate framing image, otherwise the above process is repeated until a framing image that meets the preset shooting quality is obtained or the operator actively exits the composition guidance.

[0091] For example, if the shooting guidance type includes only the first guidance type but does not include the second guidance type, the first intermediate view image is used as the shooting result image, for example, prompting the operator to shoot (eg, press the shutter button, etc.) to obtain the final image.

[0092] In this embodiment, for composition guidance, it is not necessary to identify the scene, nor is it necessary to search and match the framed image with the reference template. Instead, the position and target movement trajectory of the guidance reference object can be directly obtained through the composition prediction model, and the operator can move the guidance reference object according to the target movement trajectory, thereby avoiding the constraints and restrictions brought by fixed scenes and fixed templates, enhancing the scalability and adaptability of auxiliary shooting, and providing more flexible and rich auxiliary shooting capabilities. In addition, as described later, the composition prediction model is trained using a large number of photos recognized by professionals, so that the features recognized by professionals (such as the shooting experience of professional photographers) can be learned through machine learning methods, and modeling is performed based on the features recognized by professionals. The composition prediction model is used to automatically obtain the key composition elements that have a greater impact on the composition as the guidance reference object, and the target movement trajectory of the guidance reference object is obtained according to the learned knowledge, providing more rich and flexible auxiliary shooting, and helping the operator to complete high-level professional-level shooting.

[0093] For example, step S50 may include: in response to the shooting guidance type including the first guidance type, using the adjustment model to adjust the shooting parameters of the first intermediate framing image to obtain the second intermediate framing image; in response to the shooting guidance type not including the first guidance type, using the adjustment model to adjust the shooting parameters of the original image to obtain the second intermediate framing image; wherein the shooting parameter adjustment includes adjusting one or more of the brightness, hue, saturation, sharpness, and hue of the image.

[0094] For example, a large number of training images can be used to train the adjustment model to obtain a trained adjustment model. For the training process of the adjustment model, reference can be made to the relevant description of the subsequent training process. The trained adjustment model can automatically adjust the shooting parameters of the input image, such as adjusting the brightness, hue, saturation, hue, etc. of the input image, and optimize its shooting parameters to obtain an image that is more in line with the shooting level of a professional photographer. The shooting parameters that need to be adjusted can be set as needed, and the present disclosure does not make specific restrictions on this.

[0095] Step S60 may include: in response to the shooting guide type including the first guide type but not including the second guide type, taking the first intermediate framing image as the shooting result image; in response to the shooting guide type including the second guide type, taking the second intermediate framing image as the shooting result image.

[0096] Usually, the quality of composition has a greater impact on image effects. Therefore, composition guidance can be performed first. After obtaining a first intermediate framing image whose composition meets the preset shooting quality, the shooting parameters of the first intermediate framing image are adjusted to obtain a second intermediate image as the shooting result image. If the original image has no composition problems, the shooting parameters of the original image can be directly adjusted to obtain the second intermediate image as the shooting result image. If the original image only has composition problems and there are no problems with shooting parameters, the first intermediate image can be directly used as the shooting result image.

[0097] Figure 2 This is a flowchart of auxiliary shooting provided by at least one embodiment of the present disclosure. Figure 2 , specifically explain the specific process of the entire auxiliary shooting.

[0098] First, when the operator starts shooting, the camera is started, such as turning on the camera, and the operator frames the image. When a stable focus state is detected, the image within the current framing range of the camera is obtained as the original image. Of course, the operator is a "user" as an example for schematic description, and other framing processes can also be started, which will not be repeated here.

[0099] Afterwards, the above step S20 is used to obtain a judgment result of whether the original image meets the preset shooting quality. For example, the original image can be input into the first judgment model or neural network to obtain a judgment result of whether the original image meets the preset shooting quality. The specific process can refer to the above step S20, which will not be repeated here.

[0100] Afterwards, if the original image meets the preset shooting quality, the operator is prompted to shoot immediately, such as pressing a shutter button, etc., to obtain the final image, and the shooting process ends.

[0101] If the original image does not meet the preset shooting quality, the shooting guidance type corresponding to the original image is obtained by using the above step S30. For example, the original image can be input into the second judgment model to obtain the shooting guidance type corresponding to the original image; or, when the second judgment model and the neural network share a network, the judgment result of whether the original image meets the preset shooting quality and the shooting guidance type corresponding to the original image can be obtained at the same time in step S20. The specific process can refer to the above step S30, which will not be repeated here.

[0102] Afterwards, if the shooting guidance type includes the first guidance type, a composition guidance interaction is performed. For example, a composition guidance interaction interface is displayed to guide the operator to move the guidance reference object according to the target movement trajectory, thereby guiding the operator to re-frame and obtain a framing image. If the framing image meets the preset shooting quality, the framing image is used as the first intermediate framing image. If the framing image does not meet the preset shooting quality, the above process is repeated to guide the operator to continue to adjust the framing until a framing image that meets the preset shooting quality is obtained and used as the first intermediate framing image, or until the operator actively exits the composition guidance interaction to complete the entire composition guidance interaction. The specific process can refer to the above step S40, which will not be repeated here.

[0103] If the shooting guidance type does not include the first guidance type but includes the second guidance type, the shooting parameters of the original image are adjusted to obtain a second intermediate framing image, for example, the original image is input into the adjustment model to obtain the second intermediate framing image. The specific process can refer to the above step S50, which will not be repeated here.

[0104] If the shooting guidance type includes the first guidance type and the second guidance type, the shooting parameters of the first intermediate framing image are adjusted to obtain the second intermediate framing image, for example, the first intermediate framing image is input into the adjustment model to obtain the second intermediate framing image. The specific process can refer to the above step S50, which will not be repeated here.

[0105] If the shooting guidance type includes the first guidance type but does not include the second guidance type, the first intermediate framing image will be displayed as the shooting result image in the image preview display area of ​​the shooting device. If the operator confirms, the shooting result image will be saved, or the operator will be prompted to press the shutter to shoot, ending this round of shooting process; if the operator chooses to reshoot, the framing process will be restarted.

[0106] If the shooting guidance type includes the second guidance type, the second intermediate framing image will be displayed as the shooting result image in the image preview display area of ​​the shooting device. Similarly, if the operator confirms, the shooting result image will be saved, or the operator will be prompted to press the shutter to shoot, ending this round of shooting process; if the operator chooses to reshoot, the framing process will be restarted.

[0107] In this embodiment, an auxiliary shooting process is provided, in which the original image obtained during the operator's framing process is first evaluated to see whether it meets the preset shooting quality. If the original image is evaluated to have composition problems or shooting parameter problems, composition guidance or shooting parameter adjustment is performed. For example, the position of the guidance reference object and the target movement trajectory are directly output in the composition guidance, and the operator is guided to move the guidance reference object according to the target movement trajectory, and the framing image in a stable focus state is continuously evaluated during the movement to see whether it meets the preset shooting quality, until a framing image that meets the preset shooting quality is obtained. In addition, the adjustment model can also be used to directly adjust the shooting parameters of the original image or the first intermediate framing image. Both composition guidance and shooting parameter adjustment do not require scene recognition, nor do they require matching the template or preset parameters under the scene, so that auxiliary shooting can be achieved more flexibly, and the features recognized by professionals learned by the artificial intelligence algorithm can be output in the form of suggestions and guidance, so as to assist the operator in completing high-level professional-level shooting.

[0108] For example, the shooting guidance type corresponding to the original image and the judgment result of whether the original image meets the preset shooting quality are determined through a shared neural network, the shooting parameter adjustment is completed through the adjustment model, and the position of the guidance reference object in the original image displayed in the guidance interaction interface and the target movement trajectory of the guidance reference object are obtained through the composition prediction model. The following illustrates the training process of the shared neural network, adjustment model, and composition prediction model with reference to the accompanying drawings.

[0109] Before training, you first need to get a training set for training the network.

[0110] For example, first obtain a first training set and a second training set.

[0111] The first training set includes a plurality of first training pictures and a plurality of first standard pictures corresponding to the plurality of first training pictures one by one, and each first training picture is obtained by changing the composition structure of the first standard picture corresponding to the first training picture.

[0112] The second training set includes a plurality of second training pictures and a plurality of second standard pictures corresponding to the plurality of second training pictures one by one, and each second training picture is obtained by adjusting shooting parameters of a second standard picture corresponding to the second training picture.

[0113] For example, when obtaining the first training set, a large number of photos approved by professionals can be obtained in advance as standard pictures, and the subject in the standard picture can be marked, and the subject can be used as a reference object for guiding the composition; and the standard picture can be randomly cropped in various directions (such as up, down, left, right, etc.) to change its composition, and the picture with the changed composition is used as the first training picture. In addition, the cropping parameters need to be recorded as a supervision target during the training process.

[0114] Insufficient training data will affect the robustness of the recognition network. In order to simulate as many image states as possible on limited data to ensure the generalization of the model, the image processing method provided in at least one embodiment of the present disclosure also includes randomly cropping the standard image in multiple directions. A large number of first training images can be obtained from one standard image. The first standard images corresponding to these first training images are all the standard images to adapt to complex and changeable real scenes.

[0115] For example, when obtaining the first training set, a large number of photos that are not recognized by professionals can also be obtained as training pictures, and the subjects in the training pictures can be marked. The subjects can be used as reference objects for guiding composition; and the training pictures can be cropped in various directions (such as up, down, left, right, etc.) to optimize the composition, and the pictures with changed composition are used as the first standard pictures.

[0116] For example, when obtaining the second training set, a large number of photos approved by professionals can also be obtained in advance as standard pictures, and random shooting parameters of the standard pictures can be adjusted. For example, the shooting parameters may include contrast, brightness, hue, saturation, sharpness, hue, etc.

[0117] Since the collected images that are not recognized by professionals cannot cover all different situations such as brightness conditions and image quality, and the real scenes are complex and changeable, in order to simulate as many image states as possible on limited data to ensure the generalization of the model, the image processing method provided in at least one embodiment of the present disclosure also includes performing a variety of random shooting parameter adjustments on the standard pictures to adapt to the complex and changeable real scenes.

[0118] For example, brightness adjustment can adopt the HSV (Hue, Saturation, Value) space brightness adjustment method. For example, the second standard picture is an RGB image. First, a set of RGB pixel values ​​of each pixel in the standard picture is converted to the HSV color space, that is, the pixel value of each pixel includes a set of hue value, saturation value and brightness value; then, a value is randomly selected from a preset range as an adjustment coefficient, and the brightness value of each pixel in the HSV color space is multiplied by the adjustment coefficient to adjust the brightness of each pixel; finally, the pixel value (hue value, saturation value and brightness value) of each pixel after adjustment is converted back to the RGB color space, thereby obtaining a plurality of second training pictures of different brightness, and the plurality of second training pictures correspond to the second standard picture.

[0119] The adjustment methods of other shooting parameters can be selected according to actual needs and will not be elaborated here.

[0120] In this way, a variety of training images with different brightness and contrast can be obtained, which greatly enriches the network's training database.

[0121] For example, when obtaining the second training set, a large number of photos that are not recognized by professionals can also be obtained as training pictures. Professionals can optimize the shooting parameters of the pictures after composition adjustment, including brightness, saturation, hue, hue, etc., to obtain the second standard picture corresponding to the training picture.

[0122] After the above processing, each standard picture recognized by professionals can generate multiple first training pictures with changed composition and multiple second training pictures with degraded shooting parameters. The standard picture can be used as the first standard picture corresponding to these first training pictures, and the second standard picture corresponding to these second training pictures; in addition, each picture taken by professionals that is not recognized by professionals can generate a first standard picture with optimized composition and a second standard picture with optimized composition and overall shooting parameters. The pictures that are not recognized by professionals can be used as the first training pictures and the second training pictures. In this way, a large number of training sets can be obtained.

[0123] For the first training images in the first training set, they all have a marked guide reference object and the moving distance and moving direction of the guide reference object, which are used as training supervision targets. For the second training images in the second training set, they all have a corresponding second standard image with optimized shooting parameters as training supervision targets.

[0124] Therefore, a training set is constructed using pictures taken, optimized and annotated by professionals, as well as pictures that are not recognized by professionals, so that the network model can learn the features and shooting experience recognized by professionals in a supervised manner, and help improve the shooting level of ordinary operators.

[0125] For example, the image processing method provided by at least one embodiment of the present disclosure also includes: using the adjustment model as a generator, using the shared neural network as a discriminator, forming a generative adversarial network by the adjustment model and the shared neural network, and performing alternating iterative training on the generative adversarial network to obtain a trained shared neural network and adjustment model.

[0126] Figure 3 A training flowchart of a network model provided for at least one embodiment of the present disclosure.

[0127] like Figure 3As shown, the adjustment model plays the role of the generator in the Generative Adversarial Networks (GAN), generating images to "fool" the discriminator, and the shared neural network plays the role of the discriminator in the GAN to judge the authenticity of the images generated by the adjustment model. For example, during the training process, the generator is first allowed to continuously generate image data for judgment by the discriminator. During this process, the parameters of the discriminator are not adjusted, and only the generator is trained and the parameters are adjusted until the discriminator cannot judge the authenticity of the images generated by the generator; then, the parameters of the generator are fixed, and the discriminator is continued to be trained until the discriminator can accurately judge the authenticity of the images generated by the generator; then, the above process is continuously repeated until the generation and discrimination capabilities of the generator and the discriminator become stronger and stronger, thereby obtaining a generator and discriminator with the best generation effect.

[0128] For example, in the above alternating iterative training process, the second training set is used to train the adjustment model, and the output of the adjustment model (i.e. Figure 3 The shared neural network is trained using the output picture in (the output picture in) and the first training set, so that the shared neural network can learn both the shooting parameters and the composition. Therefore, the trained shared neural network can evaluate whether the original image meets the preset shooting quality from the two perspectives of shooting parameters and composition, and can also output the shooting guide type.

[0129] For example, the shared neural network may be a convolutional neural network including multiple convolutional layers. The present disclosure does not impose any specific restrictions on the model structure of the shared neural network.

[0130] The input of the shared neural network can be the first training image and the output image of the adjustment model. The output of the shared neural network includes two output results, one output result is a classification result, and the other output result is a first result value corresponding to the first guidance type and a second result value corresponding to the second guidance type. For example, the loss function corresponding to the shared neural network also includes two, one is a binary cross entropy loss function for judging the authenticity of the image (that is, the classification result), and the other is a binary cross entropy loss function (BinaryCrossEntropy Loss, referred to as BCE) for judging whether there are composition problems and shooting parameter problems. For example, the BCE loss function is a multi-label classification task. Two labels are set in the BCE loss function. The first label indicates whether the training image has a composition problem, and the second label indicates whether the training image has a shooting parameter problem. The specific calculation method of the binary cross entropy loss function and the BCE loss function will not be repeated here.

[0131] For example, in the process of training the shared neural network, the loss value is calculated using a binary cross entropy loss function for judging the authenticity of the picture and a binary cross entropy loss function for judging whether there are composition problems and shooting parameter problems. The two loss values ​​are used together to adjust the weights of the shared neural network so that the trained shared neural network can directly output the judgment result of whether the original image meets the preset shooting quality, as well as the shooting guidance type.

[0132] For example, the adjustment model may adopt a Unet network structure, including a decoder and an encoder, the model input is the second training image, and the model output is an output image of the same size, but the value of each pixel in the output image has been transformed. The present disclosure does not specifically limit the model structure of the adjustment model.

[0133] For example, the loss function corresponding to the adjustment model can be composed of the pixel-level reconstruction loss value ( Figure 3 not shown), feature-based perceptual loss value ( Figure 3 not shown) and adversarial loss value.

[0134] For example, the pixel-level reconstruction loss value can be calculated using the Euclidean distance (also known as the L2 loss function) between the second training image and the corresponding second standard image, or the peak signal-to-noise ratio between the second training image and the corresponding second standard image can be calculated as the reconstruction loss value.

[0135] For example, the feature-based perceptual loss value can extract features through the convolutional layer in the pre-trained VGG16 network, and then calculate the L2 distance (Euclidean distance) between the output image after feature extraction and the standard image in the feature space.

[0136] For example, the adversarial loss is based on Figure 3 The binary cross entropy loss function shown is calculated.

[0137] In this embodiment, a generative adversarial network framework is used to utilize the discriminator in the generative adversarial network (i.e., a shared neural network) to complete the preset shooting quality evaluation and shooting guidance type discrimination functions, and the generator in the generative adversarial network (i.e., an adjustment model) is used to complete the automatic adjustment function of shooting parameters, thereby integrating the two main auxiliary shooting capabilities of auxiliary composition and automatic adjustment of shooting parameters.

[0138] Of course, if the first judgment model is used to evaluate the preset shooting quality and the second judgment model is used to distinguish the shooting guidance type, the first judgment model, the second judgment model and the adjustment model can also be trained separately, which will not be repeated here.

[0139] For example, Figure 3As shown, the composition prediction model can be trained using the first training set to obtain a trained composition prediction model. The trained composition prediction model can directly output the position of the guide reference object and the target movement trajectory.

[0140] For example, the composition prediction model can adopt the model framework of the target detection model, such as Faster R-CNN, SSD and YOLO, etc. The present disclosure does not impose any specific restrictions on the model structure of the composition prediction model.

[0141] For example, Figure 3 As shown, training the composition prediction model using the first training set to obtain a trained composition prediction model can include: inputting the first training picture into the composition prediction model to obtain the position of the output guide object and the output movement trajectory corresponding to the output guide object, where the output movement trajectory can include the moving distance and moving direction of the position of the output guide object in the output picture relative to the position of the photographed subject marked in the first training picture; calculating a first loss value corresponding to the position of the output guide object and a second loss value corresponding to the output movement trajectory; adjusting the weight of the composition prediction model according to the first loss value and the second loss value; when the loss value corresponding to the composition prediction model to be trained does not meet the predetermined accuracy condition, continuing to input the first training picture to repeat the above training process; when the loss value corresponding to the composition prediction model to be trained meets the predetermined accuracy condition, obtaining a trained composition prediction model.

[0142] During the process of training the composition prediction model, the first loss value corresponding to the output position of the output guidance object and the second loss value corresponding to the output movement trajectory (movement offset) of the output guidance object are calculated simultaneously. As a result, the trained composition prediction model can directly learn the composition skills of professionals. Combined with the principle of target detection algorithm, end-to-end and flexible composition guidance capabilities are achieved. Different from the existing composition guidance solutions based on reference templates, when using the composition prediction model, the position of the guidance reference object and the target movement trajectory can be directly output without being restricted by the scene and fixed templates, providing more flexible and rich auxiliary shooting capabilities.

[0143] At least one embodiment of the present disclosure further provides an image processing device, Figure 4 A schematic block diagram of an image processing device provided for at least one embodiment of the present disclosure.

[0144] like Figure 4 As shown, the image processing device 100 may include an image acquisition unit 101, a shooting quality judgment unit 102, and an image processing unit 103. These components are interconnected via a bus system and / or other forms of connection mechanisms (not shown). Figure 4The components and structures of the image processing apparatus 100 shown are merely exemplary and not restrictive. The image processing apparatus 100 may also have other components and structures as required.

[0145] For example, these modules can be implemented by hardware (e.g., circuit) modules, software modules, or any combination of the two. The following embodiments are the same and will not be described in detail. For example, these units can be implemented by a central processing unit (CPU), a graphics processing unit (GPU), a tensor processing unit (TPU), a field programmable gate array (FPGA), or other forms of processing units with data processing capabilities and / or instruction execution capabilities and corresponding computer instructions.

[0146] For example, the image acquisition unit 101 is configured to capture the original image during the framing process. For example, the image acquisition unit 101 may include a framing device such as a lens, a camera, etc. For example, the image acquisition unit 101 may be hardware, software, firmware, or any feasible combination thereof.

[0147] For example, the shooting quality determination unit 102 is configured to determine whether the original image meets a preset shooting quality.

[0148] For example, the image processing unit 103 is configured to: in response to the original image not satisfying a preset shooting quality: provide a shooting guidance type corresponding to the original image, wherein the shooting guidance type includes at least one of a first guidance type and a second guidance type, the first guidance type is used to indicate a composition guidance relative to the original image during the framing process, and the second guidance type is used to indicate a shooting parameter adjustment relative to the original image during the framing process; in response to the first guidance type in the shooting guidance type, re-frame based on the guidance interaction given by the first guidance type to obtain a first intermediate framing image; in response to the second guidance type in the shooting guidance type, display a second intermediate framing image after the shooting parameters are adjusted; and obtain a shooting result image from the first intermediate framing image or the second intermediate framing image.

[0149] For example, when the shooting quality judgment unit 102 executes the judgment on whether the original image meets the preset shooting quality, it includes performing the following operations: inputting the original image into the first judgment model, processing the original image using the first judgment model, and obtaining a score corresponding to the original image; in response to the score being less than the first threshold, determining that the original image does not meet the preset shooting quality; in response to the score being greater than the first threshold, determining that the original image meets the preset shooting quality.

[0150] For example, when the shooting quality judgment unit 102 executes the judgment on whether the original image meets the preset shooting quality, it includes performing the following operations: using a neural network to extract features of the original image, and outputting the category of the original image, wherein the category includes a first category or a second category, the first category indicates that the original image meets the preset shooting quality, and the second category indicates that the original image does not meet the preset shooting quality.

[0151] For example, when the image processing unit 103 provides a shooting guidance type corresponding to the original image, it includes performing the following operations: inputting the original image into the second judgment model, processing the original image using the second judgment model, and obtaining a first result value corresponding to the first guidance type and a second result value corresponding to the second guidance type; in response to the first result value being greater than the second threshold, determining that the shooting guidance type corresponding to the original image includes the first guidance type; in response to the second result value being greater than the third threshold, determining that the shooting guidance type corresponding to the original image includes the second guidance type.

[0152] For example, when the image processing unit 103 performs reframing based on the guidance interaction given by the first guidance type to obtain a first intermediate framed image, it includes performing the following operations: in response to the first guidance type in the shooting guidance type, displaying the guidance interaction interface of the first guidance type, and performing guidance interaction reframing to obtain the first intermediate framed image.

[0153] When the image processing unit 103 executes in response to the first guidance type in the shooting guidance type, displays the guidance interaction interface of the first guidance type, and performs the guidance interaction reframing to obtain the first intermediate framed image, the following operations are included: displaying the position of the guidance reference object in the original image and the target movement trajectory of the guidance reference object in the guidance interaction interface, prompting the operator to move the guidance reference object according to the target movement trajectory, and guiding the operator to reframe to obtain the first intermediate framed image, wherein the guidance reference object includes the target point or target frame in the original image.

[0154] For example, the position of the guide reference object and the target movement trajectory in the original image are obtained through the composition prediction model.

[0155] The structures, functions, and training methods of the first judgment model, the second judgment model, the neural network, the composition prediction network, and other networks in the above content are the same as the corresponding network structures and functions in the above-mentioned image processing method embodiments, and will not be repeated here.

[0156] It should be noted that the image acquisition unit 101 can be used to implement Figure 1 In step S10 shown in FIG. 1 , the shooting quality judgment unit 102 may be used to implement Figure 1 In step S20 shown in FIG. 1 , the image processing unit 103 may be used to implement Figure 1Therefore, for the specific description of the functions that can be implemented by the image acquisition unit 101, the shooting quality judgment unit 102 and the image processing unit 103, reference can be made to the relevant description of steps S10 to S60 in the above-mentioned embodiment of the image processing method, and the repeated parts will not be repeated. In addition, the image processing device 100 can achieve technical effects similar to those of the above-mentioned image processing method, which will not be repeated here.

[0157] Figure 5 The following is a schematic block diagram of an electronic device provided by an embodiment of the present disclosure. Figure 5 As shown, the electronic device 200 is suitable for implementing the image processing method provided by the embodiment of the present disclosure. It should be noted that Figure 5 The components of the electronic device 200 shown are merely exemplary and non-limiting. The electronic device 200 may also have other components according to actual application requirements.

[0158] like Figure 5 As shown, the electronic device 200 may include a processing device (eg, a central processing unit, a graphics processing unit, etc.) 201, which may perform various appropriate actions and processes according to non-transitory computer-readable instructions stored in a memory to implement various functions.

[0159] For example, when the computer readable instructions are executed by the processing device 201, one or more steps in the image processing method described in any of the above embodiments may be executed. It should be noted that for a detailed description of the processing process of the image processing method, reference may be made to the relevant description in the above-mentioned embodiment of the image processing method, and the repeated parts will not be repeated.

[0160] For example, the memory may include any combination of one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) 203 and / or cache memory (cache), etc. For example, computer-readable instructions may be loaded from storage device 208 into random access memory (RAM) 203 to run computer-readable instructions. Non-volatile memory may include, for example, read-only memory (ROM) 202, hard disk, erasable programmable read-only memory (EPROM), portable compact disk read-only memory (CD-ROM), USB memory, flash memory, etc. Various applications and various data, such as style images, and various data used and / or generated by applications, etc., may also be stored in the computer-readable storage medium.

[0161] For example, the processing device 201, the ROM 202, and the RAM 203 are connected to each other via a bus 204. To the bus 204, an input / output (I / O) interface 205 is also connected.

[0162] Typically, the following devices may be connected to the I / O interface 205: an input device 206 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 207 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 208 including, for example, a magnetic tape, a hard disk, a flash memory, etc.; and a communication device 209. The communication device 209 may allow the electronic device 200 to communicate with other electronic devices wirelessly or by wire to exchange data. Although Figure 5 An electronic device 200 with various devices is shown, but it should be understood that it is not required to implement or have all the devices shown, and the electronic device 200 can be implemented or have more or fewer devices instead. For example, the processor 201 can control other components in the electronic device 200 to perform the desired functions. The processor 201 can be a device with data processing capabilities and / or program execution capabilities such as a central processing unit (CPU), a tensor processing unit (TPU) or a graphics processing unit GPU. The central processing unit (CPU) can be an X86 or ARM architecture, etc. The GPU can be directly integrated into the motherboard separately, or built into the north bridge chip of the motherboard. The GPU can also be built into the central processing unit (CPU).

[0163] Figure 6 A schematic diagram of a non-transitory computer-readable storage medium provided for at least one embodiment of the present disclosure. Figure 6 As shown, the storage medium 300 may be a non-transitory computer-readable storage medium, and one or more computer-readable instructions 301 may be non-transitory stored on the storage medium 300. For example, when the computer-readable instructions 301 are executed by a processor, one or more steps in the image processing method described above may be executed.

[0164] For example, the storage medium 300 may be applied to the above-mentioned electronic device. For example, the storage medium 300 may include a memory in the electronic device.

[0165] For example, the storage medium may include a memory card of a smart phone, a storage component of a tablet computer, a hard disk of a personal computer, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a portable compact disk read-only memory (CD-ROM), flash memory, or any combination of the above storage media, or other applicable storage media.

[0166] For example, the description of the storage medium 300 may refer to the description of the memory in the embodiment of the electronic device, and the repeated parts will not be repeated.

[0167] Figure 7 A schematic diagram of a hardware environment provided by at least one embodiment of the present disclosure. The electronic device provided by the present disclosure can be applied in an Internet system.

[0168] use Figure 7 The computer system provided in can implement the functions of the image processing device and / or electronic device involved in the present disclosure. Such computer systems may include personal computers, laptops, tablet computers, mobile phones, personal digital assistants, smart glasses, smart watches, smart rings, smart helmets and any smart portable devices or wearable devices. The specific system in this embodiment uses a functional block diagram to explain a hardware platform including a user interface. This computer device can be a general-purpose computer device or a computer device with a specific purpose. Both computer devices can be used to implement the image processing device and / or electronic device in this embodiment. The computer system may include any components that implement the information required to implement the image processing currently described. For example, the computer system can be implemented by a computer device through its hardware devices, software programs, firmware and their combinations. For convenience, Figure 7 Only one computer device is depicted in the figure, but the relevant computer functions of the information required for realizing image processing described in this embodiment can be implemented in a distributed manner by a group of similar platforms to disperse the processing load of the computer system.

[0169] like Figure 7 As shown, the computer system may include a communication port 250, connected to which is a network for realizing data communication. For example, the computer system can send and receive information and data through the communication port 250, that is, the communication port 250 can realize wireless or wired communication between the computer system and other electronic devices to exchange data. The computer system may also include a processor group 220 (i.e., the processor described above) for executing program instructions. The processor group 220 may be composed of at least one processor (e.g., a CPU). The computer system may include an internal communication bus 210. The computer system may include different forms of program storage units and data storage units (i.e., the memory or storage medium described above), such as a hard disk 270, a read-only memory (ROM) 230, and a random access memory (RAM) 240, which can be used to store various data files used for computer processing and / or communication, and possible program instructions executed by the processor group 220. The computer system may also include an input / output component 260, which is used to realize input / output data flow between the computer system and other components (e.g., a user interface 280, etc.).

[0170] Typically, the following devices can be connected to the input / output component 260: input devices such as a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices such as a display (e.g., an LCD, an OLED display, etc.), a speaker, a vibrator, etc.; storage devices including, for example, a magnetic tape, a hard disk, etc.; and a communication interface.

[0171] Although Figure 7 A computer system having various devices is shown, but it should be understood that the computer system is not required to have all of the devices shown, and the computer system may alternatively have more or fewer devices.

[0172] There are a few points to note about this disclosure:

[0173] (1) The drawings of the embodiments of the present disclosure only relate to the structures related to the embodiments of the present disclosure, and other structures may refer to the general design.

[0174] (2) For the sake of clarity, the thickness and size of layers or structures are exaggerated in the drawings used to describe the embodiments of the present invention. It is understood that when an element such as a layer, film, region, or substrate is referred to as being "on" or "under" another element, the element may be "directly" "on" or "under" the other element, or there may be intervening elements.

[0175] (3) In the absence of conflict, the embodiments of the present disclosure and the features therein may be combined with each other to obtain new embodiments.

[0176] The above description is only a specific implementation of the present disclosure, but the protection scope of the present disclosure is not limited thereto. The protection scope of the present disclosure shall be based on the protection scope of the claims.

Claims

1. An image processing method, comprising: Acquire the original image obtained during the framing process; Determining whether the original image meets a preset shooting quality; In response to the original image not meeting a preset shooting quality: providing a shooting guidance type corresponding to the original image, wherein the shooting guidance type includes at least one of a first guidance type and a second guidance type, the first guidance type is used to indicate a composition guidance relative to the original image during the framing process, and the second guidance type is used to indicate a shooting parameter adjustment relative to the original image during the framing process; In response to the first guidance type in the shooting guidance types, reframing is performed based on a guidance interaction given by the first guidance type to obtain a first intermediate framing image, wherein the guidance interaction includes displaying a guidance interaction interface, and displaying in the guidance interaction interface a position of a guidance reference object in the original image and a target moving trajectory of the guidance reference object for prompting an operator to refer to; In response to the second guidance type among the shooting guidance types, displaying a second intermediate framing image after the shooting parameters are adjusted; A shooting result image is obtained from the first intermediate framing image or the second intermediate framing image.

2. The image processing method according to claim 1, wherein: Determining whether the original image meets the preset shooting quality includes: determining whether the original image meets the preset shooting quality through scoring.

3. The image processing method according to claim 2, wherein: Judging whether the original image meets the preset shooting quality through scoring includes: Performing image processing on the original image to obtain a score corresponding to the original image; In response to the score being less than a first threshold, determining that the original image does not meet a preset shooting quality; In response to the score being greater than the first threshold, it is determined that the original image meets a preset shooting quality.

4. The image processing method according to claim 3, wherein: Performing image processing on the original image to obtain a score corresponding to the original image includes: The original image is input into a first judgment model, and the original image is processed using the first judgment model to obtain a score corresponding to the original image.

5. The image processing method according to claim 1, wherein: Determining whether the original image meets a preset shooting quality includes: Using a neural network to extract features from the original image, and outputting the category of the original image, The category includes a first category or a second category, the first category indicates that the original image meets a preset shooting quality, and the second category indicates that the original image does not meet the preset shooting quality.

6. The image processing method according to claim 1, wherein: Providing a shooting guidance type corresponding to the original image, including: Inputting the original image into a second judgment model, and processing the original image using the second judgment model to obtain a first result value corresponding to the first guidance type and a second result value corresponding to the second guidance type; In response to the first result value being greater than a second threshold, determining that the shooting guidance type corresponding to the original image includes the first guidance type; In response to the second result value being greater than a third threshold, it is determined that the shooting guide type corresponding to the original image includes the second guide type.

7. The image processing method according to claim 6, wherein: Obtaining a judgment result of whether the original image meets the preset shooting quality through a neural network, The second judgment model and the neural network share a network, The shared network is configured to simultaneously output a determination result of whether the original image meets a preset shooting quality, as well as the first result value and the second result value.

8. The image processing method according to claim 1, wherein: In response to the first guidance type among the shooting guidance types, reframing is performed based on the guidance interaction given by the first guidance type to obtain a first intermediate framing image, including: In response to the first guidance type among the shooting guidance types, a guidance interaction interface of the first guidance type is displayed, and a guidance interaction re-framing is performed to obtain the first intermediate framing image.

9. The image processing method according to claim 8, wherein: In response to the first guidance type in the shooting guidance types, displaying a guidance interaction interface of the first guidance type, and performing guidance interaction to re-frame to obtain a first intermediate framing image, including: The position of the guidance reference object in the original image and the target moving trajectory of the guidance reference object are displayed in the guidance interaction interface, the operator is prompted to move the guidance reference object according to the target moving trajectory, and the operator is guided to re-frame to obtain the first intermediate framed image, wherein the guidance reference object includes the target point or target frame in the original image.

10. The image processing method according to claim 8, wherein: Displaying the position of the guidance reference object in the original image and the target moving track of the guidance reference object in the guidance interaction interface, prompting the operator to move the guidance reference object according to the target moving track, and guiding the operator to re-frame to obtain the first intermediate framing image, including: During the movement of the guiding reference object, in response to detecting that the image obtained by framing is in a stable focus state, acquiring a framing image currently in a stable focus state; Based on the processing of the framing image, obtaining a judgment result of whether the framing image meets a preset shooting quality; In response to the framing image meeting a preset shooting quality requirement, prompting the operator to shoot to obtain the first intermediate framing image; In response to the framing image not meeting the preset shooting quality requirement, the target movement trajectory of the guiding reference object continues to be displayed, guiding the operator to re-frame until the acquired framing image meets the preset shooting quality, prompting the operator to shoot to obtain the first intermediate framing image.

11. The image processing method according to claim 8, wherein: The position of the guide reference object and the target movement trajectory in the original image are obtained by a composition prediction model; The composition prediction model is configured to receive the original image and perform target detection on the original image to obtain the position, target movement distance and target movement direction of the guiding reference object, wherein the target movement trajectory of the guiding reference object includes the target movement distance and the target movement direction.

12. The image processing method according to claim 1, before providing the shooting guide type corresponding to the original image, the image processing method further comprises: Displaying a guide mode prompt interface to prompt the operator to select whether to enter the auxiliary shooting process; In response to selecting to enter the auxiliary shooting process, subsequent processing is performed.

13. The image processing method according to claim 1, wherein: In response to the second guidance type in the shooting guidance types, displaying a second intermediate framing image after the shooting parameters are adjusted includes: In response to the shooting guidance type including the first guidance type, adjusting shooting parameters of the first intermediate framing image by using an adjustment model to obtain the second intermediate framing image; In response to the shooting guidance type not including the first guidance type, adjusting shooting parameters of the original image by using the adjustment model to obtain the second intermediate framing image; The shooting parameter adjustment includes adjusting one or more of the brightness, hue, saturation, sharpness and hue of the image.

14. The image processing method according to claim 1, wherein: Obtaining a shooting result image from the first intermediate framing image or the second intermediate framing image includes: In response to the shooting guide type including the first guide type but not including the second guide type, using the first intermediate framing image as the shooting result image; In response to the shooting guide type including the second guide type, the second intermediate view image is used as the shooting result image.

15. The image processing method according to any one of claims 1 to 14, wherein: The shooting guidance type corresponding to the original image and the judgment result of whether the original image meets the preset shooting quality are determined by a common neural network, In response to the second guidance type, the shooting parameter adjustment is completed by adjusting the model, The image processing method further comprises: The adjustment model is used as a generator, the common neural network is used as a discriminator, and the adjustment model and the common neural network constitute a generative adversarial network, The generative adversarial network is trained alternately and iteratively to obtain the trained common neural network and the adjustment model.

16. The image processing method according to claim 15, wherein: The generative adversarial network is subjected to alternating iterative training to obtain the trained neural network and the adjustment model, including: Acquire a first training set and a second training set, wherein the first training set includes a plurality of first training pictures and a plurality of first standard pictures corresponding to the plurality of first training pictures one-to-one, each first training picture is obtained by changing the composition structure of a first standard picture corresponding to the first training picture, and the second training set includes a plurality of second training pictures and a plurality of second standard pictures corresponding to the plurality of second training pictures one-to-one, each second training picture is obtained by adjusting shooting parameters of a second standard picture corresponding to the second training picture; In the alternating iterative training process, the adjustment model is trained using the second training set, and the common neural network is trained using the output of the adjustment model and the first training set.

17. The image processing method according to claim 16, wherein: The position of the guidance reference object in the original image displayed in the guidance interaction interface and the target movement trajectory of the guidance reference object are obtained through a composition prediction model, The image processing method further comprises: The composition prediction model is trained using the first training set.

18. An image processing device, comprising: An image acquisition unit, configured to acquire an original image obtained during the framing process; A shooting quality judgment unit, configured to judge whether the original image meets a preset shooting quality; Image processing unit, configured as: In response to the original image not meeting a preset shooting quality: providing a shooting guidance type corresponding to the original image, wherein the shooting guidance type includes at least one of a first guidance type and a second guidance type, the first guidance type is used to indicate a composition guidance relative to the original image during the framing process, and the second guidance type is used to indicate a shooting parameter adjustment relative to the original image during the framing process; In response to the first guidance type in the shooting guidance types, reframing is performed based on a guidance interaction given by the first guidance type to obtain a first intermediate framing image, wherein the guidance interaction includes displaying a guidance interaction interface, and displaying in the guidance interaction interface a position of a guidance reference object in the original image and a target moving trajectory of the guidance reference object for prompting an operator to refer to; In response to the second guidance type among the shooting guidance types, displaying a second intermediate framing image after the shooting parameters are adjusted; A shooting result image is obtained from the first intermediate framing image or the second intermediate framing image.

19. An electronic device comprising: A memory non-transitorily stores computer executable instructions; a processor configured to execute the computer executable instructions, Wherein, when the computer executable instructions are executed by the processor, the image processing method according to any one of claims 1-17 is implemented.

20. A non-transitory computer-readable storage medium, wherein: The non-transitory computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, implement the image processing method according to any one of claims 1 to 17.

Images