Image processing method and device, electronic equipment and medium

Abstract

The invention discloses an image processing method and device, electronic equipment and a medium, and belongs to the field of image processing. The image processing method comprises the following steps: acquiring a real-time image and multi-modal data of a shooting scene corresponding to the real-time image; sending a composition optimization request to a server, wherein the composition optimization request carries the real-time image and the multi-modal data; and receiving a composition optimization suggestion returned by the server, and outputting the composition optimization suggestion, the composition optimization suggestion being obtained based on the composition characteristics extracted by the composition model from the real-time image and the composition characteristics of the multi-modal data.

Description

Technical Field

[0001] The present application belongs to the field of image processing, and specifically relates to image processing methods, devices, electronic devices and media. Background Art

[0002] With the widespread popularity of electronic devices with image sensors, taking photos and videos has become a part of people's daily lives. However, during the image capture process, it is a challenge for users to ensure the quality of image composition. At present, the quality of image composition mostly depends on post-processing after shooting. Using image editing software to adjust the image is not only time-consuming, but also requires users to have certain professional knowledge. For users who lack professional knowledge, it is often difficult to achieve an ideal composition effect, which affects the overall quality of the image.

[0003] Therefore, how to help users optimize composition efficiently and with high quality has become a problem that needs to be solved urgently. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide an image processing method, device, electronic device and medium, which can solve the problem of how to help users optimize the composition efficiently and with high quality.

[0005] In a first aspect, an embodiment of the present application provides an image processing method, which is applied to an electronic device, and the method includes:

[0006] Acquire a real-time image captured by the electronic device and multimodal data of a shooting scene corresponding to the real-time image;

[0007] Sending a composition optimization request to a server, wherein the composition optimization request carries the real-time image and the multimodal data;

[0008] receiving the composition optimization suggestion returned by the server, and outputting the composition optimization suggestion;

[0009] The composition optimization suggestion is obtained based on the composition features by extracting the composition features of the real-time image and the multimodal data using the composition model.

[0010] In a second aspect, an embodiment of the present application provides an image processing device, including:

[0011] A real-time data acquisition module, configured to acquire a real-time image captured by the electronic device and multimodal data of a shooting scene corresponding to the real-time image;

[0012] A composition suggestion generating module is configured to send a composition optimization request to a server, wherein the composition optimization request carries the real-time image and the multimodal data;

[0013] The composition suggestion output module is configured to receive the composition optimization suggestion returned by the server and output the composition optimization suggestion.

[0014] In a third aspect, an embodiment of the present application provides an image processing method, which is applied to a server, and the method includes:

[0015] Receiving a composition optimization request sent by an electronic device, the composition optimization request carrying a real-time image and multimodal data of a shooting scene corresponding to the real-time image;

[0016] Extracting composition features of the real-time image and the multimodal data using a composition model, and obtaining composition optimization suggestions based on the composition features;

[0017] The composition optimization suggestion is returned to the electronic device.

[0018] In a fourth aspect, an embodiment of the present application provides an image processing device, the device comprising:

[0019] A composition optimization request receiving module, configured to receive a composition optimization request sent by an electronic device, wherein the composition optimization request carries a real-time image and multimodal data of a shooting scene corresponding to the real-time image;

[0020] A composition model processing module, configured to extract composition features of the real-time image and the multimodal data using a composition model, and obtain composition optimization suggestions based on the composition features;

[0021] The composition optimization suggestion returning module is configured to return the composition optimization suggestion to the electronic device.

[0022] In a fifth aspect, an embodiment of the present application provides a model training method, which is applied to a server, and the method includes:

[0023] Acquire a training data set, the training data set comprising: image samples and multimodal data samples of shooting scenes corresponding to the image samples;

[0024] The initial composition model is trained using the training data set to obtain a trained composition model, and the composition model is used to predict composition optimization suggestions.

[0025] In a sixth aspect, an embodiment of the present application provides a model training device, the device comprising:

[0026] A training data set acquisition module is configured to acquire a training data set, wherein the training data set includes: image samples and multimodal data samples of shooting scenes corresponding to the image samples;

[0027] The composition model training module is configured to use the training data set to train the initial composition model to obtain a trained composition model, and the composition model is used to predict composition optimization suggestions.

[0028] In a seventh aspect, an embodiment of the present application provides an image processing method, applied to an electronic device, the method comprising:

[0029] Acquire a real-time image and multimodal data of a shooting scene corresponding to the real-time image;

[0030] Extracting composition features of the real-time image and the multimodal data using a composition model, and obtaining composition optimization suggestions based on the composition features;

[0031] The composition optimization suggestion is output.

[0032] In an eighth aspect, an embodiment of the present application provides an image processing device, including:

[0033] A real-time data acquisition module, configured to acquire a real-time image acquired by the electronic device and multimodal data of a shooting scene corresponding to the real-time image;

[0034] A local model execution module is configured to extract composition features of the real-time image and the multimodal data using a composition model, and obtain composition optimization suggestions based on the composition features;

[0035] The composition suggestion output module is configured to output the composition optimization suggestion.

[0036] In the ninth aspect, an embodiment of the present application provides an electronic device, which includes a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the program or instructions are executed by the processor, the steps of the method described in the first aspect, the third aspect, the fifth aspect or the seventh aspect are implemented.

[0037] In the tenth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the method steps described in the first aspect, the third aspect, the fifth aspect or the seventh aspect are implemented.

[0038] In the eleventh aspect, an embodiment of the present application provides a chip, which includes a processor and a communication interface, the communication interface and the processor are coupled, and the processor is used to run programs or instructions to implement the method described in the first aspect, the third aspect, the fifth aspect or the seventh aspect.

[0039] In the twelfth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the method described in the first aspect, the third aspect, the fifth aspect or the seventh aspect.

[0040] In an embodiment of the present application, a real-time image collected by an electronic device and multimodal data of a shooting scene corresponding to the real-time image are obtained, and a composition optimization request carrying the real-time image and multimodal data is sent to a server. In this way, the server can make full use of the comprehensive context information provided by the real-time image and multimodal data, use the composition model, accurately extract the composition features of the real-time image and multimodal data, and generate composition optimization suggestions based on the composition features. In this way, the electronic device receives the composition optimization suggestions returned by the server and outputs the composition optimization suggestions, which can help users optimize the composition efficiently and with high quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 It is a schematic diagram of an application scenario of an image processing method provided in an embodiment of the present application;

[0042] Figure 2 is a flowchart of an image processing method according to an exemplary embodiment of the present application;

[0043] Figure 3 is a schematic diagram of a neural network structure according to an exemplary embodiment of the present application;

[0044] Figure 4 is a flowchart of an image processing method according to another exemplary embodiment of the present application;

[0045] Figure 5 is a structural block diagram of an image processing device according to an exemplary embodiment of the present application;

[0046] Figure 6 is a flowchart of an image processing method according to another exemplary embodiment of the present application;

[0047] Figure 7 is a flow chart of a model training method according to an exemplary embodiment of the present application;

[0048] Figure 8 is a structural block diagram of an image processing device according to another exemplary embodiment of the present application;

[0049] Fig. 9 is a structural block diagram of a model training device according to an exemplary embodiment of the present application;

[0050] Fig.10is a flowchart of an image processing method according to another exemplary embodiment of the present application;

[0051] Fig.11 is a structural block diagram of an image processing device according to another exemplary embodiment of the present application;

[0052] Fig.12 is a structural block diagram of an electronic device provided in an embodiment of the present application;

[0053] Fig.13 It is a schematic diagram of the hardware structure of an electronic device implementing an embodiment of the present application. DETAILED DESCRIPTION

[0054] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0055] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0056] First, the terms involved in the embodiments of the present application are explained.

[0057] Composition refers to the way of organizing composition elements in the picture in photography, so as to convey information and visual effects through the picture.

[0058] Multimodal data refers to data that contains multiple different types of information, which usually come from different sources or sensors. Each type of information is called a "modality".

[0059] Neural network is an algorithmic mathematical model that imitates the behavioral characteristics of animal neural networks and is composed of artificial neural nodes for distributed parallel information processing. Neural networks process information by adjusting the interconnected relationships between a large number of internal nodes.

[0060] A deep convolutional neural network is a neural network that effectively extracts the features of input data through multi-layer convolution, pooling, activation and other operations.

[0061] The self-supervised learning large model is a large model based on deep convolutional neural networks that automatically creates supervision signals from unlabeled data and learns powerful feature representations on large-scale data sets.

[0062] As the background technology, the composition quality of the image in the prior art mostly depends on the post-processing after the shooting is completed. Using image editing software to adjust the image is not only time-consuming, but also requires the user to have certain professional knowledge. How to help users optimize the composition efficiently and with high quality has become a problem to be solved urgently. In view of this, the embodiment of the present application provides an image processing method, a device, an electronic device and a medium, by acquiring a real-time image and multimodal data of the shooting scene corresponding to the real-time image, and sending a composition optimization request carrying the real-time image and the multimodal data to the server, so that the server can make full use of the comprehensive context information provided by the real-time image and the multimodal data, and use the composition model to accurately extract the composition features of the real-time image and the multimodal data, and generate a composition optimization suggestion based on the composition feature. In this way, the electronic device receives the composition optimization suggestion returned by the server and outputs the composition optimization suggestion, which can help users optimize the composition efficiently and with high quality.

[0063] The image processing method, device, electronic device and medium provided in the embodiments of the present application are described in detail below with reference to the accompanying drawings through specific embodiments and their application scenarios.

[0064] Figure 1 Schematic diagram of an application scenario of an image processing method provided in an embodiment of the present application. Figure 1 As shown, the application scenario may include: a server 10 and an electronic device 20. Among them:

[0065] The electronic device 20 is configured to obtain a real-time image and multimodal data of a shooting scene corresponding to the real-time image, and send a composition optimization request to a server.

[0066] In some embodiments, the electronic device 20 can provide the user with a triggering method for issuing a composition optimization request for the real-time image when displaying the currently captured real-time image on the user interface. For example, the user can trigger the composition optimization request by operating the controls displayed on the user interface or voice input. In other embodiments, the electronic device 20 can automatically identify the user's composition optimization intention based on the user's behavior in response to the electronic device capturing the real-time image, and automatically trigger the composition optimization request.

[0067] The server 10 is configured to extract composition features of the real-time image and the multimodal data using a composition model, generate composition optimization suggestions based on the composition features, and then return the composition optimization suggestions to the electronic device 20 .

[0068] The electronic device 20 is further configured to receive the composition optimization suggestion returned by the server 10 and output the composition optimization suggestion.

[0069] In some embodiments, the electronic device 20 may display composition optimization suggestions on the user interface, and / or output the composition optimization suggestions by voice. In other embodiments, the electronic device 20 may display auxiliary shooting suggestions generated based on the composition optimization suggestions on the user interface. In some further embodiments, the electronic device 20 may receive a first input from a user for composition optimization suggestions and / or auxiliary shooting suggestions, and the first input may include confirmation to execute as suggested, or adjustment suggestions, etc. Thus, based on the user's first input and the auxiliary shooting operation information generated based on the composition optimization suggestions, the auxiliary shooting operation and the auxiliary shooting suggestions are determined. Of course, the user may also manually optimize the composition according to the composition optimization suggestions and / or auxiliary shooting suggestions, and the embodiments of the present application are not limited to this.

[0070] It is worth noting that in the image processing method provided in the embodiment of the present application, the composition model can be deployed on the server, and the server uses the composition model to generate composition optimization suggestions. However, in some embodiments, the electronic device can also have similar functions as the server. For example, the composition model can be deployed on the electronic device, and the electronic device uses the composition model to generate composition optimization suggestions. The embodiment of the present application does not limit this.

[0071] The server 10 and the electronic device 20 are connected via a network. The network provides a medium for a communication link between the server 10 and the electronic device 20. The network may include various connection types, such as wired or wireless communication links or optical fiber cables, etc. The server 10 may include a server that provides various services, such as a server that provides composition optimization suggestion services for one or more electronic devices, a server for training a composition model that provides support for one or more electronic devices, and a server that processes data sent by an electronic device.

[0072] The electronic device 20 may include: a mobile device or a non-mobile device, such as any one or more of a smart phone, a tablet computer, a smart camera, a drone, a smart watch, a security camera, an industrial camera, a medical imaging device, a drone-mounted camera, etc. The electronic device is equipped with an image sensor such as a camera, which can collect real-time images and multimodal data and send them to a server via a wireless or wired network.

[0073] Exemplarily, the electronic device 20 can be expressed as a mobile device, and the composition model can be expressed as a self-supervised learning large model, so that based on the large model, real-time image composition optimization on the mobile terminal can be achieved, and the composition optimization suggestions output by the electronic device 20 can significantly improve the user's shooting experience.

[0074] according to Figure 1 The application scenario shown in the figure applies the scheme provided by the embodiment of the present application, by acquiring the real-time image collected by the electronic device and the multimodal data of the shooting scene corresponding to the real-time image, and sending a composition optimization request carrying the real-time image and the multimodal data to the server, so that the server can make full use of the comprehensive context information provided by the real-time image and the multimodal data, and use the composition model to accurately extract the composition features of the real-time image and the multimodal data, and obtain the composition optimization suggestions based on the composition features. Specifically, the composition model extracts the initial composition features of the real-time image and the multimodal data, and identifies the optimized composition mode of the current shooting scene based on the pre-learned composition principles, and applies the optimized composition mode to the weighted composition features by weighting the initial composition features, so that the weighted composition features meet the optimized composition mode, and then generates the composition optimization suggestions based on the weighted composition features. In this way, the electronic device receives the composition optimization suggestions returned by the server and outputs the composition optimization suggestions, which can help users to optimize the composition efficiently and with high quality.

[0075] It should be noted that the application scenarios described in the above-mentioned embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Ordinary technicians in this field can know that with the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0076] According to the above application scenarios, the following Figure 2 , an image processing method provided in an embodiment of the present application is exemplarily described. The execution subject of the image processing method may be an electronic device. It should be noted that the execution subject does not constitute a limitation on the present application.

[0077] Figure 2 FIG. 1 is a flowchart of an image processing method according to an exemplary embodiment of the present application. Figure 2 As shown, the image processing method may include the following steps:

[0078] Step 210: Acquire a real-time image and multimodal data of a shooting scene corresponding to the real-time image.

[0079] Among them, the real-time image refers to the image data collected by the image sensor of the electronic device at the current moment and not actually shot, which is usually used to display the picture of the image sensor under the current shooting field of view. When collecting real-time images, the image sensor of the electronic device can be in standby mode, preview mode or simulation mode. For example: a mobile phone can obtain real-time images collected by the camera installed on the mobile phone in preview mode. Another example: a smart camera can obtain real-time images collected in the local preview mode. Real-time images have the characteristics of immediacy and continuity. The real-time image obtained in this step can be expressed as one or more frames of continuous image data.

[0080] Multimodal data refers to data that contains multiple different types of information, which usually come from different sources or sensors. Each type of information is called a "modality". For example: electronic devices can be equipped with different types of sensors to collect different types of sensor data in the physical environment where the electronic devices are located. These different types of sensor data can form multimodal data. For another example: electronic devices can, with the user's authorization, obtain user data related to the user's shooting (such as user behavior, preferences, etc.) from data stored in the device or data stored in some platforms. The sensor data and user data currently collected by the sensor are from different sources, forming multimodal data. Exemplarily, multimodal data may include data from different sources such as environmental audio data, light intensity data, device location information, and user operation behavior.

[0081] In the embodiment of the present application, the electronic device obtains the currently captured real-time image and multimodal data related to the current shooting scene. These data are combined with the real-time image captured by the electronic device to form a comprehensive data set containing multimodal data, which provides comprehensive context information for subsequent composition optimization, improves the system's ability to understand the scene, and makes composition optimization more accurate.

[0082] Thus, in some embodiments of the present application, step 210 may include:

[0083] Acquire real-time images collected by image sensors;

[0084] The multimodal data is obtained by acquiring sensor data collected by at least one sensor of the electronic device in the shooting scene where the electronic device is located, and / or acquiring user data related to the shooting scene.

[0085] The image sensor may be a device installed on an electronic device and capable of recording or capturing image information. For example, if the electronic device is a mobile phone, the image sensor may be a camera installed on the mobile phone. For another example, if the electronic device is a smart medical device, the image sensor may be a medical imaging device.

[0086] A sensor can be a device installed on an electronic device that can detect and respond to specific physical, chemical or biological phenomena in the physical environment and convert these phenomena into electrical signals or other readable forms. For example, if the electronic device is a mobile phone, a sound sensor, a position sensor and / or a light intensity sensor can be installed on the mobile phone. The mobile phone can obtain the ambient audio data collected by the sound sensor and / or the light intensity data collected by the light intensity sensor as the data in the multimodal data.

[0087] User data may be stored in an electronic device or in any platform or server, and this application does not limit this. For example, user data may be user data related to a shooting scene, such as user shooting operation behavior data or user preference data.

[0088] In practical applications, when a user prepares to capture an image using an image sensor of an electronic device, the user may first capture a real-time image at the current moment in order to adjust the composition. In this case, the user may trigger a composition optimization request for the real-time image currently captured by the electronic device through a trigger method provided by the user interface of the electronic device. Accordingly, the electronic device may obtain multimodal data related to the currently captured real-time image and the current shooting scene.

[0089] Exemplarily, the electronic device can obtain real-time images by connecting to an image sensor. The real-time image can be represented in a two-dimensional matrix form I(t), where t is the time dimension, I(t) is the image data captured at time t, and the image data can be represented by a matrix, where the matrix elements are the light intensity values ​​of the corresponding pixels. While obtaining the real-time image, the electronic device can simultaneously obtain multimodal data related to the shooting scene, including environmental audio data A(t), light intensity data L(t), device location information P(t) and user operation behavior data U(t).

[0090] The environmental audio data A(t) can be collected by a microphone device and represented as a feature vector of an audio signal at time t;

[0091] Light intensity data L(t) can be measured by sensors and is expressed as the intensity of ambient light at time t, in lux;

[0092] The device location information P(t) can be obtained through the Global Positioning System (GPS) or other positioning devices, and is expressed as the geographic coordinates of the device at time t, expressed in longitude and latitude;

[0093] The user operation behavior data U(t) can be obtained through the touch screen or physical button input on the mobile device, and is represented as the user's operation behavior feature vector at time t.

[0094] The real-time image I(t) forms an image dataset I = {I(t)}, and the multimodal data A(t), L(t), P(t), and U(t) form a multimodal dataset: M = {A(t), L(t), P(t), U(t)}. The data in the image dataset and the multimodal dataset are combined by the correspondence of time t to obtain a set of time series data.

[0095] By acquiring sensor data and / or user data collected by electronic device sensors in their environment, these multimodal data are combined with real-time images to form a comprehensive data set containing multimodal data, which provides comprehensive contextual information for subsequent composition optimization. It can effectively get rid of the information limitations of traditional composition that relies on single image information, improve the ability to understand the scene, and make composition optimization more accurate.

[0096] Step 220: Send a composition optimization request to the server, where the composition optimization request carries the real-time image and the multimodal data.

[0097] Exemplarily, the server responds to the composition optimization request, utilizes a composition model to extract initial composition features of the real-time image and the multimodal data, performs weighted processing on the initial composition features based on the composition principles pre-learned by the composition model, and obtains composition optimization suggestions based on the weighted composition features.

[0098] It should be noted that before sending the composition optimization request in step 220, the real-time image and / or multimodal data may be preprocessed according to actual application needs. Preprocessing includes but is not limited to: image editing and adjustment (such as resizing, rotation, denoising, etc.), user data cleaning, etc. Of course, the preprocessing step may also be performed on demand after the server receives the composition optimization request, and this application does not limit this.

[0099] The composition optimization request is a request for triggering the server to call the composition model to generate composition optimization suggestions. The composition optimization request carries at least the real-time image acquired in step 210 and the corresponding multimodal data.

[0100] In some embodiments of the present application, the composition model can be deployed on a server. After receiving a composition optimization request, the server uses the composition model to extract initial composition features of real-time images and multimodal data, and based on the composition principles pre-learned by the composition model, performs weighted processing on the initial composition features, and generates composition optimization suggestions based on the weighted composition features.

[0101] In some other embodiments of the present application, the composition model may also be deployed on an electronic device, and the electronic device may call the local composition model to perform the process of generating composition optimization suggestions, but the present application does not limit this.

[0102] The composition optimization suggestion is a suggestion for optimizing the composition of the current real-time image. The composition optimization suggestion may be in any form, such as text, image, voice, or any one or more forms.

[0103] Among them, the composition model can be implemented through a variety of machine learning models, including: self-supervised learning, supervised learning model or semi-supervised learning model, and this application does not limit this. It can be understood that whether it is self-supervised learning, supervised learning model or semi-supervised learning model, by setting a suitable training data set, the training data set contains at least image samples and multimodal data samples of the shooting scene corresponding to the image samples, then after training, the model can learn the optimized composition principles in the training data set. In this way, the trained composition model can extract the initial composition features of the real-time image and the corresponding multimodal data in actual applications, and based on the pre-learned composition principles, the initial composition features are weighted, and composition optimization suggestions are generated based on the weighted composition features.

[0104] The initial composition features can be understood as the composition features extracted by the composition model after the real-time image and the corresponding multimodal data are input into the composition model and before the weighted processing. The composition features include features corresponding to multiple composition elements. Composition elements are components used to determine the composition of an image. For example, composition elements may include any number of elements such as subject, foreground, midground, background, guide lines, frame, lighting, color, contrast, texture, shape, size, direction, perspective and hierarchy. By extracting the initial composition features of the real-time image and the corresponding multimodal data, and weighting the initial composition features based on the pre-learned composition principles, the organization of these composition elements can be improved, and the visual performance of the image can be better.

[0105] The composition principle can be understood as the optimized way of organizing composition elements learned by the composition model through training. In the composition principle learned by the composition model, different composition elements can be assigned different weights based on the attention mechanism, so that the composition model performs weighted processing on the initial composition features based on the composition principle, and can organize the composition elements according to the optimized composition principle. The weights corresponding to different composition elements can be the weights learned based on the attention mechanism of the model during the training process.

[0106] In order to make the method provided in the embodiment of the present application easier to understand, the composition model in the embodiment of the present application is used as a large self-supervised learning model to exemplify the model training, model structure and model application.

[0107] Exemplarily, the composition model may be trained before step 210 or step 220. In the model training stage of the composition model, the loss value may be calculated by using a loss function corresponding to an appropriate learning mechanism according to the type and structure of the model, so as to adjust the model parameters and the weights corresponding to the composition elements. For example: in the case where the composition model is a supervised learning model, the model may be trained using labeled training data, such as by using a loss function such as cross entropy to calculate the loss value. Another example: in the case where the composition model is a large self-supervised learning model, the model may be trained using unlabeled training data, such as by using a contrast loss function of a contrast learning mechanism to calculate the loss value.

[0108] Exemplarily, in some embodiments of the present application, the model training step may include:

[0109] Acquire a training data set, the training data set comprising: image samples and multimodal data samples of shooting scenes corresponding to the image samples;

[0110] The initial composition model is trained using the training data set to obtain a trained composition model, and the composition model is used to predict composition optimization suggestions.

[0111] Taking the composition model as a self-supervised learning large model and the image sample as an unlabeled image sample as an example, before receiving the composition optimization request sent by the electronic device, the server can train the composition model in the following steps, including:

[0112] Inputting the image samples in the training data set and the corresponding multimodal data samples into the composition model;

[0113] Extracting composition features of the image sample and the corresponding multimodal data sample through the composition model;

[0114] Calculating the loss value of the composition feature of the image sample and the corresponding multimodal data sample through the loss function of the composition model;

[0115] Wherein, the loss function at least includes a contrast loss function term, and the contrast loss function term is used to calculate the contrast loss value of the composition feature of the image sample and the corresponding multimodal data sample;

[0116] The model parameters and the weights of the composition elements in the composition features are adjusted based on the loss value.

[0117] Among them, during the self-supervised learning training process, the composition model extracts the initial composition features of the image samples and the corresponding multimodal data samples, and weights the initial composition features based on the composition principles learned during the training process. Through the comparative learning mechanism, the weighted composition features of the image samples and the corresponding multimodal data samples are paired and analyzed to adjust the model parameters and the weights of the composition elements in the composition features.

[0118] The self-supervised learning model is a large machine learning model that automatically creates supervisory signals from a large amount of unlabeled training data. Since the self-supervised learning model learns information from the unlabeled data itself, no manual labeling is required, which greatly reduces the cost and time of data preparation and improves training efficiency.

[0119] In an embodiment of the present application, a training data set for training a self-supervised learning large model includes: unlabeled image samples and multimodal data samples related to the shooting scene of the image samples. The samples can be obtained in a variety of ways. For example: they can be obtained through some public data set platforms, can be obtained through Internet searches, can be obtained through manual shooting, etc. The combination of image samples and corresponding multimodal data samples is not limited. For example, the image sample {I(t)} and the multimodal data sample {M(t)} related to its shooting scene can be combined according to the time dimension to obtain a combined input data sample {I(t), M(t)}, where t represents the acquisition time of the image sample and the multimodal data. Multiple combined input data samples at different acquisition times constitute a training data set {I(t), M(t)}. The training data set may include a large number of image samples with optimized composition and their corresponding multimodal data samples. The training data set is input into the self-supervised learning large model for self-supervised learning training, and the loss function is used to optimize the parameters so that the model can learn the composition principle of the optimized composition from it.

[0120] After the combined input data samples in the training data set are input into the composition model, the self-supervised learning large model can be trained for self-supervised learning. Exemplarily, the self-supervised learning large model of the embodiment of the present application performs a pairing analysis on the weighted composition features of the image sample and the corresponding multimodal data sample through a contrast learning mechanism to calculate the loss value of the loss function, and adjusts the model parameters and the weights of the composition elements in the composition features based on the loss value.

[0121] The contrastive learning mechanism is a self-supervised learning method that trains the model by learning to distinguish the difference between similar samples and dissimilar samples. In an embodiment of the present application, the composition model performs a pairing analysis on the weighted composition features of the image sample and the corresponding multimodal data sample through the contrastive learning mechanism to calculate the loss value of the loss function, and adjusts the model parameters and the weights of the composition elements in the composition features based on the loss value, so that the composition model can learn the composition principles from the analysis of a large number of unlabeled image samples and corresponding multimodal data samples by distinguishing similar samples from dissimilar samples, extract composition features based on the learned composition principles, and perform weighted processing on the composition features to generate a composition representation with a more optimized composition.

[0122] In some embodiments of the present application, the loss function at least includes a contrast loss function term, and the contrast loss function term is used to calculate the contrast loss value of the composition feature of the image sample and the corresponding multimodal data sample;

[0123] The model parameters and the weights of the composition elements in the composition features are adjusted based on the loss value.

[0124] Among them, the contrast loss function term includes: a contrast loss value calculation part and a regularization part.

[0125] The contrast loss value calculation part is used to calculate the similarity between the image sample and its corresponding multimodal data, and compare the similarity with the similarity between all other multimodal data, maximize the similarity between positive sample pairs, and minimize the similarity between negative sample pairs, based on which the loss is evaluated.

[0126] Regularization part: Prevent overfitting through regularization terms.

[0127] In some embodiments of the present application, the loss function further includes: an image feature similarity loss function term and a multimodal feature similarity loss function term;

[0128] The image feature loss function term is used to calculate a first similarity between an initial composition feature corresponding to the image sample and a corresponding weighted composition feature through a similarity function, and determine an image feature loss value based on the first similarity;

[0129] The multimodal feature similarity loss function term is used to calculate the second similarity between the initial composition feature corresponding to the multimodal data sample and the corresponding weighted composition feature through a similarity function, and determine the multimodal feature loss value based on the second similarity.

[0130] Exemplarily, the loss function can be expressed as the following calculation formula:

[0131]

[0132] Among them, α, β, and γ are loss weight parameters, which are used to control the impact of different loss terms. I(t) is the image feature extracted by the neural network at layer l l With its estimated value The similarity between is the multimodal feature M(t) extracted at the mth layer m With its estimated value The similarity between .

[0133] Among them, the contrast loss function term can be expressed as the following calculation formula:

[0134]

[0135] Among them, sim(I i (t),M i (t)) is the image data I i (t) and multimodal data M i (t), τ is the temperature parameter, and λ is the regularization parameter, which is used to control the complexity of the model and prevent overfitting. For each model parameter θ k The gradient regularization term enables the model to have better generalization ability when facing new scenes through regularization. It is suitable for composition optimization of real-time images in various scenes, such as composition optimization of real-time images on mobile terminals.

[0136] Image Data I i (t) and multimodal data M i The similarity function between the image data I(t) and the multimodal data M(t) is used to measure the similarity between the image data I(t) and the multimodal data M(t). i (t),M i (t)) can be expressed as the following calculation formula:

[0137]

[0138] Among them, C is the number of channels, H and W are the height and width of the image respectively, and I c (t) i,j and M c (t) i,j are the pixel values ​​at the i-th and j-th positions of the image and multimodal data in the c-th channel, respectively.

[0139] The image feature similarity loss function term I(t) is the image feature extracted by the neural network at layer l l With its estimated value The similarity between them is calculated based on the principle of the similarity calculation formula, which is similar to the principle of the above sim(I(t),M(t)) calculation formula and will not be repeated here.

[0140] In the multimodal feature similarity loss function is the multimodal feature M(t) extracted at the mth layer m With its estimated value The similarity between them is calculated based on the principle of the similarity calculation formula, which is similar to the principle of the above sim(I(t),M(t)) calculation formula and will not be repeated here.

[0141] In the above embodiment, by introducing contrast loss function terms, image feature similarity loss function terms and multimodal feature similarity loss function terms into the loss function, the loss of feature prediction can be fully evaluated, the parameters of the composition model can be optimized, and the large model can recognize and enhance the composition features in the real-time image.

[0142] It should be noted that the composition model in this application can be implemented through a variety of machine learning models. For example: in some embodiments, the composition model can be implemented using a deep convolutional neural network, and in other embodiments, the composition model can be implemented using a graph neural network model, which is not limited in this application. The neural network structure of the composition model in the training phase and the model application phase can be the same, or it can be fine-tuned as needed, which is not limited in this application.

[0143] In some embodiments of the present application, the deep convolutional neural network of the composition model is as follows: Figure 3 As shown in the schematic diagram of the neural network structure, it may include: input layer 310, primary feature extraction layer 320, multi-scale feature fusion layer 330, feature enhancement layer 340, scene understanding layer 350, fully connected layer 360 and output layer 370. The neural network structure of the composition model in the training stage and the model application stage can be the same, or it can be fine-tuned as needed, and this application does not limit this.

[0144] According to the deep convolutional neural network structure of the above embodiment, in the model application stage, that is, step 220, the composition model extracts the composition features of the real-time image and the multimodal data, and obtains the composition optimization suggestions based on the composition features, which may include:

[0145] Receive real-time images and corresponding multimodal data through the input layer;

[0146] Extracting initial composition features of each composition element in the real-time image and the corresponding multimodal data through the primary feature extraction layer;

[0147] The multi-scale feature fusion layer fuses the initial composition features output by the primary feature extraction layer to obtain fused composition features;

[0148] Determining the weights corresponding to the various composition elements based on the attention mechanism through the feature enhancement layer, and weighting the fused composition features based on the weights corresponding to the various composition elements to obtain the weighted fused composition features;

[0149] The scene understanding layer analyzes the composition changes and scene dynamics in the continuous image frames and their corresponding multimodal data based on the long short-term memory network to obtain enhanced composition features.

[0150] The enhanced composition features output by the scene understanding layer are mapped to composition optimization suggestions through the fully connected layer to obtain composition optimization suggestions.

[0151] The output layer outputs composition optimization suggestions.

[0152] The input layer can be understood as a network layer for receiving input data. The dimension of the input data can be adjusted according to the computing power of the server or electronic device. In an embodiment of the present application, the input data includes image data and corresponding multimodal data. In the model training stage, the input data received by the input layer includes image samples and corresponding multimodal data samples. In the model application stage, the input data received by the input layer includes real-time images and corresponding multimodal data. For example, Figure 3 As shown, the input layer receives the input combined data set {I(t), M(t)}.

[0153] The primary feature extraction layer can be understood as a network layer used to extract initial composition features. The primary feature extraction layer can include multiple convolutional layers and ReLU activation functions. Exemplarily, the convolutional layer kernel size of the primary feature extraction layer can be designed to be 3×3, and the basic features of the edge and texture of the image are extracted through shallow convolution. For example, the calculation of the primary feature extraction layer can be expressed as the following calculation formula:

[0154]

[0155] in, is the convolution output of layer l, is the convolution kernel weight of the lth layer, I(t) i+k,j+k is the pixel value of the input image data, b l is the bias term, σ is the activation function, and the ReLU function is used.

[0156] The multi-scale feature fusion layer can be understood as a network layer used to perform multi-scale fusion processing on the initial composition features output by the primary feature extraction layer. Exemplarily, the multi-scale feature fusion layer can design a multi-branch structure based on multi-scale feature extraction, and each branch uses a convolution kernel of a different size. The convolution kernel may include, for example, 3×3, 5×5, and 7×7. By extracting and fusing features of different scales, the performance and robustness of the model can be significantly improved. For example, the calculation of the multi-scale feature fusion layer can be expressed as the following calculation formula:

[0157]

[0158] Among them, F fusion is the fusion composition feature obtained after fusion, is the convolution output of the lth layer at the sth scale, is the convolution output of the previous layer, α s is the weight parameter of features of different scales, and S is the number of scales.

[0159] The feature enhancement layer can be understood as a network layer used to perform weighted processing on the fused composition features output by the multi-scale feature fusion layer based on the attention mechanism. The feature enhancement layer can include an attention mechanism module, which assigns weights to each composition element based on the attention mechanism, and can adaptively enhance the feature expression of the composition elements, so that the weighted fused composition features can be optimized and the important visual focus can be highlighted.

[0160] The scene understanding layer can be understood as a network layer for performing a joint analysis of composition changes and scene dynamics on the fused composition features output by the feature enhancement layer. In the scene understanding layer, image data is combined with multimodal data for scene understanding. Specifically, the scene understanding layer uses a long short-term memory network to analyze the composition changes and scene dynamics in continuous image frames and their corresponding multimodal data to obtain enhanced composition features. It can be understood that both the image data and its corresponding multimodal data can have time attributes, and can be combined according to the time dimension to obtain time series data including image data and corresponding multimodal data. In the model training stage and the model application stage, the input data of the composition model can be time series data. As a result, the fused composition features input into the scene understanding layer can also be expressed as time series data, which can realize the joint analysis of composition changes and scene dynamics in continuous image frames and their corresponding multimodal data.

[0161] The fully connected layer can be understood as a network layer for integrating high-dimensional features of the enhanced composition features output by the scene understanding layer. In addition, during the model training phase, the fully connected layer is also connected to the contrastive learning module. The contrastive learning module is used to perform a pairing analysis on the weighted composition features of the image samples and the corresponding multimodal data samples (which can be understood as the high-dimensional features output by the fully connected layer in this embodiment) through a contrastive learning mechanism to calculate the loss value of the loss function, and adjust the model parameters and the weights of the composition elements in the composition features based on the loss value, so that the large model can optimize parameters in the self-supervised learning process and learn the composition principles.

[0162] In the model application stage, based on the structure of the above-mentioned deep convolutional neural network, the composition model automatically identifies the composition features in the combined data set {I(t), M(t)} based on the composition principles learned during training, analyzes the position, size and color information of each visual element in the current scene, and generates a vector representation Φ(I(t), M(t)) of the initial composition features. After that, the composition model determines the relative weight of each composition element, and weights the initial composition features through the attention mechanism model to generate the weighted composition features Ψ(I(t), M(t)). The calculation of the generated weighted composition features can be expressed as the following calculation formula:

[0163]

[0164] Among them, Ψ(I(t),M(t)) is the weighted composition vector, which represents the composition features after optimization by the composition model, α i is the weight of the i-th composition element. The weight value is automatically generated by the composition model through the attention mechanism during the self-supervised learning process and is used to adjust the importance of each composition element. i The determination of the value of makes the key visual elements occupy a more important position in the final composition, enhancing the aesthetic effect of the image. i (I(t),M(t)) is the feature representation of the i-th composition element. The features include the edge, texture, and color distribution information of the image, which are extracted by the composition model in the primary feature extraction layer and multi-scale feature fusion layer of the deep convolutional neural network. The feature representation ensures that the composition model can accurately describe the visual characteristics of each composition element. is the calculation process of attention weights, where Z is a normalization factor used to ensure that the sum of all attention weights is 1, thereby balancing the contribution of different composition elements in the final feature representation, and m is the total number of composition elements in the current scene. By analyzing the elements, the composition model can determine the relative importance of each element in the overall composition. sim(φ i (I(t),M(t)),φ j(I(t),M(t))) is the similarity function between the i-th composition element and the j-th composition element. The similarity function measures the similarity of different composition elements in visual features based on Euclidean distance, cosine similarity or other methods suitable for image feature comparison, helping the composition model to identify and strengthen the elements most relevant to the current scene composition. j To control the parameters of similarity, which are used to adjust the contribution of different composition elements to the attention weight, the composition model learns β j The value of is used to enhance the focus on important elements, thereby optimizing the final composition effect. In this way, based on the weighted composition vector Ψ(I(t), M(t)), the composition model automatically generates composition optimization suggestions.

[0165] During the model training phase, the composition model extracts initial composition features of image samples and corresponding multimodal data samples, and performs weighted processing on the initial composition features based on the composition principles learned during the training process. The processing steps are consistent with the processing steps in the model application phase described above and will not be repeated here.

[0166] Through the network structure of this embodiment, not only can the composition features be weighted based on the learned composition principles to achieve composition optimization, but the composition features can also be fused at multiple scales to improve the robustness of the model. In addition, the image data and multimodal data are jointly analyzed through the long short-term memory network to obtain enhanced composition features that include composition changes and scene dynamic representations, thereby enhancing the feature expression of the optimized composition and improving the composition optimization effect.

[0167] In some embodiments of the present application, the composition model can be a self-supervised learning large model using the above-mentioned deep convolutional neural network structure. The self-supervised learning large model can be constructed based on the self-supervised learning framework, and the above-mentioned contrast loss function is introduced for self-supervised learning. The model automatically learns and extracts composition features in images and scenes through joint analysis of unlabeled image samples and multimodal data samples, and generates composition optimization suggestions, thereby processing complex composition elements based on the contrast learning mechanism, ensuring that the model can autonomously optimize the composition of the image and improve the adaptability and generalization ability of the system in real-time scenarios.

[0168] Step 230: receiving the composition optimization suggestion returned by the server, and outputting the composition optimization suggestion.

[0169] The composition optimization suggestions may involve optimization of any one or more aspects of the composition, including but not limited to: subject position, foreground processing, background processing, guide lines, color, contrast, light intensity, texture, shooting angle, layering and depth, etc. By outputting composition optimization suggestions, users can be helped to effectively improve the visual effect of the image.

[0170] In some embodiments of the present application, the composition optimization suggestion includes at least one of: a composition frame optimization suggestion, a shooting angle optimization suggestion, and a visual focus repositioning suggestion.

[0171] Among them, the composition frame optimization suggestion is a suggestion for instructing to redraw the image boundaries so that the key visual elements are at the golden section point or the rule of thirds composition line. For example: the composition frame optimization suggestion can be expressed as "object A is placed in the left third of the picture to follow the rule of thirds."

[0172] Shooting angle optimization suggestions are suggestions for adjusting the shooting angle. For example, a shooting angle optimization suggestion can be expressed as "shoot object A from above to achieve the best shooting angle." When generating shooting angle optimization suggestions, the composition model can select the best shooting angle by adjusting and comparing the composition features under different viewing angles to generate shooting angle optimization suggestions.

[0173] Visual focus repositioning suggestions are suggestions for adjusting the focus position. For example, a visual focus repositioning suggestion can be expressed as "Please set the focus to the center of object A". When generating visual focus repositioning suggestions, the composition model can select the best focus position to enhance the visual effect of the image by identifying the most visually attractive composition elements in the scene.

[0174] By outputting at least one of the composition framework optimization suggestions, shooting angle optimization suggestions, and visual focus repositioning suggestions, it is possible to effectively help users accurately optimize the composition.

[0175] It should be noted that there are many ways to output composition optimization suggestions, which are not limited in this application. For example, the composition optimization suggestions in the form of text, images, or videos can be displayed to the user through the user interface of the electronic device, or the audio playback device of the electronic device can play the voice corresponding to the composition optimization suggestions.

[0176] Exemplarily, the composition optimization suggestion may be expressed as text, so that text such as "Place object A in the left third of the picture to follow the rule of thirds" and "Increase the blurriness of the background to make the subject more prominent" may be displayed on the user interface of the electronic device. The composition optimization suggestion is simple and clear, and easy to understand. For another example: the composition optimization suggestion may be expressed as a picture or video, so that annotation lines, boxes or other graphic elements may be added to the real-time image currently displayed by the electronic device to intuitively display the composition optimization suggestion. The composition optimization suggestion is intuitive and easy to understand, and the user can clearly see the part that needs to be adjusted. For another example: the composition optimization suggestion may be expressed as voice, so that a voice prompt may be played through the audio playback device of the electronic device to tell the user how to optimize the composition, such as the voice "Please move the camera to the left until object A is in the left third of the picture."

[0177] On the basis of outputting the composition optimization suggestion, in order to further help the user know more accurately how to operate the composition method that can realize the composition optimization suggestion, or automatically perform the shooting operation corresponding to the composition optimization suggestion, in some embodiments of the present application, the method further includes:

[0178] Based on the composition optimization suggestion, generating auxiliary shooting operation information;

[0179] Outputting auxiliary shooting suggestions according to the auxiliary shooting operation information.

[0180] Auxiliary shooting operations refer to a series of specific operations taken in the actual shooting process to achieve the composition effect corresponding to the composition optimization suggestions. Composition optimization suggestions can be understood as goals, and auxiliary shooting operations can be understood as more detailed operations to achieve the goals. Composition optimization suggestions may include suggestions on composition methods, such as guidance on composition methods such as the rule of thirds, golden section, and guide lines. Auxiliary shooting operations may include the use of specific shooting techniques and tools, such as selecting a suitable shooting angle, adjusting camera settings, and using auxiliary equipment. It can be understood that in some embodiments, the composition optimization suggestions generated by the model may also include auxiliary shooting operation information, and this application does not limit this.

[0181] By generating auxiliary shooting operation information, more specific shooting means and practice methods are provided, which can effectively help photographers achieve better composition effects in the actual shooting process, or directly and automatically perform auxiliary shooting operations.

[0182] Exemplarily, in some embodiments of the present application, generating auxiliary shooting operation information based on the composition optimization suggestion includes:

[0183] In a case where the composition optimization suggestion is a composition framework optimization suggestion, generating visual element position adjustment operation information based on positions of key visual elements in the composition data corresponding to the composition framework optimization suggestion;

[0184] In a case where the composition optimization suggestion is a visual focus relocation suggestion, generating focus position adjustment operation information based on a position of a focus in the composition data corresponding to the visual focus relocation suggestion;

[0185] In a case where the composition optimization suggestion is a shooting angle optimization suggestion, based on the optimized shooting angle in the composition data corresponding to the shooting angle optimization suggestion, analyzing the difference between the current shooting angle of the electronic device and the optimized shooting angle, and generating shooting angle adjustment operation information based on the difference;

[0186] When it is necessary to generate an aperture setting suggestion, the aperture optimization parameters are calculated based on the reference light intensity data in the composition data corresponding to the composition optimization suggestion, the current light intensity, aperture value and focal length in the real-time image, and the aperture setting operation information is generated based on the aperture optimization parameters.

[0187] Among them, when generating the operation information for adjusting the position of visual elements, the key visual elements can be determined by analyzing the composition vector Ψ(I(t), M(t)) of the optimized composition corresponding to the composition optimization suggestion, and the ideal position of the key visual elements in the image can be analyzed. For example: the composition optimization suggestion provides suggestions for optimizing the composition based on the golden section, the rule of thirds composition principle or other known composition rules. On this basis, when generating the operation information for adjusting the position of visual elements, the ideal position of the key visual elements can be identified according to the composition optimization suggestion, and based on the ideal position, specific operation content that can adjust the key visual elements to the ideal position is generated, such as "adjust the image acquisition range to move object A to the intersection 1 of the screen's nine-square grid frame lines."

[0188] When generating focus position adjustment operation information, the ideal position of the focus can be determined by analyzing the composition vector Ψ(I(t), M(t)) of the optimized composition corresponding to the composition optimization suggestion. On this basis, specific operation content that can adjust the focus to the ideal position is generated. For example, the operation information can be expressed as controlling the electronic device to execute the operation instruction of "moving the focus to the position of object A, and the position of object A is at the intersection 1 of the nine-square grid line on the screen". In this way, the focus position can be adjusted automatically or manually according to the operation content of the focus position adjustment operation.

[0189] When generating the shooting angle adjustment operation information, the difference between the current shooting angle and the optimized shooting angle can be analyzed in combination with the geometric feature information of the optimized composition corresponding to the composition optimization suggestion. For example, the geometric features of the image data and multimodal data of the optimized shooting angle and the difference between the geometric features of the image data and multimodal data of the current shooting angle can be calculated, and the shooting angle adjustment operation information can be generated based on the difference, and the operation content can include the adjustment of the camera position or angle.

[0190] When generating aperture setting operation information, the lighting conditions and depth of field requirements of the current real-time image can be analyzed, including the current light intensity, aperture value and teaching aids, and the aperture optimization parameter α can be generated based on the reference light intensity data in the composition data corresponding to the composition optimization suggestion. aperture The aperture optimization parameters are based on the light intensity data L(t) and the image composition characteristics, and the aperture size is adjusted to control the amount of light entering and the depth of field effect:

[0191]

[0192] Among them, F\# is the aperture value, f is the focal length, L(t) is the current light intensity, L0 is the reference light intensity, k is the adjustment coefficient, and by optimizing α aperture to achieve the desired depth of field effect and exposure level.

[0193] In the embodiment of the present application, outputting auxiliary shooting suggestions can help users clearly understand the operation content and quickly achieve a better composition.

[0194] The auxiliary shooting suggestion is a way to display auxiliary shooting operation information in a way that the user can understand. The auxiliary shooting suggestion is not limited to any form of expression, and can be any one or more forms such as text, picture, voice, etc. For example: the auxiliary shooting suggestion can be in the form of text "Move the focus to the position of object A, the position of object A is at the intersection 1 of the nine-square frame line on the screen".

[0195] By outputting auxiliary shooting suggestions, it is possible for users to more easily understand how to operate to optimize the composition, thereby automatically or manually performing actual shooting.

[0196] According to the above embodiment, the composition model automatically generates composition optimization suggestions, such as optimization suggestions for adjusting the composition frame, optimization suggestions for adjusting the shooting angle, optimization suggestions for repositioning the visual focus, etc., and outputs them to the user interface in real time for the user to select or adjust. With the help of the composition model based on deep learning, the composition is optimized in real time during the shooting process, thereby achieving the purpose of helping users to optimize the composition efficiently and with high quality.

[0197] In order to enable the electronic device to quickly capture an image with an optimized composition, in some embodiments of the present application, the method further includes:

[0198] receiving a first input from a user;

[0199] In response to the first input, performing an auxiliary shooting operation according to the first input and the auxiliary shooting operation information;

[0200] Image data acquired while performing the auxiliary shooting operation is output.

[0201] The user's first input may be input in a variety of ways, such as operating a control displayed on a user interface by means of a mouse or touch screen, or inputting a voice command by voice, etc. The first input may be understood as any one or more operations for confirming or adjusting the auxiliary shooting operation.

[0202] For example, when the user's first input indicates confirmation of the composition optimization suggestion and / or the auxiliary shooting operation, the auxiliary shooting operation generated based on the composition optimization suggestion can be directly used as the auxiliary operation to be executed to implement the composition corresponding to the composition optimization suggestion.

[0203] For another example, when the user's first input is to adjust the composition optimization suggestions and / or auxiliary shooting operations, the adjusted operation content can be determined based on the user's adjustment and auxiliary shooting operation information, and the adjusted operation content can be used as the auxiliary operation to be executed, so as to achieve a better composition based on the composition optimization suggestions.

[0204] The auxiliary shooting operation may be an operation of adjusting the composition or an operation of adjusting the composition and shooting. It is implemented according to actual application needs or user settings, and this application does not limit this.

[0205] The image captured in the auxiliary shooting operation may be a real-time image of a preview image or an image actually shot, depending on whether the operation performed is to adjust the composition or to adjust the composition and shoot.

[0206] In the above embodiment, after the user accepts the suggestion or adjusts the auxiliary shooting operation, the electronic device can automatically perform the operation to adjust the composition and output the image data collected after the composition is adjusted, so that the user can quickly and intuitively see the image after the composition is adjusted, reduce user operations, and improve image processing efficiency.

[0207] In order to further optimize the composition model, in some embodiments of the present application, the electronic device may further train the composition model using the image data actually taken and multimodal data related to the shooting scene after the actual shooting to optimize the composition model. In this embodiment, the method further includes:

[0208] Acquire a real-shot image actually shot by the electronic device and multimodal data of a shooting scene corresponding to the real-shot image;

[0209] A model optimization request is sent to the server, wherein the model optimization request carries multimodal data related to the real-shot image and the shooting scene corresponding to the real-shot image, so that the server uses the real-shot image and the multimodal data of the shooting scene corresponding to the real-shot image to train the composition model.

[0210] Exemplarily, a triggering method for a model optimization request can be set on a user interface of an electronic device. After the user actually captures an image using an electronic device, the electronic device can automatically, or in response to a user trigger, send the model optimization request to the server. For example: the electronic device outputs a composition optimization suggestion to the user, and the user adjusts the shooting parameters based on the actual situation and the composition optimization suggestion before shooting. Since the actual captured image is more in line with the actual situation of the shooting scene, it is a composition method further optimized by the user based on the composition optimization suggestion. Based on this, the composition model is further trained using the image data actually captured in this case and its corresponding multimodal data, which can realize autonomous optimization of the composition model, making the shooting process more intelligent and personalized with the assistance of the composition model, and achieving a more ideal composition effect.

[0211] In this embodiment, the electronic device or server can also obtain and store the composition-optimized image and its corresponding multimodal data of the shooting scene after the user accepts the composition optimization suggestion, so as to use these data to further train the composition model and optimize the composition model when the model needs to be optimized.

[0212] Through the above-mentioned embodiments, high-order scene features are extracted from the multimodal data of the real-shot images and their corresponding shooting scenes, and applied to subsequent optimization, effectively using multimodal data to characterize complex scenes, thereby providing a data-driven starting point for intelligently helping users to optimize the composition, improving the composition optimization capability of the model, and effectively improving image processing efficiency.

[0213] In an embodiment of the present application, a real-time image collected by an electronic device and multimodal data related to the shooting scene of the real-time image are obtained, and a model optimization request carrying the real-time image and multimodal data is sent to a server. In this way, the server can make full use of the comprehensive context information provided by the real-time image and multimodal data, use the composition model, extract the composition features of the real-time image and multimodal data, and perform model training for generating composition optimization suggestions based on the composition features. In this way, an optimized and trained composition model can be obtained.

[0214] Figure 4 FIG. 1 is a flowchart of an image processing method according to another exemplary embodiment of the present application. Figure 4 As shown, the image processing method may include the following steps:

[0215] Step 410: The server constructs and trains a composition model, which is a large self-supervised learning model.

[0216] The specific implementation of this step can refer to the description of the model structure, training phase and application phase of the self-supervised learning large model in step 220 in the aforementioned embodiment, which will not be repeated here.

[0217] Step 420: The electronic device obtains a real-time image and multimodal data of a shooting scene corresponding to the real-time image.

[0218] Step 430: The electronic device sends a composition optimization request to the server, so that the server generates composition optimization suggestions based on the composition optimization request using a self-supervised learning large model.

[0219] The specific implementation of the above steps 420-430 can refer to the relevant instructions on obtaining real-time images, multimodal data and composition models to generate composition optimization suggestions in steps 210-220 in the above embodiments, which will not be repeated here.

[0220] Step 440: The electronic device receives the composition optimization suggestion returned by the server.

[0221] Step 450: The electronic device outputs composition optimization suggestions.

[0222] Step 460: The electronic device adjusts the composition of the real-time image automatically or manually according to the user's input regarding the composition optimization suggestion.

[0223] Step 470: The electronic device outputs the real-time image or the image actually shot after the composition is adjusted.

[0224] The specific implementation of the above steps 440 to 470 can refer to the relevant description of step 230 in the above embodiment, which will not be repeated here.

[0225] Through the interaction between the electronic device and the server in the above embodiment, during the image capture process of the electronic device, the server analyzes the input image and multimodal data in real time, and automatically identifies and optimizes the composition elements in the image based on the deep convolutional neural network structure and contrast learning mechanism. By combining multimodal data, the model can independently judge and adjust the composition during the image acquisition process, realizing real-time optimization, reducing the user's image processing workload after shooting, improving the efficiency of image shooting, and ensuring that the ideal composition effect can still be obtained in dynamic scenes.

[0226] The image processing method provided in the embodiment of the present application can be executed by an image processing device. In the embodiment of the present application, an image processing device executing the image processing method is taken as an example to illustrate the image processing device provided in the embodiment of the present application.

[0227] Figure 5 FIG. 5 is a structural block diagram of an image processing device 500 according to an exemplary embodiment of the present application. Figure 5 As shown, the image processing device 500 may include:

[0228] The real-time data acquisition module 510 is configured to acquire the real-time image captured by the electronic device and multimodal data related to the shooting scene of the real-time image;

[0229] A composition suggestion generating module 520 is configured to send a composition optimization request to a server, wherein the composition optimization request carries the real-time image and the multimodal data;

[0230] The composition suggestion output module 530 is configured to receive the composition optimization suggestion returned by the server and output the composition optimization suggestion.

[0231] In some embodiments of the present application, the real-time data acquisition module includes:

[0232] A real-time image acquisition submodule, configured to acquire a real-time image collected by an image sensor of the electronic device;

[0233] The multimodal data acquisition submodule is configured to acquire sensor data collected by at least one sensor of the electronic device in the shooting scene where the electronic device is located, and / or user data of the user, to obtain the multimodal data.

[0234] In some embodiments of the present application, the image processing device further includes:

[0235] The operation information generating module is configured to generate auxiliary shooting operation information based on the composition optimization suggestion.

[0236] The auxiliary shooting suggestion output module is configured to output an auxiliary shooting suggestion according to the auxiliary shooting operation information.

[0237] In some embodiments of the present application, the image processing device further includes:

[0238] A user input receiving module, configured to receive a first input from a user;

[0239] The operation determination module is configured to perform an auxiliary shooting operation in response to the first input and according to the first input and the auxiliary shooting operation information.

[0240] The optimized image output module is configured to output images collected when performing the auxiliary shooting operation.

[0241] In some embodiments of the present application, the operation information generation module is configured to generate visual element position adjustment operation information based on the position of the key visual element in the composition data corresponding to the composition frame optimization suggestion when the composition optimization suggestion is a composition frame optimization suggestion;

[0242] In a case where the composition optimization suggestion is a visual focus relocation suggestion, generating focus position adjustment operation information based on a position of a focus in the composition data corresponding to the visual focus relocation suggestion;

[0243] In a case where the composition optimization suggestion is a shooting angle optimization suggestion, based on the optimized shooting angle in the composition data corresponding to the shooting angle optimization suggestion, analyzing the difference between the current shooting angle of the electronic device and the optimized shooting angle, and generating shooting angle adjustment operation information based on the difference;

[0244] When an aperture setting operation is required, the aperture optimization parameters are calculated based on the reference light intensity data in the composition data corresponding to the composition optimization suggestion, the current light intensity, aperture value and focal length in the real-time image, and the aperture setting operation information is generated based on the aperture optimization parameters.

[0245] In some embodiments of the present application, the image processing device further includes:

[0246] A model optimization data acquisition module is configured to acquire multimodal data related to a real-shot image actually shot by the electronic device and a shooting scene corresponding to the real-shot image;

[0247] The model optimization request module is configured to send a model optimization request to the server, wherein the model optimization request carries the real-shot image and multimodal data of the shooting scene corresponding to the real-shot image, and the real-shot image and the multimodal data of the corresponding shooting scene are used to train the composition model.

[0248] Figure 5 The image processing device shown obtains the real-time image collected by the electronic device and multimodal data related to the shooting scene of the real-time image, and sends a model optimization request carrying the real-time image and multimodal data to the server. In this way, the server can make full use of the comprehensive contextual information provided by the real-time image and multimodal data to continue training the composition model and optimize the composition model.

[0249] Corresponding to the above-mentioned image processing method applied to electronic equipment, as Figure 6As shown, the embodiment of the present application also provides an image processing method applied to a server. The execution subject of the image processing method can be a server. It should be noted that the execution subject does not constitute a limitation of the present application.

[0250] Figure 6 FIG. 1 is a flowchart of an image processing method according to another exemplary embodiment of the present application. Figure 6 As shown, the image processing method may include the following steps:

[0251] Step 610: Receive a composition optimization request sent by an electronic device.

[0252] The composition optimization request carries a real-time image and multimodal data of a shooting scene corresponding to the real-time image, and the real-time image is collected in real time by the electronic device.

[0253] Step 620: Using a composition model, extract composition features of the real-time image and the multimodal data, and obtain composition optimization suggestions based on the composition features.

[0254] Step 630: Return the composition optimization suggestion to the electronic device.

[0255] The specific implementation of the above steps 610-630 can refer to the relevant descriptions in steps 210-230, which will not be repeated here.

[0256] In an embodiment of the present application, the server obtains real-time images collected by the electronic device and multimodal data related to the shooting scenes of the real-time images. In this way, it is possible to fully utilize the comprehensive contextual information provided by the real-time images and multimodal data, use the composition model to accurately extract the composition features of the real-time images and multimodal data, and obtain composition optimization suggestions based on the composition features.

[0257] Exemplarily, the composition model can be used to extract the initial composition features of real-time images and multimodal data, and then the composition principles pre-learned by the composition model can be used to identify the optimized composition method of the current shooting scene. By weighting the initial composition features, the optimized composition method is applied to the weighted composition features, so that the weighted composition features conform to the optimized composition method, and then based on the weighted composition features, a composition optimization suggestion is generated. In this way, the server returns the composition optimization suggestion to the electronic device, and the electronic device outputs the composition optimization suggestion, which can help the user to optimize the composition efficiently and with high quality.

[0258] In some embodiments of the present application, the composition model is used to extract the composition features of the real-time image and the multimodal data, and composition optimization suggestions are obtained based on the composition features, including:

[0259] Receiving the real-time image and the corresponding multimodal data through an input layer;

[0260] Extracting initial composition features of each composition element in the real-time image and the corresponding multimodal data through a primary feature extraction layer;

[0261] The initial composition features output by the primary feature extraction layer are fused with multi-scale features through a multi-scale feature fusion layer to obtain fused composition features;

[0262] Determining the weights corresponding to the various composition elements based on the attention mechanism through the feature enhancement layer, and weighting the fused composition features based on the weights corresponding to the various composition elements to obtain the weighted fused composition features;

[0263] The scene understanding layer analyzes the composition changes and scene dynamics in the continuous image frames and their corresponding multimodal data based on the long short-term memory network to obtain enhanced composition features.

[0264] The enhanced composition features output by the scene understanding layer are mapped to composition optimization suggestions through a fully connected layer to obtain composition optimization suggestions.

[0265] Through the network structure and processing process of the above-mentioned embodiments, the composition model can not only weight the composition features based on the learned composition principles to achieve composition optimization, but also perform multi-scale fusion of composition features to improve the robustness of the model, and also jointly analyze the image data and multimodal data through the long short-term memory network to obtain enhanced composition features that include composition changes and scene dynamic representations, thereby enhancing the feature expression of the optimized composition and improving the composition optimization effect.

[0266] Figure 7 FIG. 1 is a flow chart of a model training method according to an exemplary embodiment of the present application. Figure 7 As shown, the model training method may include the following steps:

[0267] Step 710: Acquire a training data set, where the training data set includes: image samples and multimodal data samples of shooting scenes corresponding to the image samples.

[0268] Step 720: Using the training data set, train the initial composition model to obtain a trained composition model, where the composition model is used to predict composition optimization suggestions.

[0269] The specific implementation of the above steps 710-720 can refer to the relevant descriptions in steps 210-230, which will not be repeated here.

[0270] In some embodiments of the present application, the composition model is a self-supervised learning large model, and the image samples are unlabeled image samples;

[0271] The using the training data set to train the initial composition model to obtain the trained composition model includes:

[0272] Inputting the image samples in the training data set and the corresponding multimodal data samples into the composition model;

[0273] Extracting composition features of the image sample and the corresponding multimodal data sample through the composition model;

[0274] Calculating the loss value of the composition feature of the image sample and the corresponding multimodal data sample through the loss function of the composition model;

[0275] Wherein, the loss function at least includes a contrast loss function term, and the contrast loss function term is used to calculate the contrast loss value of the composition feature of the image sample and the corresponding multimodal data sample;

[0276] The model parameters and the weights of the composition elements in the composition features are adjusted based on the loss value.

[0277] In the above embodiment, the composition model performs pairing analysis on the composition features of image samples and corresponding multimodal data samples through a comparative learning mechanism to calculate the loss value of the loss function, and adjusts the model parameters and the weights of the composition elements in the composition features based on the loss value, so that the composition model can learn the composition principles from the analysis of a large number of unlabeled image samples and corresponding multimodal data samples by distinguishing similar samples from dissimilar samples, extract composition features based on the learned composition principles, and weightedly process the composition features to generate a composition representation with a more optimized composition.

[0278] In some embodiments of the present application, the loss function further includes: an image feature similarity loss function term and a multimodal feature similarity loss function term;

[0279] The image feature similarity loss function term is used to calculate a first similarity between an initial composition feature corresponding to the image sample and a corresponding weighted composition feature through a similarity function, and determine an image feature loss value based on the first similarity;

[0280] The multimodal feature similarity loss function term is used to calculate the second similarity between the initial composition feature corresponding to the multimodal data sample and the corresponding weighted composition feature through a similarity function, and determine the multimodal feature loss value based on the second similarity.

[0281] In the above embodiment, by introducing contrast loss function terms, image feature similarity loss function terms and multimodal feature similarity loss function terms into the loss function, the loss of feature prediction can be fully evaluated, the parameters of the composition model can be optimized, and the large model can recognize and enhance the composition features in the real-time image.

[0282] In some embodiments of the present application, the method further includes:

[0283] Receiving a model optimization request sent by an electronic device, the model optimization request carrying a real-shot image actually shot by the electronic device and multimodal data of a shooting scene corresponding to the real-shot image;

[0284] The composition model is trained using the real-shot image and multimodal data of the shooting scene corresponding to the real-shot image to obtain an optimized composition model.

[0285] Through the above-mentioned embodiments, high-order scene features are extracted from image data and multimodal data related to its shooting scene, and applied to subsequent optimization, effectively using multimodal data to characterize complex scenes, thereby providing a data-driven starting point for intelligently helping users to optimize the composition, improving the model's composition optimization capabilities, and effectively improving image processing efficiency.

[0286] It should be noted that the specific implementation of the above-mentioned image processing method applied to the server can refer to the relevant description of steps 210 to 230 in the above-mentioned embodiment of the image processing method applied to the electronic device, and will not be repeated here.

[0287] The image processing method applied to the server provided in the embodiment of the present application can be executed by another image processing device. In the embodiment of the present application, an image processing device executing the image processing method is taken as an example to illustrate the image processing device provided in the embodiment of the present application.

[0288] Figure 8 FIG. 8 is a structural block diagram of an image processing device 800 according to another exemplary embodiment of the present application. Figure 8 As shown, the image processing device may include:

[0289] The composition optimization request receiving module 810 is configured to receive a composition optimization request sent by an electronic device, wherein the composition optimization request carries a real-time image and multimodal data of a shooting scene corresponding to the real-time image, and the real-time image is collected in real time by the electronic device;

[0290] The composition model processing module 820 is configured to extract composition features of the real-time image and the multimodal data using a composition model, and generate composition optimization suggestions based on the composition features;

[0291] The composition optimization suggestion returning module 830 is configured to return the composition optimization suggestion to the electronic device.

[0292] In some embodiments of the present application, the composition model processing module includes:

[0293] A model input submodule, configured to receive the real-time image and the corresponding multimodal data through an input layer;

[0294] A primary feature extraction submodule, configured to extract initial composition features of each composition element in the real-time image and the corresponding multimodal data through a primary feature extraction layer;

[0295] A feature fusion submodule is configured to fuse the initial composition features output by the primary feature extraction layer with multi-scale features through a multi-scale feature fusion layer to obtain a fused composition feature;

[0296] The feature enhancement submodule is configured to determine the weight corresponding to each composition element based on the attention mechanism through the feature enhancement layer, and weight the fused composition feature based on the weight corresponding to each composition element to obtain the weighted fused composition feature;

[0297] The scene understanding submodule is configured to analyze the composition changes and scene dynamics in the continuous image frames and their corresponding multimodal data based on the long short-term memory network through the scene understanding layer to obtain enhanced composition features.

[0298] The suggestion generation submodule is configured to map the enhanced composition features output by the scene understanding layer to composition optimization suggestions through a fully connected layer to obtain composition optimization suggestions.

[0299] Fig. 9 is a structural block diagram of a model training device 900 according to an exemplary embodiment of the present application. Fig. 9 As shown, the model training device may include:

[0300] The training data acquisition module 910 is used to acquire a training data set, where the training data set includes: image samples and multimodal data samples of shooting scenes corresponding to the image samples.

[0301] The model training module 920 is used to train the initial composition model using the training data set to obtain a trained composition model, and the composition model is used to predict composition optimization suggestions.

[0302] In some embodiments of the present application, the composition model is a self-supervised learning large model, and the image samples are unlabeled image samples.

[0303] The model training module includes:

[0304] The model input submodule is used to input the image samples in the training data set and the corresponding multimodal data samples into the composition model.

[0305] The feature extraction submodule is used to extract the composition features of the image sample and the corresponding multimodal data sample through the composition model.

[0306] The loss calculation submodule is used to calculate the loss value of the composition feature of the image sample and the corresponding multimodal data sample through the loss function of the composition model.

[0307] Wherein, the loss function at least includes a contrast loss function term, and the contrast loss function term is used to calculate the contrast loss value of the composition feature of the image sample and the corresponding multimodal data sample;

[0308] The parameter adjustment submodule is used to adjust the model parameters and the weights of the composition elements in the composition features based on the loss value.

[0309] In some embodiments of the present application, the device further includes:

[0310] A model optimization request receiving module is configured to receive a model optimization request sent by an electronic device, wherein the model optimization request carries a real-shot image actually taken by the electronic device and multimodal data of a shooting scene corresponding to the real-shot image;

[0311] The model optimization execution module is configured to train the composition model using the real-shot image and multimodal data of the shooting scene corresponding to the real-shot image to obtain an optimized composition model.

[0312] Combine the following Fig.10 , another image processing method provided in the embodiment of the present application is exemplarily described. The execution subject of the image processing method may be an electronic device. It should be noted that the execution subject does not constitute a limitation on the present application.

[0313] Fig.10 FIG. 1 is a flowchart of an image processing method according to another exemplary embodiment of the present application. Fig.10 As shown, the image processing method may include the following steps:

[0314] Step 1010: Acquire a real-time image and multimodal data of a shooting scene corresponding to the real-time image.

[0315] Step 1020: extracting composition features of the real-time image and the multimodal data using a composition model, and obtaining composition optimization suggestions based on the composition features.

[0316] For example, a composition model can be deployed on an electronic device, and the electronic device can use the composition model to generate composition optimization suggestions.

[0317] Step 1030: Output the composition optimization suggestion.

[0318] The specific implementation of the above steps 1010-1030 can partially refer to the relevant description of steps 210-230 in the above embodiment, which will not be repeated here.

[0319] Through this embodiment, the electronic device collects real-time images and multimodal data related to the shooting scene of the real-time images, and makes full use of the comprehensive context information provided by the real-time images and the multimodal data.

[0320] The composition model is used to accurately extract the composition features of real-time images and multimodal data, and based on the composition features, composition optimization suggestions are generated. In this way, the electronic device outputs the composition optimization suggestions, which can help users optimize the composition efficiently and with high quality.

[0321] The image processing method for electronic devices provided in the embodiment of the present application may be executed by another image processing device. In the embodiment of the present application, an image processing device executing the image processing method is taken as an example to illustrate the image processing device provided in the embodiment of the present application.

[0322] Fig.11 FIG. 1 is a structural block diagram of an image processing device according to another exemplary embodiment of the present application. Fig.11 As shown, the image processing device may include:

[0323] The real-time data acquisition module 1110 is configured to acquire a real-time image and multi-modal data of a shooting scene corresponding to the real-time image.

[0324] The local model execution module 1120 is configured to extract composition features of the real-time image and the multimodal data using a composition model, and generate composition optimization suggestions based on the composition features.

[0325] The composition suggestion output module 1130 is configured to output the composition optimization suggestion.

[0326] Fig.11 The image processing device collects real-time images and multimodal data of shooting scenes corresponding to the real-time images, and makes full use of the comprehensive context information provided by the real-time images and multimodal data, and uses the composition model to accurately extract the composition features of the real-time images and multimodal data, and generates composition optimization suggestions based on the composition features. In this way, outputting the composition optimization suggestions can help users optimize the composition efficiently and with high quality.

[0327] In the embodiment of the present application, the image processing device may be an electronic device or a component in the electronic device, such as an integrated circuit or a chip. The electronic device may be an electronic device or other devices other than the electronic device. Exemplarily, the electronic device may be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc. It may also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.

[0328] The image processing device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0329] The image processing device provided in the embodiment of the present application can achieve Figure 2-8 To avoid repetition, the various processes implemented in the method embodiment are not described here.

[0330] In some embodiments of the present application, Fig.12 As shown, an embodiment of the present application further provides an electronic device 1200, including a processor 1201 and a memory 1202, wherein the memory 1202 stores programs or instructions that can be executed on the processor 1201, and when the program or instructions are executed by the processor 1201, the various steps of the above-mentioned image processing method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, they are not described here.

[0331] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0332] Fig.13 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of the present application.

[0333] The electronic device 1300 includes but is not limited to: a radio frequency unit 1301, a network module 1302, an audio output unit 1303, an input unit 1304, a sensor 1305, a display unit 1306, a user input unit 1307, an interface unit 1308, a memory 1309, and a processor 1310 and other components.

[0334] Those skilled in the art will appreciate that the electronic device 1300 may also include a power source (such as a battery) for supplying power to each component, and the power source may be logically connected to the processor 1310 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Fig.13 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0335] The processor 1310 is configured to:

[0336] Acquire the real-time image captured by the electronic device and multimodal data related to the real-time image shooting scene;

[0337] Sending a composition optimization request to a server, the composition optimization request carrying the real-time image and the multimodal data, so that the server uses a composition model to extract initial composition features of the real-time image and the multimodal data, and based on the composition principles pre-learned by the composition model, performs weighted processing on the initial composition features, and generates composition optimization suggestions based on the weighted composition features;

[0338] The composition optimization suggestion is received and returned by the server, and the composition optimization suggestion is output.

[0339] In the embodiment of the present application, the real-time image collected by the electronic device and the multimodal data related to the shooting scene of the real-time image are obtained, so that the comprehensive context information provided by the real-time image and the multimodal data can be fully utilized, and the composition features of the real-time image and the multimodal data can be accurately extracted by using the composition model, and composition optimization suggestions are generated based on the composition features. In this way, the electronic device outputs the composition optimization suggestions, which can help users optimize the composition efficiently and with high quality.

[0340] In some embodiments, the processor 1310 is further configured to:

[0341] Acquire real-time images collected by image sensors;

[0342] The multimodal data is obtained by acquiring sensor data collected by at least one sensor of the electronic device in the shooting scene where the electronic device is located, and / or acquiring user data related to the shooting scene.

[0343] In the embodiment of the present application, the electronic device obtains the currently captured real-time image and multimodal data related to the current shooting scene. These data are combined with the real-time image captured by the electronic device to form a comprehensive data set containing multimodal data, which provides comprehensive context information for subsequent composition optimization, improves the system's ability to understand the scene, and makes composition optimization more accurate.

[0344] In some embodiments, the processor 1310 is further configured to:

[0345] Based on the composition optimization suggestion, generating auxiliary shooting operation information;

[0346] Outputting auxiliary shooting suggestions according to the auxiliary shooting operation information.

[0347] By outputting auxiliary shooting suggestions, it is possible for users to more easily understand how to operate to optimize the composition, thereby automatically or manually performing actual shooting.

[0348] In some embodiments, the processor 1310 is further configured to:

[0349] receiving a first input from a user;

[0350] In response to the first input, performing an auxiliary shooting operation according to the first input and the shooting auxiliary operation information;

[0351] Outputs an image acquired while performing an assist capture operation.

[0352] In the above embodiment, after the user accepts the suggestion or adjusts the auxiliary shooting operation, the electronic device can automatically perform the operation to adjust the composition and output the image data collected after the composition is adjusted, so that the user can quickly and intuitively see the image after the composition is adjusted, reduce user operations, and improve image processing efficiency.

[0353] In some embodiments, the processor 1310 is further configured to:

[0354] In a case where the composition optimization suggestion is a composition framework optimization suggestion, generating visual element position adjustment operation information based on positions of key visual elements in the composition data corresponding to the composition framework optimization suggestion;

[0355] In a case where the composition optimization suggestion is a visual focus relocation suggestion, generating focus position adjustment operation information based on a position of a focus in the composition data corresponding to the visual focus relocation suggestion;

[0356] In a case where the composition optimization suggestion is a shooting angle optimization suggestion, based on the optimized shooting angle in the composition data corresponding to the shooting angle optimization suggestion, analyzing the difference between the current shooting angle of the electronic device and the optimized shooting angle, and generating shooting angle adjustment operation information based on the difference;

[0357] When an aperture setting operation is required, the aperture optimization parameters are calculated based on the reference light intensity data in the composition data corresponding to the composition optimization suggestion, the current light intensity, aperture value and focal length in the real-time image, and the aperture setting operation information is generated based on the aperture optimization parameters.

[0358] The above-mentioned embodiment, on the basis of outputting composition optimization suggestions, can further help users know more accurately how to perform operations to implement the composition optimization suggestions based on the various operation contents corresponding to the generated composition optimization suggestions, or automatically execute the shooting operations corresponding to the composition optimization suggestions.

[0359] In some embodiments, the processor 1310 is further configured to:

[0360] Acquire a real-shot image actually shot and multimodal data of a shooting scene corresponding to the real-shot image;

[0361] Sending a model optimization request to the server, where the model optimization request carries the real-shot image and multimodal data of a shooting scene corresponding to the real-shot image;

[0362] The real-shot image and multimodal data of the shooting scene corresponding to the real-shot image are used to train the composition model.

[0363] Through the above-mentioned embodiments, high-order scene features are extracted from image data and multimodal data related to its shooting scene, and applied to subsequent optimization, effectively using multimodal data to characterize complex scenes, thereby providing a data-driven starting point for intelligently helping users to optimize the composition, improving the model's composition optimization capabilities, and effectively improving image processing efficiency.

[0364] It should be understood that in the embodiment of the present application, the input unit 1304 may include a graphics processor (Graphics Processing Unit, GPU) 1341 and a microphone 1342, and the graphics processor 1341 processes the image data of the static picture or video obtained by the image sensor (such as a camera) in the video capture mode or the image capture mode. The display unit 1306 may include a display panel 13061, and the display panel 13061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 1307 includes a touch panel 13071 and at least one of other input devices 13072. The touch panel 13071 is also called a touch screen. The touch panel 13071 may include two parts: a touch detection device and a touch controller. Other input devices 13072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0365] The memory 1309 can be used to store software programs and various data. The memory 1309 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 1309 may include a volatile memory or a non-volatile memory, or the memory x09 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 1309 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0366] The processor 1310 may include one or more processing units; optionally, the processor 1310 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 1310.

[0367] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the above-mentioned image processing method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0368] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0369] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned image processing method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0370] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0371] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned image processing method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0372] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0373] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling an electronic device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0374] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. An image processing method, characterized in that: Applied to electronic equipment, the method comprises: Acquire a real-time image and multimodal data of a shooting scene corresponding to the real-time image; Sending a composition optimization request to a server, wherein the composition optimization request carries the real-time image and the multimodal data; receiving the composition optimization suggestion returned by the server, and outputting the composition optimization suggestion; The composition optimization suggestion is obtained by extracting composition features of the real-time image and the multimodal data using a composition model and based on the composition features.

2. The method according to claim 1, characterized in that The acquiring of the real-time image and multimodal data of the shooting scene corresponding to the real-time image includes: Acquire real-time images collected by image sensors; The multimodal data is obtained by acquiring sensor data collected by at least one sensor of the electronic device in the shooting scene where the electronic device is located, and / or acquiring user data related to the shooting scene.

3. The method according to claim 1 or 2, characterized in that: The method further comprises: Based on the composition optimization suggestion, generating auxiliary shooting operation information; Outputting auxiliary shooting suggestions according to the auxiliary shooting operation information.

4. The method according to claim 3, characterized in that The method further comprises: receiving a first input from a user; In response to the first input, performing an auxiliary shooting operation according to the first input and the auxiliary shooting operation information; Outputs an image acquired while performing an assist capture operation.

5. The method according to claim 3, characterized in that: The generating auxiliary shooting operation information based on the composition optimization suggestion includes: In a case where the composition optimization suggestion is a composition framework optimization suggestion, generating visual element position adjustment operation information based on positions of key visual elements in the composition data corresponding to the composition framework optimization suggestion; In a case where the composition optimization suggestion is a visual focus relocation suggestion, generating focus position adjustment operation information based on a position of a focus in the composition data corresponding to the visual focus relocation suggestion; In a case where the composition optimization suggestion is a shooting angle optimization suggestion, based on the optimized shooting angle in the composition data corresponding to the shooting angle optimization suggestion, analyzing the difference between the current shooting angle of the electronic device and the optimized shooting angle, and generating shooting angle adjustment operation information based on the difference; When an aperture setting operation is required, the aperture optimization parameters are calculated based on the reference light intensity data in the composition data corresponding to the composition optimization suggestion, the current light intensity, aperture value and focal length in the real-time image, and the aperture setting operation information is generated based on the aperture optimization parameters.

6. The method according to claim 1, characterized in that The method further comprises: Acquire a real-shot image actually shot and multimodal data of a shooting scene corresponding to the real-shot image; Sending a model optimization request to a server, wherein the model optimization request carries the real-shot image and multimodal data of a shooting scene corresponding to the real-shot image; The real-shot image and the multimodal data of the corresponding shooting scene are used to train the composition model.

7. An image processing device, characterized in that: include: A real-time data acquisition module is configured to acquire a real-time image captured by an electronic device and multimodal data of a shooting scene corresponding to the real-time image; A composition suggestion generating module is configured to send a composition optimization request to a server, wherein the composition optimization request carries the real-time image and the multimodal data; The composition suggestion output module is configured to receive the composition optimization suggestion returned by the server and output the composition optimization suggestion.

8. The device according to claim 7, characterized in that The device also includes: A model optimization data acquisition module is configured to acquire a real-shot image actually shot by the electronic device and multimodal data related to a shooting scene corresponding to the real-shot image; The model optimization request module is configured to send a model optimization request to the server, wherein the model optimization request carries the real-shot image and multimodal data of the shooting scene corresponding to the real-shot image, and the real-shot image and the multimodal data of the corresponding shooting scene are used to train the composition model.

9. An image processing method, characterized in that: Applied to a server, the method comprises: Receiving a composition optimization request sent by an electronic device, wherein the composition optimization request carries a real-time image and multimodal data of a shooting scene corresponding to the real-time image; Extracting composition features of the real-time image and the multimodal data using a composition model, and obtaining composition optimization suggestions based on the composition features; The composition optimization suggestion is returned to the electronic device.

10. The method according to claim 9, characterized in that The utilizing the composition model to extract the composition features of the real-time image and the multimodal data, and obtaining composition optimization suggestions based on the composition features, includes: Receiving the real-time image and the corresponding multimodal data through an input layer; Extracting initial composition features of each composition element in the real-time image and the corresponding multimodal data through a primary feature extraction layer; The initial composition features output by the primary feature extraction layer are fused with multi-scale features through a multi-scale feature fusion layer to obtain fused composition features; Determining the weights corresponding to the various composition elements based on the attention mechanism through the feature enhancement layer, and weighting the fused composition features based on the weights corresponding to the various composition elements to obtain the weighted fused composition features; The scene understanding layer analyzes the composition changes and scene dynamics in the continuous image frames and their corresponding multimodal data based on the long short-term memory network to obtain enhanced composition features. The enhanced composition features output by the scene understanding layer are mapped to composition optimization suggestions through a fully connected layer to obtain composition optimization suggestions.

11. A model training method, characterized in that: include: Acquire a training data set, the training data set comprising: image samples and multimodal data samples of shooting scenes corresponding to the image samples; The initial composition model is trained using the training data set to obtain a trained composition model, and the composition model is used to predict composition optimization suggestions.

12. The method according to claim 11, characterized in that The composition model is a self-supervised learning large model, and the image samples are unlabeled image samples; The using the training data set to train the initial composition model to obtain the trained composition model includes: Inputting the image samples in the training data set and the corresponding multimodal data samples into the composition model; Extracting composition features of the image sample and the corresponding multimodal data sample through the composition model; Calculating loss values ​​of composition features of the image sample and the corresponding multimodal data sample through the loss function of the composition model; Wherein, the loss function at least includes a contrast loss function term, and the contrast loss function term is used to calculate the contrast loss value of the composition feature of the image sample and the corresponding multimodal data sample; The model parameters and the weights of the composition elements in the composition features are adjusted based on the loss value.

13. The method according to claim 12, characterized in that The loss function also includes: an image feature similarity loss function term and a multimodal feature similarity loss function term; The image feature similarity loss function term is used to calculate a first similarity between an initial composition feature corresponding to the image sample and a corresponding weighted composition feature through a similarity function, and determine an image feature loss value based on the first similarity; The multimodal feature similarity loss function term is used to calculate the second similarity between the initial composition feature corresponding to the multimodal data sample and the corresponding weighted composition feature through a similarity function, and determine the multimodal feature loss value based on the second similarity.

14. The method according to claim 13, characterized in that The method further comprises: Receiving a model optimization request sent by an electronic device, the model optimization request carrying a real-shot image actually shot by the electronic device and multimodal data of a shooting scene corresponding to the real-shot image; The composition model is trained using the real-shot image and multimodal data of the shooting scene corresponding to the real-shot image to obtain an optimized composition model.

15. An image processing device, characterized in that: include: A composition optimization request receiving module, configured to receive a composition optimization request sent by an electronic device, wherein the composition optimization request carries a real-time image and multimodal data related to a shooting scene corresponding to the real-time image; A composition model processing module, configured to extract composition features of the real-time image and the multimodal data using a composition model, and obtain composition optimization suggestions based on the composition features; The composition optimization suggestion returning module is configured to return the composition optimization suggestion to the electronic device.

16. A model training device, characterized in that: include: A training data set acquisition module is configured to acquire a training data set, wherein the training data set includes: image samples and multimodal data samples of shooting scenes corresponding to the image samples; The composition model training module is configured to use the training data set to train the initial composition model to obtain a trained composition model, and the composition model is used to predict composition optimization suggestions.

17. An image processing method, characterized in that: Applied to electronic equipment, the method comprises: Acquire a real-time image and multimodal data of a shooting scene corresponding to the real-time image; Extracting composition features of the real-time image and the multimodal data using a composition model, and obtaining composition optimization suggestions based on the composition features; The composition optimization suggestion is output.

18. An image processing device, characterized in that: include: A real-time data acquisition module, configured to acquire a real-time image acquired by an electronic device and multimodal data of a shooting scene corresponding to the real-time image; A local model execution module is configured to extract composition features of the real-time image and the multimodal data using a composition model, and obtain composition optimization suggestions based on the composition features; The composition suggestion output module is configured to output the composition optimization suggestion.

19. An electronic device, characterized in that: It includes a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, it implements the steps of the image processing method according to any one of claims 1 to 6, claims 9 to 10, or claim 17, or the model training method according to any one of claims 11 to 14.

20. A readable storage medium, characterized in that: The readable storage medium stores programs or instructions, and when the programs or instructions are executed by the processor, the steps of the image processing method according to any one of claims 1-6, claims 9-10, or claim 17, or the model training method according to any one of claims 11-14 are implemented.

Images