Image processing method and electronic device

Abstract

Embodiments of the present application apply to the field of artificial intelligence, and provide an image processing method and an electronic device. The method comprises: determining a target subject region in which a target subject is located in a to-be-processed image according to description information obtained based on a first user operation, the description information being used to describe the target subject in the to-be-processed image, and an edge of the target subject region being an edge of the target subject in the to-be-processed image; determining a plurality of candidate images in the to-be-processed image, and respectively scoring the plurality of candidate images to obtain an image score of each candidate image, a candidate region in which each candidate image is located overlapping with the target subject region; and determining a target image from the plurality of candidate images according to the image score of each candidate image in the plurality of candidate images. Based on the technical method of the present application, a fine target image meeting user demand can be obtained, and user experience is improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and more specifically, to an image processing method and an electronic device. Background Technology

[0002] As electronic devices become increasingly feature-rich, they have become indispensable communication and entertainment tools in people's work and daily lives. People can use the photo and video recording functions of mobile phones and other electronic devices to collect media data and view the captured images. They can also use the editing functions of electronic devices to edit and process media data.

[0003] When users acquire images using electronic devices, the resulting cropped images may not meet their needs, leading to low user satisfaction. Therefore, how to crop images to obtain high-quality images that meet user requirements is a pressing issue that needs to be addressed. Summary of the Invention

[0004] This application provides an image processing method and electronic device that enables the target image determined in the image to be processed to better meet user needs and improve user experience.

[0005] In a first aspect, an image processing method is provided, comprising: acquiring an image to be processed and descriptive information obtained based on a first user operation, the descriptive information being used to describe a target subject in the image to be processed; determining a target subject region in the image to be processed based on the descriptive information, wherein the edge of the target subject region is the edge of the target subject in the image to be processed; determining a plurality of candidate images in the image to be processed, and scoring the plurality of candidate images respectively to obtain an image score for each candidate image, wherein the candidate region of each candidate image overlaps with the target subject region; and determining a target image from the plurality of candidate images based on the image score of each candidate image.

[0006] The image processing method provided in this application determines a target image in the image to be processed based on the user's description information. The target image is the image with the highest image score among multiple candidate images. The region where each candidate image is located in the image to be processed overlaps with the region of the target subject described in the user's description information in the image to be processed. Therefore, the determined target image is more in line with the user's needs and improves the user experience.

[0007] Determining the target image from the image to be processed can be understood as cropping the image to be processed. The target image can be understood as the cropped image.

[0008] In some possible implementations, the method further includes: displaying a first interface, the first interface including the image to be processed; displaying a second interface in response to a second user operation in the first interface; and obtaining the image to be processed and the description information obtained based on the first user operation, including: determining the description information according to the first user operation in the second interface.

[0009] In some possible implementations, scoring the plurality of candidate images to obtain an image score for each candidate image includes: scoring each candidate image in multiple dimensions to obtain a dimensional score for each candidate image in each dimension; determining the image score for each candidate image based on the dimensional score for each candidate image in each dimension, wherein the image score for each candidate image is positively correlated with the dimensional score for each candidate image in each dimension.

[0010] The image score of a candidate image is based on its dimensional scores in multiple dimensions. The image score of a candidate image takes into account the influence of the dimensional scores in multiple dimensions, making the target image more refined and improving the user experience.

[0011] In some possible implementations, scoring each candidate image across multiple dimensions to obtain a dimensional score for each candidate image in each dimension includes: extracting features from the multiple candidate images using an image feature extraction model to obtain image features for each candidate image; determining the dimensional score of the candidate image in the j-th dimension based on the difference between the image features of each candidate image and the j-th dimension feature corresponding to the j-th dimension among the multiple dimensions, wherein the j-th dimension feature corresponding to the j-th dimension is obtained by extracting features from the j-th dimension text using a text feature extraction model, where j is a positive integer, and the j-th dimension text is used to describe the j-th dimension.

[0012] Based on the differences between the image features of the candidate image and the dimensional features of each dimension, the dimensional score of the candidate image in each dimension is determined, making the dimensional score more accurate.

[0013] In some possible implementations, the image feature extraction model and the text feature extraction model are trained based on multiple training sample data. The multiple training sample data includes multiple training samples and label information for each training sample. The multiple training samples correspond to the multiple dimensions. Each training sample includes a first training image, a second training image, and corresponding dimensional text. The label information of each training sample indicates whether the first training image in the training sample is superior to the second training image in the training sample. The image feature extraction model and the text feature extraction model are obtained by adjusting the parameters of the initial image feature extraction model and the initial text feature extraction model respectively based on the difference between the label information of each training sample and the training information of the training sample. The training information of the training samples is based on the first training image-text difference of the training samples. The magnitude relationship between the difference between the first training image and the second training text in the training samples is determined. The training information of the training samples indicates whether the first training image in the training samples is better than the second training image in the training samples. The first training image-text difference of the training samples is the difference between the features of the first training image and the features of the training text. The second training image-text difference of the training samples is the difference between the features of the second training image and the features of the training text. The first training image features are obtained by the initial image feature extraction model from the first training image in the training samples. The second training image features are obtained by the initial image feature extraction model from the second training image in the training samples. The training text features are obtained by the initial image feature extraction model from the dimensional text in the training samples.

[0014] In image processing and computer vision, evaluating images in a particular dimension is often a subjective and complex task because it involves human perception and judgment. Directly using scores to represent the quality of each dimension can be influenced by many factors, and the scores may contain errors. However, an observer can intuitively perceive which of two images is better in a given dimension. During neural network training, using the comparison results of two images in a certain dimension as label information can improve the accuracy of the dimensional scores of candidate images obtained from the trained image feature extraction model in that dimension.

[0015] In some possible implementations, the multiple dimensions include multiple of the following image evaluation dimensions: overall image effect, lighting, content, background, foreground, and composition.

[0016] In some possible implementations, determining multiple candidate images in the image to be processed and scoring the multiple candidate images respectively includes: determining an image score for each candidate image based on a reference image selected by the user, wherein the structural similarity between the candidate image and the reference image is positively correlated with the image score of the candidate image.

[0017] In some possible implementations, determining the image score of each candidate image based on the reference image includes: for any one of the plurality of candidate images, performing feature extraction on the candidate image using an image structure feature extraction model to obtain candidate structure features of the candidate image; determining the image score of the candidate image based on the structural feature difference between the candidate structure features and the reference structure features, wherein the reference structure features are obtained by performing feature extraction on the reference image using the image structure feature extraction model, and the structural similarity between the candidate image and the reference image is negatively correlated with the structural feature difference.

[0018] The image score of the candidate image is determined based on the difference between the candidate structural features of the candidate image and the reference structural features determined based on the reference image, so that the image score is more accurate.

[0019] In some possible implementations, the text structure feature extraction model and the image structure feature extraction model are obtained by adjusting the parameters of the initial text structure feature extraction model and the initial image structure feature extraction model based on the differences between the training text structure features and the training image structure features; the training text structure features are obtained by the initial text structure feature extraction model extracting features from the training structure text, and the training image structure features are obtained by the initial image structure feature extraction model extracting features from the training candidate images; the training structure text is used to describe the structure of the training candidate images.

[0020] In some possible implementations, before determining multiple candidate images in the image to be processed and scoring the multiple candidate images respectively, the method further includes: displaying a third interface, the third interface including multiple candidate images; receiving a user selection operation in the third interface, wherein the candidate image selected by the user selection operation is the reference image.

[0021] In some possible implementations, the step of determining multiple candidate images in the image to be processed and scoring each of the multiple candidate images to obtain an image score for each candidate image includes: determining a first image set in the image to be processed, the first image set including at least one of the candidate images; performing multiple iterations, wherein the i-th iteration includes: scoring each candidate image in the i-th image set to obtain an image score for the candidate image; determining an (i+1)-th image set based on the image scores of each candidate image in the i-th image set, the (i+1)-th image set including at least one of the candidate images, i being a positive integer, the maximum value of the image score of at least one candidate image in the (i+1)-th image set being greater than or equal to the maximum value of the image score of at least one candidate image in the i-th image set, and the i-th image set being the first image set when i equals 1; and ending the iteration operation when the iteration termination condition is met.

[0022] In some possible implementations, before performing multiple iterations, the method further includes: encoding the region information of the area where each candidate image in the first image set is located to obtain a chromosome corresponding to each candidate image, wherein the first chromosome set includes the chromosome corresponding to each candidate image in the first image set; determining the (i+1)th image set based on the image score of each candidate image in the i-th image set includes: performing a target operation on the i-th chromosome set to obtain the (i+1)th chromosome set, wherein the target operation includes one or more of selection, crossover, and mutation, wherein the probability of any chromosome in the i-th chromosome set being selected as a chromosome in the (i+1)th chromosome set is positively correlated with the image score of the candidate image corresponding to any chromosome, and the i-th chromosome set is the first chromosome set when i equals 1; decoding each chromosome in the (i+1)th chromosome set to obtain the region information of the area where each candidate image in the (i+1)th chromosome set is located; determining the (i+1)th image set based on the region information of the candidate images corresponding to each chromosome in the (i+1)th chromosome set, wherein the (i+1)th image set includes the candidate images corresponding to each chromosome in the (i+1)th chromosome set.

[0023] By using a genetic algorithm, candidate images are selected from the images to be processed, thereby determining the target image with the highest image score, making the target image more accurate.

[0024] In some possible implementations, determining the target subject region in the image to be processed based on the description information obtained based on the first user operation includes: determining multiple candidate subjects in the image to be processed based on the description information; determining the target subject region based on the depth of each of the multiple candidate subjects, wherein the subject in the target subject region is the subject with the smallest depth among the multiple candidate subjects.

[0025] Generally, when taking a picture, the subject that the user focuses on is located close to the user, i.e., at a shallow depth. When there are multiple candidate subjects, the subject with the shallowest depth is selected as the target subject, making the target image determined based on the target subject more in line with the user's needs.

[0026] In some possible implementations, the candidate region where the target image is located is a portion of the image to be processed, and the method further includes: adjusting the color of the target image to achieve white balance.

[0027] When the image to be processed is a color-adjusted image, the target image is only a portion of that image, and the target image may not meet white balance requirements. Adjusting the colors of the target image can achieve white balance, thereby improving the user experience.

[0028] When the image to be processed is a color-adjusted image, only the target image is color-adjusted, which improves the efficiency of image processing and enhances the user experience.

[0029] In some possible implementations, the method is applied to an electronic device, where the image to be processed is a preview image captured by the electronic device through a camera.

[0030] In other words, acquiring the image to be processed can be done by using a camera to capture images.

[0031] Secondly, a method for training an image structure feature extraction model is provided, comprising: acquiring training structure text and training structure image, wherein the training structure text is used to describe the structure of the training structure image; extracting features from the training structure text using an initial text structure feature extraction model to obtain training text structure features; extracting features from the training structure image using an initial image structure feature extraction model to obtain training image structure features; and adjusting the parameters of the initial text structure feature extraction model and the initial image structure feature extraction model according to the difference between the training text structure features and the training image structure features, wherein the adjusted initial image structure feature extraction model is the image structure feature extraction model.

[0032] Thirdly, a method for training an image feature extraction model is provided, comprising: acquiring multiple training samples and label information for each training sample, wherein the multiple training samples correspond to multiple dimensions, each training sample includes a first training image, a second training image, and dimensional text corresponding to the corresponding dimension, and the label information of each training sample represents the first training image in the corresponding dimension as superior to the second training image in the same training sample; processing the first training image and the second training image in the multiple training samples using an initial image feature extraction model to obtain the first training image feature and the second training image feature of the first training image in each training sample; and processing the dimensional text in the multiple training samples using an initial text feature extraction model to obtain the dimensional text of each training sample. The training text features of the dimensional text in the training sample are obtained. For each training sample, the training information of the training sample is determined based on the first training image features, the second training image features, and the training text features. The training information of the training sample is determined based on the magnitude relationship between the first training image-text difference and the second training image-text difference. The first training image-text difference is the difference between the first training image features and the training text features, and the second training image-text difference is the difference between the second training image features and the training text features. The training information of the training sample indicates that the first training image in the training sample is better than the second training image in the training sample. Based on the label information of each training text and the difference between the training information of the training text, the parameters of the initial text feature extraction model and the initial image feature extraction model are adjusted. The adjusted initial image feature extraction model is the image feature extraction model.

[0033] Fourthly, an image processing apparatus is provided, including a unit for performing the method of at least one of the first to third aspects. The apparatus may be a terminal device or a chip within a terminal device.

[0034] Fifthly, an electronic device is provided, including one or more processors and a memory coupled to the one or more processors, the memory being used to store computer program code including computer instructions, wherein the one or more processors invoke the computer instructions to cause the electronic device to perform a method of at least one of the first to third aspects.

[0035] In a sixth aspect, a chip system is provided, the chip system being applied to an electronic device, the chip system including one or more processors, the one or more processors being configured to invoke computer instructions to cause the electronic device to perform the methods of at least one of the first to third aspects.

[0036] In a seventh aspect, a computer-readable storage medium is provided, the computer-readable storage medium including instructions that, when executed on an electronic device, cause the electronic device to perform the method of at least one of the first to third aspects.

[0037] Eighthly, a computer program product is provided, the computer program product comprising: computer program code, which, when executed on an electronic device, performs the method of at least one of the first to third aspects. Attached Figure Description

[0038] Figures 1 to 4 This is a schematic diagram of the graphical user interface provided in an embodiment of this application;

[0039] Figure 5 This is a schematic structural diagram of an image processing system provided in an embodiment of this application;

[0040] Figure 6 This is a schematic structural diagram of the subject selection model in the image processing system provided in the embodiments of this application;

[0041] Figure 7 This is a schematic flowchart of an image processing method provided in an embodiment of this application;

[0042] Figure 8 This is a schematic flowchart of another image processing method provided in the embodiments of this application;

[0043] Figure 9 This is a schematic flowchart of another image processing method provided in the embodiments of this application;

[0044] Figure 10 This is a schematic flowchart of another image processing method provided in the embodiments of this application;

[0045] Figure 11 This is a schematic flowchart illustrating an image structure feature extraction model training method provided in an embodiment of this application;

[0046] Figure 12 This is a schematic flowchart illustrating an image feature extraction model training method provided in an embodiment of this application;

[0047] Figure 13 This is a schematic diagram of the training process of the dimensional scoring model provided in the embodiments of this application;

[0048] Figure 14 This is a schematic structural diagram of a dimensional scoring model provided in an embodiment of this application;

[0049] Figure 15This is a schematic structural diagram of the electronic device provided in the embodiments of this application;

[0050] Figure 16 This is a schematic diagram of the software system of the electronic device provided in the embodiments of this application;

[0051] Figure 17 This is a schematic structural diagram of an image processing apparatus provided in an embodiment of this application. Detailed Implementation

[0052] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0053] It should be understood that "multiple" as mentioned in this application refers to two or more. In the description of this application, unless otherwise stated, " / " indicates "or," for example, A / B can mean A or B; "and / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist, for example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Furthermore, to facilitate a clear description of the technical solutions of this application, the terms "first," "second," etc., are used to distinguish identical or similar items with essentially the same function and effect. Those skilled in the art will understand that the terms "first," "second," etc., do not limit the quantity or execution order, and that "first," "second," etc., do not necessarily imply differences.

[0054] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0055] To facilitate understanding, before introducing the image processing method provided in the embodiments of this application, the terms or nouns involved in the embodiments of this application will be briefly introduced.

[0056] (1) Artificial Intelligence

[0057] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0058] (2) Machine Learning

[0059] Machine learning is an important branch of artificial intelligence, and deep learning is an important branch of machine learning. Deep learning refers to using multi-layer neural network structures to learn representations of various things in the real world that can be directly used for computer calculations (such as objects in images, sounds in audio, etc.) from big data.

[0060] (3) Neural Network

[0061] Neural networks can be composed of neural units, which can refer to units represented by x. s The arithmetic unit that takes an intercept of 1 as input can output the following:

[0062]

[0063] Where s = 1, 2, ..., n, n is a natural number greater than 1, W s For x s The weights are denoted by b, where b is the bias of the neural unit. f is the activation function of the neural unit, used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into an output signal. The output signal of this activation function can be used as the input to the next convolutional layer. The activation function can be the sigmoid function. A neural network is a network formed by connecting many of the above-mentioned individual neural units together, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field, which can be a region composed of several neural units.

[0064] (4) Deep Neural Networks

[0065] Deep neural networks (DNNs), also known as multi-layer neural networks, can be understood as neural networks with multiple hidden layers. Based on the position of the layers, DNNs can be divided into three categories: input layers, hidden layers, and output layers. Generally, the first layer is the input layer, the last layer is the output layer, and the layers in between are hidden layers. The layers are fully connected, meaning that any neuron in the i-th layer is connected to any neuron in the (i+1)-th layer.

[0066] In deep neural networks, more hidden layers allow the network to better depict complex real-world situations. Theoretically, the more parameters a model has, the higher its complexity and "capacity," meaning it can accomplish more complex learning tasks. Training a deep neural network is essentially the process of learning the weight matrix, with the ultimate goal of obtaining the weight matrix of all layers in the trained deep neural network (a weight matrix formed by vectors from many layers).

[0067] (5) Convolutional Neural Networks

[0068] A convolutional neural network (CNN) is a deep neural network with a convolutional structure. A CNN contains a feature extractor consisting of convolutional layers and subsampling layers. This feature extractor can be viewed as a filter, and the convolution process can be seen as using a trainable filter to convolve with an input image or a convolutional feature map. A convolutional layer is a layer of neurons in a CNN that performs convolution processing on the input signal. In a convolutional layer of a CNN, a neuron may only be connected to some of its neighboring neurons. A convolutional layer typically contains several feature maps, each composed of rectangularly arranged neural units. Neural units on the same feature map share weights, which are the convolutional kernel. Shared weights can be understood as the way image information is extracted regardless of location. The underlying principle is that the statistical information of one part of the image is the same as that of other parts. This means that image information learned in one part can also be used in another part. Therefore, the same learned image information can be used for all locations in the image. In the same convolutional layer, multiple convolutional kernels can be used to extract different image information. Generally, the more convolutional kernels there are, the richer the image information reflected by the convolution operation.

[0069] When a convolutional neural network has multiple convolutional layers, the initial convolutional layers tend to extract more general features, which can also be called low-level features. As the depth of the convolutional neural network increases, the features extracted by later convolutional layers become more and more complex, such as high-level semantic features. The higher the level of the features, the more suitable they are for the problem to be solved.

[0070] (6) Loss Function

[0071] In training a deep neural network, to ensure the output closely approximates the desired predicted value, we compare the network's prediction with the target value. Based on the difference, we update the weight vector of each layer (usually pre-configuring parameters before the initial update). For example, if the prediction is too high, the weight vector is adjusted to predict a lower value. This adjustment continues until the deep neural network predicts the target value or a value very close to it. Therefore, we need to predefine "how to compare the difference between the predicted and target values," which is the loss function or objective function. These are important equations used to measure the difference between the predicted and target values. Taking the loss function as an example, a higher output value (loss) indicates a greater difference, and training the deep neural network becomes a process of minimizing this loss.

[0072] (7) Backpropagation algorithm

[0073] Convolutional neural networks can employ backpropagation (BP) to correct the parameters in the initial super-resolution model during training, thereby reducing the reconstruction error loss. Specifically, forward propagation of the input signal to the output generates an error loss; this error loss information is then propagated back to update the parameters in the initial super-resolution model, leading to convergence of the error loss. The backpropagation algorithm is an error-loss-driven backpropagation process aimed at obtaining the optimal parameters of the super-resolution model, such as the weight matrix.

[0074] To improve the aesthetics of an image, it can be cropped.

[0075] Figure 1Image (a) illustrates a graphical user interface (GUI) for an electronic device, specifically a first album interface 1110. Upon detecting a user's click on any one of the thumbnails on the first album interface 1110, the electronic device can display, as shown below... Figure 1 The second album interface 1120 is shown in (b) above. The second album interface 1120 includes a full-size image 1121 corresponding to the thumbnail clicked by the user. The second album interface 1120 also includes an auto-cropping icon 1122.

[0076] When the electronic device detects that the user clicks the auto-cropping icon 1122, it can perform image processing on the full-size image 1121 to obtain the cropped image.

[0077] When an electronic device processes an image to obtain a cropped image, it can be displayed. Figure 1 The third album interface 1130 is shown in (c). The third album interface 1130 may include cropped images 1131.

[0078] However, the image cropping process did not take user needs into account, resulting in the cropped image 1131 not matching the user's requirements.

[0079] To improve user experience, this application provides an image processing method. In this method, an electronic device can determine descriptive information and a reference image based on user operations, and then crop the image to be processed based on the descriptive information and the reference image. Thus, during the image cropping process, user needs are considered, increasing user satisfaction with the cropped image.

[0080] The descriptive information describes the target subject in the image to be processed. The cropped target image may overlap with the target region containing the target subject. The reference image can be used to represent the image structure desired by the user. The following section combines... Figure 2 Let's take the descriptive information as an example to illustrate.

[0081] When the electronic device detects that the user has clicked the auto-crop icon 1122 in the second album interface 1120, it can display as follows: Figure 2The first image processing settings interface 1210 is shown in (a). The image initial settings interface 1210 includes an input box 1211 and an OK icon 1212. The user can enter descriptive text in the input box 1211. The image initial settings interface 1210 may also include multiple alternative texts 1212a to 1212l. The user can click on any of the alternative texts as the descriptive text. Alternatively, if the electronic device detects that a user has clicked on an alternative text, it can fill in the user-clicked alternative text in the input box 1211 as the descriptive text.

[0082] Upon detecting that the user has clicked the OK icon 1212, the electronic device can display as follows: Figure 2 The second image processing settings interface 1220 is shown in (b) of the diagram. The second image processing settings interface 1220 includes thumbnails of multiple candidate images.

[0083] When the electronic device detects that a user has clicked on a thumbnail of a candidate image, it can use that candidate image as a reference image, use the full-size image 1121 in the second album interface 1120 as the image to be processed, and perform image processing on the image to be processed based on the descriptive text and the reference image to obtain the target image. The target image is the cropped image 1131.

[0084] The image processing method provided in this application embodiment can also be applied to scenarios such as capturing exciting moments, automatic capture, and photo assistance.

[0085] The following is combined with Figure 3 The text describes the scenes captured in the video.

[0086] Figure 3 (a) shows a GUI of an electronic device, which is a third camera interface 1310 of the electronic device. The third camera interface 1310 includes a preview box 1311, a crop off icon 1312, a snapshot on icon 1313, and a camera icon 1314.

[0087] The "Capture Highlights" icon 1313 indicates that the Capture Highlights function is enabled. The "Crop Off" icon 1312 indicates that the automatic cropping function is disabled. When the Capture Highlights function is enabled and the automatic cropping function is disabled, if the electronic device detects a user clicking the camera icon 1314, it scores multiple images captured by the camera at the moment of the click, within a first time period before the click, and within a second time period after the click, to obtain a score for each captured image. The image with the highest score can be used as the captured image. The captured image can also be called a captured photo.

[0088] With the snapshot function enabled and the automatic cropping function disabled, if the electronic device detects that the user clicks the cropping off icon 1312, it can then enable the automatic cropping function. After the automatic cropping function is enabled, the electronic device can display the following: Figure 3 The fourth camera interface 1320 is shown in (b) above. The fourth camera interface 1320 includes a preview frame 1311, a crop icon 1322, a snapshot icon 1313, and a camera icon 1314. The crop icon 1322 indicates that the automatic cropping function is on.

[0089] When the system detects that the user clicks the camera icon 1314 on the fourth camera interface 1320, the electronic device crops multiple images captured by the camera at the moment the user clicks the camera icon, as well as during a first time period before the click and a second time period after the click. Each cropped image is then scored. The image with the highest score can be used as the captured image.

[0090] With the snapshot function enabled and the automatic cropping function disabled (i.e., the electronic device displays the third camera interface 1310), if the electronic device detects that the user has clicked the cropping off icon 1312, the electronic device can then enable the automatic cropping function. The electronic device can display... Figure 2 The first image processing settings interface 1210 shown in (a) can display the following when the electronic device detects that the user has clicked the OK icon 1212 in the first image processing settings interface 1210: Figure 2 The second image processing settings interface 1220 is shown in (b) above. When the electronic device detects that a user clicks on a thumbnail or skip icon 1221 of an alternative image in the second image processing settings interface 1220, it can display, as shown below... Figure 3 The fourth camera interface 1320 is shown in (b) of the image.

[0091] The fourth camera interface 1320 includes a preview frame 1311, a crop icon 1322, a snapshot icon 1313, and a camera icon 1314. The crop icon 1322 indicates that the automatic cropping function is on.

[0092] When the user clicks the camera icon 1314 in the fourth camera interface 1320, the electronic device can process all or part of the multiple images captured by the camera at the moment the user clicks the camera icon 1314, as well as the first time period before the moment of clicking and the second time period after the moment of clicking, as images to be processed to obtain the target image corresponding to each image to be processed.

[0093] Electronic device display Figure 3The fourth album interface 1330 is shown in (c). The fourth album interface 1330 may include captured images 1331.

[0094] For example, images captured by the camera at a time when the time between the capture and the click is an integer multiple of a preset time can be used as images to be processed. The image scores of multiple target images corresponding to multiple images to be processed can be compared. The target image with the highest image score can be used as the captured image.

[0095] When the electronic device detects that the user has clicked the album icon 1315 in the fourth camera interface 1320, it displays... Figure 3 The fourth album interface 1330 is shown in (c). The fourth album interface 1330 may include captured images 1331.

[0096] In scenarios with automatic image capture, when both automatic capture and automatic cropping functions are enabled, and the camera is capturing images of a scene, the electronic device can process all or part of the captured images as images to obtain the target image corresponding to each image to be processed. If the camera stops capturing images, or if the scene being captured by the camera changes, the image with the highest image score among the target images corresponding to multiple images to be processed can be used as the captured image.

[0097] When the image to be processed is an image captured by a camera, image processing is performed on the image to be processed, and the resulting target image can also be called a photographic reconstruction image.

[0098] The following is combined with Figure 4 This section explains the scenarios where photo preview assists in composition.

[0099] Figure 4 Image (a) illustrates a GUI for an electronic device, which is a first camera interface 1410. The first camera interface 1410 includes a preview box 1411 and a camera assistance icon 1412. The preview box 1411 is used to display images captured in real time by the camera in the electronic device.

[0100] The first camera interface 1410 may also include guide lines 1413a to 1413d or other guide lines in the preview frame 1411.

[0101] The preview frame 1411 is rectangular. Guide lines 1413a and 1413b are parallel to the bottom edge of the preview frame 1411, and they divide the preview frame 1411 into two equal parts. Guide lines 1413c and 1413d are parallel to the left edge of the preview frame 1411, and they divide the preview frame 1411 into three equal parts.

[0102] The guide lines in the preview frame can help users place their subject in key positions within the frame to create a more visually appealing image. However, if users don't know how to utilize these guide lines, they can simply be a visual distraction.

[0103] When the electronic device detects that the user has clicked the camera assistance off icon 1412, it can enable the camera preview and composition assistance function.

[0104] Electronic devices can display Figure 2 The first image processing settings interface 1210 shown in (a) can display the following when the electronic device detects that the user has clicked the OK icon 1212 in the first image processing settings interface 1210: Figure 2 The second image processing settings interface 1220 is shown in (b) above. When the electronic device detects that a user clicks on a thumbnail or skip icon 1221 of an alternative image in the second image processing settings interface 1220, it can display... Figure 4 The second camera interface 1420 is shown in (b) above. The second camera interface 1420 includes a preview frame 1411 and a camera assistance icon 1422.

[0105] When the electronic device displays the second camera interface 1420, the electronic device can periodically or non-periodically use the images captured by the camera as images to be processed. Figure 10 The method shown, or using Figure 5 The image processing system shown performs image processing to obtain the target image.

[0106] When the electronic device processes and obtains the target image, the second image-taking interface 1420 may also include a frame line 1423. The frame line 1423 can be understood as the edge of the target image. That is, the image located within the frame line 1423 can be understood as the target image. The target image can be understood as the best composition among the images in the preview frame 1411, i.e., the image with the highest image score.

[0107] Image processing by electronic devices takes a certain amount of time. During this processing, the scene in which the camera captures the image may change. In other words, the image in preview frame 1411 may change. When the electronic device obtains the target image, it can determine whether the first captured image (the image to be processed) has changed compared to the second captured image captured by the camera when the target image is obtained. If the determination result is negative, the electronic device can determine that the scene in which the camera captures the image has not changed.

[0108] The electronic device can calculate the perceptual similarity between the first captured image and the second captured image. If the calculated perceptual similarity is greater than or equal to a preset similarity, the electronic device can determine that the scene in which the camera captures the image has not changed, and the second image capture interface 1420 may include a frame line 1423. If the calculated perceptual similarity is less than the preset similarity, the electronic device can determine that the scene in which the camera captures the image has changed, and the second image capture interface 1420 may not include the frame line 1423. The preset similarity can be set by the user according to their needs or it can be a default value set by the electronic device. Perceptual similarity, also known as visual similarity, refers to the degree of similarity between two images in terms of visual features.

[0109] Users can adjust the camera's pose and / or focal length based on the frame line 1423 to ensure that each side of the frame line 1423 is as close as possible to the edge of the preview frame 1411, thereby improving the clarity of the target image.

[0110] When the electronic device detects that the user clicks the camera icon 1414 in the second camera interface 1420, the electronic device can store the image displayed in real time in the preview box 1411, and the electronic device can also store the target image obtained by using the image displayed in real time in the preview box 1411 as the image to be processed.

[0111] When the electronic device detects that the user has clicked the camera assistance icon 1422 on the second camera interface 1420, it can disable the camera preview and composition assistance function and display... Figure 4 The first camera interface 1410 is shown in (a) of the image.

[0112] The image processing method provided in this application embodiment can be used to... Figure 5 The image processing system shown is implemented as described.

[0113] Figure 5 This is a schematic structural diagram of an image processing system provided in an embodiment of this application.

[0114] The image processing system 400 includes a subject selection model 410, an optimization module 420, and a color adjustment module 430.

[0115] The subject selection module 410 is used to process the image to be processed, the descriptive information, and the depth image of the image to be processed to obtain the target subject region.

[0116] The description information can be information entered by the user or information determined based on the user input. The description information can include one or more of the following: description text, description area, and description point.

[0117] The descriptive text can be text used to describe the target subject. It can be user-selected preset text, such as descriptive statements from poetic Tang poems, Song lyrics, Yuan songs, modern poems, or prose. Alternatively, it can be a user's description of the target subject based on their preferences, such as keywords or statements obtained through keyword searches or editing.

[0118] Descriptive text can be used to describe the characteristics of the target subject itself. For example, descriptive text may include descriptions of gender, type of the target subject, name, etc. Descriptive text may also be used to describe the characteristics of items carried by the target subject, such as hairstyle, clothing, clothing color, and the items held. Descriptive text can also indirectly describe the target subject by describing other related objects. For example, descriptive text can describe the relative or absolute position of the target subject, or the relationship between the target subject and other subjects. For instance, descriptive text could be "a child feeding a puppy," meaning the descriptive text can indicate that the "child" is the target subject by describing the relationship between the puppy and the child.

[0119] The descriptive point can be a point located on the target body.

[0120] The description region may overlap with all or part of the target subject in the image to be processed. For example, the area of ​​the target subject within the description region may be larger than the area of ​​the target subject outside the description region; that is, the target subject is primarily located within the description region. For example, the target subject may be entirely located within the description region of the image to be processed.

[0121] The shape of the description region can be regular or irregular. When the description region has a preset regular shape, its position in the image to be processed can be described by text. The preset regular shape can be a circle, ellipse, triangle, or rectangle, etc. When the preset shape is rectangular, the description region can be called a description box.

[0122] The description region can also be represented by a description mask. A mask can be used to occlude all or part of the image being processed, thereby controlling the area or process of image processing. The description mask can be represented as a multi-valued image or a binary matrix, etc.

[0123] A descriptive mask is used to represent the approximate location of a target object in an image to be processed. For example, a descriptive mask may include multiple location points or at least one region. A mask, also called a mapping mask, can be represented as an image. The shape of the regions in the descriptive mask can be regular or irregular.

[0124] These multiple location points may include the click location where the user clicks on the target subject in the image to be processed displayed on the electronic device, or multiple points whose distance from the click location is less than or equal to a preset distance.

[0125] The at least one region may include a circle with the click location where the user clicks on the target subject in the image to be processed displayed on the electronic device as the center and a preset distance as the radius. Alternatively, the at least one region may include the area containing points on both sides of a curve that the user slides on the target subject in the image to be processed displayed on the electronic device, and the distance from the curve to the curve is less than a preset distance.

[0126] The target subject region can be used to represent the target subject in the image to be processed. The target subject represented by the target subject region can be a subject that matches the descriptive information and has a relatively shallow depth.

[0127] The optimization module 420 is used to optimize based on the target subject region to obtain the target image in the image to be processed. The target region of the target image in the image to be processed overlaps with the target subject region. When the user selects a reference image, the target image and the reference image can have a high structural similarity. Thus, in the process of image processing using the image processing system 400, the user's personalized composition needs are fully considered, improving the user experience.

[0128] The color adjustment module 430 is used to optimize the color of the target image to obtain the optimized target image.

[0129] The image to be processed can be an image captured by a camera in the electronic device where the image processing system 400 is located, an image stored in the electronic device, or an image sent by other electronic devices received by the electronic device.

[0130] The subject selection model 410 may include a subject determination model 411 and a selection module 412.

[0131] The subject determination model 411 is used to process the image to be processed and the descriptive information to obtain at least one candidate subject region.

[0132] Different candidate subject regions in the image to be processed represent different candidate subjects in the image. At least one candidate subject in the image to be processed can be understood as a subject that conforms to this descriptive information.

[0133] The selection module 412 is used to determine a target subject from at least one candidate subject based on the depth image of the image to be processed. The target subject may be one or more subjects with the shallowest depth among the at least one candidate subject. The subject with the shallowest depth can be understood as being closest to the electronic device that captured the image to be processed.

[0134] The main function of the subject selection model 410 is to encode the user-provided image to be processed and descriptive information, and to perform cross-modal information fusion and processing on the input image to be processed and the descriptive information, thereby selecting a target subject that matches the user's intent. The descriptive information can include text, meaning the subject selection model 410 can process text-based descriptive information. The descriptive information can also be understood as guiding information. The subject selection model 410 can accurately locate and segment candidate subjects in the image to be processed, and these candidate subjects match the descriptive information. The subject selection model 410 can also identify foreground candidate subjects from the segmented candidate subjects as the target subject. The subject selection model 410 can output accurate target subjects, thus supporting the normal operation of subsequent algorithm modules.

[0135] like Figure 6 As shown, the subject determination model 411 may include a first image feature extraction model 510, a descriptive information feature extraction model 520, and a feature processing model 530.

[0136] The subject determination model 411 may include the segment anything model (SAM), U-Net, or other deep neural network models.

[0137] SAM refers to a general image segmentation model that possesses general knowledge about objects, allowing it to generate masks for objects in an image based on descriptive information. This descriptive information can also be called cue information or prompts.

[0138] The first image feature extraction model 510 is used to extract features from the image to be processed in order to obtain image features.

[0139] The descriptive information feature extraction model 520 is used to extract features from descriptive information to obtain descriptive information features.

[0140] The feature processing model 530 is used to process descriptive information features and image features to generate at least one candidate subject region.

[0141] The subject determination model 411 can process data of multiple modalities, such as images and descriptive information, to determine the target subject. The data type of the image to be processed is an image. The data type of the descriptive information can include text, or data types used to represent one or more of the following: location points, masks, regions of preset shapes, etc.

[0142] When the subject determination model 411 is SAM, the structures of the descriptive information feature extraction model 520 and the feature processing model 530 in the subject determination model 411 can be found in [reference needed]. Figure 6 .

[0143] like Figure 6 As shown, the descriptive information feature extraction model 520 may include a first information feature extraction model 521 and a second information feature extraction model 522.

[0144] The first information feature extraction model 521 can be used to process the description mask in the description information to obtain the first information features.

[0145] The second information feature extraction model 522 can be used to extract features from one or more pieces of information in the descriptive information, such as descriptive text, descriptive regions, and descriptive points, to obtain second information features. The second information feature extraction model 522 can be understood as an encoder.

[0146] The descriptive information features may include first information features and / or second information features.

[0147] The feature processing model 530 may include a fusion model 521 and a feature processing sub-model 522.

[0148] The fusion model 521 is used to fuse the first information features and image features to obtain fused features.

[0149] The methods for fusing the first information feature and the image feature can be either concatenating the first information feature and the image feature, or adding the first information feature and the image feature together.

[0150] The fusion method that adds the primary information features and image features can be implemented using linear fusion or nonlinear fusion. Linear fusion includes operations such as addition and averaging. Nonlinear fusion includes attention mechanisms, pyramid pooling, deconvolution, etc.

[0151] When the descriptive information includes a descriptive mask, the feature processing sub-model 522 is used to process the second information features and the fused features to obtain at least one candidate subject region.

[0152] In the absence of a description mask in the description information, the feature processing sub-model 522 is used to process the second information features to obtain at least one candidate subject region.

[0153] The first image feature extraction model 510 can also be called an image encoder. The first image feature extraction model 510 extracts features from the image to be processed, which can also be understood as encoding the image to be processed.

[0154] The feature processing sub-model 522 can also be called the feature restoration model or mask decoder. The processing of the second information feature (or the second information feature and the fused feature) by the feature processing sub-model 522 can also be understood as the feature processing sub-model 522 decoding based on the second information feature (or the second information feature and the fused feature).

[0155] Candidate subject regions can be represented using masks. Each candidate subject region can be understood as the area in the image to be processed where the candidate subject it represents is located. The edges of the candidate subject regions can be understood as the edges of the candidate subjects in the image to be processed. In other words, in the image to be processed, there are no pixels outside the candidate subject regions that correspond to the candidate subject represented by those regions.

[0156] The first image feature extraction model 510, the first information feature extraction model 521, the second information feature extraction model 522, and the sub-generation model 522 can all be convolutional neural networks.

[0157] like Figure 6 As shown, different candidate regions can be represented by different images. Alternatively, different candidate regions can be represented as regions of different colors within the same image. Or, candidate regions can be represented by data forms other than images. Each location data point represents the pixel positions within the candidate region corresponding to that location data.

[0158] Figure 6 The image to be processed shows a person walking and another person inside a window. With the descriptive information being the text "person," the feature processing model 530 can output two candidate regions, representing the walking person and the person inside the window, respectively.

[0159] The subject-determining model 411 can be a trained neural network model.

[0160] In some embodiments, the subject determination model 411 can be trained based on multiple training segmentation samples and the label region corresponding to each training segmentation sample. Each training segmentation sample includes a training segmentation image and training descriptive information. The training descriptive information in the training segmentation sample is used to describe the label subject located in the label region corresponding to the training segmentation sample in the training segmentation image. The edge of the label region can be the edge of the label subject in the training segmentation image.

[0161] By processing the training segmentation samples using the initial subject determination model, the training regions corresponding to the training segmentation samples can be obtained. Based on the difference between the training regions corresponding to the training segmentation samples and the label regions corresponding to the training segmentation samples, the parameters of the initial subject determination model are adjusted, and the adjusted initial subject determination model is the subject determination model 411.

[0162] In other embodiments, the subject determination model 411 may be obtained through multiple stages of training. The first training description information may be used to describe the subject located in the first label region in the first training segmentation image.

[0163] In the first stage of training, multiple first training segmentation samples and a first label region corresponding to each first training segmentation sample can be obtained. Each first training segmentation sample includes a first training segmentation image and first training description information.

[0164] The edge of the first label region is the edge of the subject recorded in the first label region in the first training segmentation image. For example, the subject in the first label region can be a foreground subject in the first training segmentation image, a subject located in a prominent position in the first training segmentation image, or a subject with a large area in the first training segmentation image, etc. The subject in the first label region can be the main (or can be understood as prominent) subject in the first training segmentation image.

[0165] The first training description information may include one or more of the following: first training description text, first training description region, and first training description points. The shape of the first training description region may be a preset regular shape or other shapes. For example, a first training description region with a preset regular shape can be represented by a box, while a first training description region with other shapes can be represented by a mask. The number of first training description regions and first training description points in the first training description information can both be one or more.

[0166] The first training description information can be used to describe the main body located in the first label region in the first training segmentation image. That is, the first label region corresponding to the first training segmentation sample is the region of the main body described by the first training description information in the first training segmentation image of the first training segmentation sample. The number of first label regions corresponding to each first training segmentation sample can be one or more.

[0167] The initial segmentation model is used to process the first training segmentation sample to obtain the first training region. Based on the difference between the first training region corresponding to each first training segmentation sample and the first label region corresponding to that first training segmentation sample, the parameters of the initial segmentation model are adjusted to obtain the segmentation model adjusted in the first stage.

[0168] The difference between the first training region corresponding to each first training segmentation sample and the first label region corresponding to that first training segmentation sample can be represented as the loss value.

[0169] The first stage of training can be understood as the manual assistance stage. By manually labeling the regions corresponding to the main subjects described by the first training description information in the first training segmentation image (i.e., the first label regions), the segmentation model adjusted in the first stage is able to segment the main subjects in the image.

[0170] In the second phase of training, multiple second training segmentation samples can be obtained. These second training segmentation samples include second training segmentation images and second training descriptive information. The second training segmentation images may be the same as or different from the first training segmentation images.

[0171] The second training description information is used to describe the main body in the second training segmentation image. The second training description information may include one or more of the following: second training description text, second training description region, and second training description points. The shape of the second training description region can be a preset regular shape or other shapes.

[0172] The segmentation model adjusted in the first stage is used to process the second training segmentation sample to obtain the second training region corresponding to the second training segmentation sample.

[0173] By manually labeling the second training segmentation image, regions that correspond to the main body of the second training description text, excluding the main body recorded in the second training region, are marked. The edges of the manually labeled regions are the edges of the main body.

[0174] In the second stage, the main subject of the manually labeled region record can be understood as the secondary (or non-prominent) subject in the second training segmentation image.

[0175] Based on the difference between the second training region and the second label region corresponding to each second training segmentation sample, the parameters of the initial segmentation model are adjusted to obtain the segmentation model after the second-stage adjustment. The second label region corresponding to each second training segmentation sample includes the second training region itself and the region manually labeled based on the second training region. The difference between the second training region and the second label region corresponding to each second training segmentation sample can be represented as the loss value.

[0176] The second phase of training can be understood as the semi-automatic phase.

[0177] In the third stage of training, multiple third training segmentation samples can be obtained. These third training segmentation samples include third training segmentation images and third training descriptive information. The third training segmentation images may be the same as or different from the third training segmentation images.

[0178] The third training description information is used to describe the main body in the second training segmentation image. The third training description information may include one or more of the following: third training description text, third training description region, and third training description points. The shape of the third training description region may be a preset regular shape or other shapes.

[0179] The segmentation model adjusted in the second stage is used to process each third training segmentation sample multiple times. Each processing yields the third training region corresponding to the third training segmentation sample under that processing.

[0180] After multiple processing steps, a stable region is selected from the third training region corresponding to the third training segmentation sample as the third label region corresponding to the third training segmentation sample.

[0181] Based on the difference between the third training region corresponding to each third training segmentation sample and the third label region corresponding to that third training segmentation sample, the parameters of the initial segmentation model are adjusted. The segmentation model adjusted in the third stage is the subject determination model 411. The difference between the third training region corresponding to each third training segmentation sample and the third label region corresponding to that third training segmentation sample can be the loss value.

[0182] The third stage of training can be understood as the fully automated stage.

[0183] Traditional neural network model training requires a large amount of labeled data. However, training the subject determination model 411 through three stages reduces the amount of labeled data required and lowers manual labor costs.

[0184] During the training of a neural network model, because the model's output should be as close as possible to the desired predicted value, we can compare the current network's prediction with the target value and update the weight vector of each layer based on the difference. (Of course, there's usually an initialization process before the first update, where parameters are pre-configured for each layer in the neural network model). For example, if the model's prediction is too high, the weight vector is adjusted to predict a lower value. This adjustment continues until the neural network model can predict the target value or a value very close to it. Therefore, it's necessary to predefine "how to compare the difference between the predicted and target values," which is the loss function or objective function. These are important equations used to measure the difference between the predicted and target values. Taking the loss function as an example, a higher output value (loss) indicates a greater difference, so training the neural network model becomes a process of minimizing this loss as much as possible.

[0185] Backpropagation (BP) algorithm can be used to correct the parameters of the initial neural network model during training, thereby reducing the reconstruction error loss. Specifically, forward propagation of the input signal to the output generates error loss; this error loss information is then propagated back to update the parameters of the initial neural network model, leading to convergence of the error loss. The backpropagation algorithm is an error-loss-driven backpropagation process aimed at obtaining the optimal parameters of the neural network model, such as the weight matrix.

[0186] In some cases, the accuracy of candidate target regions obtained using the subject-specific determination model 411 is low.

[0187] On the one hand, compared with the case where the descriptive information is a descriptive point, when the descriptive information includes one or more of the descriptive text, descriptive region and descriptive mask, the subject determination model 411 can be used to process the image to be processed and the descriptive information to obtain a more accurate candidate target region. That is, the candidate subject represented by the candidate target region is more in line with the user's needs.

[0188] On the other hand, the accuracy of the region represented by the mask and / or the description region, i.e. the magnitude of the difference between the region represented by the mask and / or the description region and the candidate target region, affects the accuracy of the candidate target region.

[0189] To obtain more accurate candidate target regions, the image processing system 400 may further include a preprocessing module. The preprocessing module is used to determine descriptive region information based on user input information. The descriptive region information may include descriptive regions and / or descriptive masks. In the descriptive region information, the descriptive mask may represent at least one descriptive region. The target subject is recorded in the descriptive region represented by the descriptive region information.

[0190] User input information can be determined based on user actions. Users can perform actions on an electronic device based on the position of candidate subjects within the image to be processed, allowing the electronic device to determine the user input information. User input information can represent one or more of the following: input points, input lines, input regions, etc.

[0191] An input point can represent the location where a user clicks on the image to be processed displayed on an electronic device. An input line can be the trajectory of a user's finger sliding across the image to be processed displayed on an electronic device. An input area can be the region selected by the user on the image to be processed displayed on an electronic device.

[0192] When the user input information includes input points, the region represented by the description region information determined by the preprocessing module may include the input points.

[0193] When the user input includes an input line, the area represented by the description area information determined by the preprocessing module may overlap with the input line. For example, the ratio of the length of the portion of the input line located within the area represented by the description area information to the length of the input line itself can be greater than or equal to a preset length ratio.

[0194] When the user input information includes an input area, the area represented by the description area information determined by the preprocessing module may overlap with the input area. For example, the ratio of the area of ​​the portion of the input area located within the area represented by the description area information to the area of ​​the input area may be greater than or equal to a preset area ratio.

[0195] The preprocessing module may include an object detection model. The object detection model can be used to detect objects in the image to be processed. Object detection can be understood as detecting all objects in an image. That is, by processing the image to be processed using the object detection model, at least one object region can be obtained. The shape of the object region can be regular or irregular. For example, the edge of the object region can surround the object recorded in the object region, or the edge of the object region can be the edge of the object recorded in the object region.

[0196] In addition to processing the image, the object detection model can also obtain the category of the object recorded in each object region. That is, the object detection model can be used to generate bounding boxes for objects in an image and determine the category to which the object represented by each bounding box belongs. For example, the object detection model could be a recognizeanything model (RAM).

[0197] Based on multiple object regions output by the object detection model and user input information, a description region can be determined. The description region can be a region among the multiple object regions output by the object detection model that meets preset conditions. The preset conditions can be at least one of the following: an input point, coincidence with an input line, and coincidence with an input region. For example, the preset conditions can be that the ratio of the length of the portion of the input line located within the object region to the length of the input line is greater than or equal to a preset length ratio, and the ratio of the area of ​​the portion of the input region located within the object region to the area of ​​the input region is greater than or equal to a preset area ratio.

[0198] The objects recorded in the description region can also be understood as candidate subjects. The description region determined by the object detection model can be considered a description region.

[0199] Object detection models can be trained on training detection data. Training detection data can include multiple training detection images and corresponding label object data for each training detection image. The label object data for each training detection image includes label object regions, which are the regions where objects in the training detection images are located.

[0200] It should be understood that the number of labeled object regions in the labeled object data corresponding to the training detection image can be one or more, and the number of labeled object regions is equal to the number of objects recorded in the training detection image.

[0201] The initial object detection model is used to process the training recognition images to obtain training object data. This training object data includes the training object regions. Based on the differences between the training object data and the labeled object data, the parameters of the initial object detection model are adjusted. The adjusted initial object detection model is the final object detection model. The differences between the training object data and the labeled object data can be represented by a loss value.

[0202] In some embodiments, the label object data corresponding to each training recognition image may further include the label category for each label object region. The training object data corresponding to each training recognition image may further include the training category for each training object region.

[0203] The input points in the user input information may or may not be used as the input subject to determine the description points of model 411. The input regions in the user input information may or may not be used as the input subject to determine the description regions of model 411.

[0204] In situations with complex foregrounds or backgrounds, the number of candidate subject regions identified by the subject determination model 411 may be multiple. To improve user satisfaction, the depth image acquired during the image acquisition process can be fully utilized. The depth image can be the relative depth of various objects in the shooting scene estimated by the camera during focusing.

[0205] The selection module 412 determines, based on the depth image of the image to be processed, that the person walking has a shallower depth compared to the person inside the window, and can select the candidate subject region representing the person walking as the target subject region.

[0206] The selection module 412 can also determine the target subject region based on the detection results of visual saliency detection of the image to be processed.

[0207] Visual saliency detection, also known as image attention prediction or saliency detection, refers to the process of using intelligent algorithms to simulate the characteristics of the human visual system, predict human visual fixation points and eye movements, and extract salient regions in an image. A salient region in an image can also be understood as a region of interest (ROI). Visual saliency detection is a key image analysis technique in the field of computer vision.

[0208] The selection module 412 may include a visual saliency detection model, or the preprocessing module may include a visual saliency detection model. The visual saliency detection model may be a trained neural network model.

[0209] During the training of a visual saliency detection model, the training saliency data used can include training detection images and labeled regions of interest. Labeled regions of interest represent regions in the training detection images that are of interest to the viewer.

[0210] By processing the training images using an initial visual saliency detection model, training regions of interest (ROIs) can be obtained. Based on the difference between the training ROIs and the labeled ROIs, the parameters of the initial visual saliency detection model are adjusted. The adjusted initial detection model is the visual saliency detection model. The difference between the training ROIs and the labeled ROIs can be represented by a loss value.

[0211] The target subject region can be a candidate subject region that overlaps with the ROI. For example, the target subject region can be a candidate subject region located within the ROI. Alternatively, if the ratio between the area of ​​a candidate subject region within the ROI and the area of ​​the candidate subject region is greater than or equal to a preset area ratio threshold, the candidate subject region can be used as the target subject region.

[0212] When there are multiple candidate subject regions within a Region of Interest (ROI), the target subject region can also be determined based on the depth of these candidate regions within the ROI. The target subject region can be one or more of the shallowest regions among the multiple candidate subject regions within the ROI, determined based on the depth image.

[0213] Optimization module 420 can be executed. Figure 7 The image processing method shown processes the target subject region and the image to be processed to obtain the target subject region.

[0214] Step S601: Select multiple first candidate images from the image to be processed based on the target subject region.

[0215] The first candidate region of each first candidate image overlaps with the target subject region. Furthermore, each first candidate image can have a preset aspect ratio. In other words, the aspect ratio of the first candidate image can be a preset aspect ratio.

[0216] The length direction of the first candidate image can be the same as or different from the length or width direction of the image to be processed.

[0217] After step S601, multiple iterations can be performed. Each iteration may include steps S602 to S606. In the first iteration, i = 1.

[0218] Step S602: Using the dimensional scoring model, process the multiple i-th candidate images respectively to obtain the dimensional score of each i-th candidate image in each of the multiple dimensions.

[0219] The dimensional scoring model may include an image feature extraction model. The image feature extraction model extracts features from multiple i-th candidate images to obtain the image features of each i-th candidate image. Based on the difference between the image features of each candidate image and the j-th dimension features corresponding to the j-th dimension, the dimensional score of the candidate image in the j-th dimension can be determined, where j is a positive integer. The j-th dimension features corresponding to the j-th dimension are obtained by extracting features from the j-th dimension text using the text feature extraction model within the dimensional scoring model. The j-th dimension text is used to describe the j-th dimension.

[0220] The image feature extraction model in the dimensional scoring model can also be called the second image feature extraction model, which can be the same as or different from the first image feature extraction model 510.

[0221] Figure 14 Dimensionality scoring models can be trained neural network models. The training process for dimensionality scoring models can be found in [link to documentation]. Figure 12 Figure 13 Explanation.

[0222] When i=1, the i-th candidate image is the first candidate image.

[0223] Step S603: Calculate the structural similarity between each i-th candidate image and the reference image to obtain the structural score of the i-th candidate image.

[0224] Structural similarity (SSIM) is a metric for measuring the similarity between two images. From the perspective of image composition, structural similarity defines structural information as a dimension that reflects the structure of objects in a scene, independent of brightness and contrast.

[0225] The mean of each pixel in an image can be used to estimate the image brightness, the standard deviation of each pixel in an image can be used to estimate the contrast, and the covariance of each pixel in an image can be used as a measure of structural similarity.

[0226] The structural similarity between each i-th candidate image and the reference image can be used as the structural score of that i-th candidate image.

[0227] By using an image structure feature extraction model to extract features from multiple i-th candidate images, candidate structure features for each i-th candidate image can be obtained. The difference between the candidate structure features of each i-th candidate image and the reference structure features can represent the structural similarity between the i-th candidate image and the reference image. The reference structure features can be obtained by extracting features from the reference image using the image structure feature extraction model.

[0228] Image structure feature extraction models can be trained neural network models. Training methods for image structure feature extraction models can be found in [link to documentation]. Figure 11 Explanation.

[0229] Training an image structure feature extraction model can be based on at least one training structure dataset. Each training structure dataset includes a training structure image and training structure text. The training structure text describes the structure of the training structure image. For example, the training structure text can represent the structure category of the training structure image.

[0230] When the training structural text is the name of the structural category to which the training structural image belongs, the structural similarity between the i-th candidate image and the reference image can also be determined based on the text structural features corresponding to the structural categories to which the i-th candidate image and the reference image belong.

[0231] For each i-th candidate image, step S603 may specifically include, as follows: Figure 8 Steps S6031 to S6033 are shown.

[0232] Step S6031: Using the image structure feature extraction model, feature extraction is performed on the i-th candidate image to obtain the candidate structure features of the i-th candidate image.

[0233] Step S6032: The reference image is processed using an image structure feature extraction model to obtain the reference image structure features. The reference image structure features can be used as reference structure features.

[0234] Step S6033: Calculate the structural feature difference between the candidate structural features and the reference image structural features.

[0235] Structural feature difference can be understood as the loss value between candidate structural features and reference image structural features.

[0236] Step S6034: Determine the structural score of the i-th candidate image based on the differences in structural features.

[0237] For example, the structure score of each i-th candidate image can be determined based on the structural feature difference between the candidate structure features of the i-th candidate image and the reference image structure features of the reference image. The structure score of each i-th candidate image can be negatively correlated with the structural feature difference between the candidate structure features of the i-th candidate image and the reference image structure features of the reference image.

[0238] Alternatively, in step S603, the structure score of each i-th candidate image can be determined based on multiple i-th candidate images and the structure category selected by the user.

[0239] Different structure categories can correspond to different category structure features. The category structure features corresponding to the structure category selected by the user can be used as reference structure features.

[0240] By processing multiple i-th candidate images separately using an image structure feature extraction model, candidate structure features for each i-th candidate image can be obtained. The structure score of each i-th candidate image can be negatively correlated with the difference between the candidate structure features of that i-th candidate image and the reference structure features.

[0241] Step S604: Calculate the image score for each i-th candidate image.

[0242] The image score of each i-th candidate image is positively correlated with the dimensional score of the candidate image in each dimension, and is also positively correlated with the structure score of the candidate image.

[0243] For example, the image score of each i-th candidate image can be represented as a weighted sum of the structure score of the i-th candidate image and the dimensional score of the i-th candidate image in each dimension. The weights corresponding to the structure score and the dimensional scores in each dimension can be preset.

[0244] Step S605: Optimize the i-th candidate region based on the image scores of multiple i-th candidate images to obtain at least one (i+1)-th candidate region.

[0245] By using algorithms such as Bayesian algorithm, ant colony algorithm, particle swarm optimization algorithm, immune algorithm, and genetic algorithm, optimization can be achieved to obtain at least one second candidate region.

[0246] Each (i+1)th candidate region overlaps with the target region. Furthermore, the aspect ratio of the (i+1)th candidate region can be a preset aspect ratio.

[0247] Step S606: Determine whether the stopping condition is met.

[0248] The stopping condition can be that the number of times steps S602 to S605 are executed is greater than or equal to a preset threshold. Alternatively, the stopping condition can also be that the image score of the (i+1)th candidate image located in the (i+1)th candidate region of the image to be processed converges.

[0249] If the stopping condition is not met, i can be set to i+1, that is, the value of i is updated. The updated i is equal to the original i plus 1, and steps S602 to S606 are performed again.

[0250] At least one i-th candidate image may contain an image that is identical to the candidate images obtained in the previous multiple executions of steps S602 to S606. When performing steps S602 to S604 again, at least one i-th candidate image that is different from the i-1th candidate image obtained in the previous executions of steps S602 to S606 can be processed using steps S602 to S604, thereby obtaining the image score for each (i+1)-th candidate image.

[0251] If the stopping condition is met in step S606, step S607 can be performed.

[0252] Step S607: Determine the target image from among multiple (i+1)th candidate images.

[0253] The target image can be the image with the highest image score among the multiple (i+1)th candidate images.

[0254] The optimization module 420 may include one or more of the various models used in steps S601 to S608, such as a dimension scoring model, an image structure feature extraction model, and a text structure feature extraction model.

[0255] The following is combined with Figure 9 For cases where a genetic algorithm is used... Figure 7 The image processing method shown is explained.

[0256] After obtaining multiple first candidate images in step S601, step S901 can be performed.

[0257] Step S901: Encode the region information of the first candidate region where each first candidate image is located to obtain the chromosome corresponding to each first candidate region.

[0258] The region information of the first candidate region can represent the location of the first candidate region in the image to be processed.

[0259] When the length or width direction of each first candidate region is parallel to the length direction of the image to be processed, the region information of each first candidate region can be represented as a set of coordinate parameters {x1, y1, x2, y2}, where (x1, y1) and (x2, y2) are the coordinates of the two vertices located on the diagonal of the first candidate region, respectively.

[0260] The center point of the target entity is (xc, yc). The center point of the target entity can be located within each first candidate region.

[0261] The encoding can be in decimal form. Each encoded chromosome can be represented as a feasible solution containing the coordinate parameter set {x1, y1, x1, y2} of the first candidate region corresponding to that chromosome. Each gene position of the chromosome corresponds to one parameter in the coordinate parameter set {x1, y1, x1, y2} of the first candidate region corresponding to that chromosome.

[0262] The chromosome corresponding to each first candidate region can also be understood as the first chromosome corresponding to the first candidate image of the image to be processed located in that first candidate region.

[0263] The number of first candidate images obtained through step S601 can be understood as the number of the initial population, which can be N1, where N1 is a positive integer.

[0264] After step S901, steps S602 to S604 can be performed. Step S605, which is performed after step S604, can be step S905.

[0265] Step S905: Perform at least one of the following operations on the i-th chromosome: selection, comparison, mutation, etc., to obtain the (i+1)-th chromosome.

[0266] Each (i+1)th chromosome can correspond to an (i+1)th candidate image. The region represented by each chromosome is the (i+1)th candidate region of the (i+1)th candidate image corresponding to that chromosome in the image to be processed. Alternatively, chromosomes obtained through selection, comparison, mutation, and other operations can be selected, and chromosomes whose aspect ratios match a preset aspect ratio are used as the (i+1)th chromosomes corresponding to the (i+1)th candidate images.

[0267] The number of chromosomes obtained in step S905 each time can be N2, where N2 is a positive integer. N2 can be equal to N1 or other values.

[0268] In genetic algorithms, each generation of chromosomes can be understood as an evolutionary process. The (i+1)th chromosome obtained in each evolution can be obtained by performing operations such as selection, comparison, and mutation on the i-th chromosome. In each evolutionary step, the i-th chromosome can be understood as the chromosome of the previous generation, and the (i+1)-th chromosome can be understood as the new chromosome obtained through evolution.

[0269] The crossover process requires finding two chromosomes from the previous generation. These two chromosomes are then cut at a specific location and spliced ​​together to create a new chromosome. This new chromosome contains a certain number of genes from both chromosomes.

[0270] The selection of the two chromosomes for "crossover" from the previous generation is not random; it is typically done using a roulette wheel algorithm. After each evolutionary cycle, the fitness of each chromosome must be calculated.

[0271] The probability that each chromosome is selected for crossover can be expressed as the ratio of the fitness of a chromosome to the sum of the fitness of all chromosomes.

[0272] In other words, the fitness of a chromosome, that is, the higher the probability that a larger chromosome will be selected.

[0273] In multiple evolutionary processes, the construction of the fitness function, i.e., the method of determining fitness, is crucial for each evolution. Fitness is used to represent the evaluation of all chromosomes. Through evolution, chromosomes with low fitness can be eliminated, and only chromosomes with high fitness can be retained. Thus, after several evolutions, the quality of chromosomes will become higher and higher.

[0274] The fitness of each i-th chromosome can be positively correlated with the image score of the i-th candidate image corresponding to that chromosome. For example, the fitness s of each chromosome can be expressed as s = w1s1 + w2s2, where s1 is the average score of the dimension, s2 is the structure score, and w1 and w2 are preset coefficients.

[0275] The average dimensional score s1 of a certain i-th chromosome can be obtained by weighted summation of the dimensional scores of the i-th candidate image corresponding to that chromosome across multiple dimensions.

[0276] Users can set the coefficients w1 and w2 according to their preferences. For example, w1 can be set to 0.6 and w2 can be set to 0.4. If the user does not select a reference image, w1 can be set to 1 and w2 can be 0.

[0277] Crossover ensures that superior genes are retained in each evolutionary process, but it merely selects from the existing set of results; the number of genes remains the same, only their combination order is changed. This only guarantees that after multiple evolutions, the computational result is closer to a local optimum, but it will never reach the global optimum.

[0278] Mutation refers to randomly selecting several genes on a chromosome and then randomly modifying the values ​​of those genes, thereby introducing new genes into the existing chromosome. This breaks through the limitations of the current search and makes it easier for the algorithm to find the global optimal solution.

[0279] In each evolution, to preserve the superior chromosomes of the previous generation, the chromosomes with the highest fitness from the previous generation need to be directly copied to the next generation without modification. This can be understood as "selection". Assuming that each evolution requires the generation of N2 chromosomes, then in each evolution, N2-M chromosomes need to be generated through crossover and mutation. The remaining M chromosomes are obtained by copying the M chromosomes with the highest fitness from the previous generation, where M is a positive integer.

[0280] By using a genetic algorithm to search for candidate images, and using the weighted calculation result of the dimensional average score and the structure score as the fitness function, the best composition region can be found more quickly, that is, the target image can be determined more quickly.

[0281] If the stopping condition is determined to be met in step S606, step S607 may include steps S907 and S908.

[0282] Step S907: Identify the target chromosome with the highest image score among at least one (i+1)th chromosome.

[0283] Step S908: Decode the target chromosome to obtain the target region. The image located in the target region within the image to be processed is the target image. In other words, the (i+1)th candidate image corresponding to the target chromosome is the target image.

[0284] The following is combined with Figure 10 The image processing method provided in the embodiments of this application will be described in detail. The execution subject of the method provided in the embodiments of this application can be an electronic device, or a software / hardware module in an electronic device capable of image selection. For ease of explanation, the following embodiments will use an electronic device as an example.

[0285] Figure 10 This is a schematic flowchart of an image processing method provided in an embodiment of this application. Figure 10 The method shown includes steps S1010 to S1040.

[0286] Step S1010: Obtain the image to be processed and the description information obtained based on the first user operation. The description information is used to describe the target subject in the image to be processed.

[0287] The image to be processed can be executed Figure 10 The image captured by the camera in the electronic device shown in the method can also be an image stored in the electronic device, an image sent by other electronic devices received by the electronic device, or a frame from a video.

[0288] For example, one or more frames of an image in a video can be used as images to be processed, according to... Figure 10 The process is performed using the method shown. The time interval between two adjacent frames in the video can be equal or unequal.

[0289] The first user operation may include one or more types of operations. The first user operation may include touch input through the interface (such as clicking, swiping, dragging, etc.), voice input, or input operations through external input devices (such as keyboards or touchpads), etc.

[0290] An external input device, also known as an external input device, is a type of external device. Electronic devices can acquire information input by the user through an external input device.

[0291] An electronic device for acquiring a first user operation, and an electronic device for determining descriptive information based on the first user operation, which can interact with the execution... Figure 10 The electronic devices used in the methods shown may be the same or different electronic devices.

[0292] The descriptive information can be user input or information determined based on user input. Descriptive information may include one or more of the following: descriptive text, descriptive regions, descriptive points, etc.

[0293] In some embodiments, a user can describe a target object using their voice. The electronic device can then perform speech recognition on the received user voice and determine descriptive information based on the recognition result. The recognition result can be represented in text form.

[0294] The recognition results may include descriptive text. If the recognition results include location information, the descriptive point or descriptive region can be determined based on the location information. The descriptive text may include all parts of the recognition results other than the location information indicating the location.

[0295] Location information can represent the location of input points and / or input areas, etc.

[0296] In other embodiments, the description information may be obtained by determining the description information based on the first user operation in the second interface.

[0297] If the description information includes descriptive text, the second interface may include... Figure 2 The first image processing settings interface 1210 is shown in (a) of the image.

[0298] The first user operation may include one or more of the following: entering descriptive text in the input box 1211, clicking on alternative text, or clicking on the OK icon 1212.

[0299] The first user operation may also include, for a second interface including an image to be processed, the user selecting one or more input points, input lines, or input regions on the image to be processed in the second interface. In other words, the first user operation may also include user operations based on the second interface displayed on the electronic device.

[0300] User-selected input points represent the locations where the user clicks on the image to be processed. User-selected input lines represent the trajectory of the user's finger as it slides across the image to be processed. User-selected input regions represent the areas selected by the user on the image to be processed.

[0301] The descriptive information may include one or more of the input points, input lines, input regions, etc., selected by the user in the first user operation. Alternatively, if the first user operation includes the user selecting one or more of the input points, input lines, or input regions, the descriptive information may also include a descriptive region. The descriptive region may include the object region in the image to be processed where the input point is located. The descriptive information may also include an object region that overlaps with the input line and / or input region.

[0302] Different object regions in the image to be processed can be used to record different objects. Multiple object regions in the image to be processed can be obtained by processing the image using an object detection model.

[0303] The edges of the object region may surround the objects recorded in that object region. For example, the edges of the object region may coincide with the edges of the objects recorded in that object region. For example, the shape of the object region may be preset. It should be understood that when the shape of the objects recorded in the object region is the preset region shape, the edges of the object region may completely coincide with the edges of the objects recorded in the object region.

[0304] The second interface can be the interface displayed by the electronic device after receiving user input from the first interface.

[0305] For example, the electronic device may display a first interface, which may include an image to be processed. The first interface may be... Figure 1 The first album interface 1110 shown in (a) of the Zhongnuan website can also be... Figure 3 The fourth camera interface 1320 shown in (b) can also be Figure 4 The first camera interface 1410 or the second camera interface 1420 shown.

[0306] In response to a second user action on the first interface, the electronic device can display a second interface.

[0307] Step S1020: Based on the user's description information, determine the target subject region in the image to be processed, and the edge of the target subject region is the edge of the target subject in the image to be processed.

[0308] Based on the user's description, the target subject region can be determined in the image to be processed, and the edges of the target subject region are the edges of the target subject in the image to be processed. Therefore, the object recorded in the target subject region of the image to be processed is the target subject.

[0309] Based on the description information, at least one candidate subject can be identified in the image to be processed. Based on the depth of each candidate subject, a target subject region can be determined. The subject in the target subject region can be the subject with the lowest depth among the at least one candidate subject. The subject in the target subject region is the target subject. In other words, the target subject can be the subject among the at least one candidate subject that is closest in distance to the electronic device acquiring the image to be processed.

[0310] For example, while using a camera to capture images of a scene to obtain images to be processed, an electronic device can also capture depth information of the scene recorded in the images to be processed, thereby obtaining the depth of each subject in the scene. For example, the electronic device can capture depth images. Depth images, also known as range images, refer to images that use the distance (depth) values ​​of each point in the scene captured by the image acquisition device as pixel values.

[0311] The depth of a candidate subject can be determined by analyzing the depths of multiple points within that subject. This depth can be calculated by applying the central tendency of the data to the depths of these points. Central tendency, also known as the central location or measure of central tendency, is a representative value of a set of data. For example, the depth of a candidate subject can be the average, median, or mode of the depths of multiple points within that subject.

[0312] By using a subject determination model to process descriptive information and the image to be processed, at least one candidate subject can be obtained. The subject determination model can be a neural network model, such as SAM or U-Net. Figure 5 or Figure 6 The main body is defined by model 411.

[0313] Step S1030: Multiple candidate images are determined in the image to be processed, and each candidate image is scored to obtain an image score for each candidate image. The candidate region of each candidate image overlaps with the target subject region.

[0314] The edges of the candidate region containing the candidate image can be called the bounding box. The candidate region (proposal) can also be called the candidate composition region.

[0315] The candidate region overlaps with the target main region. For example, the ratio of the area of ​​the target main region within the candidate region to the area of ​​the target main region can be greater than or equal to a preset area ratio; alternatively, the center of the target main region can be located within the candidate region. The center of the target main region can be its geometric center.

[0316] In other words, the candidate region can be the area that includes the center of the target subject region. During the process of determining the candidate region, the center of the target subject region can be used as a reference point or anchor point.

[0317] The aspect ratio of the candidate image can be the same as or different from that of the image to be processed. The aspect ratio of the candidate image can be preset. For example, the aspect ratio of the candidate image can be the same as that of the image to be processed. Figure 10The aspect ratio of the images captured by the camera in the electronic device shown in the method is the same.

[0318] After the candidate images are identified, they can be evaluated to determine their image scores.

[0319] The image score of a candidate image can be obtained by aesthetically scoring the candidate image. For example, each candidate image is scored in multiple dimensions to obtain a dimensional score for each candidate image in each dimension. Thus, the image score of a candidate image can be determined based on the dimensional scores of each candidate image in multiple dimensions.

[0320] Step S1040: Determine the target image from the multiple candidate images based on the image score of each candidate image in the multiple candidate images.

[0321] The probability of each candidate image being the target image can be positively correlated with the image score of that candidate image.

[0322] For example, a preset number of candidate images with the highest image scores can be used as the target image, or candidate images with image scores greater than or equal to a preset image score can be used as the target image. The preset number can be greater than or equal to 1.

[0323] The image score of each candidate image can be positively correlated with the dimensional score of the candidate avatar in at least one dimension.

[0324] Different dimensions can be understood as different angles from which an image is evaluated; therefore, dimensions can also be called attributes or categories.

[0325] The number of at least one dimension can be multiple. The image score of a candidate image takes into account the influence of the image's dimensional scores across multiple dimensions, thus making it more accurate.

[0326] Scoring candidate images across multiple dimensions can be understood as performing fine-grained scoring on candidate images.

[0327] In some embodiments, different dimensions may correspond to different dimension scoring models.

[0328] In step S1030, the candidate image can be processed by the dimensional scoring model corresponding to multiple dimensions to obtain the dimensional scores of the candidate image under these multiple dimensions.

[0329] The dimension scoring model for each dimension can be obtained by using multiple training scoring data for that dimension.

[0330] The training scoring data for a specific dimension includes the image to be scored and its label score for that dimension. The initial scoring model for that dimension is used to process the image to be scored, yielding its training score for that dimension. Based on the difference between the training score and the label score for that dimension, the parameters of the initial scoring model for that dimension are adjusted. The adjusted initial scoring model for that dimension is the scoring model for that dimension.

[0331] The initial scoring models for multiple dimensions can be the same or different models.

[0332] In other embodiments, the dimensional score of a candidate image in a certain dimension can also be determined based on the image features of the candidate image and the dimensional features corresponding to that dimension.

[0333] In step S1030, an image feature extraction model can be used to extract features from the candidate image to obtain the image features of the candidate image. Based on the difference between the image features and the dimensional features corresponding to each dimension, the dimensional score of the candidate image in that dimension is determined.

[0334] The difference between the image features of the candidate image and the dimensional features corresponding to that dimension is correlated with the dimensional score of the candidate image in that dimension. The difference between the image features and the dimensional features corresponding to a certain dimension can be represented as the loss value between the image features and the dimensional features corresponding to that dimension.

[0335] In other words, when there are multiple dimensions, the dimensional score of the candidate image in the j-th dimension can be determined based on the difference between the image features of the candidate image and the j-th dimension features corresponding to the j-th dimension among the multiple dimensions, where j is a positive integer. The image features of the candidate image, the difference between the image features of the candidate image and the j-th dimension features corresponding to the j-th dimension among the multiple dimensions, and the dimensional score of the candidate image in the j-th dimension are correlated.

[0336] Based on the image features extracted from the candidate images by the image feature extraction model, the candidate images are scored in multiple dimensions, making the processing simpler and requiring fewer models, thus reducing the resource consumption of the models used in the image processing process.

[0337] The image score of a candidate image can be determined based on its dimensional score in each dimension. The image score of a candidate image is correlated with its dimensional score in each dimension.

[0338] The j-th dimension feature can be obtained by extracting features from the j-th dimension text using a text feature extraction model. The dimension text for each dimension can be pre-defined, or it can be understood as pre-defined anchor text.

[0339] In the process of image processing, text feature extraction models can be used to extract features from text in multiple dimensions to obtain dimensional features for each dimension. Alternatively, the dimensional features obtained by extracting features from text in multiple dimensions using text feature extraction models can be pre-set in the electronic device executing the image processing method.

[0340] Text feature extraction models and image feature extraction models can be jointly trained. They can be trained based on multiple training samples and the label information of each training sample. These multiple training samples correspond to multiple dimensions, and each training sample includes a first training image, a second training image, and the corresponding dimensional text. The label information of each training sample indicates whether the first training image in that training sample is superior to the second training image in that training sample. For training methods of text feature extraction models and image feature extraction models, please refer to [link to training methods]. Figure 12 Explanation.

[0341] In determining the dimensionality scores of candidate images across multiple dimensions, the dimensional features used for each dimension are determined based on the dimensional text for each dimension. The dimensional text used to determine the dimensionality scores of candidate images can be the same dimensional text used during training.

[0342] The dimension text for each dimension is used to describe that dimension. For example, the dimension text for each dimension may include the name of that dimension. For instance, the dimension text for each dimension may also include text used to rate the image in that dimension, such as "good" or "bad".

[0343] When the dimensional text of a certain dimension includes text used for positive evaluation, such as "good," "very good," or other text, the difference between the image features of a candidate image and the dimensional features of that dimension can be negatively correlated with the image score of the candidate image. That is, the smaller the difference between the image features of a candidate image and the dimensional features of that dimension, the higher the dimensional score of the candidate image in that dimension.

[0344] Conversely, when the dimensional text of a certain dimension includes text used for negative evaluation, such as "bad," "inferior," or other text, the difference between the image features of the candidate image and the dimensional features of that dimension can be positively correlated with the image score of the candidate image. In other words, the smaller the difference between the image features of the candidate image and the dimensional features of that dimension, the lower the dimensional score of the candidate image in that dimension.

[0345] If the dimensional text of at least one dimension does not include text used to evaluate the image in that dimension, then for any one of the at least one dimensions, the direction of the correlation between the image features of the candidate image, the difference between the dimensional features of that dimension, and the image score of the candidate image during the process of determining the image score of the candidate image is determined based on the training process of that at least one dimension. If the at least one dimension is processed during training with dimensional text including positive evaluation text, the correlation is negative; if the at least one dimension is processed during training with dimensional text including negative evaluation text, the correlation is positive.

[0346] The dimensional score of a candidate image in a certain dimension can be determined based on the correspondence between that dimension and the corresponding features. Different dimensions can correspond to different correspondences. The dimensional score of a candidate image in a certain dimension can be expressed as the dimensional score corresponding to the difference between the image features of the candidate image and the dimensional features of that dimension, determined according to the correspondence between those dimensions. In the case where the dimensional text of a certain dimension includes text for positive evaluation and text for negative evaluation, the correspondence for each dimension can be different.

[0347] The following example illustrates this using dimensional text, which includes text used for positive evaluations.

[0348] Candidate images are scored across multiple dimensions. These dimensions include several of the following image evaluation dimensions: overall image effect, lighting, content, background, foreground, and composition. Overall image effect refers to the overall impression and feeling the image evokes. Image content refers to the theme or information the image conveys. Foreground can be understood as the foreground elements, referring to the relatively close portion in front of the subject. Background refers to the relatively distant portion behind the subject. The background provides environmental information, atmosphere, and emotional tone. Composition refers to the layout and arrangement of the various elements within the image.

[0349] The dimension text for these multiple dimensions can be "good image", "good lighting", "good content", "good background", "good foreground", "good composition", etc.

[0350] Alternatively, these multiple dimensions may also include at least two of the following image evaluation dimensions: complementary colors, duotones, high dynamic range, image grain, light on white, long exposure, macro, motion blur, negative image, rule of thirds, shallow depth of field (shallow DOF), silhouettes, soft focus, vanishing point, etc.

[0351] Complementary colors are colors that are opposite each other on the color wheel. When they are mixed, they produce neutral gray or white.

[0352] Two-tone refers to creating a specific atmosphere or emotion for an image by combining two colors.

[0353] High-contrast images display significant differences between bright and dark areas, making details in the image stand out more and providing viewers with a strong visual experience.

[0354] Texture maps emphasize the surface texture and details of an image, such as the grain of wood or the graininess of metal. Appropriate textures can increase the realism and visual appeal of an image.

[0355] Bright white typically refers to bright, highlighted white areas in an image. These areas can attract the viewer's attention and contrast with other colors, thus highlighting specific elements in the image.

[0356] Long exposure is a photographic technique that uses a shutter speed that is kept open for an extended period to capture special effects such as flowing water and star trails. This technique can create dreamlike visual effects in images.

[0357] Macro photography focuses on capturing the details and textures of small objects, such as insects and flowers. This photographic method can reveal beauty and complexity that are normally difficult to observe.

[0358] Motion blur is a blurring effect caused by the rapid movement of the subject or camera. In some cases, this blur can add a sense of motion and speed to an image.

[0359] A negative is an image reversal effect where bright and dark areas are interchanged. This effect can produce a unique visual experience and emphasize certain details in an image.

[0360] The rule of thirds is a compositional principle that suggests dividing an image into thirds and placing important elements on these dividing lines or intersections to increase the image's balance and visual appeal.

[0361] Shallow depth of field means that only a small portion of an image is in focus, while the rest is blurred. This effect helps to highlight the subject and create a sense of depth and dimension.

[0362] A silhouette is a black outline formed when a subject is positioned relative to a background light source. Silhouette images emphasize shape and posture, creating a simple yet powerful visual effect.

[0363] Soft focus is a photographic effect in which parts of an image are intentionally blurred to create a soft, dreamlike atmosphere.

[0364] A vanishing point, also known as a disappearing or disappearing point, is a point in geometry where two or more straight lines intersect at infinity. In plane geometry, two straight lines intersect at a single point; while in solid geometry, three or more planes or curves intersect at a single point, which is called a vanishing point. Using vanishing points can give images greater depth and a sense of three-dimensionality.

[0365] The image score for each candidate image can also be determined based on the similarity between the candidate image and the reference image.

[0366] The image score of each candidate image can also be positively correlated with the structural similarity between the candidate image and the reference image.

[0367] The reference image can be an image selected by the user from multiple images stored in the electronic device, a preset image, or an image received by the electronic device.

[0368] For example, prior to step S1030, a user selection operation can be obtained. The user selection operation is used to select a reference image. The user selection operation may include one or more of the following: a user touch operation on the display screen, a user voice input, or a user inputting information through an external input device.

[0369] For example, before step S1030, the user can speak the name of the reference image. The electronic device can capture sound and perform speech recognition on the captured audio signal to obtain the name of the reference image. Based on the name of the reference image, the reference image can be identified from the images stored in the electronic device.

[0370] For example, a third interface may be displayed before step S1030. The third interface may include multiple candidate images. When a user selection operation is received from the third interface, the candidate image selected by the user selection operation can be used as a reference image.

[0371] The third interface can be Figure 2 The second image processing settings interface 1220 is shown in (b) above. A user selection operation can be an operation where the user clicks on a thumbnail of a candidate image in the second image processing settings interface 1220. The candidate image corresponding to the thumbnail clicked by the user can be understood as the candidate image selected by the user selection operation.

[0372] In step S1030, based on the reference image selected by the user, the image score of each candidate image can be determined. The structural similarity between the candidate image and the reference image is positively correlated with the image score of the candidate image.

[0373] For example, in step S1030, the structure score of each candidate image can be determined according to the correspondence between structural similarity and structure score. The structure score corresponding to each candidate image is the structure score corresponding to the structural similarity between the candidate image and the reference image. Based on the structure score of the candidate image, the image score of the candidate image can be determined. The image score of the candidate image is positively correlated with its structure score.

[0374] The structural similarity between a candidate image and a reference image can also be called the structural similarity corresponding to that candidate image.

[0375] Image structural similarity is an indicator that measures whether the structures of two images are similar.

[0376] In some embodiments, the structural similarity between candidate image x and reference image y can be based on the pixel mean μ of the candidate image. x σ x The pixel mean μ of the reference image y σ y And the correlation coefficient σ between the candidate image and the reference image. xy The parameters are determined. The structural similarity between the candidate image and the reference image can be expressed as:

[0377] SSIM(x,y)=[l(x,y)] α ·[c(x,y)] β ·[s(x,y)] γ

[0378] in

[0379]

[0380] C1, C2, C3, α, β, and γ are all preset values; C1, C2, and C3 are positive numbers close to 0 to prevent abnormal results when the denominator is zero; α, β, and γ are all greater than 0 and are used to adjust the relative importance of l(x,y), c(x,y), and s(x,y). The mean of the image reflects the image brightness, and the variance of the image reflects the image contrast.

[0381] If α=β=γ=1, C1=(K1L) 2 C2 = (K2L) 2 C3 = C2 / 2, K1 and K2 are both much less than 1, and L is the dynamic range of the image. Therefore, the structural similarity between image x and image y can be simplified as follows:

[0382]

[0383] For example, adding noise to an image can disrupt its structure. If adding noise results in a mean squared error (MSE) of 0.04 between the noisy and original images, the structural similarity between the two images can be as low as 0.15, indicating low structural similarity. However, during brightness adjustment, the overall structure of the image remains largely unchanged. Therefore, if adjusting the image brightness results in an MSE of 0.04 between the two images, the structural similarity can be as high as 0.85.

[0384] In other embodiments, an image structure feature extraction model can be used to perform feature processing on multiple candidate images to determine the structural similarity between each candidate image and the reference image.

[0385] By using an image structure feature extraction model to extract features from multiple candidate images, the candidate structure features of each candidate image can be obtained.

[0386] For any candidate image, its structural similarity can be determined based on the structural feature differences between the candidate image's structural features and the reference structural features. The structural similarity of each candidate image is negatively correlated with the structural feature differences of that candidate image. The reference structural features are obtained by extracting features from the reference image using an image structural feature extraction model.

[0387] The structural feature difference corresponding to a candidate image can be understood as the structural feature difference between the candidate structural features and the reference structural features of the candidate image.

[0388] In some embodiments, the reference structural features may be the reference image structural features of the reference image output by the image structural feature extraction model when the image structural feature extraction model extracts features from the reference image.

[0389] In other embodiments, different structural categories correspond to different category structural features. The reference structural feature can be the category structural feature corresponding to the structural category to which the reference image belongs.

[0390] An image structure feature extraction model is used to extract features from a reference image. The model outputs reference image features. The structure category corresponding to the category structure feature with the smallest difference from the reference image features is the structure category to which the reference image belongs. The category structure feature corresponding to the structure category to which the reference image belongs is the reference structure feature.

[0391] By processing the name of each structural category using a text structure feature extraction model, we can obtain the text structure features corresponding to that category. The text structure features corresponding to each structural category can also be understood as the category structure features of that category.

[0392] The structural category to which the reference image belongs can be determined by comparing the structural features of the reference image with the category structural features of each of the multiple structural categories.

[0393] The structural category to which the reference image belongs can be the structural category corresponding to the structural features that have the smallest difference from the structural features of the reference image.

[0394] Alternatively, the image category to which the reference image belongs can be determined based on the reference image's structural features and the candidate image's structural features for each candidate image. The candidate image's structural features for each candidate image can be obtained by extracting features from that candidate image using an image structural feature extraction model.

[0395] The structural category to which the reference image belongs can be the structural category with the smallest average distance to the structural features of the reference image.

[0396] The difference between the candidate structural features of any candidate image and the category structural features of the structural category to which the reference image belongs can also represent the structural similarity between the candidate image and the reference image.

[0397] In some embodiments, the reference image may be an image selected by the user from a plurality of preset candidate images. These candidate images may belong to multiple structural categories. Different structural categories may correspond to different category structural features. That is, different candidate images may correspond to different category structural features. The category structural features of the structural category to which the user-selected reference image belongs can be used as the reference structural feature.

[0398] The structural category to which the reference image belongs can be determined by the structural features of the reference image obtained by processing the reference image according to the image structural feature extraction model.

[0399] In other embodiments, the reference image may be an image selected by the user from among a plurality of preset alternative images.

[0400] After processing the reference image using an image structure feature extraction model to obtain the reference image structure features, the reference image can be added to a preset pool of candidate images according to its structure category.

[0401] Therefore, when the user selects the reference image again, there is no need to process the reference image again using the image structure feature extraction model. Instead, the reference structure features are determined based on the correspondence between the candidate image and the category structure features, thus improving processing efficiency.

[0402] Different structural categories can be understood as different compositional rules, which are basic principles and techniques that artists follow during the creative process to achieve a certain effect or express a certain theme. These basic principles and techniques can be called compositional rules. Compositional rules can help artists better organize and arrange the elements in the picture, making the work more aesthetically pleasing and expressive.

[0403] Users may lack the knowledge to create images effectively and may not be able to skillfully utilize composition rules to achieve stunning results. However, they can appreciate the beauty of images created according to certain compositional rules. By selecting reference images, users can ensure that the compositional rules used in those reference images are taken into account during image processing, and by cropping the image to be processed, the resulting target image can better meet their needs.

[0404] For multiple images belonging to a certain structural category, the degree to which they conform to the compositional rules represented by that structural category may vary. Using the category structural features corresponding to the structural category to which the reference image belongs as the reference structural features can make the selected target image conform to the compositional rules to a higher degree, and the target image more aesthetically pleasing.

[0405] Image structure feature extraction models can be trained neural network models. The training process for image structure feature extraction models can be found in [link to documentation]. Figure 11 Explanation.

[0406] For example, the text structure feature extraction model and the image structure feature extraction model are obtained by adjusting the parameters of the initial text structure feature extraction model and the initial image structure feature extraction model based on the differences between the trained text structure features and the trained image structure features. The adjusted initial text structure feature extraction model is called the text structure feature extraction model, and the adjusted initial image structure feature extraction model is called the image structure feature extraction model.

[0407] Specifically, the training text structure features are obtained by the initial text structure feature extraction model from the training structure text, while the training image structure features are obtained by the initial image structure feature extraction model from the training candidate images. The training structure text is used to describe the structure of the training candidate images.

[0408] For example, training structure text can be used to describe the structure categories of training candidate images. Different structure categories can represent different composition rules.

[0409] During step S1030, the target image can be determined by enumerating all possible candidate images. For example, multiple first candidate images can be obtained by cropping the image to be processed using a sliding window method. A sliding window refers to dividing the original image into a series of small, overlapping image blocks (windows). Processing the image region within each window allows for traversal and detection of the entire image.

[0410] However, there are many possible sizes for candidate images. Using each possible size of the candidate image as the sliding window size results in a large overall computational load for image processing.

[0411] To reduce computational cost, a first set of images can be determined from the images to be processed. This first set includes at least one candidate image. Then, based on this first set, an optimization algorithm is used to obtain more candidate images.

[0412] At least one candidate image in the first image set can be randomly determined. The number of candidate images in the first image set can be preset.

[0413] For example, multiple random regions can be randomly determined in the image to be processed, and images in the random regions that overlap with the target subject region can be used as candidate images.

[0414] In other words, after determining the first image set, multiple iterations can be performed. The i-th iteration includes: scoring each candidate image in the i-th image set to obtain an image score for that candidate image; when i is 1, the i-th image set is the first image set; and determining the (i+1)-th image set based on the image scores of each candidate image in the i-th image set. The (i+1)-th image set includes at least one candidate image, where i is a positive integer. The (i+1)-th image set and the i-th image set may include the same candidate images or different candidate images.

[0415] Candidate images in the i-th image set can also be called i-th candidate images. Candidate images in the (i+1)-th image set can also be called (i+1)-th candidate images.

[0416] In each iteration, the search strategy is improved based on the image score of each candidate image in the i-th image set, thereby making the determined (i+1)-th image set closer to the optimal solution, that is, making the candidate images in the (i+1)-th image set closer to the image with the highest image score. In other words, the (i+1)-th maximum value of the image score of at least one candidate image in the (i+1)-th image set is greater than or equal to the i-th maximum value of the image score of at least one of the candidate images in the i-th image set.

[0417] In some embodiments, the average image score of at least one candidate image in the (i+1)th image set may also be greater than or equal to the average image score of at least one candidate image in the i-th image set.

[0418] The multiple iterations can terminate when an iteration termination condition is met. The iteration termination condition can include the number of iterations being greater than or equal to a preset number of iterations. Alternatively, the termination condition can include the convergence of the average or maximum image score of at least one candidate image in the (i+1)th image set.

[0419] By incorporating the candidate image determination process into the search framework of the optimization algorithm, the target image can be obtained by searching the parameter space through the optimization algorithm.

[0420] Optimization algorithms can be Bayesian optimization algorithms or evolutionary algorithms, etc. These algorithms all have the ability to find the global optimum in the search space, rather than just the local optimum.

[0421] Bayesian optimization is a black-box function optimization algorithm. Its core idea is to gradually determine the optimal solution of a function through continuous experimentation. Bayesian optimization predicts the value of the objective function by constructing a Gaussian process model, then calculates the posterior distribution of each candidate point according to Bayes' theorem, and selects the point with the largest posterior distribution for the next iteration.

[0422] Bayesian optimization, based on Bayes' theorem, intelligently predicts the next parameter combination most likely to improve performance by establishing a probabilistic model between parameters and performance. It continuously updates the probabilistic model using historical observations, thereby improving search efficiency. Bayesian optimization constructs a posterior probability distribution by collecting historical observation data and then uses this distribution to guide the search direction. As data accumulates, the probabilistic model becomes more accurate, and the search strategy gradually converges to the optimal solution.

[0423] Evolutionary algorithms include ant colony optimization, particle swarm optimization, genetic algorithms, and immune algorithms.

[0424] Ant colony optimization (ACO) simulates the process of ants searching for food, using the accumulation and volatilization of pheromones to guide the search direction. When an ant finds a shorter path, it releases more pheromones to attract other ants to choose this path. Simultaneously, the pheromones gradually volatilize, preventing the algorithm from getting trapped in local optima. Ants tend to choose paths with higher pheromone concentrations, thus improving the search strategy.

[0425] Particle swarm optimization (PSO) simulates the behavior of animal groups such as flocks of birds or schools of fish, guiding the search direction through information sharing and cooperation among particles. Each particle represents a potential solution, adjusting its velocity and position based on its own historical best position and the best position within the swarm. PSO uses velocity and position update formulas to guide the search direction. Each particle adjusts its velocity and position based on its own historical best position and the best position within the swarm, thereby improving the search strategy.

[0426] Genetic algorithms simulate mechanisms such as natural selection, heredity, and mutation in biological evolution. They generate new solutions through operations like selection, crossover, and mutation, and evaluate their fitness to gradually approach the optimal solution. Genetic algorithms use selection to retain solutions with high fitness, crossover to generate new solutions, and mutation to increase solution diversity. These operations work together to enable the algorithm to gradually approach the optimal solution.

[0427] The immune algorithm simulates the function of a biological immune system, guiding the search direction through mechanisms such as antibody production, cloning, mutation, and memory. It utilizes the affinity between antibodies and antigens to evaluate the quality of solutions and optimizes the search strategy through the immune response process. The immune algorithm generates new solutions through antibody cloning and mutation, and retains superior solutions through affinity evaluation. Simultaneously, it remembers previously found superior solutions and uses this memory to guide the search direction in subsequent searches. These mechanisms work together to enable the algorithm to quickly approach the optimal solution.

[0428] When employing a specific optimization algorithm, the region information of the candidate image or the candidate region containing the candidate image can be encoded using the encoding method corresponding to that algorithm. This ensures that the encoding result of the candidate image or candidate region is suitable for subsequent processing by the optimization algorithm. During multiple iterations of a specific optimization algorithm, the encoding result can be decoded in each iteration to obtain the candidate images in the image set for that iteration.

[0429] Encoding the regional information of the candidate region can also be understood as encoding the candidate region.

[0430] Taking the genetic algorithm as an example, the solution process for the optimization problem of determining the target image based on image scores is explained.

[0431] After determining the first image set, the region information of the candidate region where each candidate image in the first image set is located can be encoded to obtain the chromosome corresponding to the candidate image.

[0432] Each encoded chromosome can be represented as a feasible solution to the set of coordinate parameters {x1, y1, x1, y2} containing the candidate region corresponding to that chromosome.

[0433] During the i-th iteration, at least one operation—selection, crossover, or mutation—can be performed on multiple chromosomes corresponding to multiple candidate images in the i-th image set to obtain the chromosome corresponding to each candidate image in the (i+1)-th image set. Where i is 1, the i-th chromosome set is the first chromosome set.

[0434] The probability that any chromosome in the i-th chromosome set is selected as a chromosome in the (i+1)-th chromosome set is positively correlated with the image score of the candidate image corresponding to that chromosome.

[0435] For example, the probability of obtaining the chromosome in the next iteration by crossing over with other chromosomes or by mutation can also be positively correlated with the image score of the candidate image corresponding to that chromosome.

[0436] Then, by decoding each chromosome in the (i+1)th image set, the region information of the candidate image corresponding to that chromosome can be obtained.

[0437] Based on the region information of the candidate images corresponding to each chromosome in the (i+1)th chromosome set, the (i+1)th image set can be determined, and the (i+1)th image set includes the candidate images corresponding to each chromosome in the (i+1)th chromosome set.

[0438] The i-th chromosome set can include the chromosome corresponding to each candidate image in the i-th image set. That is, during the i-th iteration, at least one operation—selection, crossover, or mutation—can be performed on at least one chromosome in the i-th chromosome set to obtain the (i+1)-th chromosome set. The (i+1)-th chromosome set can include at least one chromosome.

[0439] For details on selection, crossover, and mutation operations, please refer to [link to documentation]. Figure 9 Explanation.

[0440] Then, each chromosome in the (i+1)th chromosome set is decoded to obtain the candidate image corresponding to that chromosome. The (i+1)th image set includes the candidate image corresponding to each chromosome in the (i+1)th chromosome set.

[0441] After step S1040, the target image can be further color-adjusted to achieve white balance.

[0442] White balance is simply the balance of white. It can also be understood as restoring white objects to their true white color under any light source. To compensate for color casts that occur when shooting under specific light sources, the corresponding complementary color is enhanced, thus achieving white balance. By adjusting the colors and correcting the color temperature of the image captured by the camera, the colors of the subject can be restored, making the colors of images captured under different light conditions similar to those seen by the human eye, thus achieving white balance.

[0443] White can be understood as a bright color without a specific hue. White contains all the color components of the visible spectrum, and these color components are roughly equal in intensity, thus producing the visual effect of white. In other words, white represents a balanced mixture of all the colors of the visible spectrum, resulting in a bright, colorless visual effect.

[0444] The image to be processed can be an image that has not undergone white balance adjustment.

[0445] Alternatively, the image to be processed can be an image obtained by color adjustment of an image captured by a camera, or an image that has already achieved white balance. In other words, the image to be processed can be an image that has undergone white balance adjustment or an image with white balance settings.

[0446] The white balance achieved in the image being processed is for the entire image. A specific area of ​​the image may not achieve white balance. Therefore, for the target image, which is a portion of the image being processed, color adjustments can be made again to achieve white balance, thereby improving the user experience.

[0447] Through steps S1010 to S1040, the target image determined in the image to be processed is the image with the highest image score among multiple candidate images. The region where each candidate image is located in the image to be processed overlaps with the region of the target subject described in the user's description information in the image to be processed. Therefore, the determined target image is more in line with the user's needs and improves the user experience.

[0448] Ordinary users may lack professional photography training and find it difficult to design the aesthetic composition of their subjects when taking photos. Electronic devices, by displaying a frame in the preview window during the photo-taking process, represent the edges of the target image with the highest image score in the preview, and the target image contains the subject as described in the user's descriptive text. Therefore, the image processing method provided in this application embodiment can offer suggestions or guidance to users when taking photos.

[0449] It should be understood that Figure 10 The method shown can be processed by a central processing unit (CPU) in an electronic device, or by a CPU and a neural network processing unit (NPU) working together, or it can be processed without an NPU, using other processors suitable for neural network computing. This application does not impose any restrictions on the embodiments.

[0450] The following is combined with Figure 11 ,right Figure 10 The training method of the image structure feature extraction model used in the image processing method shown is explained.

[0451] Figure 11 This is a schematic flowchart illustrating an image structure feature extraction model training method provided in an embodiment of this application. Figure 11 The method shown includes steps S1510 to S1540.

[0452] Step S1510: Obtain training structure text and training structure image. The training structure text is used to describe the structure of the training structure image.

[0453] For example, the training structure text can represent the structure category of the training structure image. The structure category of the training structure image can be one or more of the following: center, rule of thirds, golden ratio, triangle, horizontal, vertical, diagonal, symmetric, curved, radial, vanishing point, patterns, and fill the frame. For example, the training structure text can be the name of the structure category to which the training structure image belongs. The name of the structure category can be "center," "rule of thirds," "golden ratio," etc.

[0454] The structural categories of training structural images can also be referred to as the composition rules of training structural images.

[0455] Step S1520: Use the initial text structure feature extraction model to extract features from the training structure text to obtain the training text structure features.

[0456] Step S1530: Use the initial image structure feature extraction model to extract features from the training structure image to obtain the training image structure features.

[0457] Step S1540: Based on the differences between the training text structure features and the training image structure features, adjust the parameters of the initial text structure feature extraction model and the initial image structure feature extraction model. The adjusted initial image structure feature extraction model is the image structure feature extraction model.

[0458] The adjusted initial text structure feature extraction model is the text structure feature extraction model.

[0459] The difference between the trained text structural features and the trained image structural features can be represented by the loss value.

[0460] The following is combined with Figure 12 ,right Figure 10 The training method of the image feature extraction model used in the image processing method shown is explained.

[0461] Figure 12 This application describes an image feature extraction model training method provided in an embodiment. Figure 12 The method shown includes steps S1610 to S1650.

[0462] Step S1610: Obtain multiple training samples and label information for each training sample. The multiple training samples correspond to multiple dimensions. Each training sample includes a first training image, a second training image, and the dimensional text of the corresponding dimension. The label information of each training sample indicates that, under the corresponding dimension, the first training image in the training sample is better than the second training image in the training sample.

[0463] In other words, the training image data used to train the text feature extraction model and the image feature extraction model includes multiple training samples and the label information corresponding to each training sample. Each training sample includes a first training image, a second training image, and text in the nth dimension.

[0464] The training samples include multi-dimensional text. These multiple dimensions include multiple dimensions used to score candidate images. That is, the nth dimension text in different training samples can have the same or different values ​​for n. The multiple dimensions corresponding to the multi-dimensional text in the training samples can be multiple, so that the trained image feature extraction model can be used to extract features from the candidate image to obtain the dimensional scores of the candidate image in all or some of these multiple dimensions.

[0465] The label information corresponding to each training sample indicates that, in the nth dimension of the text in the nth dimension, the first training image in the training sample is better than the second training image in the training sample.

[0466] In different training samples, the combination of the first training image and the second training image can be the same or different image combinations.

[0467] The first and second training images in the training samples can be images collected by ourselves or images from public datasets.

[0468] In each training sample, the quality of the first training image and the second training image in the nth dimension can be different.

[0469] The labels for the training samples can be obtained by multiple professional annotators. After manual annotation, the labels for each training sample undergo at least two rounds of verification to ensure high accuracy.

[0470] For example, multiple images can be scored manually across one or more dimensions. The score obtained by manual scoring for an image in a certain dimension can be called the image's label dimension score for that dimension. For example, multiple images can be scored in each dimension. The range of manually labeled label dimension scores can be the same or different for each dimension. For example, the range of label dimension scores for each dimension can all be [0, 10]. Manually labeled scores can all be natural numbers. Alternatively, manually labeled scores can also include decimals. The label dimension score for each image in a certain dimension can be the average of the scores obtained by multiple people scoring the image in that dimension.

[0471] Two images with different manually labeled label dimension scores in a certain dimension are used as the first and second training images in a training sample, respectively. The dimensional text corresponding to that dimension is used as the nth dimension text in the training sample. Based on the label dimension scores of the two images in that dimension, it can be determined which image is better in that dimension, thus determining the label information corresponding to the training sample.

[0472] The label information corresponding to each training sample can indicate whether the first training image I1 is superior to the second training image I2 in the nth dimension. The label information can be determined based on the relationship between the first label dimension score of the first training image I1 in the nth dimension and the second label dimension score of the second training image I2 in the nth dimension. If the first label dimension score of the first training image I1 in the nth dimension is greater than the second label dimension score of the second training image I2 in the nth dimension, the label information indicates that the first training image I1 is superior to the second training image I2 in the nth dimension. If the first label dimension score of the first training image I1 in the nth dimension is less than or equal to the second label dimension score of the second training image I2 in the nth dimension, the label information indicates that the first training image I1 is not superior to the second training image I2 in the nth dimension. It should be understood that if the first label dimension score of the first training image I1 in the nth dimension is equal to the second label dimension score of the second training image I2 in the nth dimension, then the second training image I2 is comparable to the first training image I1 in the nth dimension, meaning they have the same level of superiority or inferiority.

[0473] The process of manually scoring multiple images in each dimension can be understood as the process of manually labeling training samples.

[0474] When selecting images to determine training samples, for a certain dimension, two images with the label dimension score of that dimension can be randomly selected as the first training image and the second training image in the training samples.

[0475] Alternatively, two images with similar scores in each dimension (e.g., the difference in scores is less than or equal to a preset score difference) can be used as the first and second training images in the training samples. For example, two images with similar but different scores in each dimension (e.g., the difference in scores is greater than 0 and less than or equal to a preset score difference) can be used as the first and second training images in the training samples.

[0476] In other words, for each training sample, the difference between the first label dimension score of the first training image in the nth dimension and the second label dimension score of the second training image in the nth dimension is less than a preset score difference threshold. Label information can be determined based on the relationship between the first and second label dimension scores. Images with higher label dimension scores are considered better images.

[0477] Therefore, in each dimension, the dimension scores determined by the image feature extraction model and the text feature extraction model can more accurately reflect the small differences in quality between two images in a certain dimension, making the dimension scores determined by the image feature extraction model and the text feature extraction model more accurate when using the same number of training samples.

[0478] The distribution of label dimension scores in each dimension of the multiple images selected as the first and second training images in a plurality of samples can be uniform. That is, in each dimension, the number of images whose label dimension scores belong to each score range is roughly equal.

[0479] In other words, among multiple images manually labeled with sub-label scores, a number of images of roughly equal quantity are selected from each score range according to the distribution of the sub-label scores, and these are used as the first training images. Then, images whose sub-label score in a certain dimension differs from that of the first training images from a preset score difference threshold are selected as the second training images.

[0480] This avoids data imbalance and improves the accuracy of the dimension scores of images determined by the image feature extraction model.

[0481] In multiple training samples, the difference in the number of dimensional texts belonging to different dimensions can be less than or equal to a preset difference. In other words, the number of dimensional texts belonging to different dimensions in the training samples can be approximately equal. This improves the accuracy of the dimensionality scores of images determined by the image feature extraction model.

[0482] When the dimensional text in a certain dimension includes text used for positive evaluation, the training information corresponding to the training samples indicates whether the first training image in that training sample is superior to the second training image in that training sample within that dimension. If the difference between the first training image and the text is greater than or equal to the difference between the second image and the text, the determined training information indicates that the first training image is not superior to the second training image in that certain dimension. If the difference between the first training image and the text is less than the difference between the second image and the text, the determined training information indicates that the first training image is superior to the second training image in that certain dimension.

[0483] Conversely, if the dimensional text in a certain dimension includes text used for negative evaluation, and the difference between the first training image and the text is less than or equal to the difference between the second image, the determined training information indicates that the first training image is not superior to the second training image in that certain dimension. If the difference between the first training image and the text is greater than the difference between the second image, the determined training information indicates that the first training image is superior to the second training image in that certain dimension.

[0484] If the dimensional text in at least one dimension does not include text used to evaluate the image in that dimension, such as only including the dimension name, then during training, it can be processed as if each dimensional text includes text with positive evaluations, or as if each dimensional text includes text with negative evaluations.

[0485] Step S1620: The initial image feature extraction model is used to process the first training image in the multiple training samples and the second training image in the multiple training samples respectively, so as to obtain the first training image feature of the first training image in each training sample and the second training image feature of the second training image in each training sample.

[0486] Step S1630: Using the initial text feature extraction model, process the dimensional text in multiple training samples to obtain the training text features of the dimensional text in each training sample.

[0487] In other words, during training, for each training sample, the initial image feature extraction model is used to extract features from the first and second training images of that training sample, respectively, to obtain the features of the first and second training images. Similarly, the initial text feature extraction model is used to extract features from the nth dimension of the text in that training sample, resulting in the training text features.

[0488] Step S1640: For each training sample, the training information of the training sample is determined based on the first training image features, the second training image features, and the training text features. The training information of the training sample is determined based on the magnitude relationship between the first training image-text difference and the second training image-text difference. The first training image-text difference is the difference between the first training image features and the training text features, and the second training image-text difference is the difference between the second training image features and the training text features. The training information of the training sample indicates that the first training image in the training sample is better than the second training image in the training sample.

[0489] The difference between the first training image features and the training text features can be used as the first training image-text difference. The difference between the second training image features and the training text features can be used as the second training image-text difference.

[0490] For each training text, the training information corresponding to the training sample can be determined based on the relationship between the differences between the first training image and the second training image.

[0491] For example, when the difference between the first training image and the second training image is greater than or equal to the difference between the second training image and the first training image, the training information can indicate that the first training image is not better than the second training image in the nth dimension; when the difference between the first training image and the second training image is less than the difference between the second training image and the first training image, the training information can indicate that the first training image is better than the second training image in the nth dimension.

[0492] Step S1650: Based on the difference between the label information of each training text and the training information of that training text, adjust the parameters of the initial text feature extraction model and the initial image feature extraction model. The adjusted initial image feature extraction model is then called the image feature extraction model.

[0493] The adjusted initial image feature extraction model is a text feature extraction model.

[0494] The difference between the training information and the sample labels can be represented by a loss value. This difference can be understood as triplet rank loss. In other words, the adjustment of the parameters of the initial second image sub-model 812 and the initial second text sub-model 822 can be based on triplet rank loss. Triplet rank loss is a loss function in deep learning, mainly used for training samples with small differences. Training based on triplet rank loss is actually a form of metric learning, learning relative distances rather than actual values.

[0495] The initial image feature extraction model may include a pre-trained first image sub-model. The initial text feature extraction model may include a pre-trained first text sub-model.

[0496] In some embodiments, adjusting the parameters of the initial image feature extraction model may include adjusting the parameters of the first image sub-model. Adjusting the parameters of the initial text feature extraction model may include adjusting the parameters of the first text sub-model.

[0497] In other embodiments, the initial image feature extraction model may include an initial second image sub-model and a pre-trained first image sub-model. The initial text feature extraction model may include an initial second text sub-model and a pre-trained first text sub-model. Adjusting the parameters of the initial image feature extraction model can be understood as adjusting the parameters of the initial second image sub-model. Adjusting the parameters of the initial text feature extraction model can be understood as adjusting the parameters of the initial second text sub-model.

[0498] Pre-training the first image sub-model and the first text sub-model can be based on multiple pre-trained image and text datasets. Each pre-trained image and text dataset includes a pre-trained image and pre-trained text. Each pre-trained text is used to describe the pre-trained image in at least one dimension. For example, the pre-trained text "The buildings in the foreground of this photo are really beautiful" describes the image in the foreground dimension, while the pre-trained text "The background of this image looks cluttered" describes the image in the background dimension.

[0499] By processing the pre-training images using the initial first image sub-model, pre-training image features can be obtained. Similarly, by processing the pre-training images using the initial first text sub-model, pre-training text features can be obtained. Based on the differences between the pre-training image features and the pre-training text features, the parameters of both the initial first image sub-model and the initial first text sub-model are adjusted. The adjusted initial first image sub-model and the adjusted initial first text sub-model are then termed the first image sub-model and the first text sub-model, respectively.

[0500] like Figure 13 As shown in (a), during the pre-training process, the initial first image sub-model 811 can be used to extract features from the pre-training image to obtain the pre-training initial image features, and the initial first text sub-model 821 can be used to extract features from the pre-training text to obtain the pre-training initial text features. The pre-training text is used to evaluate the pre-training image in one or more dimensions.

[0501] A combination of pre-trained text and pre-trained images can be called an aesthetic review data pair. Publicly available aesthetic datasets can include aesthetic review data pairs. That is, aesthetic review data pairs in publicly available datasets can be used for pre-training of a first image sub-model and a first text sub-model. Publicly available aesthetic datasets may include, for example, aesthetic visual analysis (AVA) datasets, large-scale artificial intelligence open network (LAION) aesthetic evaluation datasets, photo review datasets, etc.

[0502] The data for aesthetic reviews is abundant and easy to obtain.

[0503] Based on the pre-training differences between the initial image features and the initial text features, the parameters of the initial first image sub-model 811 and the initial first text sub-model 821 are adjusted. The adjusted initial first image sub-model 811 is... Figure 13 The first image sub-model 711 shown in (b) is adjusted to the initial first text sub-model 821. Figure 13 The first text sub-model 721 is shown in (b) of the diagram. The pre-training difference between the initial image features and the initial text features can be represented by a loss value, which can be called the image-text contrast loss.

[0504] The purpose of pre-training is to achieve coarse-grained alignment, so that the first image sub-model 711 and the first text sub-model 721 can align the natural language description text with the image.

[0505] The first image sub-model 711 and the first text sub-model 721 can be understood as follows: Figure 14 The backbone network of the dimension scoring model 700 is shown. The pre-training of the first image sub-model 711 and the first text sub-model 721 is understood to be achieved through an image-text contrast learning strategy.

[0506] After obtaining the pre-trained first image sub-model 711 and first text sub-model 721, the training process can begin. Figure 14 The training of the second image sub-model 712 and the second text sub-model 722 in the dimensional scoring model 700 shown can be based on training image and text data. Through the training process, fine-tuning of the dimensional scoring model 700 can be achieved.

[0507] During training, such as Figure 13As shown in (b) in the figure, the initial dimension scoring model may include a first image sub-model 711, an initial second image sub-model 812, a first text sub-model 721, and an initial second text sub-model 822.

[0508] The first image sub-model 711 in the initial image feature extraction model can be used to extract features from the first training image I1 to obtain the initial first training image features. The first image sub-model 711 can also be used to extract features from the second training image I2 to obtain the initial second training image features.

[0509] The initial second image sub-model 812 in the initial image feature extraction model can be used to extract features from the initial first training image features to obtain the first training image features. The initial second image sub-model 812 can also be used to extract features from the initial second training image features to obtain the second training image features. The initial second image sub-model 812 can also be referred to as a learnable mapping layer.

[0510] The first text sub-model 721 in the initial text feature extraction model can be used for the nth dimension text T. n Feature extraction is performed to obtain initial training text features. The initial second text sub-model 822 in the initial text feature extraction model can be used to extract features from the initial training text features to obtain training text features. The initial second text sub-model 822 can also be called the category mapping layer.

[0511] The difference between the first training image features and the training text features can be represented as the first training image-text difference. The difference between the second training image features and the training text features can be represented as the second training image-text difference. Both the first and second training image-text differences can be represented by a loss value.

[0512] Based on the differences between the first training image and the second training image, training information can be determined, thereby completing the training of the second image sub-model and the second text sub-model.

[0513] Taking text with positive evaluations as an example, adjusting the parameters of the initial second image sub-model 812 and the initial second text sub-model 822 based on the difference between training information and sample labels can make the difference between the training image features and training text features corresponding to images with higher label dimension scores in the nth dimension in the first training image I1 and the second training image I2 smaller. This means that the loss between the training image features and training text features of images with higher dimension scores in the nth dimension is less than the loss between the training image features and training text features of images with lower dimension scores in the nth dimension.

[0514] like Figure 14As shown, the image feature extraction model 710 may include a second image sub-model 712 and a pre-trained first image sub-model 711, and the text feature extraction model 720 may include a second text sub-model 722 and a pre-trained first text sub-model 721. The first image sub-model 711, the second image sub-model 712, the first text sub-model 721, and the second text sub-model 722 may all be convolutional neural networks.

[0515] In the image feature extraction model, the first image sub-model 711 is used to extract features from the candidate image to obtain initial candidate image features, and the second image sub-model 712 is used to extract features from the initial candidate image features to obtain candidate image features.

[0516] In the text feature extraction model, the first text sub-model 721 is used to extract features from the dimensional text of any dimension to obtain the initial dimensional text features of that dimension, and the second text sub-model 722 is used to extract features from the initial dimensional text features of that dimension to obtain the dimensional features of that dimension.

[0517] The initial image feature extraction model includes a pre-trained first image sub-model, and the initial text feature extraction model includes a pre-trained first text sub-model. This can reduce the number of training samples and label information required during the training process and the manual cost of labeling multiple training samples.

[0518] It should be understood that Figure 10 , Figure 11 , Figure 12 The methods shown can be performed by the same or different electronic devices.

[0519] For example, execute Figure 11 , Figure 12 The electronic devices used in the illustrated method can be the same or different training devices. Training devices can be, for example, servers or cloud computing. Execution Figure 10 The electronic device used in the method described can be called an execution device. An execution device can be a terminal, such as a mobile phone, tablet, laptop, AR / VR device, in-vehicle terminal, etc., or it can be a server or cloud-based device.

[0520] The execution device can be configured with I / O interfaces for data interaction with external devices. Users can transmit input data to the execution device through client devices. The execution device performs... Figure 10 The method shown yields the processing result (i.e., the target image). The executing device can then return the target image to the client device.

[0521] Alternatively, the user can input the input data on the execution device. The input data may include the image to be processed and descriptive information, and may also include a reference image. The execution device can then display the target image.

[0522] It should be understood that the above examples are provided to help those skilled in the art understand the embodiments of this application, and are not intended to limit the embodiments of this application to the specific values ​​or scenarios illustrated. Those skilled in the art can obviously make various equivalent modifications or changes based on the above examples, and such modifications or changes also fall within the scope of the embodiments of this application.

[0523] The above text combined Figures 1 to 14 The image processing method of the embodiments of this application is described in detail below, and will be combined with Figures 15 to 17 This document describes in detail the apparatus embodiments of this application. It should be understood that the image processing apparatus in the embodiments of this application can execute the image processing methods described in the foregoing embodiments of this application. That is, the specific working processes of the various products described below can be referred to the corresponding processes in the foregoing method embodiments.

[0524] Figure 15 A hardware system for an electronic device applicable to this application is shown.

[0525] Electronic device 100 can be a mobile phone, smart screen, tablet computer, wearable electronic device, in-vehicle electronic device, augmented reality (AR) device, virtual reality (VR) device, laptop computer, ultra-mobile personal computer (UMPC), netbook, personal digital assistant (PDA), projector, etc. This application embodiment does not limit the specific type of electronic device 100.

[0526] Electronic device 100 may include processor 110, external memory interface 120, internal memory 121, universal serial bus (USB) interface 130, charging management module 140, power management module 141, battery 142, antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, sensor module 180, button 190, motor 191, indicator 192, camera 193, display screen 194, and subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0527] Figure 15 The connection relationships between the modules shown are merely illustrative and do not constitute a limitation on the connection relationships between the modules of the electronic device 100. Optionally, the modules of the electronic device 100 may also adopt a combination of various connection methods described in the above embodiments.

[0528] It should be noted that, Figure 15 The structure shown does not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include... Figure 15 The components shown may include more or fewer components, or the electronic device 100 may include... Figure 15 The components shown may be a combination of certain components, or the electronic device 100 may include... Figure 15 Sub-components of some of the components shown. Figure 15 The components shown can be implemented in hardware, software, or a combination of software and hardware.

[0529] Processor 110 may include one or more processing units. For example, processor 110 may include at least one of the following processing units: application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, video codec, digital signal processor (DSP), baseband processor, and neural network processing unit (NPU). These different processing units may be independent devices or integrated devices.

[0530] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from this memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0531] Electronic device 100 can implement display functions through GPU, display screen 194 and application processor.

[0532] A GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU performs mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0533] The display screen 194 can be used to display images or videos. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a mini light-emitting diode (Mini LED), a micro light-emitting diode (Micro LED), a micro OLED, or a quantum dot light-emitting diode (QLED). In some embodiments, the electronic device 100 may include one or N displays 194, where N is a positive integer greater than 1.

[0534] Electronic device 100 can perform shooting functions through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0535] The ISP (Image Signal Processor) is used to process data fed back from the camera 193. For example, when taking a picture, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, and the camera's photosensitive element transmits this electrical signal to the ISP for processing, transforming it into an image visible to the naked eye. The ISP can perform algorithmic optimization of image noise, brightness, and color. The ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In some embodiments, the ISP can be set in the camera 193.

[0536] Camera 193 is used to capture still images or videos. An object is projected onto a photosensitive element by generating an optical image through the lens. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then passed to an ISP for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP for processing. The DSP converts the digital image signal into a standard red-green-blue (RGB), YUV, or other image signal format. In some embodiments, the electronic device 100 may include one or N cameras 193, where N is a positive integer greater than 1.

[0537] Digital signal processors (DSPs) are used to process digital signals. Besides digital image signals, they can also process other digital signals. For example, when electronic device 100 selects a frequency, the DSP can perform Fourier transforms on the frequency energy.

[0538] Video codecs are used to compress or decompress digital video. Electronic device 100 may support one or more video codecs. Thus, electronic device 100 can play or record video in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG 2, MPEG 3, and MPEG 4.

[0539] An NPU (Neural Processing Unit) is a processor that borrows from the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, to rapidly process input information and continuously learn. NPUs can enable intelligent cognitive functions in electronic devices, such as image recognition, facial recognition, speech recognition, and text understanding.

[0540] The external storage interface 120 can be used to connect an external memory card, such as a secure digital (SD) card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external storage interface 120 to perform data storage functions. For example, music, video, and other files can be stored on the external memory card.

[0541] Internal memory 121 can be used to store computer executable program code, including instructions. Internal memory 121 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function (e.g., sound playback and image playback). The data storage area may store data created during the use of electronic device 100 (e.g., audio data and phonebook). Furthermore, internal memory 121 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, and universal flash storage (UFS). Processor 110 executes various processing methods of electronic device 100 by running instructions stored in internal memory 121 and / or instructions stored in memory disposed in the processor.

[0542] Touch sensor 180K, also known as a touch device, can be disposed on display screen 194. The touch sensor 180K and display screen 194 together form a touchscreen, also known as a touch screen. Touch sensor 180K is used to detect touch operations applied to or near it. Touch sensor 180K can transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through display screen 194. In other embodiments, touch sensor 180K may also be disposed on the surface of electronic device 100, and in a different location from display screen 194.

[0543] Figure 15 The hardware system of the electronic device 100 has been described in detail. The software system of the electronic device 100 is described below. The software system can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This application embodiment takes a layered architecture as an example to exemplarily describe the software system of the electronic device 100.

[0544] like Figure 16 As shown, a layered architecture software system is divided into several layers, each with a clear role and division of labor. Layers communicate with each other through software interfaces. In some embodiments, the software system can be divided into four layers, from top to bottom: the application layer, the application framework layer, the Android Runtime and system libraries, and the kernel layer.

[0545] The application layer may include applications such as camera, calendar, call, map, navigation, WLAN, Bluetooth, music, video, SMS, and photo album. Applications such as camera and photo album can be used to execute the image processing methods provided in this application.

[0546] The application framework layer provides an application programming interface (API) and programming framework for applications in the application layer. The application framework layer may include some predefined functions.

[0547] For example, the application framework layer includes a window manager, content providers, a view system, a phone manager, a resource manager, and a notification manager.

[0548] The Android Runtime consists of core libraries and a virtual machine. The Android Runtime is responsible for the scheduling and management of the Android system.

[0549] The core library consists of two parts: one part is the functionalities that need to be called by the Java language, and the other part is the Android core library.

[0550] The application layer and application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and application framework layer as binary files. The virtual machine is used to perform functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

[0551] The system library can include multiple functional modules, such as: surface manager, media libraries, 3D graphics processing library, and 2D graphics engine.

[0552] The kernel layer is the layer between hardware and software. It can include driver modules such as display drivers, camera drivers, audio drivers, and sensor drivers.

[0553] It should be noted that the above is passed Figure 15 An example is given to illustrate the structure of an electronic device. Figure 16 An example of a software architecture diagram for an electronic device is provided; however, this application does not impose any limitations on it.

[0554] Figure 17 This is a schematic diagram of the image processing apparatus provided in the embodiments of this application.

[0555] The image processing apparatus 1700 includes an acquisition unit 1710 and a processing unit 1720.

[0556] In some embodiments, the image processing device 1700 may perform Figure 10 The image processing method shown.

[0557] The acquisition unit 1710 is used to acquire an image to be processed and descriptive information obtained based on a first user operation, wherein the descriptive information is used to describe the target subject in the image to be processed.

[0558] The processing unit 1720 is configured to determine, based on the description information, a target subject region in the image to be processed, wherein the edge of the target subject region is the edge of the target subject in the image to be processed.

[0559] The processing unit 1720 is further configured to determine multiple candidate images in the image to be processed, and score the multiple candidate images respectively to obtain an image score for each candidate image, wherein the candidate region of each candidate image overlaps with the target subject region.

[0560] The processing unit 1720 is further configured to determine a target image from the plurality of candidate images based on the image score of each candidate image among the plurality of candidate images.

[0561] Optionally, the image processing apparatus 1700 also includes a display unit.

[0562] The display unit is used to display a first interface, the first interface including the image to be processed.

[0563] The display unit is also used to display a second interface in response to a second user operation on the first interface.

[0564] The acquisition unit 1710 is specifically used to determine the description information based on the first user operation in the second interface.

[0565] Optionally, the processing unit 1720 is further configured to score each candidate image in multiple dimensions to obtain a dimensional score for each candidate image in each dimension.

[0566] The processing unit 1720 is further configured to determine the image score of each candidate image based on the dimensional score of each candidate image in each dimension, wherein the image score of each candidate image is positively correlated with the dimensional score of the candidate image in each dimension.

[0567] Optionally, the processing unit 1720 is further configured to extract features from the plurality of candidate images respectively using an image feature extraction model to obtain image features for each candidate image.

[0568] The processing unit 1720 is further configured to determine the dimension score of the candidate image in the j-th dimension based on the difference between the image features of each candidate image and the j-th dimension features corresponding to the j-th dimension in the plurality of dimensions. The j-th dimension features corresponding to the j-th dimension are obtained by extracting features from the j-th dimension text of the j-th dimension through a text feature extraction model, where j is a positive integer and the j-th dimension text is used to describe the j-th dimension.

[0569] Optionally, the image feature extraction model and the text feature extraction model are trained based on multiple sample training data. The multiple sample training data includes multiple training samples and label information for each training sample. The multiple training samples correspond to the multiple dimensions. Each training sample includes a first training image, a second training image, and dimensional text for the corresponding dimension. The label information for each training sample indicates whether the first training image in the training sample is superior to the second training image in the training sample.

[0570] The image feature extraction model and the text feature extraction model are obtained by adjusting the parameters of the initial image feature extraction model and the initial text feature extraction model respectively based on the difference between the label information of each training sample and the training information of the training sample. The training information of the training sample is determined based on the magnitude relationship between the first training image-text difference and the second training image-text difference of the training sample. The training information of the training sample indicates whether the first training image in the training sample is better than the second training image in the training sample. The first training image-text difference of the training sample is the difference between the first training image feature and the training text feature. The second training image-text difference of the training sample is the difference between the second training image feature and the training text feature. The first training image feature is obtained by the initial image feature extraction model extracting features from the first training image in the training sample. The second training image feature is obtained by the initial image feature extraction model extracting features from the second training image in the training sample. The training text feature is obtained by the initial image feature extraction model extracting features from the dimensional text in the training sample.

[0571] Optionally, the multiple dimensions include multiple of the following image evaluation dimensions: overall image effect, lighting, content, background, foreground, and composition.

[0572] Optionally, the processing unit 1720 is further configured to determine the image score of each candidate image based on the reference image selected by the user, wherein the structural similarity between the candidate image and the reference image is positively correlated with the image score of the candidate image.

[0573] Optionally, the processing unit 1720 is further configured to, for any one of the plurality of candidate images, perform feature extraction on the arbitrary candidate image through an image structure feature extraction model to obtain the candidate structure features of the arbitrary candidate image.

[0574] The processing unit 1720 is further configured to determine the image score of any candidate image based on the structural feature difference between the candidate structural features and the reference structural features of any candidate image, wherein the reference structural features are obtained by extracting features from the reference image using an image structural feature extraction model, and the structural similarity between the candidate image and the reference image is negatively correlated with the structural feature difference.

[0575] Optionally, the text structure feature extraction model and the image structure feature extraction model are obtained by adjusting the parameters of the initial text structure feature extraction model and the initial image structure feature extraction model based on the differences between the training text structure features and the training image structure features; the training text structure features are obtained by the initial text structure feature extraction model extracting features from the training structure text, and the training image structure features are obtained by the initial image structure feature extraction model extracting features from the training candidate images; the training structure text is used to describe the structure of the training candidate images.

[0576] Optionally, the image processing apparatus 1700 also includes a display unit and a receiving unit.

[0577] The display unit is used to display a third interface, which includes multiple alternative images.

[0578] The receiving unit is used to receive the user selection operation in the third interface, wherein the candidate image selected by the user selection operation is the reference image.

[0579] Optionally, the processing unit 1720 is further configured to determine a first image set in the image to be processed, the first image set including at least one of the candidate images.

[0580] The processing unit 1720 is further configured to perform multiple iterations. The i-th iteration includes: scoring each candidate image in the i-th image set to obtain an image score for the candidate image; determining an (i+1)-th image set based on the image scores of each candidate image in the i-th image set, wherein the (i+1)-th image set includes at least one of the candidate images, where i is a positive integer, and the maximum value of the image score of at least one candidate image in the (i+1)-th image set is greater than or equal to the maximum value of the image score of at least one candidate image in the i-th image set; and, when i equals 1, the i-th image set is the first image set; and ending the iteration operation when the iteration termination condition is met.

[0581] Optionally, the processing unit 1720 is further configured to encode the region information of the region where each candidate image in the first image set is located, so as to obtain the chromosome corresponding to each candidate image, wherein the first chromosome set includes the chromosome corresponding to each candidate image in the first image set.

[0582] The processing unit 1720 is further configured to perform a target operation on the i-th chromosome set to obtain the (i+1)-th chromosome set. The target operation includes one or more of selection, crossover, and mutation. The probability that any chromosome in the i-th chromosome set is selected as a chromosome in the (i+1)-th chromosome set is positively correlated with the image score of the candidate image corresponding to any chromosome. When i equals 1, the i-th chromosome set is the first chromosome set.

[0583] The processing unit 1720 is further configured to decode each chromosome in the (i+1)th chromosome set to obtain the region information of the candidate image region corresponding to each chromosome in the (i+1)th chromosome set.

[0584] The processing unit 1720 is further configured to determine the (i+1)th image set based on the region information of the candidate images corresponding to each chromosome in the (i+1)th chromosome set, wherein the (i+1)th image set includes the candidate images corresponding to each chromosome in the (i+1)th chromosome set.

[0585] Optionally, the processing unit 1720 is further configured to determine a plurality of candidate subjects in the image to be processed based on the description information.

[0586] The processing unit 1720 is further configured to determine the target subject region based on the depth of each of the plurality of candidate subjects, wherein the subject in the target subject region is the subject with the smallest depth among the plurality of candidate subjects.

[0587] Optionally, the processing unit 1720 is further configured to perform color adjustment on the target image to achieve white balance in the target image.

[0588] Optionally, the image processing device 1700 is located in an electronic device, and the image to be processed is a preview image captured by the electronic device through a camera.

[0589] In other embodiments, the image processing device 1700 may perform... Figure 11 The training method for the image structure feature extraction model is shown.

[0590] The acquisition unit 1710 is used to acquire training structure text and training structure image, wherein the training structure text is used to describe the structure of the training structure image.

[0591] The processing unit 1720 is used to extract features from the training structure text using the initial text structure feature extraction model in order to obtain the training text structure features.

[0592] The processing unit 1720 is also used to extract features from the training structure image using the initial image structure feature extraction model to obtain the training image structure features.

[0593] The processing unit 1720 is also used to adjust the parameters of the initial text structure feature extraction model and the initial image structure feature extraction model according to the difference between the training text structure features and the training image structure features, and the adjusted initial image structure feature extraction model is the image structure feature extraction model.

[0594] In yet other embodiments, the image processing apparatus 1700 may perform... Figure 12 The image feature extraction model training method is shown.

[0595] The acquisition unit 1710 is used to acquire multiple training samples and label information for each training sample. The multiple training samples correspond to multiple dimensions. Each training sample includes a first training image, a second training image, and dimensional text for the corresponding dimension. The label information for each training sample indicates that, under the corresponding dimension, the first training image in the training sample is superior to the second training image in the training sample.

[0596] The processing unit 1720 is used to process the first training image in multiple training samples and the second training image in multiple training samples respectively using the initial image feature extraction model, so as to obtain the first training image feature of the first training image in each training sample and the second training image feature of the second training image in each training sample.

[0597] The processing unit 1720 is used to process the dimensional text in multiple training samples using the initial text feature extraction model to obtain the training text features of the dimensional text in each training sample.

[0598] The processing unit 1720 is configured to, for each training sample, determine the training information of the training sample based on the first training image features, the second training image features, and the training text features. The training information of the training sample is determined based on the magnitude relationship between the first training image-text difference and the second training image-text difference. The first training image-text difference is the difference between the first training image features and the training text features, and the second training image-text difference is the difference between the second training image features and the training text features. The training information of the training sample indicates that the first training image in the training sample is better than the second training image in the training sample.

[0599] The processing unit 1720 is used to adjust the parameters of the initial text feature extraction model and the initial image feature extraction model based on the difference between the label information of each training text and the training information of the training text. The adjusted initial image feature extraction model is then used as the image feature extraction model.

[0600] It should be noted that the image processing device 1700 described above is embodied in the form of a functional unit. The term "unit" here can be implemented in software and / or hardware, and there is no specific limitation on this.

[0601] For example, a "unit" can be a software program, a hardware circuit, or a combination of both that implements the above functions. The hardware circuit may include an application-specific integrated circuit (ASIC), electronic circuitry, a processor (e.g., a shared processor, a proprietary processor, or a group processor) and memory for executing one or more software or firmware programs, integrated logic circuitry, and / or other suitable components that support the described functions.

[0602] Therefore, the units of the various examples described in the embodiments of this application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0603] This application also provides a chip, which includes a data interface and one or more processors. When the one or more processors execute instructions, they read instructions stored in memory through the data interface to implement the image processing method described in the above method embodiments.

[0604] This application also provides an image processing system, including an execution device and a training device.

[0605] The one or more processors can be general-purpose processors or special-purpose processors. For example, the one or more processors can be central processing units (CPUs), digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, such as discrete gate, transistor logic devices, or discrete hardware components.

[0606] The chip can be used as a component of a terminal device or other electronic device. For example, the chip can be located in electronic device 100.

[0607] Processors and memory can be configured separately or integrated together. For example, processors and memory can be integrated onto a system-on-a-chip (SoC) in a terminal device. That is, the chip can also include memory.

[0608] The memory may store a program, which can be run by a processor to generate instructions, causing the processor to execute the image processing method described in the above method embodiments according to the instructions.

[0609] Optionally, the memory may also store data. Optionally, the processor may also read data stored in the memory, which may be stored at the same memory address as the program, or the data may be stored at a different memory address than the program.

[0610] For example, the memory can be used to store related programs of the image processing method provided in the embodiments of this application, and the processor can be used to call the related programs of the image processing method stored in the memory to implement the image processing method of the embodiments of this application.

[0611] For example, the memory can be used to store the relevant program of the image structure feature extraction model training method provided in the embodiments of this application, and the processor can be used to call the relevant program of the image structure feature extraction model training method stored in the memory to implement the image structure feature extraction model training method of the embodiments of this application.

[0612] For example, the memory can be used to store the relevant program of the image feature extraction model training method provided in the embodiments of this application, and the processor can be used to call the relevant program of the image feature extraction model training method stored in the memory to implement the image feature extraction model training method of the embodiments of this application.

[0613] This chip can be installed in electronic devices.

[0614] This application also provides a computer program product that, when executed by a processor, implements one or more of the image processing method, image structure feature extraction model training method, and image feature extraction model training method described in any of the method embodiments of this application.

[0615] The computer program product can be stored in memory, for example, it is a program. The program is eventually converted into an executable object file that can be executed by the processor after processes such as preprocessing, compilation, assembly and linking.

[0616] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a computer, implements one or more of the image processing method, image structure feature extraction model training method, and image feature extraction model training method described in any of the method embodiments of this application. The computer program may be a high-level language program or an executable object program.

[0617] The computer-readable storage medium is, for example, memory. Memory can be volatile or non-volatile, or it can include both volatile and non-volatile memory. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), which serves as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).

[0618] The embodiments of this application may involve the use of user data. In practical applications, user-specific personal data may be used in the scheme described herein within the scope permitted by applicable laws and regulations, provided that it complies with the applicable laws and regulations of the country (e.g., with the user's explicit consent, with the user being properly notified, etc.).

[0619] In the description of this application, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance, or a specific order or sequence. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances.

[0620] In this application, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or multiple items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0621] It should be understood that in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0622] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0623] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0624] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for example, the division of units is merely a logical functional division, and other division methods may exist in actual implementation; for example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0625] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0626] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0627] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An image processing method, characterized in that, The method includes: Acquire an image to be processed and descriptive information obtained based on a first user operation, wherein the descriptive information is used to describe the target subject in the image to be processed; Based on the description information, a target subject region is determined in the image to be processed, and the edge of the target subject region is the edge of the target subject in the image to be processed. Multiple candidate images are identified in the image to be processed, and each candidate image is scored to obtain an image score for each candidate image. The candidate region of each candidate image overlaps with the target subject region. The target image is determined from the plurality of candidate images based on the image score of each candidate image. The step of scoring the plurality of candidate images to obtain an image score for each candidate image includes: The image feature extraction model is used to extract features from the multiple candidate images to obtain the image features of each candidate image; Based on the difference between the image features of each candidate image and the j-th dimension features corresponding to the j-th dimension in multiple dimensions, the dimension score of each candidate image in the j-th dimension is determined. The j-th dimension features corresponding to the j-th dimension are obtained by extracting features from the j-th dimension text of the j-th dimension through a text feature extraction model. j is a positive integer. The j-th dimension text is used to describe the j-th dimension. Based on the dimensional score of each candidate image in each of the multiple dimensions, an image score for each candidate image is determined, wherein the image score of any candidate image is positively correlated with the dimensional score of the candidate image in each dimension.

2. The method according to claim 1, characterized in that, The method further includes: Display a first interface, which includes the image to be processed; In response to a second user action on the first interface, the second interface is displayed; The step of acquiring the image to be processed and the description information obtained based on the first user operation includes: determining the description information according to the first user operation in the second interface.

3. The method according to claim 1 or 2, characterized in that, The image feature extraction model and the text feature extraction model are trained based on multiple sample training data. The multiple sample training data includes multiple training samples and label information for each training sample. The multiple training samples correspond to the multiple dimensions. Each training sample includes a first training image, a second training image, and the corresponding dimension text. The label information of each training sample indicates whether the first training image in the training sample is better than the second training image in the training sample. The image feature extraction model and the text feature extraction model are obtained by adjusting the parameters of the initial image feature extraction model and the initial text feature extraction model respectively based on the difference between the label information of each training sample and the training information of the training sample. The training information of the training sample is determined based on the magnitude relationship between the first training image-text difference and the second training image-text difference of the training sample. The training information of the training sample indicates whether the first training image in the training sample is better than the second training image in the training sample. The first training image-text difference of the training sample is the difference between the first training image feature and the training text feature. The second training image-text difference of the training sample is the difference between the second training image feature and the training text feature. The first training image feature is obtained by the initial image feature extraction model extracting features from the first training image in the training sample. The second training image feature is obtained by the initial image feature extraction model extracting features from the second training image in the training sample. The training text feature is obtained by the initial image feature extraction model extracting features from the dimensional text in the training sample.

4. The method according to claim 1 or 2, characterized in that, The multiple dimensions include several of the following image evaluation dimensions: overall image effect, lighting, content, background, foreground, and composition.

5. The method according to claim 1 or 2, characterized in that, The step of determining multiple candidate images from the image to be processed and scoring each of the multiple candidate images includes: Based on the reference image selected by the user, an image score is determined for each candidate image, and the structural similarity between the candidate image and the reference image is positively correlated with the image score of the candidate image.

6. The method according to claim 5, characterized in that, The step of determining the image score of each candidate image based on the reference image includes: For any one of the multiple candidate images, feature extraction is performed on the candidate image using an image structure feature extraction model to obtain the candidate structure features of the candidate image. Based on the structural feature difference between the candidate structural features and the reference structural features of any candidate image, the image score of any candidate image is determined. The reference structural features are obtained by extracting features from the reference image using an image structural feature extraction model. The structural similarity between any candidate image and the reference image is negatively correlated with the structural feature difference.

7. The method according to claim 6, characterized in that, The text structure feature extraction model and the image structure feature extraction model are obtained by adjusting the parameters of the initial text structure feature extraction model and the initial image structure feature extraction model based on the differences between the training text structure features and the training image structure features. The training text structure features are obtained by the initial text structure feature extraction model extracting features from the training structure text, and the training image structure features are obtained by the initial image structure feature extraction model extracting features from the training candidate images. The training structure text is used to describe the structure of the training candidate images.

8. The method according to claim 5, characterized in that, Before determining multiple candidate images from the image to be processed and scoring each of the multiple candidate images, the method further includes: A third interface is displayed, which includes multiple alternative images; The system receives a user selection operation from the third interface, where the selected candidate image is the reference image.

9. The method according to any one of claims 1, 2, 6-8, characterized in that, The step of determining multiple candidate images from the image to be processed and scoring each of the multiple candidate images to obtain an image score for each candidate image includes: A first image set is determined from the image to be processed, the first image set including at least one of the candidate images; Perform multiple iterations, where the i-th iteration includes: Each candidate image in the i-th image set is scored to obtain an image score for the candidate image; Based on the image score of each candidate image in the i-th image set, the (i+1)-th image set is determined. The (i+1)-th image set includes at least one candidate image. i is a positive integer. The maximum value of the image score of at least one candidate image in the (i+1)-th image set is greater than or equal to the maximum value of the image score of at least one candidate image in the i-th image set. When i equals 1, the i-th image set is the first image set. The iteration operation ends when the iteration termination condition is met.

10. The method according to claim 9, characterized in that, Before performing multiple iterations, the method further includes: encoding the region information of the region where each candidate image in the first image set is located to obtain the chromosome corresponding to each candidate image, wherein the first chromosome set includes the chromosome corresponding to each candidate image in the first image set; The step of determining the (i+1)th image set based on the image score of each candidate image in the i-th image set includes: A target operation is performed on the i-th chromosome set to obtain the (i+1)-th chromosome set. The target operation includes one or more of selection, crossover, and mutation. The probability that any chromosome in the i-th chromosome set is selected as a chromosome in the (i+1)-th chromosome set is positively correlated with the image score of the candidate image corresponding to any chromosome. When i equals 1, the i-th chromosome set is the first chromosome set. Decode each chromosome in the (i+1)th chromosome set to obtain the region information of the candidate image region corresponding to each chromosome in the (i+1)th chromosome set; The (i+1)th image set is determined based on the region information of the candidate images corresponding to each chromosome in the (i+1)th chromosome set. The (i+1)th image set includes the candidate images corresponding to each chromosome in the (i+1)th chromosome set.

11. The method according to any one of claims 1, 2, 6-8, and 10, characterized in that, The step of determining the target subject region in the image to be processed based on the description information obtained based on the first user operation includes: Based on the description information, multiple candidate subjects are identified in the image to be processed; The target subject region is determined based on the depth of each of the plurality of candidate subjects, wherein the subject in the target subject region is the subject with the smallest depth among the plurality of candidate subjects.

12. The method according to any one of claims 1, 2, 6-8, and 10, characterized in that, The candidate region where the target image is located is a part of the image to be processed. The method further includes: adjusting the color of the target image to achieve white balance.

13. The method according to any one of claims 1, 2, 6-8, and 10, characterized in that, The method is applied to an electronic device, where the image to be processed is a preview image captured by the electronic device through a camera.

14. An electronic device, characterized in that, The electronic device includes: one or more processors, and memory; The memory is coupled to the one or more processors, the memory being used to store computer program code, the computer program code including computer instructions, the one or more processors invoking the computer instructions to cause the electronic device to perform the method as described in any one of claims 1 to 13.

15. A chip system, characterized in that, The chip system is applied to an electronic device, the chip system including one or more processors, the one or more processors being used to invoke computer instructions to cause the electronic device to perform the method as described in any one of claims 1 to 13.

16. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes instructions that, when executed on an electronic device, cause the electronic device to perform the method as described in any one of claims 1 to 13.

Images