Image processing method, model training method and related device

Abstract

The present application provides an image processing method, a model training method and related devices. In the method, an electronic device inputs a first image into a multi-task learning model, and determines, by the multi-task learning model, scene corresponding feature information and object corresponding feature information from the first image corresponding feature information. Then, the multi-task learning model outputs a scene classification result according to the scene corresponding feature information, and outputs a target detection result according to the object corresponding feature information. The electronic device determines a target exposure parameter according to the shooting scene indicated by the scene classification result and / or the shooting object indicated by the target detection result, and displays the second image after collecting the second image based on the target exposure parameter. Thus, the electronic device can determine the scene classification result and the target detection result required for automatic exposure by one model, thereby reducing the computational pressure of the electronic device and improving the performance of the electronic device.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence, and in particular, to an image processing method, a model training method, and related devices. BACKGROUND

[0002] Automatic exposure (AE) is a widely used technology in photography and videography. For example, an electronic device adjusts exposure parameters (e.g., sensitivity, aperture size, exposure time, etc.) according to ambient light to obtain an image or video with appropriate brightness through automatic exposure technology.

[0003] Generally, an electronic device needs to determine exposure parameters through corresponding manners for specific shooting scenes (e.g., snow scenes, night scenes, etc.) and specific shooting objects (e.g., human faces, etc.), so the electronic device can determine the category of the shooting scene and the category of the shooting object before determining the exposure parameters.

[0004] However, determining multiple shooting scenes and multiple shooting objects respectively can affect the power consumption and memory usage efficiency of the electronic device, thereby reducing the use performance of the electronic device. SUMMARY

[0005] The image processing method, the model training method, and the related devices provided by the embodiments of the present application can determine the scene classification result and the target detection result required by automatic exposure through a multi-task learning model, thereby reducing the computational pressure of the electronic device and improving the performance of the electronic device.

[0006] In a first aspect, the present application provides an image processing method, which comprises:

[0007] obtaining a first image, wherein the first image is collected based on an initial exposure parameter;

[0008] inputting the first image into a multi-task learning model, and outputting a scene classification result and a target detection result through the multi-task learning model, wherein the scene classification result is used to represent a shooting scene corresponding to the first image, and the target detection result is used to represent a shooting object contained in the first image;

[0009] determining a target exposure parameter according to the scene classification result and / or the target detection result;

[0010] displaying a second image, wherein the second image is collected based on the target exposure parameter.

[0011] The initial exposure parameter can be an exposure parameter used by the electronic device by default after starting a camera application, can be an exposure parameter manually selected by a user and received by the electronic device, or can be an exposure parameter determined by the electronic device based on the brightness of a previous frame of image.

[0012] In the above method, the electronic device can determine the shooting scene and the shooting object corresponding to the first image by using one multi-task learning model. Compared with the prior art in which multiple algorithms are called to perform a scene classification task and / or a target detection task, the present application has smaller power consumption and uses less memory when running one model, thereby improving the use performance of the electronic device and shortening the processing time of automatic exposure. In addition, the electronic device can simplify the steps of determining the exposure parameter based on the result output by one model. Moreover, when there is an incorrect result in the result output by the multi-task learning model, the electronic device can update the subsequent algorithm of the multi-task learning model based on the incorrect result, thereby continuously improving the automatic exposure effect. Meanwhile, compared with the prior art in which more algorithms are deployed, the present application can reduce the cost and the maintenance burden of the algorithms.

[0013] In a possible implementation of the first aspect, the inputting the first image into the multi-task learning model and outputting a scene classification result and a target detection result by the multi-task learning model comprises:

[0014] inputting the first image into the multi-task learning model and determining feature information corresponding to the first image by the multi-task learning model, wherein the feature information corresponding to the first image comprises feature information corresponding to the shooting scene and the shooting object;

[0015] determining the feature information corresponding to the shooting scene and the feature information corresponding to the shooting object according to the feature information corresponding to the first image;

[0016] determining the scene classification result based on the feature information corresponding to the shooting scene by a first output layer in the multi-task learning model;

[0017] determining the target detection result based on the feature information corresponding to the shooting object by a second output layer in the multi-task learning model.

[0018] In the existing scene classification scheme or target detection scheme, the feature information required by the task is usually directly extracted from the input image, and then the extracted feature information is processed to obtain the result, without feature decoupling. The embodiment of the present application decouples the feature information required by the task from the feature information corresponding to the first image, so that the feature information corresponding to the shooting scene not only contains the relevant information of the shooting scene, but also contains the relevant information of the shooting object, and similarly, the feature information corresponding to the shooting object not only contains the relevant information of the shooting object, but also contains the relevant information of the shooting scene, that is, there is shared feature information in the feature information corresponding to the shooting scene and the feature information corresponding to the shooting object. The electronic device determines the scene classification result based on the feature information corresponding to the shooting scene and the shared feature information, which can enhance the understanding of the multi-task learning model for the first image compared with the electronic device determining the scene classification result based only on the feature information corresponding to the shooting scene, thereby outputting a scene classification result with higher accuracy. Similarly, the electronic device can obtain a target detection result with higher accuracy based on the feature information corresponding to the shooting object and the shared feature information.

[0019] In a possible implementation of the first aspect, the determining the feature information corresponding to the shooting scene and the feature information corresponding to the shooting object according to the feature information corresponding to the first image comprises:

[0020] determining a first weight and a second weight;

[0021] determining the feature information corresponding to the shooting object according to the feature information corresponding to the first image and the first weight;

[0022] determining the feature information corresponding to the shooting scene according to the feature information corresponding to the first image and the second weight.

[0023] In the above method, in addition to being used to represent the percentage of a certain factor (for example, the feature information corresponding to the shooting scene or the feature information corresponding to the shooting object) in the whole (for example, the feature information corresponding to the first image), the weight is also used to represent the importance or contribution of the factor in the whole. Therefore, the feature decoupling according to the first weight and the second weight in the embodiment of the present application can be understood as being performed according to the importance of the feature information corresponding to the shooting scene or the feature information corresponding to the shooting object in the feature information corresponding to the first image. For example, taking the first weight as an example, the first weight is used to represent the importance of the feature information corresponding to the shooting object relative to the feature information corresponding to the first image, and the determination of the feature information corresponding to the shooting object according to the first weight and the feature information corresponding to the first image can enhance the useful feature information (information related to the shooting object) and suppress the useless feature information (feature information irrelevant to the shooting object), thereby making the determined feature information of the shooting object have a higher correlation with the shooting object.

[0024] In a possible implementation of the first aspect, the feature information comprises feature information in a channel dimension and feature information in a spatial dimension, and the determining the first weight and the second weight comprises:

[0025] performing first processing on the feature information corresponding to the first image in the channel dimension to determine a first processing result, wherein the first processing result is used to represent a weight of the feature information corresponding to the photographed object relative to the feature information corresponding to the first image in the channel dimension;

[0026] performing second processing on the feature information corresponding to the first image in the spatial dimension to determine a second processing result, wherein the second processing result is used to represent a weight of the feature information corresponding to the photographed object relative to the feature information corresponding to the first image in the spatial dimension;

[0027] determining the first weight according to the first processing result and the second processing result, wherein the first weight is used to represent a weight of the feature information corresponding to the photographed object relative to the feature information corresponding to the first image in the channel dimension and in the spatial dimension;

[0028] determining the second weight according to the first weight.

[0029] Exemplarily, the first processing on the feature information corresponding to the first image in the channel dimension can refer to the following expression:

[0030]

[0031] wherein, is the feature information corresponding to the first image. is average pooling processing on the feature corresponding to the first image, is maximum pooling processing on the feature corresponding to the first image, is a full connection layer structure, is an activation function.

[0032] Exemplarily, the second processing on the feature information corresponding to the first image in the spatial dimension can refer to the following expression:

[0033]

[0034] wherein, is the feature information corresponding to the first image. is average pooling processing on the feature information corresponding to the first image, is maximum pooling processing on the feature information corresponding to the first image, for the convolution operation, and for the convolution operation, is an activation function.

[0035] For example, the first weight can be determined according to the following expression:

[0036]

[0037] wherein, is the feature information corresponding to the first image, is the first weight, is the first processing result, is the second processing result. indicates that the size of the first processing result and the size of the second processing result are both expanded to corresponding sizes, and then the expanded first processing result and the expanded second processing result are accumulated and averaged to obtain a result.

[0038] In the above method, the feature information corresponding to the first image extracted by the electronic device includes feature information in the channel dimension and feature information in the spatial dimension. The electronic device can process the feature information corresponding to the first image in the channel dimension and in the spatial dimension, respectively, to obtain the weight of the feature information corresponding to the photographed object in the channel dimension relative to the feature information corresponding to the first image, and the weight of the feature information corresponding to the photographed object in the spatial dimension relative to the feature information corresponding to the first image.

[0039] In one implementation, different channels can have different contributions to different features. For example, some channels can contain more key information (e.g., information related to the photographed object), while other channels can contain noise or redundant information (e.g., information unrelated to the photographed object). Therefore, the electronic device can determine the weight of each channel containing key information (e.g., information related to the photographed object), and by giving higher weights to channels containing more key information and lower weights to channels containing less key information, the features related to the photographed object can be enhanced and the features unrelated to the photographed object can be suppressed in the channel dimension. Similarly, the features related to the photographed object can be enhanced and the features unrelated to the photographed object can be suppressed in the spatial dimension. Thus, the first weight and the second weight are determined based on the weight in the channel dimension and the weight in the spatial dimension, so as to decouple the feature information corresponding to the photographed object from the feature information corresponding to the first image.

[0040] In one possible implementation of the first aspect, the determining, by the first output layer in the multi-task learning model, of the scene classification result based on the feature information corresponding to the photographed scene comprises:

[0041] input the feature information corresponding to the shooting scene into the first output layer in the multi-task learning model, and output the scene classification result through the first output layer, wherein the scene classification result includes a first judgment result and a category corresponding to the shooting scene, and the first judgment result is used to represent whether the shooting scene belongs to a preset scene.

[0042] For example, the first output layer includes a scene discriminator and a scene classifier. The scene discriminator is used to determine whether the shooting scene corresponding to the first image is a preset scene, and obtain a first judgment result. The scene classifier is used to determine the category corresponding to the shooting scene.

[0043] The preset scene includes, but is not limited to, a snow scene, a stage, a grassland, a night scene, and the like.

[0044] In the above method, the electronic device outputs the first judgment result and the category corresponding to the shooting scene in the form of two separate data. When the electronic device determines the target exposure parameter based on the scene classification result, the first judgment result can be analyzed first. For example, if the first judgment result is that the shooting scene does not belong to the preset scene, the electronic device does not analyze the category corresponding to the shooting scene, but further analyzes the target detection result, which can simplify the amount of data processed by the electronic device and further speed up the automatic exposure.

[0045] In a possible implementation of the first aspect, the determining the target exposure parameter according to the scene classification result and / or the target detection result includes:

[0046] In a case where the first judgment result is that the shooting scene belongs to the preset scene, the target exposure parameter is determined according to the category corresponding to the shooting scene.

[0047] In the above method, since the shooting scene contains all visible elements of the first image, the shooting scene is related to the information of the whole first image. When the shooting scene belongs to the preset scene, it means that the electronic device needs to adjust the overall brightness of the image, and at this time, the electronic device also synchronously adjusts the brightness of the shooting object contained in the first image. The electronic device does not need to consider the category of the shooting object, and can adjust the brightness of the image to an appropriate state according to the category of the shooting scene. This saves the computing resources of the electronic device and speeds up the automatic exposure of the electronic device.

[0048] In a possible implementation of the first aspect, the determining the target detection result based on the feature information corresponding to the shooting object through the second output layer in the multi-task learning model includes:

[0049] input the feature information corresponding to the photographed object into the second output layer in the multi-task learning model, and output the target detection result through the second output layer, wherein the target detection result comprises a category of the photographed object and position information of the photographed object in the first image.

[0050] For example, the second output layer comprises a target detector, which follows the design principle of decoupled heads and can simultaneously detect the category of the photographed object and the position of the photographed object.

[0051] In the above method, the target detector follows the design principle of decoupled heads and includes a part of network structure shared by the target detectors, so as to realize the simplification of the target detector structure, balance the representation ability of the target detector to the operator and the calculation overhead of the hardware in the electronic device, and make the results output by the shared structure respectively used for determining the type of the photographed object and determining the position information of the photographed object in the first image, thereby reducing the operation pressure of the electronic device and saving the processing time of the electronic device.

[0052] In a possible implementation of the first aspect, the determining of the target exposure parameter according to the scene classification result and / or the target detection result comprises:

[0053] In a case where the first determination result indicates that the photographed scene does not belong to the preset scene, when the category of the photographed object belongs to a preset category, the target exposure parameter is determined according to the category of the photographed object and the position information of the photographed object in the first image.

[0054] In the above method, when the electronic device determines that the category of the photographed object belongs to a preset category, it indicates that the electronic device needs to adjust the exposure parameter at this time. The electronic device determines the target exposure parameter based on the photographed object, can collect a second image with moderate brightness based on the target exposure parameter, and thus adjusts the brightness of the first image in a targeted manner, thereby improving the exposure quality of the details of the first image.

[0055] In a second aspect, the present application provides a model training method, which comprises:

[0056] obtaining a training set of a multi-task learning model, wherein the training set comprises one or more of the following: a training image, a scene classification label corresponding to the training image, and a target detection label, wherein the scene classification label is used to represent a photographed scene corresponding to the training image, and the target detection label is used to represent a photographed object contained in the training image;

[0057] The training image is input into the multi-task learning model, the scene classification label is output by a first output layer of the multi-task learning model, and the target detection label is output by a second output layer of the multi-task learning model, so as to train the multi-task learning model to obtain a trained multi-task learning model, wherein the multi-task learning model is used to determine a shooting scene corresponding to an image and a shooting object contained in the image.

[0058] In the above method, the scene classification label is output by the first output layer, so that the ability of the multi-task learning model in scene classification can be trained. The target detection label is output by the second output layer, so that the ability of the multi-task learning model in target detection can be trained, thereby improving the ability of the multi-task learning model in determining a shooting scene and a shooting object.

[0059] In a possible implementation of the second aspect, the training of the multi-task learning model to obtain the trained multi-task learning model comprises:

[0060] The multi-task learning model is used to determine feature information corresponding to the training image, wherein the feature information corresponding to the training image comprises feature information corresponding to the shooting scene and the shooting object;

[0061] The third feature information is determined according to the feature information corresponding to the training image, wherein the third feature information is feature information corresponding to the shooting scene in the feature information corresponding to the training image, and the fourth feature information is feature information corresponding to the shooting object in the feature information corresponding to the training image;

[0062] The multi-task learning model is trained based on the third feature information and the fourth feature information to obtain the trained multi-task learning model.

[0063] In the above method, the feature information corresponding to the image input into the multi-task learning model can comprise feature information shared by multiple tasks, or can comprise feature information specific to a certain task. The feature information corresponding to the training image is decoupled into the third feature information and the fourth feature information, so that the feature information corresponding to any one of the multiple tasks can also comprise feature information shared with other tasks on the basis of determining feature information specific to each task in the multiple tasks. Thus, any one of the multiple tasks can also strengthen the understanding of the image based on the shared feature information on the basis of processing the feature information specific to the task, thereby improving the training effect of the multi-task learning model.

[0064] In a possible implementation of the second aspect, the training of the multi-task learning model based on the third feature information and the fourth feature information to obtain the trained multi-task learning model comprises:

[0065] determining, by the first output layer, a scene classification result corresponding to the training image according to the third feature information;

[0066] determining, by the second output layer, a target detection result corresponding to the training image according to the fourth feature information;

[0067] training the multi-task learning model based on a result of comparing the scene classification result corresponding to the training image with the scene classification label and a result of comparing the target detection result corresponding to the training image with the target detection label, to obtain the trained multi-task learning model.

[0068] In the above method, the scene classification result output by the first output layer is compared with the scene classification label, and the target detection result output by the second output layer is compared with the target detection label, so that the parameters of the multi-task learning model can be optimized in a targeted manner, thereby improving the training effect.

[0069] In a third aspect, an electronic device is provided, and the electronic device comprises one or more processors and one or more memories. The one or more memories are coupled to the one or more processors. The one or more memories are configured to store computer program codes, and the computer program codes comprise computer instructions. The one or more processors are configured to invoke the computer instructions to cause the electronic device to perform the image processing method, the training method of the model described in the first aspect or any possible implementation of the first aspect.

[0070] In a fourth aspect, a chip or a chip system is provided, and the chip or the chip system comprises at least one processor and a communication interface. The communication interface and the at least one processor are connected through a circuit. The at least one processor is configured to run computer programs or instructions to perform the image processing method, the training method of the model described in the first aspect or any possible implementation of the first aspect. The communication interface in the chip can be an input / output interface, a pin, or a circuit.

[0071] In a possible implementation, the chip or the chip system described in the embodiments of the present application further comprises at least one memory. The at least one memory stores instructions. The memory can be a storage unit inside the chip, such as a register, a cache, or the like. Alternatively, the memory can be a storage unit of the chip, such as a read-only memory, a random access memory, or the like.

[0072] In a fifth aspect, an embodiment of the present application provides a computer storage medium, which stores a computer program. The computer program is executed by a processor to cause a computer to perform the image processing method, the model training method described in the first aspect or any possible implementation manner of the first aspect.

[0073] In a sixth aspect, an embodiment of the present application provides a computer program product. When the computer program product is run on a communication device, the communication device performs the image processing method, the model training method described in the first aspect or any possible implementation manner of the first aspect.

[0074] It should be understood that the description of technical features, technical solutions, advantages or similar language in the present application does not imply that all features and advantages can be realized in any single embodiment. On the contrary, it can be understood that the description of a feature or advantage means that the specific technical feature, technical solution or advantage is included in at least one embodiment. Therefore, the description of technical features, technical solutions or advantages in the specification does not necessarily refer to the same embodiment. Furthermore, the technical features, technical solutions and advantages described in the embodiments can be combined in any appropriate manner. Those skilled in the art will understand that the embodiments can be implemented without one or more specific technical features, technical solutions or advantages of a specific embodiment. In other embodiments, additional technical features and advantages can be identified in specific embodiments that do not embody all embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0075] The following describes the drawings used in the embodiments of the present application.

[0076] Figure 1 is a structural schematic diagram of an electronic device provided by an embodiment of the present application;

[0077] Figure 2 is a software architecture schematic diagram of an electronic device provided by an embodiment of the present application;

[0078] Figure 3A is a system desktop of an electronic device provided by an embodiment of the present application;

[0079] Figures 3B-3D is a preview interface of a group of camera applications provided by an embodiment of the present application;

[0080] Figure 4A is a system desktop of an electronic device provided by an embodiment of the present application;

[0081] Figure 4B 、 Figure 4C is a preview interface of another group of camera applications provided by an embodiment of the present application;

[0082] Figure 5 is a flowchart of an image processing method provided by an embodiment of the present application;

[0083] Figure 6 is a structural diagram of a multi-task learning model provided by an embodiment of the present application;

[0084] Figure 7 is a flowchart of feature decoupling provided by an embodiment of the present application;

[0085] Figure 8 is a flowchart of determining a target exposure parameter provided by an embodiment of the present application. DETAILED DESCRIPTION

[0086] The terms used in the following embodiments of the present application are only for the purpose of describing specific embodiments and are not intended to be limiting of the present application. As used in the specification and the appended claims of the present application, the singular forms “a,” “an” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms “and / or” as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0087] Hereinafter, the terms “first” and “second” are only used for the purpose of description, and cannot be understood as implying or suggesting relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined with “first” and “second” can explicitly or implicitly include one or more of the features, and in the description of the embodiments of the present application, the meaning of “a plurality of” is two or more, unless otherwise specified.

[0088] In order to facilitate the clear description of the technical solutions of the embodiments of the present application, the following briefly introduces some terms involved in the embodiments of the present application.

[0089] 1. Multi-task Learning (MTL): Multi-task learning is a machine learning method that improves the generalization ability and learning efficiency of a model by training multiple related tasks simultaneously. In multi-task learning, the model shares some parameters and feature representations to transfer knowledge between different tasks. For example, in the embodiments of the present application, the electronic device trains a multi-task learning model that can handle scene classification tasks and target detection tasks in parallel through multi-task learning, so that the automatic exposure function can be realized through one multi-task learning model.

[0090] 2、Scene classification: Scene classification is an important task in computer vision and machine learning, which aims to identify and classify different scenes or environments presented in an image. This process involves using algorithms to analyze the features of an image to determine the type of scene the image represents. For example, in the embodiments of the present application, the electronic device performs scene classification on the captured image to obtain the shooting scene, to determine whether the shooting scene belongs to the preset scene, and if it belongs to the preset scene, the specific scene category can also be determined based on the scene classification task.

[0091] 3、Object recognition: Object recognition aims to detect and identify specific objects in an image or video. It not only involves identifying the category of the object, but also often needs to determine the location of the object in the image, and can further achieve object tracking and analysis. For example, in the embodiments of the present application, the electronic device performs object recognition on the captured image to obtain the shooting object, to determine whether the shooting object is a shooting object of a preset category, and if it is a shooting object of a preset category, the specific object category and the position information of the shooting object relative to the image can also be determined based on the object recognition task.

[0092] 4、Exposure convergence: In computer vision, exposure convergence can be used to describe how to adjust the exposure parameters of the camera in a dynamic environment to obtain the best exposure effect. For example, in the embodiments of the present application, the electronic device determines the target brightness based on the shooting scene and / or the shooting object of the input image (e.g., the first image), determines the target exposure parameter according to the brightness of the input image and the target brightness, and then captures the second image based on the target exposure parameter, so that the brightness of the second image is equal to or close to the target brightness, thereby achieving exposure convergence.

[0093] 5、18% neutral gray: The brightness and color of an object are determined by the reflectivity of the object to light, for example, the reflectivity of a pure black object is 0, the reflectivity of a pure white object is 100%, and the reflectivity of an object in a neutral gray is 18%. On the color spectrum, the central neutral gray is the average of all grays, so it is generally believed that the average reflectivity of objects in nature is 18% neutral gray. Therefore, the electronic device can take 18% neutral gray as the target brightness, and adjusting the brightness of the image to 18% neutral gray can make multiple photographed objects present an average brightness, thereby reducing the situation of overexposure or underexposure of the image.

[0094] 6、Exposure parameter: Exposure parameter is one of the three main factors that affect the brightness and quality of an image in photography and videography, which is called exposure triangle, and they are aperture, shutter speed and sensitivity. By reasonably adjusting the aperture, shutter speed and sensitivity, the exposure, sharpness and noise of the image can be effectively controlled, thereby improving the imaging quality of the image. In the embodiments of the present application, the electronic device can achieve automatic exposure by determining the exposure parameter.

[0095] To facilitate understanding of the embodiments of the present application, the technical problems to be solved by the present application are analyzed and proposed as follows.

[0096] With the popularity of electronic devices (e.g., smart terminal devices), users use electronic devices to collect images as daily routines. In order to improve the quality of images, the electronic device can determine exposure parameters (e.g., sensitivity, aperture size, exposure time, etc.) according to ambient light through an automatic exposure (AE) algorithm, so as to collect images with appropriate brightness based on the exposure parameters, and reduce the overexposure or underexposure of images.

[0097] Generally, the electronic device sets the exposure parameters with the goal of adjusting the brightness of the image to 18% of the middle gray. However, in some boundary cases, the electronic device may be difficult to collect images with appropriate brightness based on 18% of the middle gray, and it is necessary to re-determine the target brightness based on the current shooting scene and / or shooting object, so as to determine the exposure parameters based on the re-determined target brightness.

[0098] For example, the boundary case includes the case where the shooting scene contains a large area of dark area or a large area of bright area. For example, in the shooting scene of snow scene, there is a large area of bright area, at this time, the brightness of the image collected by the electronic device is too high (i.e., higher than 18% of the middle gray), and the electronic device reduces the luminous flux to reduce the brightness of the image, so that the brightness of the image approaches 18% of the middle gray (target brightness). Since the electronic device reduces the luminous flux, it may cause the final collected image to be underexposed, and the imaging effect is poor. In order to solve the problem that the electronic device is difficult to collect images with appropriate brightness based on 18% of the middle gray in the preset scene (e.g., snow scene), the target brightness can be re-determined based on the preset scene (e.g., a target brightness higher than 18% of the middle gray is determined for the shooting scene of snow scene), and then the exposure parameters suitable for the shooting scene are determined based on the target brightness, so as to reduce the overexposure or underexposure of images.

[0099] For example, the boundary condition includes a case where the shooting object contains a preset target, including but not limited to a human face, an animal, a plant, a ball, and the like, which the electronic device may focus on during shooting. Since the electronic device may need the preset target to present a better imaging effect (for example, an exposure effect with moderate brightness) when the shooting object contains the preset target. However, the proportion of the area of the preset target in the total area of the picture may be small, and the brightness of the large area of the picture other than the preset target has a greater impact on the electronic device controlling the exposure parameter, which may cause the exposure parameter determined by the electronic device to be unsuitable for the preset target, so that the exposure effect of the preset target in the captured image is poor, affecting the user's shooting experience. To solve the problem that the electronic device is difficult to capture an image with moderate brightness based on 18% of the middle gray when the shooting object contains a preset target, the target brightness can be determined based on the shooting object (for example, the preset target), and then the exposure parameter suitable for the shooting object is determined based on the target brightness, thereby reducing the overexposure or underexposure of the image.

[0100] Therefore, before adjusting the exposure parameter, the electronic device can first identify the shooting scene and the shooting object. If the current shooting scene and / or the shooting object belong to the above boundary condition, the electronic device can determine the target brightness based on the identified shooting scene and / or shooting object, so as to determine the exposure parameter suitable for the shooting scene and / or the shooting object based on the target brightness, thereby reducing the overexposure or underexposure of the image and obtaining an image with moderate brightness.

[0101] Based on the continuous development of modern computer vision and image processing fields, the electronic device is preconfigured with a plurality of algorithms related to scene classification and target detection. For example, in the field of scene classification, there are a snow scene classification algorithm for identifying a snow scene, a stage classification algorithm for identifying a stage, and the like. For another example, in the field of target detection, there are an intelligent face mask algorithm for detecting a human face, a flower detection algorithm for identifying a flower, and the like. It can be seen that the above algorithms are all for single task identification or detection, for example, the intelligent face mask algorithm is used for detecting a human face, and the snow scene classification algorithm is used for judging whether the shooting scene is a snow scene. This is because the above algorithms may be generated in the development process of the image processing field. For example, when a human face in an image needs to be detected, a related manufacturer mines features based on the human face, trains a human face detection model, and finally deploys the trained human face detection model online. The electronic device can call the human face detection model to realize the human face detection function. Therefore, with the development of the image processing field, a plurality of algorithms for single task are designed.

[0102] Generally, in the automatic exposure scheme of the electronic device, the electronic device can call one or more algorithms in the above algorithms as needed to perform a scene classification task and / or a target detection task, which has a certain flexibility. However, when the number of algorithms called by the electronic device is large, the power consumption of simultaneously running multiple algorithms by the electronic device is large, and the memory occupied by simultaneously running multiple algorithms is large, which can cause the performance of the electronic device to decrease, thereby affecting the processing time of the electronic device when performing automatic exposure. In addition, if the electronic device calls multiple algorithms, the multiple algorithms can output corresponding results at different times, which affects the time length of the electronic device for determining the exposure parameter based on the above results, and the determination of the exposure parameter based on the results output by the multiple algorithms can increase the complexity of the electronic device in determining the exposure parameter. Moreover, since the multiple algorithms are independently developed and isolated from each other, when there is an error result in the results output by the multiple algorithms respectively, the electronic device is difficult to determine the algorithm outputting the error result, thereby being difficult to update the subsequent algorithm of the algorithm outputting the error result, and affecting the effect of the electronic device performing automatic exposure. At the same time, the deployment of more algorithms by the electronic device increases the cost of the electronic device and the maintenance burden of the algorithms.

[0103] In summary, the electronic device calling multiple algorithms for automatic exposure has problems of large power consumption, large memory occupation, high complexity in determining the exposure parameter, great difficulty in optimization, high cost, etc. How to effectively integrate multiple independent algorithms to improve the efficiency of determining the exposure parameter and optimize the exposure effect of the image is a problem to be solved in the research of automatic exposure algorithms.

[0104] Therefore, an image processing method is provided in the embodiments of the present application. After the electronic device acquires a first image based on an initial exposure parameter, the electronic device inputs the first image into a multi-task learning model, and outputs a scene classification result and a target detection result through the multi-task learning model. Then, the electronic device determines a target exposure parameter according to a shooting scene indicated by the scene classification result and / or a shooting object indicated by the target detection result, and displays a second image acquired by the electronic device based on the target exposure parameter. In the embodiments of the present application, the electronic device can determine the scene classification result and the target detection result through one model (i.e., the multi-task learning model), which reduces the number of models compared with the prior art in which multiple models are needed to determine the scene classification result and the target detection result, reduces the operation pressure of the electronic device, and further improves the performance of the electronic device.

[0105] The image processing method of the embodiments of the present application will be described below in combination with the hardware structure and software structure of a mobile phone.

[0106] In some embodiments, the electronic device includes, but is not limited to, a mobile phone, a tablet (PAD), a personal digital assistant (PDA), a handheld device with a camera function, a computing device, a vehicle-mounted device, or a wearable device. The form of the electronic device is not limited in the embodiments of the present application.

[0107] Please refer to Figure 1 , Figure 1 is a structural schematic diagram of an electronic device provided by the embodiments of the present application, as Figure 1 shown, the electronic device 100 can include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a loudspeaker 170A, a receiver 170B, a microphone 170C, a headset interface 170D, a sensor module 180, a key 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 can include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0108] It can be understood that the structure shown in the embodiments of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 can include more or fewer components than shown, or combine certain components, or split certain components, or different component arrangements. The components shown can be implemented in hardware, software, or a combination of software and hardware.

[0109] The processor 110 can generate operation control signals according to instruction operation codes and timing signals, and complete the control of fetching and executing instructions.

[0110] The processor 110 can also have a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory can hold instructions or data that the processor 110 has just used or is using repeatedly. If the processor 110 needs to use the instructions or data again, it can be called directly from the memory. This avoids repeated access and reduces the waiting time of the processor 110, thus improving the efficiency of the system.

[0111] In some embodiments, the processor 110 inputs a first image collected by the camera 193 into the multi-task learning model, extracts feature information corresponding to the first image from the first image through the multi-task learning model, where the feature information corresponding to the first image includes feature information corresponding to a shooting scene and a shooting object. Then, the multi-task learning model decouples the feature information corresponding to the first image into feature information corresponding to the shooting scene and feature information corresponding to the shooting object. The multi-task learning model obtains a scene classification result based on the feature information corresponding to the shooting scene, where the scene classification result includes a judgment result about whether the shooting scene corresponding to the first image belongs to a preset scene, and a specific scene category when it belongs to the preset scene. The multi-task learning model obtains a target detection result based on the feature information corresponding to the shooting object, where the target detection result includes a judgment result about whether the shooting object in the first image belongs to a preset category, and a specific object category when it belongs to the preset category. The processor 110 determines a target exposure parameter according to the scene classification result and the target detection result, so that the camera 193 can collect an image based on the target exposure parameter.

[0112] In some embodiments, the processor 110 can include one or more interfaces. The interfaces can include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, and a subscriber identity module (SIM) interface.

[0113] The UART interface is a universal serial bus for asynchronous communication. The bus can be a bidirectional communication bus. It converts data to be transmitted between serial communication and parallel communication. In some embodiments, the UART interface is commonly used to connect the processor 110 and the wireless communication module 160.

[0114] The MIPI interface can be used to connect the processor 110 and peripheral devices such as the display screen 194 and the camera 193. The MIPI interface includes a camera serial interface (CSI), a display serial interface (DSI), and the like.

[0115] In some embodiments, the processor 110 and the display screen 194 communicate through the DSI interface to realize the display function of the electronic device 100.

[0116] The wireless communication function of the electronic device 100 can be realized through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor, and the baseband processor, and the like.

[0117] The antenna 1 and the antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in the electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be multiplexed to improve the utilization rate of the antennas. For example, the antenna 1 can be multiplexed as a diversity antenna of a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0118] The wireless communication module 160 can provide a wireless communication solution including wireless local area networks (WLAN) (such as a wireless fidelity (Wi-Fi) network), Bluetooth (BT), a global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared technology (IR), and the like, which are applied to the electronic device 100.

[0119] The electronic device 100 realizes the display function through the GPU, the display screen 194, and the application processor, and the like. The GPU is a microprocessor for image processing, which is connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 can include one or more GPUs, which execute program instructions to generate or change display information.

[0120] The internal memory 121 can be used to store computer executable program codes including instructions. The internal memory 121 can include a program storage area and a data storage area. The program storage area can store an operating system, at least one application required for at least one function (e.g., a sound play function, an image play function, etc.), and the like. The data storage area can store data (e.g., audio data, a phonebook, etc.) created during use of the electronic device 100, and the like. In addition, the internal memory 121 can include a high-speed random access memory, and can further include a non-volatile memory such as at least one of a magnetic disk storage device, a flash memory device, a universal flash storage (UFS), and the like. The processor 110 performs various function applications and data processing of the electronic device 100 by executing instructions stored in the internal memory 121 and / or instructions stored in a memory disposed in the processor.

[0121] In an embodiment of the present application, the internal memory 121 can store a multi-task learning model. The multi-task learning model can be used to determine whether a photographing scene of an image input to the multi-task learning model belongs to a preset scene, and to determine whether the image input to the multi-task learning model includes a photographing object of a preset category.

[0122] The pressure sensor 180A is used to sense a pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A can be disposed in the display screen 194. There are many types of pressure sensors 180A, such as a resistive pressure sensor, an inductive pressure sensor, a capacitive pressure sensor, and the like. The capacitive pressure sensor can include at least two parallel plates having a conductive material. When a force is applied to the pressure sensor 180A, the capacitance between the electrodes changes. The electronic device 100 determines the intensity of the pressure according to the change in capacitance. When a touch operation is applied to the display screen 194, the electronic device 100 detects the intensity of the touch operation according to the pressure sensor 180A. The electronic device 100 can also calculate the position of the touch according to the detection signal of the pressure sensor 180A. In some embodiments, touch operations applied to the same touch position but having different touch operation intensities can correspond to different operation instructions.

[0123] Touch sensor 180K, also known as a "touch device," can be located on display screen 194. The touch sensor 180K and display screen 194 together form a touchscreen, also known as a "touchscreen." Touch sensor 180K detects touch operations applied to or near it. The touch sensor can transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through display screen 194. In other embodiments, touch sensor 180K may also be located on the surface of electronic device 100, in a different position than display screen 194.

[0124] The software system of electronic devices (such as mobile phones) can adopt a layered architecture, transaction-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This application uses the layered architecture of the Android system as an example to illustrate the software architecture of a mobile phone. Please refer to... Figure 2 , Figure 2 This is a schematic diagram of the software architecture of the electronic device provided in the embodiments of this application.

[0125] like Figure 2 As shown, the layered architecture divides the software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into five layers, from top to bottom: application layer, application framework layer, hardware abstraction layer (HAL), driver layer, and hardware layer. Wherein:

[0126] The application layer can include a series of packages. For example, applications may include camera, gallery, and other applications. A camera application may include, but is not limited to, a UI module, a photo-taking module, a gallery module, etc. The UI module, such as the cameraUI module, is primarily responsible for human-computer interaction in the camera application, such as controlling the preview interface and its display, and receiving and responding to user actions within the preview interface. The photo-taking module provides functions such as taking photos and focusing. The gallery module can be used to store photos taken by the user in the electronic device's file system or a specific database for retrieval by applications such as the gallery.

[0127] The application framework layer provides application programming interfaces (APIs) and programming frameworks for applications in the application layer, and mainly involves a camera framework, which can include camera access interfaces such as a camera extension library and a camera service, and plays a role of presenting and enabling. The camera application can interact with the camera application through the application API, and can also interact with the HAL through the HAL interface definition language (HIDL). The application framework layer can also include a window manager, and the camera application and the gallery application can show the taken photos to the user under the support of the window manager.

[0128] The hardware abstraction layer (HAL) is an interface layer between the application framework layer and the driver layer, and provides a virtual hardware platform for the operating system. For example, the hardware abstraction layer can include a camera hardware abstraction layer and an auto exposure module. The camera hardware abstraction layer can provide virtual hardware for camera device 1 (a first camera), camera device 2 (a second camera), and more camera devices.

[0129] The auto exposure module stores a plurality of image processing algorithms. For example, in an embodiment of the present application, the auto exposure module can include an auto exposure algorithm and the like. The auto exposure module can determine a target exposure parameter in combination with the auto exposure algorithm and the shooting parameters reported by the camera module and / or other sensors.

[0130] In one implementation, the auto exposure module can perform scene classification and target detection on the first image according to the auto exposure algorithm to determine whether the shooting scene of the first image is a preset scene, and whether the first image contains a shooting object of a preset category. The first image is a frame of image collected by the electronic device through the camera device.

[0131] Specifically, the automatic exposure module can extract feature information corresponding to the first image from the first image according to a multi-task learning model in the automatic exposure algorithm, where the feature information corresponding to the first image includes feature information corresponding to a shooting scene and a shooting object. Then, the multi-task learning model decouples the feature information corresponding to the first image into feature information corresponding to the shooting scene and feature information corresponding to the shooting object. The multi-task learning model obtains a scene classification result based on the feature information corresponding to the shooting scene, where the scene classification result includes a judgment result about whether the shooting scene corresponding to the first image belongs to a preset scene, and a specific scene category in the preset scene. The multi-task learning model obtains a target detection result based on the feature information corresponding to the shooting object, where the target detection result includes a judgment result about whether the shooting object in the first image belongs to a preset category, and a specific object category when belonging to the preset category.

[0132] In an implementation, after obtaining the scene classification result and the target detection result, the automatic exposure module can further determine a target exposure parameter based on the scene classification result and the target detection result according to the automatic exposure algorithm, to drive the camera module to collect an image based on the target exposure parameter.

[0133] The driving layer is a layer between hardware and software, and includes driving of various hardware. The driving layer can include a camera device driver, a digital signal processor driver, and an image processor driver, etc. The camera device driver is used to drive an image sensor of one or more cameras in the camera module to collect an image, and drive an image signal processor to pre-process the image. The digital signal processor driver is used to drive a digital signal processor to process the image. The image processor driver is used to drive a graphics processor to process the image.

[0134] The hardware layer can include the camera module, the image signal processor, the digital signal processor, the image processor, and the memory.

[0135] The camera module can include an image sensor (for example, image sensor 1, image sensor 2, etc.) of one or more cameras.

[0136] The memory includes a plurality of storage units, and the electronic device can identify a shooting scene and a shooting object in the first image through a model in the memory. For example, in the existing scheme, the electronic device identifies the shooting scene and the shooting object in the first image through a plurality of models through a plurality of storage units. In the embodiment of the present application, the electronic device runs the multi-task learning model through one storage unit, which can realize identification of the shooting scene and the shooting object in the first image, and relieve the storage pressure of the electronic device.

[0137] The following illustrates the workflow of the software and hardware of the electronic device in the scenario of starting the camera application.

[0138] In the embodiments of the present application, when the touch sensor 180K receives the touch operation, a corresponding hardware interrupt is sent to the kernel layer, and the kernel layer processes the user operation into a raw input event (including touch coordinates, timestamp of touch operation, etc.), and identifies the control corresponding to the input event. Taking the touch operation as an example, the user operation on the camera application, the camera application calls the camera access interface of the application framework layer, starts the camera application, and then sends an instruction to start the camera capability by calling the camera device in the camera hardware abstraction layer, such as camera device 1. The camera hardware abstraction layer sends the instruction to the camera device driver in the driver layer, which can start the sensor corresponding to the camera device, such as sensor 1, and then collects image light signals through sensor 1 to obtain a first image.

[0139] In one implementation, the image signal processor returns the first image to the hardware abstraction layer through the camera device driver, and the hardware abstraction layer returns the first image data to the camera application through the camera interface. Subsequently, the camera application can show the first image to the user with the support of the window manager.

[0140] Meanwhile, the hardware abstraction layer can also send the first image to the automatic exposure module. Based on the support of the image signal processor and the digital signal processor, the automatic exposure module in the hardware abstraction layer can determine whether the shooting scene of the first image is a preset scene and whether the first image contains a shooting object of a preset category in combination with the automatic exposure algorithm and the shooting parameters reported by the camera module and / or other sensors. If the shooting scene of the first image is the preset scene, the automatic exposure module determines the target exposure parameter based on the shooting scene of the first image. If the shooting scene of the first image is not the preset scene and the first image contains a shooting object of the preset category, the automatic exposure module determines the target exposure parameter based on the shooting object of the first image. If the shooting scene of the first image is not the preset scene and the first image does not contain a shooting object of the preset category, the automatic exposure module does not output the target exposure parameter.

[0141] The camera device driver can start the sensor corresponding to the camera device again, such as sensor 1, and then collect image light signals based on the target exposure parameter through sensor 1 to obtain a second image.

[0142] The image signal processor returns the second image to the hardware abstraction layer through the camera device driver, and the hardware abstraction layer returns the second image data to the camera application through the camera interface. Subsequently, the camera application can show the second image to the user with the support of the window manager, and at this time, the exposure effect of the second image is better than that of the first image.

[0143] In a possible implementation, the hardware abstraction layer can also send the second image to an auto exposure module. Based on the support of the image signal processor and the digital signal processor, the auto exposure module in the hardware abstraction layer can determine the target exposure parameter corresponding to the second image in combination with an auto exposure algorithm, so as to capture an image based on the target exposure parameter corresponding to the second image until an image with moderate brightness (the brightness is equal to or close to the target brightness) is captured.

[0144] The image processing method provided by the embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0145] The image processing method provided by the embodiments of the present application can be applied to an electronic device with a hardware structure as shown in Figure 1 and a software structure as shown in Figure 2 Or more or less components than shown, or combination of certain components, or splitting of certain components, or different arrangement of components, and the like of the electronic device of the hardware structure and the software structure.

[0146] In the embodiments of the present application, the shooting scene in which the electronic device captures an image corresponds to the overall information of the image, which can affect the overall brightness of the image. The shooting object contained when the electronic device captures an image corresponds to the local information of the image, which can affect the local brightness of the image. Therefore, the embodiments of the present application adjust the overall brightness of the image based on the shooting scene, and / or adjust the local brightness of the image based on the shooting object, so that the captured image presents a high-quality exposure effect from the overall or the local.

[0147] The process of the electronic device performing auto exposure for a preset scene will be described below with reference to Figures 3A-3D .

[0148] Figure 3A is a system desktop of an electronic device provided by the embodiments of the present application, Figures 3B-3D is a preview interface of a group of camera applications provided by the embodiments of the present application.

[0149] Figure 3A is a system desktop 301 of an electronic device 100 provided by the embodiments of the present application. As shown in Figure 3A , the system desktop 301 can include a status bar 302, a page indicator 303, and a plurality of application icons.

[0150] The status bar 302 can include one or more signal strength indicators of mobile communication signals (may also be referred to as cellular signals), for example, a 5th Generation Mobile Communication Technology (5G) signal strength indicator, a wireless fidelity (Wi-Fi) signal strength indicator, a battery status indicator, a time indicator (for example, 8:00), and the like.

[0151] The page indicator 303 can be used to indicate the positional relationship of the currently displayed page with other pages.

[0152] The plurality of application icons can include a time application icon (for example, 08:00), a date application icon (for example, January 1, Friday), a weather application icon (for example, 5℃), an application market application icon, a memo application icon, a mall application icon, a browser application icon, a phone application icon, an information application icon, a camera application icon, a settings application icon, and the like. Without being limited to the above-mentioned icons, the system desktop 301 can also include other application icons, which are not listed one by one here. The plurality of application icons can be distributed in a plurality of pages. The page indicator 303 can be used to indicate which one of the plurality of pages carrying a plurality of application icons is currently browsed by the user. The user can browse other pages through left and right sliding touch operations.

[0153] It can be understood that, Figure 3A The user interface described above and the subsequent user interface examples exemplarily show a possible user interface style of an electronic device such as a mobile phone, and should not constitute a limitation on the embodiments of the present application.

[0154] As Figure 3A indicated, the electronic device 100 can receive a user operation for starting the camera application, for example, an operation of clicking the desktop icon of the camera application, and in response to the operation, the electronic device can display a preview interface of the camera application as shown in Figure 3B

[0155] Figure 3B The user interface of the photographing and display service provided by the embodiments of the present application is exemplarily shown, also referred to as the preview interface 304. In Figure 3B ​In the preview interface 304 shown, the electronic device 100 displays an image 305, a function selection area 306, a mode selection area 307, a zoom function 308, a gallery 309, a shooting button 310, and a switching button 311. The function selection area 306 includes at least one function setting button. Exemplarily, the function setting button includes, but is not limited to: a vision button, a flash button, an Artificial Intelligence (AI) camera button, a color button, and a settings button. Optionally, if the electronic device 100 detects that a user touches any function setting button in the function selection area 306, it sets the corresponding function according to the touched function setting button. For example, if it detects that a user touches the flash button, it sets the flash. Exemplarily, the mode selection area 307 includes at least one shooting mode button. Shooting mode buttons include, but are not limited to: an aperture mode button, a night scene mode button, a portrait mode button, a photo mode button, a video mode button, a professional mode button, and a more mode button. Optionally, if the electronic device 100 detects that the user has touched any shooting mode button in the mode selection area 307, it switches to the corresponding shooting mode according to the touched shooting mode button. For example, the gallery 309 is used to review captured images; specifically, the gallery 309 provides the user with an entry point to review captured images. The user clicks on the gallery 309 to enter the interface for reviewing captured photos. For example, the shooting button 310 is used to take a photo, and the switch button 311 is used to switch between the front and rear cameras during shooting.

[0156] like Figure 3B As shown, electronic device 100 responds to user operation from Figure 3A The system desktop 301 shown switches to Figure 3B After the preview interface 304 is displayed, the electronic device 100 can receive user operations to switch modes, such as swiping left / right in the mode selection area 307, and change the currently used shooting mode according to the operation. By default, the electronic device first uses the "take a picture" mode. In one implementation, if the shooting mode currently used by the electronic device 100 is not the "take a picture" mode, when it receives a left / right swipe operation in the mode selection area 307 and makes the float stop at the "take a picture" option, the electronic device 100 can switch to the "take a picture" mode.

[0157] like Figure 3C As shown, in "Photo" mode, image frames captured by the camera are displayed in the preview interface 304, for example, image 305. Optionally, the preview interface 304 can display images processed by image processing algorithms corresponding to different modes in real time, so that users can perceive the photo effects corresponding to different shooting modes in real time.

[0158] In the image 305, it can be seen that the image 305 corresponds to a snow scene, and contains a large area of bright region. For example, the brightness of the image collected by the electronic device 100 is high (i.e., higher than the middle gray scale of 18%), and the electronic device reduces the light flux to reduce the brightness of the image, so that the brightness of the image is close to the middle gray scale of 18% (the initial target brightness). Therefore, the final collected image 305 is underexposed, and the brightness is dark.

[0159] In a possible implementation, after collecting the image 305, the electronic device 100 can input the image 305 into the multi-task learning model. The scene classification result output by the multi-task learning model includes a result that the image 305 belongs to a preset scene and a result that the scene type corresponding to the image 305 is a snow scene, and the target detection result output by the multi-task learning model includes a result that the image 305 contains a shooting object (for example, a plant in the image 305) of a preset category and a result of the position information of the shooting object in the image 305. Then, based on the scene classification result that the shooting scene of the image 305 belongs to the preset scene, the electronic device 100 can determine the target exposure parameter based on the scene type corresponding to the image 305, and optionally, the target exposure parameter is determined based on the target detection result.

[0160] For example, the electronic device 100 obtains the target brightness based on the snow scene (for example, determines the target brightness higher than the middle gray scale of 18%), and then obtains the target exposure parameter based on the brightness of the image 305 and the target brightness. Optionally, the electronic device 100 collects the image 313 based on the target exposure parameter, and the image 313 is displayed in the preview interface 312 as shown in Figure 3D .

[0161] As shown in Figure 3D , the image 313 displayed in the preview interface 312 is collected by the electronic device 100 based on the target exposure parameter. It can be seen that the brightness of the image 313 is higher than that of the image 305, and the exposure degree of the image 313 is greater than that of the image 305, thereby solving the problem of underexposure in the preset scene of the snow scene.

[0162] The process of the electronic device for automatically exposing the shooting object will be described below. Figures 4A-4C

[0163] Figure 4A is a system desktop of an electronic device provided by an embodiment of the present application, Figure 4B , Figure 4C is another preview interface of a camera application provided by an embodiment of the present application.

[0164] Figure 4A ​The system desktop 301 of an electronic device is provided in the embodiments of the present application. As shown in Figure 4A , the system desktop 301 can include a status bar 302, a page indicator 303, and a plurality of application icons. The status bar 302, the page indicator 303, and the plurality of application icons can refer to the related description in Figure 3A , and will not be described here again.

[0165] It can be understood that Figure 4A the user interface introduced subsequently exemplarily shows a possible user interface style of the electronic device taking a mobile phone as an example, and should not constitute a limitation on the embodiments of the present application.

[0166] As shown in Figure 4A , the electronic device 100 can receive a user operation of opening a camera application, for example, clicking a desktop icon of the camera application, and in response to the operation, the electronic device can display a preview interface as shown in Figure 4B .

[0167] Figure 4B The user interface of the photographing and display service provided in the embodiments of the present application, also referred to as the preview interface 401, is exemplarily shown. In the preview interface 401 as shown in Figure 4B , the electronic device 100 displays an image 402, a function selection area 306, a mode selection area 307, a zoom function 308, a gallery 309, a photographing button 310, and a switching button 311. The function selection area 306, the mode selection area 307, the zoom function 308, the gallery 309, the photographing button 310, and the switching button 311 can refer to the related description in Figure 3B , and will not be described here again.

[0168] Among them, as can be seen from the image 402, the photographing object of the image 402 includes a white bird 4021 and leaves, the white bird 4021 is a bright area, the leaves are dark areas, and the area of the region where the white bird 4021 is located is smaller than the area of the region where the leaves are located. Optionally, the brightness of the picture collected by the electronic device 100 will be affected by the region where the leaves are located and will be low (i.e., lower than the intermediate gray scale of 18%). The electronic device can increase the luminous flux to increase the brightness of the picture, so that the brightness of the picture is close to the intermediate gray scale of 18% (the initial target brightness). Therefore, it will cause the overexposure of the white bird 4021 in the finally collected image 402, so that the brightness of the white bird 4021 is high, resulting in poor imaging quality of the image 402.

[0169] In a possible implementation, after the electronic device 100 collects the image 402, the electronic device 100 can input the image 402 into the multi-task learning model, and the output of the multi-task learning model includes: the image 402 does not belong to the preset scene, and the output of the target detection result includes: the image 402 contains a shooting object of a preset category (for example, the white bird 4021), and the position information of the shooting object (for example, the white bird 4021) in the image 402. Then, based on the scene classification result that the shooting scene of the image 402 does not belong to the preset scene, the electronic device 100 can continue to analyze the target detection result, and based on the target detection result, it is determined that the image 402 contains a shooting object of a preset category, and the electronic device 100 can determine the target exposure parameter based on the shooting object.

[0170] For example, the electronic device 100 determines the target brightness (for example, a target brightness lower than the middle gray of 18%) based on the animal (for example, the white bird 4021) shooting object, and then determines the target exposure parameter based on the brightness of the image 402 and the target brightness. Alternatively, the electronic device 100 collects the image 404 based on the target exposure parameter, which is displayed in the preview interface 403 as shown in Figure 4C .

[0171] As shown in Figure 4C , the image 404 displayed in the preview interface 403 is collected by the electronic device 100 based on the target exposure parameter. It can be seen that the brightness of the white bird 4041 in the image 404 is lower than that of the white bird 4021 in the image 402, and the exposure degree of the image 404 is lower than that of the image 402, thereby solving the problem of overexposure of the area where the white bird 4021 is located in the image 402.

[0172] The above Figures 3A-3D 、 Figures 4A-4C introduces the interface schematic diagram of the electronic device when performing automatic exposure. The process of the electronic device determining the shooting scene and the shooting object and performing automatic exposure based on the shooting scene and / or the shooting object will be described in detail below. Figure 5

[0173] Please refer to Figure 5 , Figure 5 is a flowchart of an image processing method provided by an embodiment of the present application. As shown in Figure 5 , the flowchart includes S501 to S504. First, the specific implementation of the electronic device performing automatic exposure through the multi-task learning model will be described in combination with the exemplary flowchart shown in Figure 5 .

[0174] In step S501, a first image is obtained.

[0175] ​Specifically, after the electronic device starts the camera application, a frame of image can be collected at a certain interval, and then the frame of image is displayed in the preview interface of the camera application, so as to provide the user with the change of the shooting scene and / or the shooting object in real time, so that the user adjusts the composition or saves the currently displayed image based on the image displayed in the preview interface. For example, the continuous multiple frames of images collected by the electronic device at a certain interval can be referred to as a preview stream, and the preview stream image displayed in the preview interface by the electronic device can be an image collected by the electronic device, or an image processed (such as exposure processing, noise reduction processing, color correction processing, etc.). Optionally, the first image and the second image in the embodiment of the present application are a frame of image in the preview stream. For example, the preview interface can refer to the preview interface 304 shown in Figure 3B and Figure 3C the preview interface 312 shown in Figure 3D the preview interface 401 shown in Figure 4B and the preview interface 403 shown in Figure 4C The first image can refer to the image 305 shown in Figure 3B and Figure 3C and the image 402 shown in Figure 4B

[0176] For example, the first image can be an image collected by the electronic device based on initial exposure parameters, for example, the initial exposure parameters can be the exposure parameters used by the electronic device by default after starting the camera application, or the exposure parameters manually selected by the user received by the electronic device, or the exposure parameters determined by the electronic device based on the brightness of the last frame of image, and the present application does not limit this.

[0177] In step S502, the first image is input into the multi-task learning model, and the scene classification result and the target detection result are output through the multi-task learning model.

[0178] Specifically, the multi-task learning model is used to process multiple tasks in parallel, for example, in the present application, the multi-task learning model is used to process the scene classification task and the target detection task of the first image in parallel, so as to obtain the scene classification result and the target detection result corresponding to the first image.

[0179] ​Exemplarily, the scene classification result is used to represent a shooting scene in which the electronic device captures the first image. For example, the scene classification result can include a result of judging a category of the shooting scene, such as a result of the electronic device judging whether the category of the shooting scene belongs to a preset scene through the multi-task learning model. Further, when the electronic device judges that the category of the shooting scene belongs to the preset scene, a specific category of the shooting scene can also be obtained through the multi-task learning model. The present embodiment does not limit the shooting scene when the electronic device is shooting, nor does it limit the preset scene. The preset scene includes but is not limited to a scene that is not suitable for exposure processing according to the 18% intermediate gray, such as the snow scene shown in FIG. 1, and the shooting scenes of a stage, a night scene, and a lawn, which are not limited here. Figure 3B

[0180] Exemplarily, the target detection result is used to represent a shooting object contained in the first image. For example, the target detection result can include a result of identifying a category of the shooting object, such as a result of the electronic device identifying whether a shooting object of a preset category exists in the first image through the multi-task learning model. Further, when the electronic device identifies that the first image contains a shooting object of a preset category, the position information of the shooting object of the preset category in the first image can also be obtained through the multi-task learning model. The present embodiment does not limit the shooting object when the electronic device is shooting, nor does it limit the preset category of the shooting object. The preset category of the shooting object includes but is not limited to an object that is not suitable for exposure processing according to the 18% intermediate gray, such as the animal category of the shooting object shown in FIG. 1, and the shooting objects of the categories of a face, a plant, and a ball, which are not limited here. Figure 4B

[0181] In one possible implementation, the electronic device inputs the first image into the multi-task learning model, determines the feature information corresponding to the first image through the multi-task learning model, wherein the feature information corresponding to the first image includes feature information corresponding to the shooting scene and the shooting object, and then determines the feature information corresponding to the shooting scene and the feature information corresponding to the shooting object according to the feature information corresponding to the first image. The electronic device determines the scene classification result based on the feature information corresponding to the shooting scene through the first output layer in the multi-task learning model, and determines the target detection result based on the feature information corresponding to the shooting object through the second output layer in the multi-task learning model.

[0182] ​​Specifically, the first image can contain multiple types of feature information, for example, feature information related to the scene classification task, feature information related to the target detection task, and other feature information unrelated to the scene classification task and the target detection task. Optionally, the other feature information unrelated to the scene classification task and the target detection task is redundant information, and the electronic device can selectively extract the feature information related to the scene classification task and the target detection task and remove the feature information unrelated to the scene classification task and the target detection task when performing feature extraction on the first image by using the multi-task learning model. The process in which the electronic device extracts the feature information corresponding to the first image can be referred to as the introduction of the feature extraction module 601 in Figure 6 Then, the electronic device decouples the feature information corresponding to the first image into feature information corresponding to the shooting scene and feature information corresponding to the shooting object, uses the feature information corresponding to the shooting scene for the scene classification task to obtain a scene classification result, and uses the feature information corresponding to the shooting object for the target detection task to obtain a target detection result. In this way, the accuracy of the electronic device in performing the scene classification task and the target detection task based on the feature information of the first image can be improved, the computational complexity can be reduced, and the operation speed can be improved.

[0183] In one implementation, the electronic device determines the first weight and the second weight, and then determines the feature information corresponding to the shooting object according to the feature information corresponding to the first image and the first weight, and determines the feature information corresponding to the shooting scene according to the feature information corresponding to the first image and the second weight.

[0184] Specifically, since the feature information corresponding to the first image contains feature information corresponding to the shooting scene and feature information corresponding to the shooting object, to decouple the feature information corresponding to the shooting scene and the feature information corresponding to the shooting object from the feature information corresponding to the first image, the electronic device can determine the feature information corresponding to the shooting scene based on the weight (the second weight) of the feature information corresponding to the shooting scene relative to the feature information corresponding to the first image, and determine the feature information corresponding to the shooting object based on the weight (the first weight) of the feature information corresponding to the shooting object relative to the feature information corresponding to the first image.

[0185] The weight is used to represent not only the percentage of a certain factor (e.g., the feature information corresponding to the shooting scene or the feature information corresponding to the shooting object) in the whole (e.g., the feature information corresponding to the first image), but also the importance or contribution of the factor in the whole. Therefore, the feature decoupling is performed according to the first weight and the second weight, which can be understood as performing feature decoupling according to the importance of the feature information corresponding to the shooting scene or the feature information corresponding to the shooting object in the feature information corresponding to the first image. For example, taking the first weight as an example, the first weight is used to represent the importance of the feature information corresponding to the shooting object relative to the feature information corresponding to the first image. The feature information corresponding to the shooting object is determined according to the first weight and the feature information corresponding to the first image, which can enhance the useful feature information (information related to the shooting object) and suppress the useless feature information (feature information irrelevant to the shooting object), so that the determined feature information of the shooting object has a higher correlation with the shooting object. For example, the process of determining the first weight and the second weight by the electronic device can be referred to the description of Figure 7 .

[0186] In a possible implementation, the electronic device performs first processing on the feature information corresponding to the first image in the channel dimension to determine a first processing result, and performs second processing on the feature information corresponding to the first image in the spatial dimension to determine a second processing result. Then, the first weight is determined according to the first processing result and the second processing result, and the second weight is determined according to the first weight.

[0187] The feature information corresponding to the first image extracted by the electronic device includes feature information in the channel dimension and feature information in the spatial dimension. The electronic device can process the feature information corresponding to the first image in the channel dimension and in the spatial dimension respectively, to obtain the weight of the feature information corresponding to the shooting object relative to the feature information corresponding to the first image in the channel dimension, and the weight of the feature information corresponding to the shooting object relative to the feature information corresponding to the first image in the spatial dimension. Then, the first weight and the second weight are determined based on the first processing result and the second processing result, so that the first weight represents the weight of the feature information corresponding to the shooting object relative to the feature information corresponding to the first image in the channel dimension and in the spatial dimension, and the second weight represents the weight of the feature information corresponding to the shooting scene relative to the feature information corresponding to the first image in the channel dimension and in the spatial dimension.

[0188] Exemplarily, the electronic device can perform first processing on the feature information corresponding to the first image in the channel dimension based on a channel attention mechanism to obtain a first processing result. The first processing result is used to represent the weight of the feature information corresponding to the shooting object relative to the feature information corresponding to the first image in the channel dimension. For details, refer to step S701.

[0189] Exemplarily, the electronic device can perform second processing on the feature corresponding to the first image in the spatial dimension based on a spatial attention mechanism to obtain a second processing result. The second processing result is used to represent the weight of the feature information corresponding to the shooting object relative to the feature information corresponding to the first image in the spatial dimension. For details, refer to step S702.

[0190] In a possible implementation, the electronic device inputs the feature information corresponding to the shooting scene into the first output layer in the multi-task learning model, and outputs the scene classification result through the first output layer.

[0191] The scene classification result includes the first judgment result and the category of the shooting scene, and the first judgment result is used to represent whether the shooting scene belongs to the preset scene.

[0192] Specifically, the electronic device inputs the feature information corresponding to the shooting scene into the first output layer in the multi-task learning model, and determines whether the shooting scene of the first image belongs to the preset scene through the first output layer to obtain the first judgment result. If the shooting scene belongs to the preset scene, the electronic device can also output the scene category corresponding to the shooting scene through the first output layer. For details of the process of determining the scene classification result by the electronic device, refer to the description of the multi-task processing module 603 in Figure 6

[0193] In a possible implementation, the electronic device inputs the feature information corresponding to the shooting object into the second output layer in the multi-task learning model, and outputs the target detection result through the second output layer, wherein the target detection result includes the category of the shooting object and the position information of the shooting object in the first image.

[0194] Specifically, the electronic device inputs the feature information corresponding to the shooting object into the second output layer in the multi-task learning model, and determines whether the first image contains a shooting object of a preset category through the second output layer. If the shooting object of the preset category is contained, the electronic device can also output the category of the shooting object and the position information of the shooting object relative to the first image through the second output layer. For details of the process of determining the target detection result by the electronic device, refer to the description of the multi-task processing module 603 in Figure 6

[0195] ​​In step S503, the target exposure parameter is determined according to the scene classification result and the target detection result.

[0196] Specifically, after obtaining the scene classification result and the target detection result, the electronic device can determine the target brightness according to the scene classification result and the target detection result. Then, the target exposure parameter is determined by taking the target of adjusting the brightness of the first image to the target brightness.

[0197] In a possible implementation, in a case where the first determination result is that the shooting scene belongs to the preset scene, the target exposure parameter is determined according to the category corresponding to the shooting scene.

[0198] Specifically, since the shooting scene contains all visible elements of the first image, the shooting scene is related to the information of the whole first image. When the shooting scene belongs to the preset scene, it indicates that the electronic device needs to adjust the overall brightness of the image, and at this time, the electronic device also needs to adjust the brightness of the shooting object contained in the first image. The electronic device does not need to consider the category of the shooting object, and can adjust the brightness of the image to an appropriate state according to the category of the shooting scene.

[0199] In a possible implementation, in a case where the first determination result is that the shooting scene does not belong to the preset scene, when the category of the shooting object belongs to the preset category, the target exposure parameter is determined according to the category of the shooting object and the position information of the shooting object in the first image.

[0200] Specifically, when the electronic device determines that the shooting scene does not belong to the preset scene according to the first determination result, the electronic device determines whether the category of the shooting object belongs to the preset category according to the target detection result. When the electronic device determines that the category of the shooting object belongs to the preset category, it indicates that the electronic device needs to adjust the exposure parameter at this time, and the electronic device can determine the target exposure parameter based on the category of the shooting object and the position information of the shooting object in the first image.

[0201] It should be noted that in the above embodiments, the electronic device preferentially determines the target exposure parameter based on the shooting scene. In addition to the above embodiments, the electronic device can also preferentially determine the target exposure parameter based on the shooting object. For example, in a case where the category of the shooting object belongs to the target category, when the proportion of the area of the shooting object in the total area of the first image is greater than or equal to the proportion threshold, at this time, the electronic device can preferentially determine the target exposure parameter based on the shooting object. This is because when the proportion of the area of the shooting object in the total area of the first image is greater than or equal to the proportion threshold, it indicates that the area of the shooting object is large, and the brightness of the shooting object has a large influence on the overall brightness of the image, so the electronic device can consider the brightness of the shooting object.

[0202] In addition, in an implementation, when the difference between the brightness of the photographed object and the brightness of the background in the image is large, the electronic device can be difficult to eliminate the brightness difference between the photographed object and the background by adjusting the brightness of the entire image. The electronic device can process the brightness of the region where the photographed object is located based on the position information of the photographed object in the first image, to reduce the brightness difference between the photographed object and the background, so that the entire image presents a moderate brightness.

[0203] In step S504, the second image is displayed.

[0204] Specifically, after the electronic device determines the target exposure parameter based on the photographed scene and / or the photographed object, the electronic device can collect the second image based on the target exposure parameter. The target exposure parameter includes parameters such as sensitivity, aperture size, shutter speed, etc. The electronic device can adjust the sensitivity of the image sensor to adjust the sensitivity to light, adjust the aperture size to adjust the amount of light entering the camera, and adjust the shutter speed to adjust the exposure time, thereby controlling the exposure of the camera to collect a second image with moderate brightness and reduce the overexposure or underexposure of the second image.

[0205] For example, the second image can refer to Figure 3D the image 313 shown in FIG. 3, and Figure 4C the image 404 shown in FIG. 4.

[0206] In the following, the structure of the multi-task learning model will be introduced to illustrate a possible implementation process of the electronic device processing the first image through the multi-task learning model. Figure 6

[0207] Please refer to Figure 6 , Figure 6 is a structural diagram of a multi-task learning model provided by an embodiment of the present application. As Figure 6 shown, the multi-task learning model includes a feature extraction module 601, a feature decoupling module 602, and a multi-task processing module 603.

[0208] For example, the feature extraction module 601 is configured to extract features of an image (e.g., the first image) input into the multi-task learning model, to obtain feature information corresponding to the first image. Optionally, image feature extraction refers to processing and analyzing the information contained in the image, and extracting the information that is not easily affected by random factors as the features of the image, so as to realize the representation of the original features of the image as a set of features with obvious physical or statistical significance. Image features refer to a series of mathematical sets that can represent the content or characteristics of the image.

[0209] ​In an implementation, the feature extraction module 601 extracts feature information corresponding to a task according to the task processed by the electronic device through the multi-task learning model. For example, the electronic device processes a scene classification task and an object detection task through the multi-task learning model, and the feature extraction module 601 can extract feature information according to the scene classification task and the object detection task. Among them, the scene classification task is used to determine the shooting scene when the electronic device collects the first image, and the feature information related to the scene classification task is the feature information related to the shooting scene. Since the shooting scene contains all visible elements of the first image, the shooting scene is related to the information of the whole first image, and the electronic device needs to identify the shooting scene based on the features representing the whole first image. For example, the features representing the whole first image include but are not limited to statistical features, which can be effectively applied in image classification, segmentation, retrieval and enhancement tasks by quantifying the pixel value, color distribution and structure of the image. The electronic device can determine the style characteristics of the first image by analyzing the statistical features of the first image. For example, the contrast, brightness and tone distribution of the first image can be understood through the gray level histogram in the statistical features, and the distribution of colors in the first image can be determined through the color histogram in the statistical features, so that the style of the first image can be more comprehensively described based on the information related to the global first image such as color and contrast, so that the accuracy of the determined scene classification result is higher. Therefore, the feature extraction module 601 can extract feature information related to the whole first image from the first image for subsequent scene classification tasks.

[0210] In an implementation, the target detection task is used for detecting a photographed object contained in the first image, and the feature information related to the target detection task is feature information related to the photographed object. Since the area of the region where the photographed object is located is smaller than the total area of the first image, the photographed object is related to the information of the local part of the first image, and the electronic device needs to detect the photographed object based on the features representing the local part of the first image. For example, the features representing the local part of the first image include but are not limited to texture features, edge features, shape features, and spatial relationship features, and the like. For example, the texture features are used to describe the surface properties of the scene corresponding to the image or the image region. The edge features represent the regions with significant gray level changes in the image, which usually correspond to the boundaries or shapes of objects. The shape features are used to describe the geometric shapes of objects, including the information such as the boundaries, contours, areas, perimeters, shape complexities, and the like of objects. The spatial relationship features describe the relative positions, directions, and distances between objects, which not only focus on the shapes and sizes of objects, but also consider the distribution and arrangement of objects in space. Therefore, the electronic device can identify the category of the photographed object, determine the position of the photographed object, or determine the contour of the photographed object based on the features representing the local part of the first image, so that the accuracy of the determined target detection result is higher. Thus, the feature extraction module 601 can extract feature information related to the local part of the first image from the first image for the subsequent target detection task.

[0211] Exemplarily, the structure of the feature extraction module 601 includes, but is not limited to, a convolution layer, a pooling layer, a normalization layer, an activation layer, and the like, where the application embodiments do not limit the connection relationship, position, and quantity of the above-mentioned convolution layer, pooling layer, normalization layer, and activation layer. In one implementation, the feature extraction model can first perform sliding window scanning on the first image through the filter (also known as the convolution kernel) of the convolution layer, multiply the filter and the local region of the first image, and sum to obtain the output of the convolution layer, so as to extract features (such as features corresponding to the shooting scene and features corresponding to the shooting object) from the first image. Then, the feature extraction module 601 inputs the output of the convolution layer into the pooling layer, reduces the spatial size of the feature map extracted from the convolution layer through the pooling layer, while retaining important information (information related to the shooting scene and the shooting object). The feature extraction module 601 inputs the output of the pooling layer into the normalization layer, and performs linear transformation on the batch data obtained from the pooling layer through the normalization layer, so that the distribution of the above-mentioned batch data is more stable, which is beneficial to the feature extraction module 601 to speed up the convergence speed of the above-mentioned batch data. Finally, the feature extraction module 601 inputs the output of the normalization layer into the activation layer to obtain the feature information corresponding to the first image. Wherein, the activation layer can introduce a nonlinear factor, so that the multi-task learning model can learn more complex features (such as nonlinear features), and improve the extraction ability of the multi-task learning model for complex features. Therefore, the combination of the convolution layer and the pooling layer of the feature extraction module 601 can effectively reduce the number of parameters of the multi-task learning model, and enhance the generalization ability of the multi-task learning model on different images. Through the normalization layer and the activation layer, the training effect and the generalization ability of the multi-task learning model can be enhanced, and the performance of the multi-task learning model can be improved.

[0212] In one possible implementation, the multi-task learning model can extract the feature information corresponding to the first image through a multi-task decoder (MTLDecoder). Wherein, the multi-task decoder includes a feature extraction module 601 and a feature decoupling module 602, which are used to extract features from the image (such as the first image) input into the multi-task decoder, and then map the extracted features to multiple output tasks (such as the scene classification task and the target detection task in the multi-task processing module 603), so that the multi-task learning model can process multiple output tasks in parallel.

[0213] In the embodiment of the present application, the format of the image input by the electronic device into the multi-task decoder is not limited, for example, the format of the image collected by the sensor of the electronic device can be YUV format, and the electronic device can convert the image in YUV format into RGB color mode (RGB) format, and then input the image in RGB format into the multi-task decoder, so that the electronic device extracts features of the image in RGB format through the feature extraction module 601 in the multi-task decoder. For example, the process of feature extraction by the electronic device through the multi-task decoder can refer to the following expression (1):

[0214] (1)

[0215] In the above expression (1), is an image (for example, a first image) input by the electronic device into the multi-task decoder, is a feature (for example, a feature corresponding to the first image) extracted by the electronic device through the multi-task decoder, is a function expression for feature extraction of the first image by the electronic device through the multi-task decoder.

[0216] Exemplarily, the feature decoupling module 602 is configured to decouple the feature information (for example, the feature information corresponding to the first image) output by the feature extraction module 601 into feature information corresponding to a plurality of tasks in the multi-task processing module 603 respectively, for example, the multi-task processing module 603 includes a scene classification task and a target detection task, and the feature decoupling module 602 can decouple the feature information corresponding to the first image into feature information corresponding to a photographed object and feature information corresponding to a photographed scene, wherein the feature information corresponding to the photographed scene is applicable to the scene classification task, and the feature information corresponding to the photographed object is applicable to the target detection task.

[0217] It can be understood that, in multi-task learning, the feature information (for example, the feature information corresponding to the first image) corresponding to the image input by the electronic device into the multi-task learning model can include feature information shared by a plurality of tasks, or can include feature information specific to a certain task, so that the feature decoupling module 602 is configured to determine feature information specific to each task in the plurality of tasks, so that the feature information corresponding to any one task in the plurality of tasks can also include feature information shared with other tasks. For example, the feature decoupling module 602 determines that the feature information corresponding to the photographed scene includes not only feature information specific to the scene classification task, but also feature information shared by the scene classification task and the target detection task. Thus, any one task in the plurality of tasks can also enhance the understanding of the first image based on the shared feature information on the basis of processing the feature information specific to the task, so as to output a result with higher accuracy.

[0218] The following will be combined with Figure 7 The process of feature decoupling of the feature decoupling module is described in detail.

[0219] Please refer to Figure 7 , Figure 7 is a flowchart of feature decoupling provided by the embodiment of the application, which takes determining the weight of the feature information corresponding to the shooting object relative to the feature information corresponding to the first image as an example to illustrate the process of feature decoupling. It can be understood that the process of feature decoupling can also be based on Figure 7 The implementation shown in the figure first determines the weight of the feature information corresponding to the shooting scene relative to the feature information corresponding to the first image, and then performs feature decoupling. Here, no longer be described. As shown in Figure 7 The flowchart includes S701 to S705, and the specific content is as follows.

[0220] S701, determine the weight of the feature information corresponding to the shooting object relative to the feature information corresponding to the first image in the channel dimension.

[0221] Specifically, the feature information corresponding to the first image includes a feature map, and the size of the feature map is , where C is the number of channels of the feature map, which belongs to the channel dimension. As can be seen, the feature information corresponding to the first image contains multiple different channels. Generally, different channels may have different contributions to different features, for example, some channels may contain more key information (for example, information related to the shooting object), while other channels may contain noise or redundant information (for example, information unrelated to the shooting object). Therefore, the electronic device can determine the weight of each channel containing key information (for example, information related to the shooting object) through the channel attention mechanism, and by giving higher weight to the channel containing more key information and lower weight to the channel containing less key information, the features related to the shooting object can be enhanced and the features unrelated to the shooting object can be suppressed in the channel dimension, so as to achieve the purpose of decoupling the feature information corresponding to the shooting object from the feature information corresponding to the first image.

[0222] Wherein, the channel attention mechanism belongs to one of the attention mechanisms (Attention Mechanism), which can automatically focus on the most relevant information in the massive data by simulating the selective attention of human vision, thereby improving the efficiency and accuracy of the model. Further, the channel attention mechanism focuses on judging the importance of different channels and determining which channel features are more discriminative for the task.

[0223] Exemplarily, the result of the electronic device extracting the feature information corresponding to the first image based on the channel attention mechanism is as follows expression (2):

[0224] (2)

[0225] wherein, is the feature information corresponding to the first image, which can be seen from expression (1). is the average pooling processing of the feature corresponding to the first image, is the maximum pooling processing of the feature corresponding to the first image, is the full connection layer structure, is the activation function.

[0226] Specifically, since the channel attention mechanism is processed in the channel dimension, first, the spatial dimension needs to be pooled to compress the spatial size to obtain the vector in the channel dimension. For example, in expression (2), the electronic device compresses the spatial size by performing average pooling processing and maximum pooling processing on the feature information corresponding to the first image, so that the obtained first feature information is the feature information of the first image in the channel dimension. Then, the electronic device processes the first feature information through the full connection layer to obtain the weight containing information related to the shooting object in each channel, which can be regarded as the attention weight of each channel, used to measure the contribution degree of each channel to the feature information corresponding to the shooting object. Finally, the electronic device maps the output of the full connection layer through the activation function to obtain the first processing result, so that the first processing result includes the weight of the feature information corresponding to the shooting object in each channel relative to the first feature information. At this time, the size of the first processing result is , containing information in the channel dimension.

[0227] S702, determine the weight of the feature information corresponding to the shooting object relative to the feature information corresponding to the first image in the spatial dimension.

[0228] Specifically, the feature information corresponding to the first image includes a feature map, and the size of the feature map is , H is the height of the feature map, which belongs to the spatial dimension, and W is the width of the feature map, which belongs to the spatial dimension. Based on the height and width of the feature map, a plurality of spatial positions of the feature map can be determined. Generally, different spatial positions can have different contributions to different features, for example, some spatial positions can contain more key information (for example, information related to the photographed object), while other spatial positions can contain noise or redundant information (for example, information unrelated to the photographed object). Therefore, the electronic device can determine the weight of each spatial position containing key information (for example, information related to the photographed object) through the spatial attention mechanism, and by giving higher weights to spatial positions containing more key information and lower weights to spatial positions containing less key information, the features related to the photographed object can be enhanced and the features unrelated to the photographed object can be suppressed in the spatial dimension, thereby achieving the purpose of decoupling the feature information corresponding to the photographed object from the feature information corresponding to the first image.

[0229] wherein the spatial attention mechanism is an important attention mechanism in deep learning, which is used to focus on specific areas when processing data such as images or videos, in order to improve the performance and efficiency of the model. For example, the electronic device can calculate the attention weight of each pixel point in the first image through the spatial attention mechanism, and then concentrate the attention of the model (such as a multi-task learning model) on the most important area in the first image, so that the model can more effectively process complex visual tasks and improve its performance and generalization ability.

[0230] For example, the electronic device extracts the feature information corresponding to the first image based on the spatial attention mechanism is as follows:

[0231] (3)

[0232] wherein is the feature information corresponding to the first image, which can be specifically referred to expression (1). is the average pooling processing of the feature information corresponding to the first image, is the maximum pooling processing of the feature information corresponding to the first image, is the convolution operation on and is an activation function.

[0233] Specifically, since the spatial attention mechanism is processed in the spatial dimension, it is necessary to perform pooling processing in the channel dimension to compress the channel size first. For example, in expression (3), the electronic device performs average pooling processing and maximum pooling processing on the feature information corresponding to the first image based on the channel dimension, and obtains two feature maps with a size of ​The feature maps are then concatenated based on channels to obtain a single feature map. The feature map. Then, for this... The feature maps are used for convolution operations (for example, the size of the convolution kernel can be...). ), making this The feature map is reduced to a one-dimensional feature map to obtain the second feature information. The size of the second feature information is... This includes the spatial feature information of the first image. Furthermore, the electronic device can process the second feature information through convolution operations to obtain weights at each spatial location containing information related to the captured object. These weights can be considered attention weights at each spatial location, used to measure the contribution of each spatial location to the feature information corresponding to the captured object. Finally, the electronic device normalizes the output of the convolution operation to a probability distribution through an activation function, obtaining the second processing result. This second processing result includes the weights of the feature information corresponding to the captured object at each spatial location relative to the second feature information, ensuring that the sum of the weights at each spatial location is 1. The size of the second processing result is then [size missing]. It contains information in the spatial dimension.

[0234] S703, determine the first weight and the second weight.

[0235] Specifically, the first processing result is the weight in the channel dimension, and the second processing result is the weight in the spatial dimension. The feature information corresponding to the first image includes both channel-dimensional and spatial-dimensional feature information. To ensure that the feature information corresponding to the subject, decoupled from the feature information corresponding to the first image, also includes both channel-dimensional and spatial-dimensional feature information, the electronic device needs to obtain a first weight based on the first and second processing results. This first weight represents the weight of the feature information corresponding to the subject relative to the feature information corresponding to the first image in both the channel and spatial dimensions. This allows the electronic device to decouple the feature information corresponding to the subject from the feature information corresponding to the first image based on the first weight when executing S704.

[0236] In one possible implementation, since the size of the first processing result is The size of the second processing result is The size of the feature information corresponding to the first image is Therefore, the electronic device can first reduce the size of the first processing result from... Expand to This is equivalent to copying the feature map H or W times in the spatial dimension, resulting in an expanded feature map with a height and width of 1. The size of the second processing result is then changed from... Expand to , which is equivalent to copying C times in the channel dimension, and the value of the expanded channel number is 1. Then, the electronic device performs accumulation and averaging operations on the expanded first processing result and the expanded second processing result to obtain the first weight. The first weight may be obtained according to the following expression (4):

[0237] (4)

[0238] , wherein is the feature information corresponding to the first image, which can be specifically seen from expression (1). is the first processing result, which can be specifically seen from expression (2), is the second processing result, which can be specifically seen from expression (3). represents that the sizes of the first processing result and the second processing result are both expanded to , and the result obtained by performing accumulation and averaging operations on the expanded first processing result and the expanded second processing result.

[0239] For example, since the feature information corresponding to the first image includes the feature information corresponding to the photographed object and the feature information corresponding to the photographed scene, the sum of the weight (i.e., the first weight) of the feature information corresponding to the photographed object relative to the feature information corresponding to the first image and the weight (i.e., the second weight) of the feature information corresponding to the photographed scene relative to the feature information corresponding to the first image is 1. Therefore, when the electronic device determines the first weight , the second weight can be determined as according to the sum of the first weight and the second weight being 1. It should be noted that if the feature information corresponding to the first image further includes other feature information in addition to the feature information corresponding to the photographed object and the feature information corresponding to the photographed scene, the electronic device can determine the corresponding weight in the same manner as steps S701 and S702, and then perform feature decoupling.

[0240] S704, determining the feature information corresponding to the photographed object based on the first weight.

[0241] Specifically, after determining the first weight, the electronic device can determine the feature information corresponding to the photographed object according to the feature information corresponding to the first image and the first weight. For example, the first weight and the feature information corresponding to the first image are multiplied, so that the feature information related to the photographed object in the feature information corresponding to the first image is amplified, and the feature information unrelated to the photographed object is weakened, thereby decoupling the feature information corresponding to the photographed object from the feature information corresponding to the first image.

[0242] For example, the feature information corresponding to the photographed object can be seen from the following expression (5):

[0243] (5)

[0244] wherein, is the feature information corresponding to the first image, which can be seen from expression (1). is the first weight, which can be seen from expression (4). indicates that the feature information corresponding to the first image is weighted and fused with the first weight.

[0245] It should be noted that, may also be understood as a mask. Wherein the mask refers to a specific region in an array or an image, which is used to select, hide or modify part of the content of the data. For example, in the first weight, the first weight can be used to select the feature information corresponding to the shooting object from the feature information corresponding to the first image, and hide other feature information, so that the multi-task learning model can focus on the information related to the shooting object.

[0246] S705, determining the feature information corresponding to the shooting scene based on the second weight.

[0247] Specifically, after determining the second weight, the electronic device can determine the feature information corresponding to the shooting scene according to the feature information corresponding to the first image and the second weight. For example, the second weight and the feature information corresponding to the first image are multiplied, so that the feature information related to the shooting scene in the feature information corresponding to the first image is amplified, and the feature information unrelated to the shooting scene is weakened, so as to decouple the feature information corresponding to the shooting scene from the feature information corresponding to the first image.

[0248] For example, the feature information corresponding to the shooting scene can be seen from the following expression (6):

[0249] (6)

[0250] wherein, is the feature information corresponding to the first image, which can be seen from expression (1). is the second weight, indicates that the feature information corresponding to the first image is weighted and fused with the second weight.

[0251] It should be noted that, may also be understood as a mask. For example, the second weight can be used to select the feature information corresponding to the shooting scene from the feature information corresponding to the first image, and hide other feature information, so that the multi-task learning model can focus on the information related to the shooting scene.

[0252] Exemplarily, according to the expression (5) and the expression (6), the feature information corresponding to the first image can also be expressed as the following expression (7):

[0253] (7)

[0254] wherein, is the feature information corresponding to the first image, which can be specifically referred to the expression (1). is the first weight, which can be specifically referred to the expression (4), is the second weight, is the feature information corresponding to the shooting object, is the feature information corresponding to the shooting scene.

[0255] It should be noted that the electronic device decouples the feature information corresponding to the first image into the feature information corresponding to the shooting scene and the feature information corresponding to the shooting object, so that the multi-task learning model can better capture the feature information required by different tasks when processing multiple tasks in parallel, thereby improving the accuracy of the task.

[0256] In addition, in the existing scene classification scheme or target detection scheme, the feature information required by the task is usually directly extracted from the input image, and then the extracted features are processed to obtain the result, without involving feature decoupling. For example, in the existing face detection task, the electronic device directly extracts the feature information corresponding to the face from the input image, and then performs the target detection task based on the feature information corresponding to the face, thereby obtaining the face detection result. The embodiment of the present application decouples the feature information required by the task from the feature information corresponding to the first image based on the weight, so that the feature information corresponding to the shooting scene not only contains the related information of the shooting scene, but also contains the related information of the shooting object, and similarly, the feature information corresponding to the shooting object not only contains the related information of the shooting object, but also contains the related information of the shooting scene, that is, there is shared feature information in the feature information corresponding to the shooting scene and the feature information corresponding to the shooting object. The electronic device determines the scene classification result based on the feature information corresponding to the shooting scene and the shared feature information, which can strengthen the understanding of the multi-task learning model for the first image compared with the electronic device determining the scene classification result based only on the feature information corresponding to the shooting scene, thereby outputting a scene classification result with higher accuracy. Similarly, the electronic device can obtain a target detection result with higher accuracy based on the feature information corresponding to the shooting object and the shared feature information.

[0257] Please continue to refer to Figure 6 , after the feature decoupling module 602 obtains the feature information corresponding to the shooting scene and the feature information corresponding to the shooting object, the feature information and the feature information corresponding to the shooting object can be input into the multi-task processing module 603.

[0258] The multi-task processing module 603 is configured to process multiple tasks in parallel, such as the scene classification task or the target detection task in the embodiments of the present application.

[0259] As shown in Figure 6 The multi-task processing module 603 includes a scene discriminator and a scene classifier, which are configured to perform the scene classification task. The scene discriminator and the scene classifier belong to the first output layer of the multi-task learning model.

[0260] For example, the scene discriminator is configured to determine whether the shooting scene corresponding to the first image is a preset scene, and obtain a first determination result. In one implementation, the scene discriminator is a multi-layer fully connected structure, wherein the multi-layer fully connected structure is configured to make a decision, and the first determination result can be obtained according to the feature information corresponding to the shooting scene. In another implementation, the scene discriminator can be a combination of a multi-layer convolutional layer and a fully connected layer, and the structure of the scene discriminator is not limited in the embodiments of the present application.

[0261] For example, the electronic device inputs the feature information corresponding to the shooting scene into the scene discriminator, and the prediction process of the scene discriminator is as follows expression (8):

[0262] (8)

[0263] wherein, is the first determination result, which can be a binary classification label of 0 or 1. For example, if the first determination result is 0, it means that the shooting scene does not belong to the preset scene, and if the determination result is 1, it means that the shooting scene belongs to the preset scene. is a second weight, is the feature information corresponding to the shooting scene, is the network structure of the scene discriminator, is the prediction process of the scene discriminator.

[0264] For example, the scene classifier is configured to determine the category of the shooting scene, and the design principle of the scene classifier can be the same as that of the scene discriminator. For example, the scene classifier can be a multi-layer fully connected structure or a combination of a multi-layer convolutional layer and a fully connected layer, and the structure of the scene classifier is not limited in the embodiments of the present application.

[0265] For example, the electronic device inputs the feature information corresponding to the shooting scene into the scene classifier, and the prediction process of the scene classifier is as follows expression (9):

[0266] (9)

[0267] wherein, For the category corresponding to the shooting scene, for example, if the shooting scene belongs to a preset scene, the category corresponding to the shooting scene output by the scene classifier belongs to one or more of the preset scenes. If the shooting scene does not belong to the preset scene, the scene classifier does not output a specific scene category. For the second weight, For the feature information corresponding to the shooting scene, For the network structure of the scene classifier, For the prediction process of the scene classifier.

[0268] As Figure 6 shown, the multi-task processing module 603 further includes a target detector, which is configured to perform a target detection task. The target detector belongs to the second output layer of the multi-task learning model.

[0269] For example, the target detector is configured to output a target detection result based on the feature information corresponding to the shooting object, wherein the target detection result includes a category corresponding to the shooting object and position information of the shooting object in the first image.

[0270] Unlike the scene discriminator and the scene classifier, the target detector follows the design principle of decoupled detection heads, and can simultaneously detect the category of the shooting object and the position of the shooting object, for example Figure 6 The target detector includes a shared structure, a classification branch, and a regression branch. The shared structure is a part of the network structure shared by the target detector, which can be used to balance the representation ability of the target detector to the operator and the calculation overhead of the hardware in the electronic device, so that the result output by the shared structure can be directly used for the classification branch and the regression branch, reducing the operation pressure of the electronic device and saving the processing time of the electronic device.

[0271] For example, the electronic device inputs the feature information corresponding to the shooting object into the target detector, and the prediction process of the target detector is as follows expression (10):

[0272] (10)

[0273] Wherein, The target detection result includes the category of the shooting object output by the classification branch, and the position information of the shooting object in the first image output by the regression branch. The target feature of the shared structure, The network structure of the target detector, The prediction process of the target detector.

[0274] The classification branch is used for predicting the category of the photographed object, and the structure of the classification branch can be a fully connected layer or a convolutional layer, which is not limited herein. For example, if the classification branch identifies that the photographed object belongs to a preset category, the classification branch can display a candidate box of the category on the photographed object, and output a category probability of the candidate box, where the category probability is used to represent the probability that the photographed object belongs to the category.

[0275] The regression branch is used for determining the position information of the photographed object in the first image, for example, the position information includes the coordinate information of the candidate box in the classification branch (for example, the center point coordinates, width and height of the candidate box, etc.). The structure of the regression branch can be a fully connected layer or a convolutional layer, which is not limited herein.

[0276] The following Figure 8 introduces a process of determining a target exposure parameter according to a photographed scene and / or a photographed object.

[0277] Please refer to Figure 8 , Figure 8 is a flowchart for determining a target exposure parameter provided by an embodiment of the present application. As shown in Figure 8 , the flowchart includes steps S801 to S805, and the specific process is as follows:

[0278] S801, obtaining a scene classification result and a target detection result.

[0279] Specifically, after the electronic device inputs the first image into the multi-task learning model, the electronic device obtains the scene classification result and the target detection result output by the multi-task learning model, and then determines the target exposure parameter according to the scene classification result and the target detection result based on steps S801 to S805.

[0280] The scene classification result includes a first judgment result and a category corresponding to the photographed scene, and the target detection result includes a category of the photographed object and position information of the photographed object in the first image.

[0281] S802, determining whether the photographed scene belongs to a preset scene.

[0282] When the electronic device determines that the photographed scene belongs to the preset scene according to the first judgment result in the scene classification result, the electronic device performs step S803 without considering the target detection result, thereby saving the computing resources of the electronic device and accelerating the speed of automatic exposure of the electronic device. When the electronic device determines that the photographed scene does not belong to the preset scene according to the first judgment result, the electronic device performs step S804 to determine the target exposure parameter according to the target detection result.

[0283] S803, determining the target exposure parameter based on the category of the photographed scene.

[0284] Specifically, when the electronic device determines that the shooting scene belongs to the preset scene according to the first determination result, the electronic device can first determine the target brightness based on the category of the shooting scene, and then determine the target exposure parameter based on the difference between the brightness of the first image and the target brightness, so that the brightness of the image collected by the electronic device based on the target exposure parameter is equal to or close to the target brightness, thereby realizing exposure convergence. The brightness of the first image can be obtained from the image signal output by the image sensor of the electronic device.

[0285] The preset scene includes but is not limited to snow scene, stage, grassland, night scene, etc. Taking the shooting scene corresponding to the first image as the night scene as an example, since the night scene contains a large area of dark area, the brightness of the picture collected by the electronic device is low (lower than 18% of the middle gray scale), and the electronic device increases the light flux to increase the brightness of the picture, which will cause the exposure of the first image collected at this time to be excessive. Therefore, when the electronic device determines that the shooting scene of the first image is a night scene, the electronic device can determine the target exposure parameter by reducing the target brightness, so that the exposure intensity of the image collected by the electronic device based on the target exposure parameter is less than the first image, thereby reducing the exposure degree.

[0286] S804, determine whether the category of the shooting object belongs to the preset category.

[0287] Specifically, in the case that the electronic device determines that the shooting scene does not belong to the preset scene according to the first determination result, when the electronic device determines that the category of the shooting object belongs to the preset category, the electronic device can execute step S805 to determine the target exposure parameter corresponding to the shooting object. When the electronic device determines that the category of the shooting object does not belong to the preset category, at this time, the shooting scene does not belong to the preset scene and the category of the shooting object does not belong to the preset category, which means that the shooting scene and the shooting object at this time can present moderate exposure based on the target brightness of 18% of the middle gray scale, and there is no need to adjust the exposure parameter. Moreover, if the scene classification result and / or the target detection result are incorrect, the electronic device does not adjust the exposure parameter, which can avoid the possibility of incorrect adjustment, thereby ensuring the stability and reliability of automatic exposure.

[0288] For example, in step S502, the electronic device can output the candidate box of the shooting object and the category probability of the candidate box through the target detector. The electronic device can determine the category of the shooting object based on the category probability of the candidate box, for example, when the category probability is greater than or equal to a threshold value, the electronic device determines that the category of the shooting object belongs to the category corresponding to the candidate box, and the electronic device can execute step S805 to adjust the exposure parameter. When the category probability is less than the threshold value, the electronic device determines that the category of the shooting object does not belong to the category corresponding to the candidate box, and the electronic device does not adjust the exposure parameter.

[0289] S805, determine the target exposure parameter based on the category and the position information of the shooting object.

[0290] Specifically, when the category of the shooting object belongs to the preset category, the electronic device can determine the target brightness according to the category of the shooting object and the position information of the shooting object in the first image. Then, the target exposure parameter is determined based on the difference between the brightness of the first image and the target brightness, so that the brightness of the image captured by the electronic device based on the target exposure parameter is equal to or close to the target brightness, thereby realizing exposure convergence.

[0291] It can be understood that since the automatic exposure of the electronic device is a process of adjusting the brightness of the image to the target brightness, the electronic device may need to adjust the exposure parameter multiple times to adjust the brightness of the image to the target brightness. The electronic device can determine the shooting scene and the shooting object of the input image based on the method provided in the embodiments of the present application multiple times through the multi-task learning model, then determine the target exposure parameter based on the shooting scene and / or the shooting object, and capture the image again based on the target exposure parameter, until the image with the target brightness is captured, to provide the user with a preview image with better quality.

[0292] The above embodiments are the process of automatic exposure of the electronic device through the multi-task learning model, and the training process of the multi-task learning model will be specifically introduced below.

[0293] The multi-task learning model can be trained on a first device. The first device can be a device with data transceiving capability, data storage capability and data processing capability, and can be a physical device such as a host, a rack server, a blade server, etc., or a virtual device such as a virtual machine, a container, etc. Further, the first device can be a server or a server cluster composed of multiple servers.

[0294] In one possible implementation, the first device obtains a training set of the multi-task learning model, and the training set includes one or more of the following: a training image, a scene classification label corresponding to the training image, and a target detection label. Then, the first device trains the multi-task learning model by taking the training image as the input of the multi-task learning model, taking the scene classification label corresponding to the training image as the output of a first output layer in the multi-task learning model, and taking the target detection label as the output of a second output layer in the multi-task learning model, to obtain a trained multi-task learning model.

[0295] The structure of the multi-task learning model can be referred to Figure 6 Figure 6 ​It can be seen that the multi-task learning model comprises a scene discriminator, a scene classifier and a target detector. The scene discriminator and the scene classifier belong to a first output layer of the multi-task learning model, the scene discriminator is configured to determine whether a shooting scene of an input image belongs to a preset scene, and the scene classifier is configured to determine which one of the preset scenes the shooting scene belongs to. Therefore, the scene discriminator and the scene classifier are used for a scene classification task, and the first device can train the scene discriminator and the scene classifier according to a scene classification label. The target detector belongs to a second output layer of the multi-task learning model, the target detector is configured to determine whether a shooting object belonging to a preset category exists in the input image, and if the shooting object belonging to the preset category exists, the target detector can further determine position information of the shooting object in the input image. Therefore, the target detector is used for a target detection task, and the first device can train the target detector according to a target detection label.

[0296] For example, the training set of the multi-task learning model can be obtained by the first device from other electronic devices, or can be constructed by the first device. Taking the construction of the training set by the first device as an example, first, the first device obtains a plurality of images containing a preset scene and / or a shooting object of a category being a preset category, wherein the images can be collected by the above-mentioned electronic device, or can be images from other electronic devices, so as to ensure the diversity of the training set. Then, the first device labels the plurality of images obtained above to obtain a scene classification label and / or a target detection label. The scene classification label is mainly used to label the category of the shooting scene corresponding to the image. For example, the scene classification label of the shooting scene of a snow scene is labeled as “0”, the scene classification label of the shooting scene of a stage is labeled as “1”, the scene classification label of the shooting scene of a grassland is labeled as “2”, and so on. The target detection label is mainly used to label the category of the shooting object contained in the image and the position of the shooting object in the image, for example, the first device determines the position of the shooting object in the image by labeling a label box in the region where the shooting object is located. Thus, after the first device labels the collected images, a training set containing training images (i.e., the collected images), a scene classification label and a target detection label is obtained.

[0297] In a possible implementation, the first device inputs the training images in the training set into the multi-task learning model, determines the feature information corresponding to the training images through the multi-task learning model, and then decouples the feature information corresponding to the training images into third feature information and fourth feature information through the multi-task learning model. Finally, the multi-task learning model is trained based on the third feature information and the fourth feature information to obtain a trained multi-task learning model.

[0298] The structure of the multi-task learning model can be referred to Figure 6As shown in Figure 6 The multi-task learning model includes a feature extraction module 601, a feature decoupling module 602, and a multi-task processing module 603. First, the first device extracts features of the training image through the feature extraction module 601 to obtain feature information corresponding to the training image. In order to train the scene classification task and the target detection task of the multi-task learning model, the feature information corresponding to the training image includes feature information corresponding to the shooting scene related to the scene classification task, and feature information corresponding to the shooting object related to the target detection task. Then, the first device decouples the feature information corresponding to the training image into third feature information and fourth feature information through the feature decoupling module 602, wherein the third feature information is the feature information corresponding to the shooting scene in the feature information corresponding to the training image, and the fourth feature information is the feature information corresponding to the shooting object in the feature information corresponding to the training image. Then, the first device trains the multi-task processing module 603 based on the third feature information and the fourth feature information.

[0299] In a possible implementation, as shown in Figure 6 The multi-task processing module 603 includes a scene discriminator, a scene classifier, and a target detector. The first device can obtain a scene classification result through the multi-task processing module 603, for example, the first device determines whether the shooting scene of the training image belongs to a preset scene according to the third feature information through the scene discriminator, and determines which category of the preset scene the shooting scene belongs to through the scene classifier. The first device can also obtain a target detection result through the multi-task processing module 603, for example, the first device outputs the category of the shooting object and the position information of the shooting object in the training image according to the fourth feature information through the target detector. The first device can compare the scene classification result with the scene classification label to obtain a first comparison result. The target detection result is compared with the target detection label to obtain a second comparison result. Then, the parameters of the multi-task learning model are adjusted based on the first comparison result and the second comparison result. The first device processes other training images in the training set through the multi-task learning model according to the above steps again until the first comparison result is less than a first comparison threshold and the second comparison result is less than a second comparison threshold, at which time the first device completes the training of the multi-task learning model, so that the multi-task learning model has the ability to process the scene classification task and the target detection task in parallel.

[0300] It should be understood that each step in the above method embodiments provided by the present application can be completed by integrated logic circuits of hardware in the processor or instructions in the form of software. The method steps disclosed in combination with the embodiments of the present application can be directly embodied as hardware processor execution, or executed by a combination of hardware and software modules in the processor.

[0301] The application further provides an electronic device, which can include a memory and a processor. The memory can be used to store a computer program, and the processor can be used to call the computer program in the memory to enable the electronic device to perform the method in any one of the above embodiments.

[0302] The application further provides a chip system, which includes at least one processor for implementing the functions involved in the method performed by the electronic device in any one of the above embodiments.

[0303] In a possible design, the chip system further includes a memory for storing program instructions and data, and the memory is located in or outside the processor.

[0304] The chip system can be composed of a chip, or can include a chip and other discrete devices.

[0305] Optionally, the processor in the chip system can be one or more. The processor can be implemented by hardware or software. When implemented by hardware, the processor can be a logic circuit, an integrated circuit, etc. When implemented by software, the processor can be a general-purpose processor, which is enabled by reading software codes stored in the memory.

[0306] Optionally, the memory in the chip system can also be one or more. The memory can be integrated with the processor, or can be arranged separately from the processor, and the embodiments of the application do not make any limitation in this aspect. For example, the memory can be a non-transient processor, such as a read-only memory (ROM), which can be integrated with the processor on the same chip, or can be arranged on different chips respectively, and the embodiments of the application do not make any limitation on the type of the memory and the arrangement manner of the memory and the processor.

[0307] Exemplarily, the chip system can be a field programmable gate array (FPGA), can be an application specific integrated circuit (ASIC), can also be a system on chip (SoC), can also be a central processor unit (CPU), can also be a network processor (NP), can also be a digital signal processor (DSP), can also be a micro controller unit (MCU), can also be a programmable logic device (PLD) or other integrated chip.

[0308] The present application also provides a computer program product, comprising a computer program (also referred to as code or instruction), which, when executed, causes a computer to perform the method performed by the electronic device in any one of the above embodiments.

[0309] The present application also provides a computer readable storage medium, which stores a computer program (also referred to as code or instruction). When the computer program is executed, it causes a computer to perform the method performed by the electronic device in any one of the above embodiments.

[0310] The embodiments of the present application can be combined in any manner to achieve different technical effects.

[0311] In the above embodiments, all or part of the processes can be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, all or part of the processes can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes described in the present application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions can be transferred from one website, computer, server or data center to another website, computer, server or data center through wired (such as coaxial cable, optical fiber, digital subscriber line) or wireless (such as infrared, wireless, microwave, etc.) manner. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. that includes one or more available media sets. The available media can be magnetic media (such as floppy disk, hard disk, magnetic tape), optical media (such as DVD), or semiconductor media (such as Solid State Disk), etc.

[0312] Those of ordinary skill in the art can understand that all or part of the processes in the above embodiments can be instructed by a computer program to complete the relevant hardware, and the program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the above method embodiments. The aforementioned storage medium includes ROM or random access memory (RAM), magnetic disk or optical disk, and various media that can store program codes.

[0313] In summary, the above only describes the embodiments of the technical solutions of the present application, and is not used to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made according to the disclosure of the present application shall be included in the protection scope of the present application.

Claims

1. An image processing method, characterized in that, The method includes: Acquire a first image, wherein the first image is acquired based on initial exposure parameters; The first image is input into a multi-task learning model, and the multi-task learning model outputs scene classification results and object detection results. The scene classification results include a first judgment result and the category of the shooting scene corresponding to the first image. The first judgment result is used to characterize whether the shooting scene belongs to a preset scene. The preset scene includes at least one of snow scene, stage, grass and night scene. The object detection results are used to characterize the category of the shooting object contained in the first image and the location information of the shooting object in the first image. When the first determination result indicates that the shooting scene belongs to the preset scene, the target exposure parameters are determined according to the category corresponding to the shooting scene. When the first determination result indicates that the shooting scene does not belong to the preset scene, if the category of the shooting object belongs to the preset category, the target exposure parameter is determined according to the category of the shooting object and the position information of the shooting object in the first image; if the category of the shooting object does not belong to the preset category, the exposure parameter is not adjusted. Display a second image, wherein the second image was acquired based on the target exposure parameters.

2. The method according to claim 1, characterized in that, The step of inputting the first image into a multi-task learning model and outputting scene classification results and object detection results through the multi-task learning model includes: The first image is input into the multi-task learning model, and the feature information corresponding to the first image is determined by the multi-task learning model. The feature information corresponding to the first image includes the feature information corresponding to the shooting scene and the shooting object. Based on the feature information corresponding to the first image, determine the feature information corresponding to the shooting scene and the feature information corresponding to the shooting object; The scene classification result is determined by the first output layer of the multi-task learning model based on the feature information corresponding to the shooting scene. The target detection result is determined by the second output layer of the multi-task learning model based on the feature information corresponding to the photographed object.

3. The method according to claim 2, characterized in that, The step of determining the feature information corresponding to the shooting scene and the feature information corresponding to the shooting object based on the feature information corresponding to the first image includes: Determine the first weight and the second weight; The feature information corresponding to the shooting object is determined based on the feature information corresponding to the first image and the first weight; The feature information corresponding to the shooting scene is determined based on the feature information corresponding to the first image and the second weight.

4. The method according to claim 3, characterized in that, The feature information includes feature information in the channel dimension and feature information in the spatial dimension. Determining the first weight and the second weight includes: The feature information corresponding to the first image is processed in the channel dimension to determine the first processing result, wherein the first processing result is used to characterize the weight of the feature information corresponding to the shooting object relative to the feature information corresponding to the first image in the channel dimension. The feature information corresponding to the first image is subjected to a second processing in the spatial dimension to determine a second processing result, wherein the second processing result is used to characterize the weight of the feature information corresponding to the shooting object relative to the feature information corresponding to the first image in the spatial dimension. The first weight is determined based on the first processing result and the second processing result, wherein the first weight is used to characterize the weight of the feature information corresponding to the shooting object relative to the feature information corresponding to the first image in the channel dimension and the spatial dimension. The second weight is determined based on the first weight.

5. The method according to any one of claims 2 to 4, characterized in that, The step of determining the scene classification result based on the feature information corresponding to the shooting scene through the first output layer of the multi-task learning model includes: The feature information corresponding to the shooting scene is input into the first output layer of the multi-task learning model, and the scene classification result is output through the first output layer. The scene classification result includes a first judgment result and the category corresponding to the shooting scene. The first judgment result is used to characterize whether the shooting scene belongs to a preset scene.

6. The method according to claim 5, characterized in that, The step of determining the target detection result based on the feature information corresponding to the photographed object through the second output layer of the multi-task learning model includes: The feature information corresponding to the photographed object is input into the second output layer of the multi-task learning model, and the target detection result is output through the second output layer. The target detection result includes the category of the photographed object and the position information of the photographed object in the first image.

7. A method for training a model, characterized in that, The method includes: Obtain a training set for a multi-task learning model, wherein the training set includes one or more of the following: training images, scene classification labels corresponding to the training images, and object detection labels, wherein the scene classification labels are used to characterize the shooting scene corresponding to the training images, and the object detection labels are used to characterize the shooting objects contained in the training images; The training image is used as input to the multi-task learning model, the scene classification label is used as output of the first output layer of the multi-task learning model, and the object detection label is used as output of the second output layer of the multi-task learning model. The multi-task learning model is trained to obtain a trained multi-task learning model. The multi-task learning model outputs the scene classification result and the object detection result of the first image. The scene classification result includes a first judgment result and the category corresponding to the shooting scene of the first image. The first judgment result is used to characterize whether the shooting scene belongs to a preset scene. The preset scene includes at least one of snow scene, stage, grass, and night scene. The object detection result includes the category of the shooting object and the position information of the shooting object in the first image. When the first judgment result indicates that the shooting scene belongs to the preset scene, the category corresponding to the shooting scene is used to determine the target exposure parameter. When the first judgment result indicates that the shooting scene does not belong to the preset scene, when the category of the shooting object belongs to the preset category, the category of the shooting object and the position information of the shooting object in the first image are used to determine the target exposure parameter; when the category of the shooting object does not belong to the preset category, the exposure parameter is not adjusted.

8. The method according to claim 7, characterized in that, The step of training the multi-task learning model to obtain the trained multi-task learning model includes: The feature information corresponding to the training image is determined by the multi-task learning model, wherein the feature information corresponding to the training image includes the feature information corresponding to the shooting scene and the shooting object; The third feature information and the fourth feature information are determined based on the feature information corresponding to the training image, wherein the third feature information is the feature information corresponding to the shooting scene in the feature information corresponding to the training image, and the fourth feature information is the feature information corresponding to the shooting object in the feature information corresponding to the training image; The multi-task learning model is trained based on the third and fourth feature information to obtain the trained multi-task learning model.

9. The method according to claim 8, characterized in that, The step of training the multi-task learning model based on the third feature information and the fourth feature information to obtain the trained multi-task learning model includes: The first output layer determines the scene classification result corresponding to the training image based on the third feature information; The second output layer determines the target detection result corresponding to the training image based on the fourth feature information; The multi-task learning model is trained by comparing the scene classification result and the scene classification label corresponding to the training image, and by comparing the target detection result and the target detection label corresponding to the training image, to obtain the trained multi-task learning model.

10. An electronic device, characterized in that, The electronic device includes: one or more processors and one or more memories; wherein the one or more memories are coupled to the one or more processors, the one or more memories are used to store computer program code, the computer program code including computer instructions, and the one or more processors invoke the computer instructions to cause the electronic device to perform the method as described in any one of claims 1 to 9.

11. A chip system, characterized in that, The chip system is applied to an electronic device, the chip system including one or more processors, the processors being configured to invoke computer instructions to cause the electronic device to perform the method as described in any one of claims 1 to 9.

12. A computer program product containing instructions, characterized in that, When the computer program product is run on an electronic device, it causes the electronic device to perform the method as described in any one of claims 1 to 9.

13. A computer-readable storage medium comprising instructions, characterized in that, When the instructions are executed on an electronic device, the electronic device causes the electronic device to perform the method as described in any one of claims 1 to 9.

Images