Image processing method and device and electronic equipment

Abstract

The invention discloses an image processing method and device and electronic equipment. The method applied to an image processing chip comprises the steps that the image processing chip receives sensor mode information sent by an AP, wherein the sensor mode information is information of a sensor mode entered when a camera is started; loading an artificial intelligence (AI) model according to the sensor mode information; wherein the camera comprises N sensor modes, each sensor mode corresponds to M photographing modes, and each photographing mode has a corresponding AI model; the N sensor modes have priorities, and the M photographing modes have priorities; obtaining an image frame according to a photographing instruction and photographing mode information sent by the AP; and processing the image frame by using an AI model corresponding to the photographing mode information.

Description

Technical Field

[0001] The present application belongs to the field of image display, and specifically relates to an image processing method, device and electronic equipment. Background Art

[0002] Mobile phone imaging has been one of the important research directions for major manufacturers in recent years as a means of technological innovation and improving user experience. As the current terminal functions and features are concentrated in the same application processor (AP), and there are many sensor modes and camera modes related to photography, the image processing process is cumbersome and occupies a large chip processing space. However, the processing space of the AP is limited, which affects the image processing efficiency. Summary of the invention

[0003] The purpose of the embodiments of the present application is to provide an image processing method, device and electronic device to solve the problem of low image processing efficiency when taking photos by existing terminals.

[0004] In order to solve the above technical problems, this application is implemented as follows:

[0005] In a first aspect, an embodiment of the present application provides an image processing method, which is applied to an image processing chip, comprising:

[0006] The image processing chip receives the sensor mode information sent by the application processor AP, where the sensor mode information is information about the sensor mode entered when the camera is turned on;

[0007] According to the sensor mode information, an artificial intelligence (AI) model is loaded; wherein the camera includes N sensor modes, each of the sensor modes corresponds to M photographing modes, and each photographing mode has a corresponding AI model; the N sensor modes have priorities, and the M photographing modes have priorities;

[0008] Acquire an image frame according to the photographing instruction and photographing mode information sent by the AP;

[0009] The image frame is processed using the AI ​​model corresponding to the photographing mode information.

[0010] In a second aspect, an embodiment of the present application provides an image processing method, which is applied to an AP, including:

[0011] The AP chip obtains the sensor mode information entered when the camera is turned on;

[0012] Sending the sensor mode information to an image processing chip;

[0013] According to the sensor mode information, the AI ​​model is transmitted to the image processing chip, wherein the camera includes N sensor modes, each of the sensor modes corresponds to M photographing modes, and each photographing mode has a corresponding AI model; the N sensor modes have priorities, and the M photographing modes have priorities;

[0014] When receiving a click input from the user, a photographing instruction and photographing mode information are sent to the image processing chip.

[0015] In a third aspect, an embodiment of the present application provides an image processing device, which is applied to an image processing chip, including:

[0016] A first receiving module, used to receive sensor mode information sent by the AP, where the sensor mode information is information about the sensor mode entered when the camera is turned on;

[0017] A first processing module, configured to load an AI model according to the sensor mode information; wherein the camera includes N sensor modes, each of the sensor modes corresponds to M photographing modes, and each photographing mode has a corresponding AI model; the N sensor modes have priorities, and the M photographing modes have priorities;

[0018] A first acquisition module, used to acquire an image frame according to the photographing instruction and photographing mode information sent by the AP;

[0019] The second processing module is used to process the image frame using the AI ​​model corresponding to the shooting mode information.

[0020] In a fourth aspect, an embodiment of the present application provides an image processing device, applied to an AP, including:

[0021] The second acquisition module is used to obtain the sensor mode information entered when the camera is turned on;

[0022] A first sending module, used for sending the sensor mode information to the image processing chip;

[0023] A second sending module is used to transmit the AI ​​model to the image processing chip according to the sensor mode information, wherein the camera includes N sensor modes, each of the sensor modes corresponds to M photographing modes, and each photographing mode has a corresponding AI model; the N sensor modes have priorities, and the M photographing modes have priorities;

[0024] The third sending module is used to send a photo-taking instruction and photo-taking mode information to the image processing chip when receiving a photo-taking input from the user.

[0025] In a fifth aspect, an embodiment of the present application provides an electronic device, comprising a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the program or instructions are executed by the processor, the steps of the method described in the first aspect are implemented, or the steps of the method described in the second aspect are implemented.

[0026] In a sixth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented, or the steps of the method described in the second aspect are implemented.

[0027] In the seventh aspect, an embodiment of the present application further provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the method described in the first aspect, or to implement the method described in the second aspect.

[0028] In an eighth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the method as described in the first aspect, or to implement the method as described in the second aspect.

[0029] In an embodiment of the present application, an image processing chip for user image processing is added to the electronic device. When the user takes a photo, the AP of the electronic device can send the sensor mode information entered by the user to the image processing chip, and the image processing chip loads one or more AI models corresponding to the photo mode according to the sensor mode information. After the user clicks to take a photo, the AP sends the photo instruction and the photo mode information used by the user to the image processing chip; the image processing chip obtains the image frame according to the photo mode information, and uses the AI ​​model corresponding to the photo mode to process the image frame. This embodiment sets the image processing process in the newly added image processing chip, and the corresponding AI model is pre-loaded in the image processing chip before the user clicks to take a photo. The corresponding AI model can be used to quickly process the image after the user clicks to take a photo, which saves the processing space of the AP and can improve the image processing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is one of the flowcharts of the image processing method of the embodiment of the present application;

[0031] Figure 2 This is the second flowchart of the image processing method according to the embodiment of the present application;

[0032] Figure 3 is one of the structural schematic diagrams of the image processing device according to the embodiment of the present application;

[0033] Figure 4 This is the second structural diagram of the image processing device according to the embodiment of the present application;

[0034] Figure 5 It is one of the structural schematic diagrams of the electronic device according to the embodiment of the present application;

[0035] Figure 6 This is the second structural schematic diagram of the electronic device according to the embodiment of the present application. DETAILED DESCRIPTION

[0036] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0037] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0038] The network processing method, device and electronic device provided in the embodiments of the present application are described in detail below with reference to the accompanying drawings through specific embodiments and their application scenarios.

[0039] like Figure 1 As shown, an embodiment of the present application provides an image processing method, which is applied to an image processing chip, and the method includes:

[0040] Step 101: An image processing chip receives sensor mode information sent by an application processor AP, where the sensor mode information is information about the sensor mode entered when a camera is turned on.

[0041] In this embodiment, the electronic device may include an AP and an image processing chip. The AP is the main chip of the electronic device and may also be called an application chip. The image processing chip may be an external chip, and may also be understood as a newly added chip on the electronic device for image processing. The AP and the image processing chip are connected to perform data transmission.

[0042] Optionally, the image processing chip may be an external chip equipped with a neural network processor (Neural-network Processing Unit, NPU). For example, if the NPU is set in the image processing chip, the NPU may perform image processing.

[0043] When a user opens a camera on an electronic device for the first time, he or she may enter various sensor modes according to the user's operation. For example, the user may enter "portrait" mode, "photographing" mode, "video recording" mode, etc., which may correspond to different sensor modes. The AP sends the sensor mode information entered by the user to the image processing chip.

[0044] Step 102: Load an artificial intelligence (AI) model according to the sensor mode information; wherein the camera includes N sensor modes, each of the sensor modes corresponds to M photographing modes, the N sensor modes have priorities, each photographing mode has a corresponding AI model; and the M photographing modes have priorities;

[0045] In this embodiment, after the image processing chip receives the sensor mode information sent by the AP, it loads the AI ​​model corresponding to the photo mode. Among them, the camera may contain multiple sensor modes, such as the image readout mode of the first camera, the region of interest ROI mode of the first camera, the image readout mode of the second camera, the ROI mode of the second camera, etc. For each sensor mode, multiple photo modes are corresponding, and the photo modes corresponding to each sensor mode may be the same or different. Each photo mode has a corresponding AI model, such as a night scene mode, a sports mode, etc. The night scene mode corresponds to an AI model, and the sports mode corresponds to an AI model. The AI ​​model is used to implement image processing in the photo mode. The image processing chip can only load the AI ​​model of the photo mode corresponding to the sensor mode entered by the user, or it can load the AI ​​model of the photo mode corresponding to multiple sensor modes including the sensor mode entered by the user. When loading the AI ​​model corresponding to the photo mode, the AI ​​models corresponding to all photo modes in each sensor mode can be loaded, or only the AI ​​models corresponding to some commonly used photo modes corresponding to each sensor mode can be loaded.

[0046] Optionally, the N sensor modes have loading priorities, and the M photographing modes corresponding to each sensor mode have loading priorities. It should be noted that the priority rankings of the photographing modes corresponding to different sensor modes may be the same or different.

[0047] The image processing chip loads the AI ​​model of the photographing mode corresponding to the sensor mode, which can be understood as receiving the AI ​​model of the photographing mode corresponding to the sensor mode sent by the AP, or can be understood as obtaining the AI ​​model of the photographing mode corresponding to the sensor mode from the AP. After loading the AI ​​model, the image processing chip stores the AI ​​model and can generate an identification ID corresponding to each AI model.

[0048] Step 103: Acquire an image frame according to the photographing instruction and photographing mode information sent by the AP;

[0049] In this embodiment, when the user clicks to take a photo, the AP receives the user's photo input, and according to the drive of the image processing chip, the AP controls the camera APP to send a photo instruction and photo mode information to the image processing chip, and the photo mode information is used to indicate the photo mode used when the user clicks to take a photo. The photo mode information may include a photo mode identifier.

[0050] Different shooting modes may correspond to different frame capture requirements (i.e., requirements for obtaining image frames), for example, the frame capture requirement corresponding to the night scene mode is to capture 6 frames forward, and the frame capture requirement corresponding to the sports mode is to capture 7 frames backward. The image processing chip obtains image frames according to the corresponding frame capture requirements based on the shooting mode used by the user.

[0051] Step 104: Use the AI ​​model corresponding to the photographing mode information to process the image frame.

[0052] In this embodiment, the image processing chip can determine the photographing mode used by the user according to the photographing mode information, determine the AI ​​model corresponding to the photographing mode in the loaded AI models, and use the determined AI model to process the image frame.

[0053] Optionally, after the image processing chip processes the image frame, the processed image frame is sent to the AP, and the AP performs other processing, such as cropping, adding text, etc., to finally generate a photo.

[0054] In an embodiment of the present application, an image processing chip for user image processing is added to the electronic device. When the user takes a photo, the AP of the electronic device can send the sensor mode information entered by the user to the image processing chip, and the image processing chip loads one or more AI models corresponding to the photo mode according to the sensor mode information. After the user clicks to take a photo, the AP sends the photo instruction and the photo mode information used by the user to the image processing chip; the image processing chip obtains the image frame according to the photo mode information, and uses the AI ​​model corresponding to the photo mode to process the image frame. This embodiment sets the image processing process in the newly added image processing chip, and the corresponding AI model is pre-loaded in the image processing chip before the user clicks to take a photo. The corresponding AI model can be used to quickly process the image after the user clicks to take a photo, which saves the processing space of the AP and can improve the image processing efficiency.

[0055] As an optional embodiment, the step of loading the AI ​​model according to the sensor mode information includes:

[0056] According to the sensor mode information, in the order of priority of N sensor modes from high to low and the order of priority of M camera modes from high to low, loading the AI ​​model corresponding to the camera mode, M and N are greater than or equal to 2, and the N sensor modes include the target sensor mode indicated by the sensor mode information;

[0057] or,

[0058] According to the sensor mode information, load the AI ​​model corresponding to the photographing mode under the target sensor mode indicated by the sensor mode information.

[0059] In this embodiment, when the image processing chip loads the AI ​​model, if it loads the AI ​​models of multiple camera modes corresponding to multiple sensor modes, it can be loaded in order of sensor mode priority from high to low and camera mode priority from high to low. Optionally, the image processing chip can first determine the priorities of multiple sensor modes and multiple camera modes, and load each AI model in order according to the priority order, for example, requesting the AP chip to transmit each AI model in order.

[0060] Optionally, the image processing chip can load the AI ​​models of all shooting modes corresponding to all sensor modes in sequence according to priority, so that no matter what shooting mode the user uses, the image can be processed quickly without the need to obtain the AI ​​model separately, thereby improving image processing efficiency.

[0061] Alternatively, the image processing chip can load AI models of all photographing modes (or some photographing modes, such as commonly used photographing modes) corresponding to some sensor modes. If the AI ​​model of the photographing mode actually used by the user for subsequent photographing has been loaded, the corresponding AI model can be used directly for image processing. If it has not been loaded, the AI ​​model corresponding to the photographing mode actually used can be reloaded. This can save storage space of the image processing chip on the one hand, and on the other hand, because commonly used AI models are stored, image processing efficiency can be improved for most photographing situations.

[0062] Alternatively, the image processing chip can load only the AI ​​model of the photo mode corresponding to the sensor mode entered by the user according to the sensor mode information, and perform high-speed image processing for the user's photo, which can further reduce the space and cost of the image processing chip.

[0063] As an optional embodiment, the method further includes:

[0064] Determining the priority of the sensor mode according to the sensor mode information;

[0065] The priority of the sensor mode is determined according to the sensor mode entered when the camera is turned on and the zoom ratio corresponding to the sensor mode; the target sensor mode indicated by the sensor mode information has the highest priority;

[0066] The sensor mode includes at least one of the following: an image readout mode for the first camera, a region of interest (ROI) mode for the first camera, an image readout mode for the second camera, and an ROI mode for the second camera.

[0067] In this embodiment, the image processing chip can determine the priority order of multiple sensor modes according to the sensor mode entered by the user and the zoom magnification corresponding to each sensor mode. Assume that the sensor modes include four types: image readout mode (bining) for the first camera, region of interest ROI mode for the first camera, image readout mode (bining) for the second camera, and ROI mode for the second camera. The first camera can be a main camera, the image readout mode for the first camera can be expressed as 1X bining, and the ROI mode for the first camera can be expressed as 1X ROI mode; the second camera can be a tele camera, the image readout mode for the second camera can be expressed as 3X bining, and the ROI mode for the second camera can be expressed as 3X ROI mode. Here, "X" represents the zoom magnification. Different sensor modes may correspond to different zoom magnifications, for example, the zoom magnification in the "portrait" mode is different from the zoom magnification in the "camera" mode.

[0068] When determining the priorities of multiple sensor modes, the sensor mode entered by the user is determined, and the sensor mode entered by the user has the highest priority. For example, if the sensor mode entered by the user is 1X bining, the priorities of the four sensor modes are ranked from high to low as 1X bining>1X ROI>3X bining>3X ROI.

[0069] The priority order of multiple sensor modes is as follows:

[0070] (1) If the sensor mode entered by the user is 1X, 1X binning and 1X ROI are the highest priority, and 3X binning and 3X ROI are the second priority. Conversely, if the sensor mode entered by the user is 3X, 3X binning and 3X ROI are the first priority, and 1X bining and 1X ROI are the second priority.

[0071] In this case, the AI ​​model of the camera mode corresponding to the 1X bining mode is loaded first, then the AI ​​model of the camera mode corresponding to the 1XROI (2x zoom) mode is loaded, then the AI ​​model of the camera mode corresponding to the 3X bining (3x zoom) mode is loaded, and finally the AI ​​model of the camera mode corresponding to the 3X ROI (6x zoom) mode is loaded. Because the user's zoom time is relatively long, which is in seconds, the AI ​​models under the following three sensor modes have sufficient time to load.

[0072] (2) If the sensor mode entered by the user is 1X binning mode, 1X binning is placed before 1X ROI. If the sensor mode entered is 1X ROI, 1X ROI is placed before 1X bining.

[0073] The priority of the photographing mode corresponding to each sensor mode can be determined according to the user's usage of the photographing mode. For example, if the user uses the night scene mode the most when taking pictures, the night scene mode has the highest priority, and the priorities of other photographing modes are determined in turn. The priority of the photographing mode can also be sorted according to the requirements of each photographing mode for obtaining image frames. For example, the priority of the photographing mode is determined according to the frame capture situation of each photographing mode. For example: the priority of the photographing mode for taking frames forward is higher than the priority of the photographing mode for taking frames backward, and the priority of the photographing mode with a small number of frames is higher than the priority of the photographing mode with a large number of frames.

[0074] As an optional embodiment, according to the sensor mode information, in the order of priority of the N sensor modes from high to low and the order of priority of the M camera modes from high to low, the AI ​​model corresponding to the camera mode is loaded, including:

[0075] According to the order of priority of the N sensor modes from high to low and the order of priority of the M camera modes from high to low, the AI ​​models corresponding to at least two camera modes corresponding to each sensor mode are loaded in sequence;

[0076] The photographing modes corresponding to each sensor mode include at least two of the following:

[0077] Night mode;

[0078] High Dynamic Range Imaging (HDR) mode;

[0079] Sports mode;

[0080] Polar night mode.

[0081] In this embodiment, when loading the AI ​​model, the image processing chip can load the AI ​​models of at least two camera modes corresponding to each sensor mode. The at least two camera modes can be camera modes commonly used by users. The camera modes may include at least two of the following: night scene mode, HDR mode, sports mode, polar night mode, etc. It should be noted that the camera modes include but are not limited to the four described above. This camera mode is only an example and may also include any other camera modes.

[0082] The priority of multiple photo modes can be determined according to the frequency of use by the user, and the priority of the photo mode corresponds to the priority of loading the AI ​​model. Taking the sensor mode including 1X binning, 1X ROI, 3X bining, 3X ROI, and each sensor mode including the above four photo modes as an example, the priority of the photo mode and the corresponding frame capture requirements (i.e., the requirements for obtaining image frames) of each photo mode are shown in Table 1 below.

[0083] Table 1: Photo mode priority

[0084]

[0085] As an optional embodiment, acquiring an image frame according to the photographing instruction and photographing mode information sent by the AP includes:

[0086] According to the photographing instruction and the photographing mode information, the image frame of the original image is acquired; wherein the photographing mode information includes: the photographing mode selected by the user and the image frame acquisition requirements corresponding to the photographing mode; different photographing modes correspond to different image frame acquisition requirements.

[0087] In this embodiment, when the user clicks to take a photo, the AP sends the photo instruction and the photo mode information used by the user to the image processing chip. The image processing chip obtains the image frame according to the photo mode used by the user.

[0088] For the priority of the photo modes in Table 1 above, both the night scene and HDR modes require forward frame capture. After the user clicks to take a photo, the NPU photo acceleration needs to be executed immediately. Both the sports mode and the polar night mode require backward frame capture. For example, at 30fps, grabbing 7 and 8 frames backward will add an additional 231ms and 262ms respectively. The AI ​​model can continue to be transmitted while acquiring the image frame. After the frame capture is completed, the AI ​​model is loaded and the NPU photo acceleration can continue.

[0089] Optionally, the method also includes: upon receiving a photo-taking instruction and photo-taking mode information sent by the AP, if the AI ​​model corresponding to the photo-taking mode indicated by the photo-taking mode information is not loaded in the image processing chip, then loading the AI ​​model corresponding to the photo-taking mode indicated by the photo-taking mode information.

[0090] In this embodiment, the image processing chip determines the corresponding AI model from the already loaded AI models according to the photographing mode used by the user, and if the AI ​​model corresponding to the photographing mode used by the user is not loaded, the image processing chip loads the AI ​​model. For example: when the image processing chip loads the AI ​​model in step 102, only the AI ​​models corresponding to some photographing modes corresponding to multiple sensor modes are loaded, or only the AI ​​models corresponding to some photographing modes corresponding to the sensor mode entered by the user are loaded, and the AI ​​model corresponding to the photographing mode actually used by the user when taking pictures is not loaded, then the AI ​​model corresponding to the photographing mode actually used when taking pictures is obtained from the AP.

[0091] Optionally, the image frame is processed using an AI model corresponding to the photographing mode information, including:

[0092] When the AI ​​model corresponding to the photographing mode indicated by the photographing mode information is loaded, the image frame is processed using the AI ​​model in the network processor NPU of the image processing chip.

[0093] In this embodiment, when the image processing chip is performing image processing, if the corresponding AI model has not been loaded, the image processing is performed after the loading is completed. The image processing chip includes an NPU, and the loaded AI model can be stored in the NPU, and the image processing is performed in the NPU.

[0094] The following example illustrates the implementation process of the image processing method in the embodiment of the present application.

[0095] Example 1: The image processing chip loads the AI ​​models of all camera modes corresponding to all sensor modes.

[0096] Assume that there are four sensor modes: 1X bining, 1X ROI, 3X bining, 3X ROI, and each sensor mode contains four photo modes: night scene, HDR, sports, and polar night. The AI ​​models corresponding to the 16 photo modes are divided into four groups, and the priority order is determined according to the application scenario (as shown in Table 1). The AP transmits all the above AI models at one time and saves them to the double data rate (DDR) memory of the image processing chip. If the loading is not completed in extreme cases, complete the frame capture action first, wait for the AI ​​model to be loaded, and then enter the NPU for image processing.

[0097] Step 11: Open the camera and enter the preview. This step is a prerequisite for taking photos.

[0098] Step 12: According to the user's operation of opening the camera, obtain the sensor mode entered by the user.

[0099] When a user turns on the camera and enters it for the first time, he or she may enter various modes according to the user's operation. The sensor modes can be divided into four types: 1X bining, 1X ROI, 3X bining, and 3X ROI. The sensormode information entered by the user is sent to the image processing chip (such as an external integrated circuit (IC) or an external chip).

[0100] Step 13: Load AI models according to priority based on sensor mode information.

[0101] The image processing chip (such as an external IC) divides all AI model files into four categories and four priorities according to the sensor mode entered by the user.

[0102] For example, if the sensor mode entered by the user is 1X bining, the priority is sorted from high to low as 1Xbining>1X ROI>3X bining>3X ROI.

[0103] Among them, every time the AP transmits an AI model, it will notify the image processing chip (such as an external IC) of the model ID that has just been transmitted.

[0104] In this step, the policy principle for setting the model priority is: the requirements for obtaining image frames in night scene mode and HDR mode both include forward frame capture. After the user clicks to take a photo, NPU image processing needs to be performed immediately. Both sports mode and polar night mode capture frames backward. For example, in the case of 30fps, 7 and 8 frames are captured backward (231ms and more than 262ms), respectively. The AI ​​model can continue to be transmitted while capturing frames. After the frame capture is completed, the AI ​​model is also loaded, and the NPU photo acceleration can continue.

[0105] Step 14: The user clicks to take a photo.

[0106] When the user clicks to take a photo, the camera APP will send the photo command and related meta information (such as photo mode information) to the image processing chip (such as the external IC) through the driver of the external IC.

[0107] Step 15: The image processing chip (such as an external IC) completes frame capture (ie, obtains the image frame).

[0108] The image processing chip (such as an external IC) receives the photo-taking instruction and related meta information of step 14, wherein the meta information includes the photo-taking mode (such as night scene, HDR, sports, polar night) and the frame grabbing requirements of the photo-taking (how many frames to grab forward, how many frames to grab backward). The frame grabbing action (capturing the original RAW image) is first completed inside the image processing chip (such as an external IC) according to the meta information, and the original RAW image is locked.

[0109] Step 16: Select the corresponding AI model ID.

[0110] According to the photo mode transmitted in step 14, determine the corresponding AI model ID.

[0111] Step 17: Determine whether the corresponding AI model has been loaded.

[0112] According to the AI ​​model ID obtained in step 16, it is determined whether the corresponding AI model has been transmitted.

[0113] Step 18: Continue with step 17. If the AI ​​model has not been loaded, wait for it to be loaded.

[0114] In this step, if the AI ​​model has not been transmitted, wait for the corresponding model to be transmitted. This situation occurs when the load is extremely high and does not occur under normal circumstances.

[0115] Step 19: Continuing from step 17, if the AI ​​model is loaded, the image frame is directly processed in the NPU.

[0116] Usually, the AI ​​model selected in step 16 has been loaded, and the image frame can be directly processed in the NPU.

[0117] Step 20: Photo taking is complete.

[0118] After the image processing chip (such as an external IC) completes the image processing, the image is sent to the AP for other post-processing and finally a JPEG photo is generated.

[0119] In this embodiment, when the user turns on the camera, all possible AI models that may be used are transmitted to the image processing chip (such as an external IC) at one time and saved in the DDR of the image processing chip. In this way, there is no need to retransmit the model in various switching situations, ensuring that the NPU photo acceleration will not be delayed, ensuring the best user experience.

[0120] Example 2: The image processing chip loads the AI ​​models corresponding to some camera modes corresponding to all sensor modes.

[0121] Assume that there are four sensor modes: 1X bining, 1X ROI, 3X bining, 3X ROI, and each sensor mode contains four photo modes: night scene, HDR, sports, and polar night. When the user turns on the camera, only 8 groups (a total of 16 groups) of model files are loaded at one time. These 8 groups of model files are the AI ​​models of the night scene mode and HDR mode corresponding to the four sensor modes of 1X bining, 1X ROI, 3X bining, and 3X ROI. When the AI ​​model corresponding to the sports mode or polar night mode is needed, it will be re-transmitted. In this way, only 9 model files (8 fixed models + 1 instantaneous model) need to be saved in the image processing chip.

[0122] Step 21: Open the camera and enter the preview. This step is a prerequisite for taking photos.

[0123] Step 22: According to the user's operation of turning on the camera, the sensor mode entered by the user is obtained.

[0124] When a user turns on the camera and enters it for the first time, he or she may enter various modes depending on the user's operation. The sensor mode can be divided into four types: 1X bining, 1X ROI, 3X bining, and 3X ROI. The sensormode information entered by the user is sent to the image processing chip (such as an external IC).

[0125] Step 23: Load AI models according to priority based on sensor mode information.

[0126] The image processing chip (such as an external IC) divides all AI model files into four categories and four priorities according to the sensor mode entered by the user.

[0127] For example, if the sensor mode entered by the user is 1X bining, the priority is sorted from high to low as 1Xbining>1X ROI>3X bining>3X ROI.

[0128] Among them, every time the AP transmits an AI model, it will notify the image processing chip (such as an external IC) of the model ID that has just been transmitted.

[0129] In this step, the policy principle for setting the model priority is: the requirements for obtaining image frames in night scene mode and HDR mode both include forward frame capture. After the user clicks to take a photo, NPU image processing needs to be performed immediately. Both sports mode and polar night mode capture frames backward. For example, in the case of 30fps, 7 and 8 frames are captured backward, respectively, with an additional increase (231ms and more than 262ms). The AI ​​model can continue to be transmitted while capturing frames. After the frame capture is completed, the AI ​​model is also loaded, and NPU photo acceleration can continue.

[0130] Compared with Example 1, in this step, each sensor mode only loads the AI ​​model corresponding to the night scene and HDR photography modes.

[0131] Step 24: The user clicks to take a photo.

[0132] When the user clicks to take a photo, the camera APP will send the photo command and related meta information (such as photo mode information) to the image processing chip (such as the external IC) through the driver of the external IC.

[0133] Step 25: Determine whether the camera mode used by the user is sports mode or polar night mode.

[0134] Step 26: Transfer the corresponding AI model file.

[0135] If the condition in step 25 is true, that is, whether the shooting mode used by the user is sports or polar night mode, then step 26 is executed, otherwise this step does not need to be executed.

[0136] If the shooting mode used by the user in step 25 is the sports mode, the AI ​​model file corresponding to the sports mode in the current sensor mode is transmitted.

[0137] If the shooting mode used by the user in step 25 is the polar night mode, the AI ​​model file corresponding to the polar night mode in the current sensor mode is transmitted.

[0138] Optionally, the transmitted AI model file corresponding to the motion mode or polar night mode in the current sensor mode will overwrite the model memory in the image processing chip.

[0139] Step 27: Select the corresponding AI model ID.

[0140] According to the photo taking mode transmitted in step 24, the corresponding AI model ID is determined.

[0141] Step 28: The image processing chip (such as an external IC) completes frame capture (ie, obtains the image frame).

[0142] The image processing chip (such as an external IC) receives the photo-taking instruction and related meta information of step 24, wherein the meta information includes the photo-taking mode (such as night scene, HDR, sports, polar night) and the frame grabbing requirements of the photo-taking (how many frames to grab forward, how many frames to grab backward). The frame grabbing action (capturing the original RAW image) is first completed inside the image processing chip (such as an external IC) according to the meta information, and the original RAW image is locked.

[0143] Step 29: Determine whether the corresponding AI model has been loaded.

[0144] According to the AI ​​model ID obtained in step 27, it is determined whether the corresponding AI model has been transmitted.

[0145] Step 30: Continue with step 29. If the AI ​​model has not been loaded, wait for it to be loaded.

[0146] In this step, if the AI ​​model has not been transmitted, wait for the corresponding model to be transmitted. This situation occurs when the load is extremely high and does not occur under normal circumstances.

[0147] Step 31: Continuing from step 29, if the AI ​​model is loaded, the image frame is directly processed in the NPU.

[0148] Typically, the AI ​​model selected in step 27 has been loaded, and the image frame can be directly processed in the NPU.

[0149] Step 32: Photo taking is complete.

[0150] After the image processing chip (such as an external IC) completes the image processing, the image is sent to the AP for other post-processing and finally a JPEG photo is generated.

[0151] Compared with Example 1, this embodiment can save DDR space in the image processing chip and effectively reduce the cost of the external image processing chip.

[0152] Optionally, based on the second example, the DDR space of the image processing chip can be further reduced, and the cost of the image processing chip can be further reduced. For example, if the sensor mode entered by the user is 1X bining, the image processing chip only loads and stores the AI ​​models of the night scene mode and HDR mode corresponding to the 1X bining mode. If the sensor mode is switched, the AI ​​models of the night scene mode and HDR mode corresponding to the switched sensor mode are transmitted. The other steps are the same as those in the second example and will not be repeated here.

[0153] In the embodiment of the present application, the external image processing based on NPU camera acceleration, due to the large number of camera modes, the AP needs to transfer a large number of large model files to the image processing chip. Common camera modes include ordinary night scene (grabbing forward frames), ordinary HDR (grabbing 5 frames forward and 2 frames backward), sports (grabbing 7 frames backward), polar night (grabbing 8 frames backward) and other modes. The sensor mode is divided into 1X binning, 1X ROI, 3X binning, 3X ROI. These modes correspond to the binning mode of the main camera, the ROI quad mode of the main camera, the binning mode of the tele camera, and the ROI quad mode of the tele camera. Each sensor mode corresponds to 4 camera modes, which corresponds to 4 AI models, for a total of 16 models. Before the user turns on the camera and takes a photo, the AP chip transmits the AI ​​models corresponding to multiple camera modes to the image processing chip through the Secure Digital Input / Output (SDIO) interface, which will not affect the image processing rate after taking the photo, thereby realizing camera acceleration.

[0154] In an embodiment of the present application, an image processing chip for user image processing is added to the electronic device. When the user takes a photo, the AP of the electronic device can send the sensor mode information entered by the user to the image processing chip, and the image processing chip loads one or more AI models corresponding to the photo mode according to the sensor mode information. After the user clicks to take a photo, the AP sends the photo instruction and the photo mode information used by the user to the image processing chip; the image processing chip obtains the image frame according to the photo mode information, and uses the AI ​​model corresponding to the photo mode to process the image frame. This embodiment sets the image processing process in the newly added image processing chip, and the corresponding AI model is pre-loaded in the image processing chip before the user clicks to take a photo. The corresponding AI model can be used to quickly process the image after the user clicks to take a photo, which saves the processing space of the AP and can improve the image processing efficiency.

[0155] like Figure 2 As shown, the embodiment of the present application also provides an image processing method, which is applied to an AP, including:

[0156] Step 201, the AP obtains sensor mode information entered when the camera is turned on;

[0157] Step 202: Send the sensor mode information to the image processing chip;

[0158] Step 203: According to the sensor mode information, provide the image processing chip AI model, wherein the camera includes N sensor modes, each of the sensor modes corresponds to M photographing modes, and each photographing mode has a corresponding AI model; the N sensor modes have priorities, and the M photographing modes have priorities;

[0159] Step 204: When receiving a photographing input from the user, a photographing instruction and photographing mode information are sent to the image processing chip.

[0160] In this embodiment, the electronic device may include an AP and an image processing chip. The AP is the main chip of the electronic device and may also be called an application chip. The image processing chip may be an external chip, and may also be understood as a newly added chip on the electronic device for image processing. The AP and the image processing chip are connected to perform data transmission.

[0161] When a user opens a camera on an electronic device for the first time, he or she may enter various sensor modes according to the user's operation. For example, the user may enter "portrait" mode, "photographing" mode, "video recording" mode, etc., which may correspond to different sensor modes. The AP sends the sensor mode information entered by the user to the image processing chip.

[0162] After the AP sends the sensor mode information, it transmits the AI ​​model corresponding to the photo mode to the image processing chip. Among them, the camera may contain multiple sensor modes, such as the image readout mode of the first camera, the region of interest ROI mode of the first camera, the image readout mode of the second camera, the ROI mode of the second camera, etc. For each sensor mode, multiple photo modes are corresponding, and the photo modes corresponding to each sensor mode can be the same or different. Each photo mode has a corresponding AI model, such as a night scene mode, a sports mode, etc. The night scene mode corresponds to an AI model, and the sports mode corresponds to an AI model. The AI ​​model is used to implement image processing in the photo mode. The AP can only transmit the AI ​​model of the photo mode corresponding to the sensor mode entered by the user, or it can transmit the AI ​​model of the photo mode corresponding to multiple sensor modes including the sensor mode entered by the user. When transmitting the AI ​​model corresponding to the photo mode, the AI ​​models corresponding to all photo modes corresponding to each sensor mode can be transmitted, or only the AI ​​models corresponding to some commonly used photo modes corresponding to each sensor mode can be transmitted.

[0163] Optionally, the N sensor modes have transmission priorities, and the M photographing modes corresponding to each sensor mode have transmission priorities. It should be noted that the priority rankings of the photographing modes corresponding to different sensor modes may be the same or different.

[0164] When the user clicks to take a photo, the AP sends a photo instruction and photo mode information to the image processing chip according to the driver of the image processing chip, and the photo mode information is used to indicate the photo mode used when the user clicks to take a photo. The photo mode information may include a photo mode identifier.

[0165] The image processing chip can determine the photographing mode used by the user according to the photographing mode information, determine the AI ​​model corresponding to the photographing mode in the loaded AI models, and then use the determined AI model to process the image frame.

[0166] In an embodiment of the present application, an image processing chip for user image processing is added to the electronic device. When the user takes a photo, the AP of the electronic device can send the sensor mode information entered by the user to the image processing chip, and the image processing chip loads one or more AI models corresponding to the photo mode according to the sensor mode information. After the user clicks to take a photo, the AP sends the photo instruction and the photo mode information used by the user to the image processing chip, so that the image processing chip can obtain the image frame according to the photo mode information, and use the AI ​​model corresponding to the photo mode to process the image frame. This embodiment sets the image processing process in the newly added image processing chip, and the corresponding AI model is pre-loaded in the image processing chip before the user clicks to take a photo. The corresponding AI model can be used to quickly process the image after the user clicks to take a photo, which saves the processing space of the AP and can improve the image processing efficiency.

[0167] As an optional embodiment, transmitting the AI ​​model to the image processing chip according to the sensor mode information includes:

[0168] According to the sensor mode information, in the order of priority of N sensor modes from high to low and the order of priority of M camera modes from high to low, transmitting the AI ​​model corresponding to the camera mode, M and N are greater than or equal to 2, and the N sensor modes include the target sensor mode indicated by the sensor mode information;

[0169] or,

[0170] According to the sensor mode information, an AI model corresponding to the photographing mode under the target sensor mode indicated by the sensor mode information is transmitted.

[0171] In this embodiment, when the AP transmits the AI ​​model, if it transmits the AI ​​models of multiple camera modes corresponding to multiple sensor modes, it can be transmitted in order of sensor mode priority from high to low and camera mode priority from high to low. Optionally, the AP can first determine the priorities of multiple sensor modes and multiple camera modes, and transmit each AI model in order according to the priority order.

[0172] Optionally, the AP can transmit the AI ​​models of all shooting modes corresponding to all sensor modes in sequence according to priority, so that no matter what shooting mode the user uses, the image processing chip can quickly process the image without the need to transmit the AI ​​model separately, thereby improving image processing efficiency.

[0173] Alternatively, the AP may transmit AI models of all photographing modes (or some photographing modes, such as commonly used photographing modes) corresponding to some sensor modes. If the AI ​​model of the photographing mode actually used by the user for subsequent photographing has been transmitted, the image processing chip can directly use the corresponding AI model for image processing. If it has not been transmitted, the AI ​​model corresponding to the photographing mode actually used can be retransmitted. This can save storage space of the image processing chip on the one hand, and on the other hand, because commonly used AI models are stored, image processing efficiency can be improved for most photographing situations.

[0174] Alternatively, the AP can transmit only the AI ​​model of the photo mode corresponding to the sensor mode entered by the user based on the sensor mode information, and perform high-speed image processing for the user's photo, which can further reduce the space and cost of the image processing chip.

[0175] As an optional embodiment, the method further includes:

[0176] Determining the priority of the sensor mode according to the sensor mode information;

[0177] The priority of the sensor mode is determined according to the sensor mode entered when the camera is turned on and the zoom ratio corresponding to the sensor mode; the target sensor mode indicated by the sensor mode information has the highest priority;

[0178] The sensor mode includes at least one of the following: an image readout mode for the first camera, a ROI mode for the first camera, an image readout mode for the second camera, and a ROI mode for the second camera.

[0179] In this embodiment, the AP may determine the priority order of multiple sensor modes according to the sensor mode entered by the user and the zoom magnification corresponding to each sensor mode. Assume that the sensor modes include four types: image readout mode (bining) for the first camera, region of interest ROI mode for the first camera, image readout mode (bining) for the second camera, and ROI mode for the second camera. The first camera may be a main camera, the image readout mode for the first camera may be expressed as 1X bining, and the ROI mode for the first camera may be expressed as 1XROI mode; the second camera may be a telescopic camera, the image readout mode for the second camera may be expressed as 3X bining, and the ROI mode for the second camera may be expressed as 3X ROI mode. Here, "X" represents the zoom magnification. Different sensor modes may correspond to different zoom magnifications, for example, the zoom magnification in the "portrait" mode is different from the zoom magnification in the "video" mode.

[0180] When determining the priorities of multiple sensor modes, the sensor mode entered by the user is determined, and the sensor mode entered by the user has the highest priority. For example, if the sensor mode entered by the user is 1X bining, the priorities of the four sensor modes are ranked from high to low as 1X bining>1X ROI>3X bining>3X ROI.

[0181] (1) If the sensor mode entered by the user is 1X, 1X binning and 1X ROI are the highest priority, and 3X binning and 3X ROI are the second priority. Conversely, if the sensor mode entered by the user is 3X, 3X binning and 3X ROI are the first priority, and 1X bining and 1X ROI are the second priority.

[0182] In this case, the AI ​​model of the camera mode corresponding to the 1X bining mode is loaded first, then the AI ​​model of the camera mode corresponding to the 1XROI (2x zoom) mode is loaded, then the AI ​​model of the camera mode corresponding to the 3X bining (3x zoom) mode is loaded, and finally the AI ​​model of the camera mode corresponding to the 3X ROI (6x zoom) mode is loaded. Because the user's zoom time is relatively long, which is in seconds, the AI ​​models under the following three sensor modes have sufficient time to load.

[0183] (2) If the sensor mode entered by the user is 1X binning mode, 1X binning is placed before 1X ROI. If the sensor mode entered is 1X ROI, 1X ROI is placed before 1X bining.

[0184] The priority of the photographing mode corresponding to each sensor mode can be determined according to the user's usage of the photographing mode. For example, if the user uses the night scene mode the most when taking pictures, the night scene mode has the highest priority, and the priorities of other photographing modes are determined in turn. The priority of the photographing mode can also be sorted according to the requirements of each photographing mode for obtaining image frames. For example, the priority of the photographing mode is determined according to the frame capture situation of each photographing mode. For example: the priority of the photographing mode for taking frames forward is higher than the priority of the photographing mode for taking frames backward, and the priority of the photographing mode with a small number of frames is higher than the priority of the photographing mode with a large number of frames.

[0185] As an optional embodiment, according to the sensor mode information, in the order of priority of the N sensor modes from high to low and the order of priority of the M camera modes from high to low, the AI ​​model corresponding to the camera mode is transmitted, including:

[0186] In descending order of priority of the N sensor modes and in descending order of priority of the M camera modes, sequentially transmitting AI models corresponding to at least two camera modes corresponding to each sensor mode;

[0187] The photographing modes corresponding to each sensor mode include at least two of the following:

[0188] Night mode;

[0189] HDR mode;

[0190] Sports mode;

[0191] Polar night mode.

[0192] In this embodiment, when transmitting the AI ​​model, the AP may transmit the AI ​​models of at least two photo modes corresponding to each sensor mode. The at least two photo modes may be photo modes commonly used by users. The photo modes may include at least two of the following: night scene mode, HDR mode, sports mode, polar night mode, etc. It should be noted that the photo modes include but are not limited to the four described above. The photo modes are for example only and may also include any other photo modes.

[0193] The priority of multiple photo modes can be determined according to the frequency of use by the user, and the priority of the photo mode corresponds to the priority of the transmission AI model. Taking the sensor mode including 1X binning, 1X ROI, 3X bining, 3X ROI, and each sensor mode includes the above four photo modes as an example, the priority of the photo mode and the corresponding frame capture requirements (i.e., the requirements for obtaining image frames) of each photo mode are shown in Table 1.

[0194] When the user clicks to take a photo, the AP sends the photo command and the photo mode information used by the user to the image processing chip. The image processing chip obtains the image frame according to the photo mode used by the user, and processes the image frame using the transmitted AI mode.

[0195] When the image processing chip is performing image processing, if the corresponding AI model has not been loaded, the image processing is performed after the loading is completed. The image processing chip includes an NPU, and the loaded AI model can be stored in the NPU, and the image processing is performed in the NPU.

[0196] In an embodiment of the present application, an image processing chip for user image processing is added to the electronic device. When the user takes a photo, the AP of the electronic device can send the sensor mode information entered by the user to the image processing chip, and the image processing chip loads one or more AI models corresponding to the photo mode according to the sensor mode information. After the user clicks to take a photo, the AP sends the photo instruction and the photo mode information used by the user to the image processing chip, so that the image processing chip can obtain the image frame according to the photo mode information, and use the AI ​​model corresponding to the photo mode to process the image frame. This embodiment sets the image processing process in the newly added image processing chip, and the corresponding AI model is pre-loaded in the image processing chip before the user clicks to take a photo. The corresponding AI model can be used to quickly process the image after the user clicks to take a photo, which saves the processing space of the AP and can improve the image processing efficiency.

[0197] like Figure 3 As shown, the embodiment of the present application further provides an image processing device 300, which is applied to an image processing chip of an electronic device, including:

[0198] A first receiving module 310 is used to receive sensor mode information sent by the AP, where the sensor mode information is information about the sensor mode entered when the camera is turned on;

[0199] The first processing module 320 is used to load an artificial intelligence AI model according to the sensor mode information; wherein the camera includes N sensor modes, each of the sensor modes corresponds to M photographing modes, and each photographing mode has a corresponding AI model; the N sensor modes have priorities, and the M photographing modes have priorities;

[0200] A first acquisition module 330, configured to acquire an image frame according to the photographing instruction and photographing mode information sent by the AP;

[0201] The second processing module 340 is used to process the image frame using the AI ​​model corresponding to the photographing mode information.

[0202] Optionally, the first processing module is specifically used to:

[0203] According to the sensor mode information, in the order of priority of N sensor modes from high to low and the order of priority of M camera modes from high to low, loading the AI ​​model corresponding to the camera mode, M and N are greater than or equal to 2, and the N sensor modes include the target sensor mode indicated by the sensor mode information;

[0204] or,

[0205] According to the sensor mode information, load the AI ​​model corresponding to the photographing mode under the target sensor mode indicated by the sensor mode information.

[0206] Optionally, the device further comprises:

[0207] A first determining module, configured to determine the priority of the sensor mode according to the sensor mode information;

[0208] The priority of the sensor mode is determined according to the sensor mode entered when the camera is turned on and the zoom ratio corresponding to the sensor mode; the target sensor mode indicated by the sensor mode information has the highest priority;

[0209] The sensor mode includes at least one of the following: an image readout mode for the first camera, a region of interest ROI mode for the first camera, an image readout mode for the second camera, and a ROI mode for the second camera.

[0210] Optionally, the first processing module is specifically used to:

[0211] According to the order of priority of the N sensor modes from high to low and the order of priority of the M camera modes from high to low, the AI ​​models corresponding to at least two camera modes corresponding to each sensor mode are loaded in sequence;

[0212] The photographing modes corresponding to each sensor mode include at least two of the following:

[0213] Night mode;

[0214] HDR mode;

[0215] Sports mode;

[0216] Polar night mode.

[0217] Optionally, the first acquisition module is specifically used to:

[0218] Acquire an image frame of the original image according to the photographing instruction and the photographing mode information;

[0219] The photographing mode information includes: a photographing mode selected by a user and image frame acquisition requirements corresponding to the photographing mode; different photographing modes correspond to different image frame acquisition requirements.

[0220] Optionally, the device further comprises:

[0221] The third processing module is used to load the AI ​​model corresponding to the photographing mode indicated by the photographing mode information if the AI ​​model corresponding to the photographing mode indicated by the photographing mode information is not loaded in the image processing chip when the photographing instruction and photographing mode information sent by the AP are received.

[0222] Optionally, the second processing module is specifically used to:

[0223] When the AI ​​model corresponding to the photographing mode indicated by the photographing mode information is loaded, the image frame is processed using the AI ​​model in the network processor NPU of the image processing chip.

[0224] In an embodiment of the present application, an image processing chip for user image processing is added to the electronic device. When the user takes a photo, the AP of the electronic device can send the sensor mode information entered by the user to the image processing chip, and the image processing chip loads one or more AI models corresponding to the photo mode according to the sensor mode information. After the user clicks to take a photo, the AP sends the photo instruction and the photo mode information used by the user to the image processing chip; the image processing chip obtains the image frame according to the photo mode information, and uses the AI ​​model corresponding to the photo mode to process the image frame. This embodiment sets the image processing process in the newly added image processing chip, and the corresponding AI model is pre-loaded in the image processing chip before the user clicks to take a photo. The corresponding AI model can be used to quickly process the image after the user clicks to take a photo, which saves the processing space of the AP and can improve the image processing efficiency.

[0225] like Figure 4 As shown, the embodiment of the present application further provides an image processing device 400, which is applied to an AP of an electronic device, including:

[0226] The second acquisition module 410 is used to acquire sensor mode information entered when the camera is turned on;

[0227] A first sending module 420, configured to send the sensor mode information to an image processing chip;

[0228] A second sending module 430 is used to transmit the AI ​​model to the image processing chip according to the sensor mode information, wherein the camera includes N sensor modes, each of the sensor modes corresponds to M photographing modes, and each photographing mode has a corresponding AI model; the N sensor modes have priorities, and the M photographing modes have priorities;

[0229] The third sending module 440 is used to send a photo-taking instruction and photo-taking mode information to the image processing chip when receiving a photo-taking input from the user.

[0230] Optionally, the second sending module is specifically used for:

[0231] According to the sensor mode information, in the order of priority of N sensor modes from high to low and the order of priority of M camera modes from high to low, transmitting the AI ​​model corresponding to the camera mode, M and N are greater than or equal to 2, and the N sensor modes include the target sensor mode indicated by the sensor mode information;

[0232] or,

[0233] According to the sensor mode information, an AI model corresponding to the photographing mode under the target sensor mode indicated by the sensor mode information is transmitted.

[0234] Optionally, the device further comprises:

[0235] A second determining module, configured to determine the priority of the sensor mode according to the sensor mode information;

[0236] The priority of the sensor mode is determined according to the sensor mode entered when the camera is turned on and the zoom ratio corresponding to the sensor mode; the target sensor mode indicated by the sensor mode information has the highest priority;

[0237] The sensor mode includes at least one of the following: an image readout mode for the first camera, a ROI mode for the first camera, an image readout mode for the second camera, and a ROI mode for the second camera.

[0238] Optionally, the second sending module is specifically used for:

[0239] Transmitting the AI ​​models corresponding to at least two photographing modes in each sensor mode in sequence according to the order of priority of the N sensor modes from high to low and the order of priority of the M photographing modes from high to low;

[0240] The photographing modes corresponding to each sensor mode include at least two of the following:

[0241] Night mode;

[0242] HDR mode;

[0243] Sports mode;

[0244] Polar night mode.

[0245] The image processing device in the embodiment of the present application may be a component in an electronic device, such as an integrated circuit or a chip. The electronic device may be a terminal, or may be other devices other than a terminal. Exemplarily, the electronic device may be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc. It may also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.

[0246] The image processing device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0247] The image processing device provided in the embodiment of the present application can achieve Figure 1 to Figure 2 To avoid repetition, the various processes implemented by the method embodiment are not described here.

[0248] Optional, such as Figure 5As shown, the embodiment of the present application further provides an electronic device 500, which may be an image processing chip or an AP. The electronic device includes a processor 501, a memory 502, and a program or instruction stored in the memory 502 and executable on the processor 501. When the program or instruction is executed by the processor 501, each process of the above-mentioned image processing method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0249] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0250] Figure 6 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of the present application.

[0251] The electronic device 600 includes but is not limited to: a radio frequency unit 601, a network module 602, an audio output unit 603, an input unit 604, a sensor 605, a display unit 606, a user input unit 607, an interface unit 608, a memory 609, and a processor 610.

[0252] Those skilled in the art will appreciate that the electronic device 600 may also include a power source (such as a battery) for supplying power to each component, and the power source may be logically connected to the processor 610 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Figure 6 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0253] Wherein, in the case where the electronic device is an image processing chip:

[0254] The interface unit 608 is used to: receive sensor mode information sent by the application program AP, where the sensor mode information is information about the sensor mode entered when the camera is turned on;

[0255] According to the sensor mode information, an artificial intelligence AI model is loaded; wherein the camera includes N sensor modes, each of the sensor modes corresponds to M photographing modes, and each photographing mode has a corresponding AI model; the N sensor modes have priorities, and the M photographing modes have priorities;

[0256] The processor 610 is used to: obtain an image frame according to the photo-taking instruction and photo-taking mode information sent by the AP; and process the image frame using the AI ​​model corresponding to the photo-taking mode information.

[0257] Optionally, the interface unit is specifically used for:

[0258] According to the sensor mode information, in the order of priority of N sensor modes from high to low and the order of priority of M camera modes from high to low, loading the AI ​​model corresponding to the camera mode, M and N are greater than or equal to 2, and the N sensor modes include the target sensor mode indicated by the sensor mode information;

[0259] or,

[0260] According to the sensor mode information, load the AI ​​model corresponding to the photographing mode under the target sensor mode indicated by the sensor mode information.

[0261] Optionally, the processor is further configured to:

[0262] Determining the priority of the sensor mode according to the sensor mode information;

[0263] The priority of the sensor mode is determined according to the sensor mode entered when the camera is turned on and the zoom ratio corresponding to the sensor mode; the target sensor mode indicated by the sensor mode information has the highest priority;

[0264] The sensor mode includes at least one of the following: an image readout mode for the first camera, a region of interest ROI mode for the first camera, an image readout mode for the second camera, and a ROI mode for the second camera.

[0265] Optionally, the interface unit is specifically used for:

[0266] According to the order of priority of the N sensor modes from high to low and the order of priority of the M camera modes from high to low, the AI ​​models corresponding to at least two camera modes corresponding to each sensor mode are loaded in sequence;

[0267] The photographing modes corresponding to each sensor mode include at least two of the following:

[0268] Night mode;

[0269] High dynamic range imaging HDR mode;

[0270] Sports mode;

[0271] Polar night mode.

[0272] Optionally, the processor is further configured to: acquire an image frame of the original image according to the photographing instruction and the photographing mode information;

[0273] The photographing mode information includes: a photographing mode selected by a user and image frame acquisition requirements corresponding to the photographing mode; different photographing modes correspond to different image frame acquisition requirements.

[0274] Optionally, the interface unit is further used for:

[0275] When receiving the photo-taking instruction and the photo-taking mode information sent by the AP, if the AI ​​model corresponding to the photo-taking mode indicated by the photo-taking mode information is not loaded in the image processing chip, the AI ​​model corresponding to the photo-taking mode indicated by the photo-taking mode information is loaded.

[0276] Optionally, the processor is specifically configured to:

[0277] When the AI ​​model corresponding to the photographing mode indicated by the photographing mode information is loaded, the image frame is processed using the AI ​​model in the network processor NPU of the image processing chip.

[0278] Wherein, in the case where the electronic device is an AP:

[0279] The processor 610 is used to: obtain sensor mode information entered when the camera is turned on;

[0280] The interface unit 608 is used to: send the sensor mode information to the image processing chip;

[0281] According to the sensor mode information, the AI ​​model is transmitted to the image processing chip, wherein the camera includes N sensor modes, each of the sensor modes corresponds to M photo modes, and each photo mode has a corresponding AI model; the N sensor modes have priorities, and the M photo modes have priorities;

[0282] When receiving a photographing input from the user, a photographing instruction and photographing mode information are sent to the image processing chip.

[0283] Optionally, the interface unit is specifically used for:

[0284] According to the sensor mode information, in the order of priority of N sensor modes from high to low and the order of priority of M camera modes from high to low, transmitting the AI ​​model corresponding to the camera mode, M and N are greater than or equal to 2, and the N sensor modes include the target sensor mode indicated by the sensor mode information;

[0285] or,

[0286] According to the sensor mode information, an AI model corresponding to the photographing mode under the target sensor mode indicated by the sensor mode information is transmitted.

[0287] Optionally, the processor is further configured to:

[0288] Determining the priority of the sensor mode according to the sensor mode information;

[0289] The priority of the sensor mode is determined according to the sensor mode entered when the camera is turned on and the zoom ratio corresponding to the sensor mode; the target sensor mode indicated by the sensor mode information has the highest priority;

[0290] The sensor mode includes at least one of the following: an image readout mode for the first camera, a ROI mode for the first camera, an image readout mode for the second camera, and a ROI mode for the second camera.

[0291] Optionally, the interface unit is specifically used for:

[0292] Transmitting the AI ​​models corresponding to at least two photographing modes in each sensor mode in sequence according to the order of priority of the N sensor modes from high to low and the order of priority of the M photographing modes from high to low;

[0293] The photographing modes corresponding to each sensor mode include at least two of the following:

[0294] Night mode;

[0295] HDR mode;

[0296] Sports mode;

[0297] Polar night mode.

[0298] It should be understood that in the embodiment of the present application, the input unit 604 may include a graphics processor (GPU) 6041 and a microphone 6042, and the graphics processor 6041 processes the image data of a static picture or video obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The display unit 606 may include a display panel 6061, and the display panel 6061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 607 includes a touch panel 6071 and at least one of other input devices 6072. The touch panel 6071 is also called a touch screen. The touch panel 6071 may include two parts: a touch detection device and a touch controller. Other input devices 6072 may include, but are not limited to, a physical keyboard, function keys (such as a volume control button, a switch button, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0299] The memory 609 can be used to store software programs and various data. The memory 609 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 609 may include a volatile memory or a non-volatile memory, or the memory 609 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 609 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0300] The processor 610 may include one or more processing units; optionally, the processor 610 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 610.

[0301] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the above-mentioned image processing method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0302] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0303] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned image processing method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0304] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0305] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned image processing method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0306] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0307] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

Claims

1. An image processing method, characterized in that: Applied to image processing chips, including: The image processing chip receives the sensor mode information sent by the application processor AP, where the sensor mode information is information about the sensor mode entered when the camera is turned on; According to the sensor mode information, an artificial intelligence AI model is loaded; wherein the camera includes N sensor modes, each of the sensor modes corresponds to M photographing modes, and each photographing mode has a corresponding AI model; the N sensor modes have priorities, and the M photographing modes have priorities; Acquire an image frame according to the photographing instruction and photographing mode information sent by the AP; The image frame is processed using the AI ​​model corresponding to the photographing mode information.

2. The method according to claim 1, characterized in that The step of loading the AI ​​model according to the sensor mode information includes: According to the sensor mode information, in the order of priority of N sensor modes from high to low and the order of priority of M camera modes from high to low, loading the AI ​​model corresponding to the camera mode, M and N are greater than or equal to 2, and the N sensor modes include the target sensor mode indicated by the sensor mode information; or, According to the sensor mode information, load the AI ​​model corresponding to the photographing mode under the target sensor mode indicated by the sensor mode information.

3. The method according to claim 2, characterized in that The method further comprises: Determining the priority of the sensor mode according to the sensor mode information; The priority of the sensor mode is determined according to the sensor mode entered when the camera is turned on and the zoom ratio corresponding to the sensor mode; the target sensor mode indicated by the sensor mode information has the highest priority; The sensor mode includes at least one of the following: an image readout mode for the first camera, a region of interest ROI mode for the first camera, an image readout mode for the second camera, and a ROI mode for the second camera.

4. The method according to claim 2 or 3, characterized in that: According to the sensor mode information, in the order of priority of the N sensor modes from high to low and the order of priority of the M camera modes from high to low, the AI ​​model corresponding to the camera mode is loaded, including: According to the order of priority of the N sensor modes from high to low and the order of priority of the M camera modes from high to low, the AI ​​models corresponding to at least two camera modes corresponding to each sensor mode are loaded in sequence; The photographing modes corresponding to each sensor mode include at least two of the following: Night mode; High dynamic range imaging HDR mode; Sports mode; Polar night mode.

5. The method according to claim 1, characterized in that Acquiring an image frame according to the photographing instruction and photographing mode information sent by the AP, including: Acquire an image frame of the original image according to the photographing instruction and the photographing mode information; The photographing mode information includes: a photographing mode selected by a user and image frame acquisition requirements corresponding to the photographing mode; different photographing modes correspond to different image frame acquisition requirements.

6. The method according to claim 5, characterized in that The method further comprises: When receiving the photo-taking instruction and the photo-taking mode information sent by the AP, if the AI ​​model corresponding to the photo-taking mode indicated by the photo-taking mode information is not loaded in the image processing chip, the AI ​​model corresponding to the photo-taking mode indicated by the photo-taking mode information is loaded.

7. The method according to claim 1, 5 or 6, characterized in that: Processing the image frame using the AI ​​model corresponding to the photographing mode information includes: When the AI ​​model corresponding to the photographing mode indicated by the photographing mode information is loaded, the image frame is processed using the AI ​​model in the network processor NPU of the image processing chip.

8. An image processing method, characterized in that: Applied to APs, including: The AP obtains the sensor mode information entered when the camera is turned on; Sending the sensor mode information to an image processing chip; According to the sensor mode information, the AI ​​model is transmitted to the image processing chip, wherein the camera includes N sensor modes, each of the sensor modes corresponds to M photographing modes, and each photographing mode has a corresponding AI model; the N sensor modes have priorities, and the M photographing modes have priorities; When receiving a photographing input from the user, a photographing instruction and photographing mode information are sent to the image processing chip.

9. The method according to claim 8, characterized in that The transmitting the AI ​​model to the image processing chip according to the sensor mode information includes: According to the sensor mode information, in the order of priority of N sensor modes from high to low and the order of priority of M camera modes from high to low, transmitting the AI ​​model corresponding to the camera mode, M and N are greater than or equal to 2, and the N sensor modes include the target sensor mode indicated by the sensor mode information; or, According to the sensor mode information, an AI model corresponding to the photographing mode under the target sensor mode indicated by the sensor mode information is transmitted.

10. The method according to claim 9, characterized in that The method further comprises: Determining the priority of the sensor mode according to the sensor mode information; The priority of the sensor mode is determined according to the sensor mode entered when the camera is turned on and the zoom ratio corresponding to the sensor mode; the target sensor mode indicated by the sensor mode information has the highest priority; The sensor mode includes at least one of the following: an image readout mode for the first camera, a ROI mode for the first camera, an image readout mode for the second camera, and a ROI mode for the second camera.

11. The method according to claim 9 or 10, characterized in that: According to the sensor mode information, in the order of priority of the N sensor modes from high to low and the order of priority of the M camera modes from high to low, the AI ​​model corresponding to the camera mode is transmitted, including: Transmitting the AI ​​models corresponding to at least two photographing modes in each sensor mode in sequence according to the order of priority of the N sensor modes from high to low and the order of priority of the M photographing modes from high to low; The photographing modes corresponding to each sensor mode include at least two of the following: Night mode; HDR mode; Sports mode; Polar night mode.

12. An image processing device, characterized in that: Applied to image processing chips, including: A first receiving module, used to receive sensor mode information sent by the AP, where the sensor mode information is information about the sensor mode entered when the camera is turned on; A first processing module is used to load an AI model according to the sensor mode information; wherein the camera includes N sensor modes, each of the sensor modes corresponds to M photographing modes, and each photographing mode has a corresponding AI model; the N sensor modes have priorities, and the M photographing modes have priorities; A first acquisition module, used to acquire an image frame according to the photographing instruction and photographing mode information sent by the AP; The second processing module is used to process the image frame using the AI ​​model corresponding to the shooting mode information.

13. The device according to claim 12, characterized in that The first processing module is specifically used for: According to the sensor mode information, in the order of priority of N sensor modes from high to low and the order of priority of M camera modes from high to low, loading the AI ​​model corresponding to the camera mode, M and N are greater than or equal to 2, and the N sensor modes include the target sensor mode indicated by the sensor mode information; or, According to the sensor mode information, load the AI ​​model corresponding to the photographing mode under the target sensor mode indicated by the sensor mode information.

14. The device according to claim 13, characterized in that The device also includes: A first determining module, configured to determine the priority of the sensor mode according to the sensor mode information; The priority of the sensor mode is determined according to the sensor mode entered when the camera is turned on and the zoom ratio corresponding to the sensor mode; the target sensor mode indicated by the sensor mode information has the highest priority; The sensor mode includes at least one of the following: an image readout mode for the first camera, a region of interest ROI mode for the first camera, an image readout mode for the second camera, and a ROI mode for the second camera.

15. The device according to claim 13 or 14, characterized in that The first processing module is specifically used for: According to the order of priority of the N sensor modes from high to low and the order of priority of the M camera modes from high to low, the AI ​​models corresponding to at least two camera modes corresponding to each sensor mode are loaded in sequence; The photographing modes corresponding to each sensor mode include at least two of the following: Night mode; HDR mode; Sports mode; Polar night mode.

16. The device according to claim 12, characterized in that The first acquisition module is specifically used for: Acquire an image frame of the original image according to the photographing instruction and the photographing mode information; The photographing mode information includes: a photographing mode selected by a user and image frame acquisition requirements corresponding to the photographing mode; different photographing modes correspond to different image frame acquisition requirements.

17. An image processing device, characterized in that: Applied to APs, including: The second acquisition module is used to obtain the sensor mode information entered when the camera is turned on; A first sending module, used for sending the sensor mode information to the image processing chip; A second sending module is used to transmit the AI ​​model to the image processing chip according to the sensor mode information, wherein the camera includes N sensor modes, each of the sensor modes corresponds to M photographing modes, and each photographing mode has a corresponding AI model; the N sensor modes have priorities, and the M photographing modes have priorities; The third sending module is used to send a photo-taking instruction and photo-taking mode information to the image processing chip when receiving a photo-taking input from the user.

18. The device according to claim 17, characterized in that The second sending module is specifically used for: According to the sensor mode information, in the order of priority of N sensor modes from high to low and the order of priority of M camera modes from high to low, transmitting the AI ​​model corresponding to the camera mode, M and N are greater than or equal to 2, and the N sensor modes include the target sensor mode indicated by the sensor mode information; or, According to the sensor mode information, an AI model corresponding to the photographing mode under the target sensor mode indicated by the sensor mode information is transmitted.

19. The device according to claim 18, characterized in that The device also includes: A second determining module, configured to determine the priority of the sensor mode according to the sensor mode information; The priority of the sensor mode is determined according to the sensor mode entered when the camera is turned on and the zoom ratio corresponding to the sensor mode; the target sensor mode indicated by the sensor mode information has the highest priority; The sensor mode includes at least one of the following: an image readout mode for the first camera, a ROI mode for the first camera, an image readout mode for the second camera, and a ROI mode for the second camera.

20. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the image processing method as described in any one of claims 1 to 7 are implemented, or the steps of the image processing method as described in any one of claims 8 to 11 are implemented.

Images