Shooting method and device, electronic equipment and medium

Abstract

The invention discloses a shooting method and device, electronic equipment and a medium, and belongs to the technical field of communication. The electronic equipment comprises an image sensor, a front image signal processor ISP and an application processor, the image sensor is connected with the front ISP, and the front ISP is connected with the application processor; the method comprises the steps of collecting original image data through the image sensor under the condition that a shooting instruction is received; performing first image processing on the original image data by adopting an AI image processing algorithm through the front ISP to obtain first image data; and performing second image processing on the first image data through the application processor, and outputting target image data.

Description

Technical Field

[0001] The present application belongs to the field of communication technology, and specifically relates to a shooting method, device, electronic equipment and medium. Background Art

[0002] With the popularity of electronic devices such as mobile phones and the convenience of electronic devices in shooting, users are increasingly relying on electronic devices for daily shooting, and the demand for shooting functions is also increasing. The shooting effect and shooting experience of electronic devices are closely related to the processing power of their image systems. Therefore, how to improve the processing power of image systems has become particularly important.

[0003] At present, electronic devices are mainly developed based on application processors (AP), and image systems need to rely on artificial intelligence (AI) image processing algorithms to improve the image processing effect after shooting. However, due to the limitations of the AP platform's own computing power and system power consumption, some AI image processing algorithms with high computing power requirements cannot be deployed, which ultimately affects the overall image processing effect. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a shooting method, device, electronic device and medium, which can solve the problem of poor overall processing effect of the existing shooting processing solution based on the application processor AP.

[0005] In a first aspect, an embodiment of the present application provides a shooting method, which is performed by an electronic device, wherein the electronic device includes an image sensor, a front image signal processor ISP, and an application processor, wherein the image sensor is connected to the front ISP, and the front ISP is connected to the application processor; the method includes:

[0006] When receiving a shooting instruction, collecting raw image data through the image sensor;

[0007] Performing first image processing on the original image data by using an artificial intelligence (AI) image processing algorithm through the front-end ISP to obtain first image data;

[0008] The application processor performs second image processing on the first image data and outputs target image data.

[0009] In a second aspect, an embodiment of the present application provides an electronic device, comprising: N image sensors, a selection module, a pre-image signal processor ISP and an application processor, wherein the image sensor is connected to the pre-ISP, and the pre-ISP is connected to the application processor;

[0010] The front ISP is provided with an ASIC chip and a MIPI switching bus network; the N first input terminals of the selection module are connected to the N image sensors in a one-to-one correspondence, the second terminal of the selection module is connected to the MIPI switching bus network, and the MIPI switching bus network is connected to the application processor; the MIPI switching bus network is also connected to the ASIC chip for communication;

[0011] The MIPI switching bus network is used to determine whether to transmit the original image data to the ASIC chip for processing according to the shooting scene and shooting parameters.

[0012] In a third aspect, an embodiment of the present application provides a shooting device, which is arranged in an electronic device, and includes an image sensor, a front image signal processor ISP and an application processor, wherein the image sensor is connected to the front ISP, and the front ISP is also connected to the application processor; wherein:

[0013] The image sensor is used to collect raw image data when receiving a shooting instruction;

[0014] The front ISP is used to perform first image processing on the original image data using an AI image processing algorithm to obtain first image data;

[0015] The application processor is used to perform second image processing on the first image data and output target image data.

[0016] In a fourth aspect, an embodiment of the present application provides an electronic device, comprising a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the programs or instructions are executed by the processor, the steps of the shooting method described in the first aspect are implemented.

[0017] In a fifth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the shooting method described in the first aspect are implemented.

[0018] In a sixth aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the shooting method as described in the first aspect.

[0019] In a seventh aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the shooting method as described in the first aspect.

[0020] In an embodiment of the present application, the electronic device includes an image sensor, a pre-image signal processor ISP and an application processor, wherein the image sensor is connected to the pre-ISP, and the pre-ISP is also connected to the application processor; upon receiving a shooting instruction, the image sensor collects raw image data; the pre-ISP uses an AI image processing algorithm to perform a first image processing on the raw image data to obtain first image data; the application processor performs a second image processing on the first image data to output target image data. In this way, by using an external pre-ISP to deploy some AI image processing algorithms, the pre-ISP can be used to accelerate the preliminary processing of the captured image data, and the application processor can further process the image data after the preliminary processing by the ISP, which can effectively improve the image shooting effect and image processing performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 is a schematic diagram of the structure of a multimedia processing system of an electronic device provided in an embodiment of the present application;

[0022] Figure 2 It is one of the flow charts of the shooting method provided in the embodiment of the present application;

[0023] Figure 3 This is the second flowchart of the shooting method provided in the embodiment of the present application;

[0024] Figure 4 It is a framework structure diagram of a high-performance dedicated multimedia chip of an electronic device provided in an embodiment of the present application;

[0025] Figure 5 It is a framework structure diagram of a multimedia chip with a multi-layer MIPI switching bus network arranged in an electronic device provided in an embodiment of the present application;

[0026] Figure 6 is a structural diagram of an electronic device provided in an embodiment of the present application;

[0027] Figure 7 It is a hardware structure diagram of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0028] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0029] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way are interchangeable where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0030] In order to make the embodiments of the present application clearer, the relevant technical knowledge involved in the embodiments of the present application is further described in detail below, and the purpose and improvement points of the embodiments of the present application are briefly introduced as follows:

[0031] AP: Application Processor, application processor;

[0032] ISP: Image Signal Processor, image signal processing unit;

[0033] PRE_ISP: Pre-ISP processor, which processes the image data output by the camera, i.e. the image sensor, before the AP multimedia processing unit;

[0034] ORB: Oriented FAST and Rotated BRIEF, a commonly used image feature detection algorithm. The ORB algorithm is divided into two parts: feature point extraction and feature point description;

[0035] NPU: Neural Processing Unit, neural processing unit.

[0036] At present, multimedia systems, especially functions such as video / image shooting, are very important for electronic devices such as smartphones, tablets, and car displays. The quality of the camera function, that is, the effect and experience of taking photos, is closely related to the multimedia system capabilities of electronic devices. In particular, the computing power requirements of current AI algorithms are increasing. Therefore, improving the processing power of multimedia systems is currently the key.

[0037] Current electronic devices are developed based on application processors (APs). The rapid development of AI processing algorithms has greatly improved the photo / video processing effects of mobile phones. In particular, in terms of image detail processing, AI algorithms have more intelligent processing results than traditional computer vision (CV) algorithms, and their processing results are more in line with people's processing habits and requirements. However, AI processing algorithms place higher requirements on the computing power and performance of mobile phone processors. Currently, due to the limitations of the computing power and system power consumption of the AP platform itself, some image processing AI algorithms with higher computing power requirements cannot be deployed, resulting in a decrease in the overall image processing effect and a low image formation rate. In addition, due to the limitations of the AP platform's multiple processing units sharing chip memory and bus resources, in user overload scenarios, the AI ​​algorithm runs for a long time on the AP platform, resulting in a large overall delay in photo taking, affecting the user experience.

[0038] In view of the technical problems existing in the above-mentioned prior art, the embodiment of the present application proposes a system solution in which a high-performance front-end dedicated ISP chip is externally mounted on the application processor AP side, such as Figure 1 As shown, in terms of system architecture, the solution adopts a front-end PRE architecture, that is, the external multimedia chip, namely the front-end PRE_ISP chip, is on the front side of the AP, and between the image sensor (Sensor) and the AP, and processes the image data in the original RAW domain. Specifically, the photo AI processing algorithm, image preprocessing algorithm (such as image frame selection and ORB fast registration preprocessing algorithm), etc. can be superimposed on the front ISP chip to improve the shooting effect; in terms of system interface, the embodiment of the present application can adopt a flexible multi-layer mobile industry processor interface (Mobile Industry Processor Interface, MIPI) switching bus network to realize the working mode switching and timing control of the external AI acceleration chip in different scenarios; the embodiment of the present application also implements the photo processing enhancement algorithm in the multimedia chip using a dedicated ASIC circuit to reduce processing delays and improve system performance.

[0039] The following is a detailed description of the shooting method and the multimedia chip architecture of the electronic device provided in the embodiments of the present application through specific embodiments and their application scenarios in conjunction with the accompanying drawings.

[0040] See also Figure 1 and Figure 2 , Figure 1 A schematic diagram of the structure of a multimedia processing system of an electronic device provided in an embodiment of the present application, Figure 2 This is a flowchart of a shooting method provided in an embodiment of the present application, and the method is executed by an electronic device, such as Figure 1As shown, the electronic device includes an image sensor 101, a pre-image signal processor ISP102 and an application processor 103, wherein the image sensor 101 is connected to the pre-ISP102, and the pre-ISP102 is also connected to the application processor 103; Figure 2 As shown, the method comprises the following steps:

[0041] Step 201 : upon receiving a shooting instruction, collecting raw image data through the image sensor 101 .

[0042] In the embodiment of the present application, the above-mentioned image sensor 101 can be understood as a camera enabled during shooting; the above-mentioned front ISP 102 is an external image processing chip, and the above-mentioned application processor 103 is an AP chip.

[0043] Specifically, when receiving a shooting instruction, the camera, i.e., the image sensor 101, enters a working state and collects image data. Specifically, multiple frames of image data may be collected, so as to be further processed by the front ISP 102 for frame selection, registration, and even image synthesis, and finally form image data with enhanced shooting effect. Among them, the shooting instruction may be a photo taking instruction or a video recording instruction. It should be noted that the image data collected by the image sensor 101 is the original image data, i.e., RAW domain data that has not undergone any image processing.

[0044] In addition, the raw image data collected by the image sensor 101 can be transmitted to the front-end ISP 102 via a relevant data interface such as a MIPI interface.

[0045] Step 202: The front-end ISP 102 uses an artificial intelligence (AI) image processing algorithm to perform a first image processing on the original image data to obtain first image data.

[0046] In an embodiment of the present application, in order to accelerate the real-time processing of captured image data, a portion of the AI ​​image processing algorithm can be deployed on the front-end ISP 102 to reduce the burden on the application processor 103 while improving the image processing capability of the captured image data and improving processing efficiency.

[0047] Specifically, in this step, after receiving the raw image data transmitted by the image sensor 101, the pre-ISP 102 can use the AI ​​image processing algorithm deployed thereon to perform a first image processing on the raw image data, that is, preliminary image data, to obtain the first image data after processing, wherein the first image processing may be RAW domain noise reduction, dynamic range enhancement such as HDR processing, image synthesis, etc.

[0048] Step 203: Perform second image processing on the first image data through the application processor 103 to output target image data.

[0049] Specifically, other image processing algorithms can be superimposed on the AP platform, i.e., the application processor 103, such as image synthesis, image quality enhancement, filters and other AI image processing algorithms. In specific applications, the corresponding AI image processing algorithms can be used in combination with the actual shooting scene to process the captured image data.

[0050] In this step, after receiving the preliminary processed image data output by the front ISP 102, the application processor 103 can apply the corresponding AI image processing algorithm to perform further image processing on the processed image data, such as image quality enhancement, image synthesis, special filters, etc.

[0051] After the application processor 103 performs the second image processing, the target image data that has been finally processed is outputted, and the target image data can be transmitted to the display screen 104 for display, so that the user can preview the shooting effect after the image processing.

[0052] In some embodiments, before step 202, the method further includes:

[0053] The pre-ISP 102 performs frame selection processing on the original image data by adopting a preset frame selection strategy, and performs registration processing on the selected image frames by using an image feature detection algorithm;

[0054] The step 202 includes:

[0055] Performing a first image processing on the registered image data by the front ISP 102 to obtain the first image data, wherein the first image processing includes at least one of noise reduction, dynamic range enhancement, demosaicing and image synthesis;

[0056] The step 203 includes:

[0057] The application processor 103 performs second image processing on the first image data and outputs target image data.

[0058] That is, in some embodiments, pre-processing algorithms such as frame selection and registration, as well as some AI image processing algorithms other than the pre-processing algorithms such as frame selection and registration, such as one or more algorithms of AI noise reduction, high dynamic range imaging (HDR), AI de-mosaicing, and AI image synthesis, can be superimposed on the front ISP 102. Accordingly, other image processing algorithms, such as AI image processing algorithms such as image quality enhancement, blurring, and brightening, can be superimposed on the AP platform, i.e., the application processor 103.

[0059] Therefore, the pre-ISP 102 can also perform the first image processing on the registered image data, that is, the first step of image enhancement processing, such as noise reduction, HDR, de-mosaicing and image synthesis, to obtain the first image data after processing. The first image data is then transmitted to the application processor 103 for the second image processing, that is, the second step of image enhancement processing, such as image synthesis, image quality enhancement, blurring, brightening, etc., to obtain the final target image data after processing.

[0060] Through this implementation, the overall computing power and processing performance of the system can be improved by superimposing algorithms such as image preprocessing and image AI enhancement processing on the front ISP102, thereby improving the image processing effect and shooting efficiency during shooting.

[0061] Alternatively, if Figure 4 As shown, the front ISP 102 includes an application specific integrated circuit (ASIC) chip 110; the ASIC chip 110 includes a shooting preprocessing unit 111 and a shooting acceleration processing unit 112, the image sensor 101 is connected to the shooting preprocessing unit 111, the shooting preprocessing unit 111 is connected to the shooting acceleration processing unit 112, and the shooting acceleration processing unit 112 is connected to the application processor 103;

[0062] The performing first image processing on the original image data by using the AI ​​image processing algorithm through the front ISP 102 to obtain the first image data includes:

[0063] The shooting preprocessing unit 111 adopts the preset frame selection strategy to perform frame selection processing on the original image data, and uses the image feature detection algorithm to perform registration processing on the selected image frames;

[0064] The image data after registration is subjected to image enhancement processing by the shooting acceleration processing unit 112 to obtain the first image data, wherein the image enhancement processing includes at least one of noise reduction, dynamic range enhancement, demosaicing and image synthesis.

[0065] In some embodiments, Figure 4As shown, the front ISP 102 can be implemented by a dedicated ASIC chip 110, and the ASIC chip 110 is integrated with a shooting preprocessing unit 111 and a shooting acceleration processing unit 112, wherein the shooting preprocessing unit 111 is implemented by a dedicated ASIC acceleration hardware circuit, and the shooting acceleration processing unit 112 is implemented by a dedicated NPU acceleration unit, that is, both are implemented by dedicated hardware circuits, which can accelerate the processing of captured images. Among them, the shooting preprocessing unit 111 can also be called a photo acceleration preprocessing unit, and the shooting acceleration processing unit 112 can also be called a dedicated AI accelerator.

[0066] Specifically, the frame selection processing of the original image data according to the preset frame selection strategy can be implemented in the shooting preprocessing unit 111, and the registration processing of the selected image frame can be implemented using image feature detection algorithms such as ORB. In the shooting acceleration processing unit 112, image enhancement processing such as AI noise reduction, AI dynamic range enhancement, AI de-mosaicing, AI image synthesis, etc. can be implemented on the registered image data, and the processed data is output to the application processor 103 for a second image enhancement processing such as image quality enhancement processing to obtain the final target image data.

[0067] Among them, the preset frame selection strategy provides a strategy for how to select a reference frame and how to capture a frame, so that the original image data can be subjected to frame selection processing according to the preset frame selection strategy, so as to select several image frames from multiple original image frames captured by the camera for subsequent processing, such as selecting the 1st, 3rd, and 5th frames according to the preset frame selection strategy.

[0068] The above-mentioned use of ORB and other algorithms to achieve registration processing of the selected image frames can be to use image feature detection algorithms such as the ORB algorithm to perform image registration processing on the selected image frames. Specifically, the selected image frames can be aligned in spatial position so that the same features in these images are in the same position as much as possible.

[0069] It should be noted that the above-mentioned frame selection and registration processing can be collectively referred to as preprocessing. In specific implementation, preprocessing algorithms such as frame selection and registration can be superimposed on the shooting preprocessing unit 111 of the front ISP 102, so that the original image data collected by the image sensor 101 can be preprocessed by the shooting preprocessing unit 111, such as frame selection and registration, thereby reducing the image processing pressure on the application processor 103, i.e., the AP platform.

[0070] In addition, the image data pre-processed by the front ISP 102 can be transmitted to the application processor 103 through a relevant data interface such as a MIPI interface for further image enhancement processing.

[0071] It should be noted that, in some embodiments, Figure 4As shown, the front ISP 102 also includes a storage unit 113, which may be a high-performance on-chip dedicated dynamic random access memory (DRAM) storage unit, or a high-performance double data rate synchronous dynamic random access memory (DDR) storage unit;

[0072] The shooting pre-processing unit 111 is connected to the storage unit 113 , and the storage unit 113 is also connected to the shooting acceleration processing unit 112 .

[0073] Specifically, the image data processed by the shooting pre-processing unit 111 can be written into the storage unit 113 for buffering, and the shooting acceleration processing unit 112 accesses the storage unit 113 when needed and reads the image data from the storage unit 113 for further processing.

[0074] In this way, a storage and computing integrated architecture based on the front-end ISP chip can be realized. By integrating a high-performance on-chip dedicated storage unit on the dedicated ASIC chip 110, the delay of the AI ​​accelerator in reading and writing memory access can be reduced, thereby improving AI processing performance.

[0075] Optionally, the adopting a preset frame selection strategy to perform frame selection processing on the original image data includes:

[0076] Determining a reference frame in the original image data;

[0077] Taking the reference frame as a reference, M image frames are captured forward and / or backward from the original image data, wherein the selected image frames include the reference frame and the M image frames, and M is an integer greater than 1.

[0078] Specifically, the preset frame selection strategy may include a set number of captured frames, a preset reference frame selection method, and a frame capture strategy for capturing image frames forward and / or backward based on the reference frame. For example, the preset frame selection strategy stipulates that the first frame is the reference frame, a total of 3 frames are captured, and 3 frames are captured forward from the reference frame, so that the reference frame in the original image data can be determined according to the preset frame selection strategy, and several frames can be captured forward / backward from the reference frame according to the frame capture strategy. It should be noted that when forward frame capture is adopted, the front ISP chip can continuously capture image data output by the camera and store historically collected image data. When selecting frames, several frames can be captured from the historically collected image data based on the current reference frame.

[0079] It should be noted that the above-mentioned frame grabbing strategy of grabbing image frames forward and / or backward based on the reference frame can specifically be, in some embodiments, a frame grabbing strategy of grabbing image frames forward based on the reference frame, such as grabbing 3 frames forward from the reference frame; in other embodiments, a frame grabbing strategy of grabbing image frames backward based on the reference frame can also be adopted, such as grabbing 3 frames backward from the reference frame; in some other embodiments, a frame grabbing strategy of grabbing image frames forward and backward based on the reference frame can also be adopted, that is, these two frame grabbing strategies can be combined, such as grabbing 3 frames forward and backward respectively from the reference frame.

[0080] Through this implementation, a preset frame selection strategy can be used to implement frame selection processing, thereby ensuring the effect of image enhancement processing based on the selected frames.

[0081] Combine the following Figure 3 The embodiment shown here is used to illustrate the first embodiment of the present application:

[0082] This implementation proposes a system solution in which a high-performance front-end ISP multimedia camera AI acceleration chip is mounted on the AP. In terms of system architecture, the solution adopts a pre-processing architecture, that is, the external multimedia chip is placed before the AP, and performs AI processing on the image data in the RAW domain, such as Figure 3 As shown, the specific steps include:

[0083] 301. The camera collects image / video data according to the configured frame rate and outputs it to the PRE_ISP external chip through the high-performance MIPI interface;

[0084] 302. After receiving the image RAW data, the dedicated front-end ISP multimedia chip performs frame selection processing according to a preset frame selection strategy (including the number of captured frames and forward / backward based on the reference frame);

[0085] 303. Perform image registration processing on the selected data frame using the ORB algorithm;

[0086] 304. Output the processing results to a dedicated AI acceleration hardware unit for processing, such as RAW domain noise reduction, HDR processing, image synthesis, etc.;

[0087] 305. After the dedicated front-end ISP multimedia chip completes the post-processing of the image, the image data and feature image are transmitted back to the AP through different MIPIs;

[0088] 306. After receiving the image data output by the dedicated front-end ISP multimedia chip, the AP platform superimposes a corresponding image processing algorithm, such as an image quality enhancement algorithm, and writes the image data into the camera storage unit.

[0089] This implementation method effectively improves the image photography effect and performance of electronic devices, especially mobile phones, by plugging in a high-performance dedicated camera AI acceleration chip, namely the front-end ISP chip, and deploying a high-performance AI camera acceleration algorithm.

[0090] Optionally, before the shooting pre-processing unit 111 performs frame selection processing on the original image data using a preset frame selection strategy, the method further includes:

[0091] Performing pixel correction processing on the original image data through the shooting preprocessing unit 111, wherein the pixel correction includes at least one of bad pixel correction and black level correction;

[0092] The shooting preprocessing unit 111 adopts a preset frame selection strategy to perform frame selection processing on the original image data, including:

[0093] The shooting preprocessing unit 111 adopts the preset frame selection strategy to perform frame selection processing on the corrected original image data.

[0094] In some embodiments, the camera pre-processing unit 111 can also be used to first perform pixel correction processing on the original image data collected by the camera, such as completing bad pixel correction, black level correction, etc., to obtain complete original image data without bad pixels, and then perform frame selection, alignment and other processing on the corrected image data.

[0095] Specifically, Defective Pixel Correction (DPC) mainly deals with pixel defects caused by manufacturing process problems. These defective pixels are greatly different from the surrounding pixels, and may appear as white dots in a completely dark environment, or black dots in a high-brightness environment. Bad pixels are divided into static bad pixels and dynamic bad pixels. The former always remains constant, while the latter changes due to conditions such as exposure or temperature. The bad pixel correction process includes two steps: bad pixel detection and correction. Detection is usually performed by comparing the difference between the center pixel and the adjacent pixels. If the difference exceeds the preset threshold, it is judged as a bad pixel. Correction repairs these defective pixels through methods such as interpolation replacement.

[0096] Black Level Correction (BLC) deals with the problem of image dark area distortion caused by insufficient sensor accuracy or circuit thermal radiation. When the lens module is completely blocked and no light is incident, the sensor can still read non-zero voltage data. This data does not represent the actual image information, but is caused by sensor accuracy problems or electrons excited by circuit thermal radiation. Black level correction corrects these inaccurate dark area data to ensure the accuracy of the dark area of ​​the image, thereby improving image quality.

[0097] Combine the following Figure 4 The high-performance dedicated multimedia chip framework shown is used to illustrate the second embodiment of this application:

[0098] This implementation proposes a solution to implement image AI processing using hardware circuits in a multimedia dedicated chip to reduce AI algorithm processing delay and power consumption. The main process is as follows:

[0099] 401. The camera sensor transmits the image RAW domain data to the dedicated front-end ISP multimedia chip through the MIPI interface; wherein, Figure 4 The first interface and the second interface are both MIPI interfaces.

[0100] 402. A dedicated front-end ISP multimedia chip pre-processes the image through a dedicated ASIC acceleration hardware circuit, including a front-end basic ISP hardware circuit unit, to complete bad pixel correction, black level correction, etc.;

[0101] 403, dedicated front ISP multimedia chip, which performs AI noise reduction, AI de-mosaicing, AI dynamic range enhancement and other processing on images through a dedicated NPU acceleration unit;

[0102] 404. Dedicated front-end ISP multimedia chip, which reduces the latency of AI accelerator reading and writing memory access and improves AI processing performance by integrating high-performance on-chip dedicated DRAM storage units;

[0103] 405. The dedicated multimedia chip transmits the processed data back to the AP via the MIPI interface. Figure 4 The third interface and the fourth interface are both MIPI interfaces.

[0104] In this implementation, by using dedicated ASIC circuits, AI accelerators and DDR memory, the deployment of high-computing AI models in processors of electronic devices such as mobile phones becomes a reality, effectively improving the energy efficiency and latency performance of electronic devices, especially mobile phone multimedia processing.

[0105] Alternatively, if Figure 5 As shown, the front ISP 102 is provided with an ASIC chip 110 and a MIPI switching bus network 120; the number of image sensors 101 is N, and the electronic device further includes a selection module 105, the N first ends of the selection module 105 are connected to the N image sensors 101 in a one-to-one correspondence, the second end of the selection module 105 is connected to the MIPI switching bus network 120, and the MIPI switching bus network 120 is connected to the application processor 103; the MIPI switching bus network 120 is also connected to the ASIC chip 110 for communication;

[0106] Before the front ISP 102 uses the AI ​​image processing algorithm to perform the first image processing on the original image data, the method further includes:

[0107] According to the shooting scene, the control selection module 105 selects at least one of the N image sensors 101 as a target image sensor;

[0108] Collecting raw image data through the target image sensor;

[0109] In the case where it is determined to turn on the ASIC chip 110 according to the shooting scene and the shooting parameters, the original image data is transmitted to the ASIC chip 110 through the MIPI switching bus network 120;

[0110] The first image processing of the original image data by using the AI ​​image processing algorithm through the front ISP 102 includes:

[0111] The ASIC chip 110 uses an AI image processing algorithm to perform a first image processing on the original image data.

[0112] That is, in some embodiments, a flexible multi-layer MIPI switching bus network is proposed to realize the working mode switching and timing control of the external AI acceleration chip in different scenarios, so as to flexibly realize the function switching in different scenarios to achieve the optimal combination of camera overall performance, power consumption and effect. The main system block diagram is as follows: Figure 5 shown.

[0113] The electronic device may include multiple image sensors 101, that is, equipped with multiple camera Sensors, such as Sensor0, Sensor1, Sensor2 and Sensor3, which may be a front camera, a rear camera, an infrared camera, a rear ultra-wide camera, etc. Multiple sensors are controlled by the selection module 105 to be enabled or not. Specifically, the target sensor to be enabled can be flexibly selected according to the actual usage scenario requirements. For example, two of the sensors can be flexibly selected as the input of the photo AI acceleration plug-in chip, such as selecting the rear main camera and the rear ultra-wide camera.

[0114] The MIPI switching bus network 120 can support physical layer bypass and dynamic switching functions, that is, it can support direct transparent transmission of raw image data to the application processor 103, and support switching of raw image data between the ASIC chip 110 and the application processor 103. The camera system can choose whether to turn on the photo AI acceleration plug-in chip according to shooting parameters such as the shooting scene and the focal length used by the enabled camera, that is, whether to turn on the ASIC chip 110 to process the raw image data. When the option is turned on, the MIPI switching bus network 120 transmits the raw image data to the ASIC chip 110 for processing, such as performing frame selection, registration, and other processing on the raw image data through the ASIC chip 110.

[0115] The MIPI switching bus network 120 also supports data link layer bypass and dynamic switching functions. The AI ​​acceleration plug-in chip can continuously capture the image data output by the sensor and perform forward frame capture processing based on the reference frame to improve the photo-taking effect, while outputting the image data to the AP for preview.

[0116] The MIPI switching bus network 120 also supports the use of dedicated mipi channels and supports custom data frame packaging, and sends the output ghost image and meta information, that is, the intermediate processing result information, to the AP for processing, which can ensure a higher transmission efficiency.

[0117] Optionally, when the ASIC chip 110 is not turned on, the ASIC chip 110 is in a dormant state.

[0118] That is, in some embodiments, when the ASIC chip 110 is not turned on, that is, when the AI ​​acceleration plug-in chip is not turned on, the ASIC chip 110 can be put into a sleep state. Specifically, the shooting pre-processing unit 111 and the shooting acceleration processing unit 112 of the plug-in ASIC chip 110 are in a power-off state, thereby achieving the purpose of reducing system power consumption.

[0119] Optionally, the method further comprises:

[0120] When it is determined not to start the ASIC chip 110 according to the shooting scene and shooting parameters, the original image data is transparently transmitted to the application processor 103 through the MIPI switching bus network 120, and the original image data is processed by the application processor 103.

[0121] When it is determined not to turn on the ASIC chip 110 based on shooting parameters such as the shooting scene and the focal length used by the enabled camera, that is, when the external chip camera function is not turned on, the MIPI switching bus network 120 can output the image data output by the sensor directly to the application processor 103, i.e., AP, through the physical layer bypass interface, and the AP completes the image processing.

[0122] In this way, the flexible multi-layer MIPI switching bus network can realize the working mode switching and timing control of the external AI acceleration chip, and then flexibly realize the function switching in different scenarios, and achieve the comprehensive optimization of the overall performance, power consumption and effect of the camera.

[0123] Optionally, in the case of determining to turn on the ASIC chip 110 according to the shooting scene and the shooting parameters, transmitting the original image data to the ASIC chip 110 through the MIPI switching bus network 120 includes:

[0124] When the ASIC chip 110 is not currently turned on and it is determined that the ASIC chip 110 needs to be turned on according to the shooting scene and shooting parameters, the target image sensor is kept in an activated state, and the original image data is transmitted to the ASIC chip 110 through the MIPI switching bus network 120.

[0125] That is, the plug-in chip system can support seamless switching. When the next frame of image needs to turn on the AI ​​acceleration plug-in chip for taking pictures, there is no need to turn off the camera. The target image sensor can be kept in the started state, and seamless switching to the AI ​​acceleration plug-in chip for image processing can be achieved.

[0126] Combine the following Figure 5 The multimedia dedicated chip system framework with a multi-layer MIPI switching bus network is shown to illustrate the third embodiment of the present application:

[0127] This implementation proposes a flexible multi-layer MIPI switching bus network to achieve working mode switching and timing control of external AI acceleration chips in different scenarios, so as to flexibly achieve function switching in different scenarios, so as to achieve the optimal combination of overall camera performance, power consumption and effect. The main descriptions are as follows:

[0128] 1. The system hardware circuit design can flexibly select two sensors as the input of the photo AI acceleration plug-in chip according to the usage scenario requirements of the specific product, such as selecting the rear main camera and the rear ultra-wide camera;

[0129] 2. The internal MIPI switching network of the external chip supports physical layer bypass and dynamic switching functions. The camera system can choose whether to turn on the AI ​​acceleration external chip for taking photos according to the scene, focal length, etc. When the chip photo function is not turned on, the image data output by the Sensor is output to the AP through the physical layer bypass interface and the AP completes the image processing; and the system supports seamless switching, that is, when the next frame of the image needs to turn on the AI ​​acceleration external chip for taking photos, there is no need to turn off the camera and seamlessly switch to the AI ​​acceleration external chip for image processing; at the same time, when the AI ​​acceleration external chip is not turned on, the image preprocessing and AI acceleration processing units of the external chip are in the power-off state, which minimizes the system power consumption.

[0130] 3. The MIPI switching network inside the external chip supports data link layer bypass and dynamic switching functions. The AI ​​acceleration external chip can continuously capture the image data output by the sensor and perform forward frame capture processing based on the reference frame to improve the photo effect. At the same time, the image data is output to the AP for preview;

[0131] 4. The internal MIPI switching network of the external chip supports the use of dedicated mipi channels, and supports custom data frame packaging, sending the output ghost image and meta information to the AP for processing.

[0132] In this implementation, on the system interface, we use a flexible multi-layer MIPI switching bus network to achieve working mode switching and timing control of the external AI acceleration chip in different scenarios, so as to flexibly realize function switching in different scenarios to achieve the optimal combination of camera overall performance, power consumption and effect.

[0133] It should be noted that in actual applications, more hardware accelerators, DSP and other dedicated processor units can be added to the PRE architecture to improve the system processing flexibility.

[0134] The shooting method in the embodiment of the present application is executed by an electronic device, and the electronic device includes an image sensor, a pre-image signal processor ISP and an application processor, wherein the image sensor is connected to the pre-ISP, and the pre-ISP is also connected to the application processor; when receiving a shooting instruction, the image sensor collects raw image data; the pre-ISP uses an AI image processing algorithm to perform a first image processing on the raw image data to obtain first image data; the application processor performs a second image processing on the first image data to output target image data. In this way, by using an external pre-ISP to deploy some AI image processing algorithms, the pre-ISP can be used to accelerate the preliminary processing of the captured image data, and then the application processor can further perform image processing on the image data after the preliminary processing by the ISP, which can effectively improve the image shooting effect and image processing performance.

[0135] The embodiment of the present application mainly deals with photo acceleration, that is, through a high-performance NPU hardware circuit and an AI photo processing algorithm, multiple images are synthesized into one image to achieve enhanced photo effects. At the same time, in the image preprocessing module, continuous images can be captured forward or backward based on the reference frame, and the captured images can be registered using the ORB algorithm to reduce the impact of camera shake. In the system interface, we use a flexible multi-layer MIPI switching bus network to achieve working mode switching and timing control of the external AI acceleration chip in different scenarios.

[0136] See also Figure 5 , Figure 5 The framework structure diagram of an electronic device provided in an embodiment of the present application in which a multi-layer MIPI switching bus network is arranged in a multimedia chip. Figure 5 As shown, the electronic device includes: N image sensors 101, a selection module 105, a pre-image signal processor ISP102 and an application processor 103, the image sensor 101 is connected to the pre-ISP102, and the pre-ISP102 is also connected to the application processor 103;

[0137] The front ISP 102 is provided with an ASIC chip 110 and a MIPI switching bus network 120; the N first ends of the selection module 105 are connected to the N image sensors 101 in a one-to-one correspondence, the second end of the selection module 105 is connected to the MIPI switching bus network 120, and the MIPI switching bus network 120 is connected to the application processor 103; the MIPI switching bus network 120 is also connected to the ASIC chip 110 for communication;

[0138] The MIPI switching bus network 120 is used to determine whether to transmit the original image data to the ASIC chip 110 for processing according to the shooting scene and shooting parameters.

[0139] This embodiment is a structural embodiment of an electronic device corresponding to the aforementioned method embodiment. Its specific introduction can refer to the relevant description in the aforementioned embodiment. The electronic device can achieve the same technical effect as the aforementioned embodiment. To avoid repetition, it will not be repeated here.

[0140] The shooting method provided in the embodiment of the present application can be executed by a shooting device. In the embodiment of the present application, the shooting method is executed by a shooting device as an example to illustrate the shooting device provided in the embodiment of the present application.

[0141] See also Figure 1 , Figure 1The structural diagram of the shooting device provided in the embodiment of the present application is provided in an electronic device, including an image sensor 101, a front ISP 102 and an application processor 103, wherein the image sensor 101 is connected to the front ISP 102, and the front ISP 102 is also connected to the application processor 103; wherein,

[0142] The image sensor 101 is used to collect raw image data when receiving a shooting instruction;

[0143] The front ISP 102 is used to perform first image processing on the original image data by using an artificial intelligence AI image processing algorithm to obtain first image data;

[0144] The application processor 103 is used to perform image enhancement processing on the first image data and output target image data.

[0145] Alternatively, if Figure 4 As shown, the front ISP 102 includes an application-specific integrated circuit ASIC chip 110; the ASIC chip 110 includes a shooting preprocessing unit 111 and a shooting acceleration processing unit 112, the image sensor 101 is connected to the shooting preprocessing unit 111, the shooting preprocessing unit 111 is connected to the shooting acceleration processing unit 112, and the shooting acceleration processing unit 112 is connected to the application processor 103;

[0146] The shooting preprocessing unit 111 is used to perform frame selection processing on the original image data using a preset frame selection strategy, and perform registration processing on the selected image frames using an image feature detection algorithm;

[0147] The shooting acceleration processing unit 112 is used to perform image enhancement processing on the registered image data to obtain first image data, wherein the image enhancement processing includes at least one of noise reduction, dynamic range enhancement, demosaicing and image synthesis.

[0148] Optionally, the shooting pre-processing unit 111 is further used to perform pixel correction processing on the original image data, wherein the pixel correction includes at least one of bad pixel correction and black level correction.

[0149] Optionally, the front-end ISP 102 is used for:

[0150] Determining a reference frame in the original image data;

[0151] Taking the reference frame as a reference, M image frames are captured forward and / or backward from the original image data, wherein the selected image frames include the reference frame and the M image frames, and M is an integer greater than 1.

[0152] Alternatively, if Figure 5As shown, the front ISP 102 is provided with an ASIC chip 110 and a MIPI switching bus network 120; the number of image sensors 101 is N, and the shooting device further includes a selection module 105, the N first ends of the selection module 105 are connected to the N image sensors 101 in a one-to-one correspondence, the second end of the selection module 105 is connected to the MIPI switching bus network 120, and the MIPI switching bus network 120 is connected to the application processor 103; the MIPI switching bus network 120 is also connected to the ASIC chip 110 for communication;

[0153] The photographing device also includes:

[0154] The application processor 103 is used to control the selection module 105 to select at least one of the N image sensors as a target image sensor according to the shooting scene;

[0155] The target image sensor is used to collect raw image data;

[0156] The MIPI switching bus network 120 is used to transmit the original image data to the ASIC chip 110 when it is determined to turn on the ASIC chip 110 according to the shooting scene and shooting parameters;

[0157] The ASIC chip 110 is used to perform a first image processing on the original image data by using an AI image processing algorithm.

[0158] Optionally, the photographing device further includes:

[0159] The MIPI switching bus network 120 is further used to transparently transmit the original image data to the application processor 103 when it is determined not to turn on the ASIC chip 110 according to the shooting scene and shooting parameters;

[0160] The application processor 103 is used to process the original image data.

[0161] Optionally, when the ASIC chip 110 is not turned on, the ASIC chip 110 is in a dormant state.

[0162] Optionally, when the ASIC chip is not currently turned on and it is determined according to the shooting scene and shooting parameters that the ASIC chip needs to be turned on, the target image sensor remains in an activated state.

[0163] The shooting device is set in an electronic device, including an image sensor, a pre-image signal processor ISP and an application processor, wherein the image sensor is connected to the pre-ISP, and the pre-ISP is also connected to the application processor; wherein the image sensor is used to collect raw image data when receiving a shooting instruction; the pre-ISP is used to perform a first image processing on the raw image data using an AI image processing algorithm to obtain first image data; the application processor is used to perform a second image processing on the first image data to output target image data. In this way, by using an external pre-ISP to deploy some AI image processing algorithms, the pre-ISP can be used to accelerate the preliminary processing of the captured image data, and then the application processor can perform further image processing on the image data preliminarily pre-processed by the ISP, which can effectively improve the image shooting effect and image processing performance.

[0164] The shooting device in the embodiment of the present application can be an electronic device, or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or other devices other than a terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc. It can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., and the embodiment of the present application does not make specific limitations.

[0165] The shooting device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0166] The shooting device provided in the embodiment of the present application can achieve Figure 2 or Figure 3 The various processes implemented by the method embodiment can achieve the same technical effect, and to avoid repetition, they will not be described here.

[0167] Alternatively, if Figure 6 As shown, an embodiment of the present application further provides an electronic device 600, including a processor 601 and a memory 602, wherein the memory 602 stores programs or instructions that can be executed on the processor 601, and when the program or instructions are executed by the processor 601, the various steps of the above-mentioned shooting method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, they are not described again here.

[0168] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0169] Figure 7 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of the present application.

[0170] The electronic device 100 includes, but is not limited to, components such as a radio frequency unit 701, a network module 702, an audio output unit 703, an input unit 704, a sensor 705, a display unit 706, a user input unit 707, an interface unit 708, a memory 709, and a processor 710. The electronic device also includes an image sensor, a pre-image signal processor ISP, and an application processor, wherein the image sensor is connected to the pre-ISP, and the pre-ISP is connected to the application processor.

[0171] Those skilled in the art will appreciate that the electronic device 700 may also include a power source (such as a battery) for supplying power to each component, and the power source may be logically connected to the processor 710 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Figure 7 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0172] Wherein, the image sensor is used to collect raw image data when receiving a shooting instruction;

[0173] The front ISP is used to perform first image processing on the original image data using an artificial intelligence (AI) image processing algorithm to obtain first image data;

[0174] The application processor is used to perform second image processing on the first image data and output target image data.

[0175] Optionally, the front-end ISP is used to include an application-specific integrated circuit ASIC chip; the ASIC chip includes a shooting preprocessing unit and a shooting acceleration processing unit, the image sensor is connected to the shooting preprocessing unit, the shooting preprocessing unit is connected to the shooting acceleration processing unit, and the shooting acceleration processing unit is connected to the application processor;

[0176] The shooting pre-processing unit is used to perform a first image processing on the original image data by using an AI image processing algorithm to obtain first image data;

[0177] The shooting acceleration processing unit is further used to perform image enhancement processing on the registered image data to obtain the first image data, wherein the image enhancement processing includes at least one of noise reduction, dynamic range enhancement, de-mosaicing and image synthesis.

[0178] Optionally, the shooting pre-processing unit is further used to perform pixel correction processing on the original image data, wherein the pixel correction includes at least one of bad pixel correction and black level correction;

[0179] The shooting pre-processing unit is used to adopt the preset frame selection strategy to perform frame selection processing on the corrected original image data.

[0180] Optionally, the front-end ISP is further used for:

[0181] Determining a reference frame in the original image data;

[0182] Taking the reference frame as a reference, M image frames are captured forward and / or backward from the original image data, wherein the selected image frames include the reference frame and the M image frames, and M is an integer greater than 1.

[0183] Optionally, the front ISP is provided with an ASIC chip and a MIPI switching bus network; the number of the image sensors is N, the electronic device further comprises a selection module, the N first ends of the selection module are connected to the N image sensors in a one-to-one correspondence, the second end of the selection module is connected to the MIPI switching bus network, the MIPI switching bus network is connected to the application processor; the MIPI switching bus network is also connected to the ASIC chip for communication;

[0184] The processor 710 is used to control the selection module to select at least one of the N image sensors as a target image sensor according to a shooting scene;

[0185] The target image sensor is used to collect raw image data;

[0186] The MIPI switching bus network is used to transmit the original image data to the ASIC chip when it is determined to turn on the ASIC chip according to the shooting scene and shooting parameters;

[0187] The ASIC chip is used to perform a first image processing on the original image data using an AI image processing algorithm.

[0188] Optionally, the MIPI switching bus network is also used to transparently transmit the original image data to the application processor and process the original image data through the application processor when it is determined not to turn on the ASIC chip based on the shooting scene and shooting parameters.

[0189] Optionally, when the ASIC chip is not turned on, the ASIC chip is in a dormant state.

[0190] Optionally, the processor 710 is further configured to keep the target image sensor in an activated state when the ASIC chip is not currently activated and it is determined according to the shooting scene and shooting parameters that the ASIC chip needs to be activated;

[0191] The MIPI switching bus network is used to transmit the original image data to the ASIC chip.

[0192] It should be understood that in the embodiment of the present application, the input unit 704 may include a graphics processor (Graphics Processing Unit, GPU) 7041 and a microphone 7042, and the graphics processor 7041 processes the image data of the static picture or video obtained by the image capture device (such as a camera) in the video capture mode or the image capture mode. The display unit 706 may include a display panel 7061, and the display panel 7061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 707 includes a touch panel 7071 and at least one of other input devices 7072. The touch panel 7071 is also called a touch screen. The touch panel 7071 may include two parts: a touch detection device and a touch controller. Other input devices 7072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0193] The memory 709 can be used to store software programs and various data. The memory 709 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 709 may include a volatile memory or a non-volatile memory, or the memory 709 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 709 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0194] The processor 710 may include one or more processing units; optionally, the processor 710 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 710.

[0195] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the above-mentioned shooting method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0196] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0197] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned shooting method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0198] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0199] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned shooting method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0200] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0201] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0202] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A shooting method, characterized in that: The method is performed by an electronic device, the electronic device includes an image sensor, a pre-image signal processor ISP and an application processor, the image sensor is connected to the pre-ISP, and the pre-ISP is also connected to the application processor; the method includes: When receiving a shooting instruction, collecting raw image data through the image sensor; Performing first image processing on the original image data by using an artificial intelligence (AI) image processing algorithm through the front ISP to obtain first image data; The application processor performs second image processing on the first image data and outputs target image data.

2. The method according to claim 1, characterized in that The front ISP includes an application-specific integrated circuit ASIC chip; the ASIC chip includes a shooting preprocessing unit and a shooting acceleration processing unit, the image sensor is connected to the shooting preprocessing unit, the shooting preprocessing unit is connected to the shooting acceleration processing unit, and the shooting acceleration processing unit is connected to the application processor; The performing first image processing on the original image data by using the AI ​​image processing algorithm through the front ISP to obtain the first image data includes: The shooting preprocessing unit adopts a preset frame selection strategy to perform frame selection processing on the original image data, and uses an image feature detection algorithm to perform registration processing on the selected image frames; The image data after registration is subjected to image enhancement processing by the shooting acceleration processing unit to obtain the first image data, wherein the image enhancement processing includes at least one of noise reduction, dynamic range enhancement, de-mosaicing and image synthesis.

3. The method according to claim 2, characterized in that Before the shooting preprocessing unit adopts a preset frame selection strategy to perform frame selection processing on the original image data, the method further includes: Performing pixel correction processing on the original image data by the shooting preprocessing unit, wherein the pixel correction includes at least one of bad pixel correction and black level correction; The performing frame selection processing on the original image data by adopting a preset frame selection strategy through the shooting preprocessing unit includes: The shooting preprocessing unit adopts the preset frame selection strategy to perform frame selection processing on the corrected original image data.

4. The method according to claim 2, characterized in that: The adopting a preset frame selection strategy to perform frame selection processing on the original image data includes: Determining a reference frame in the original image data; Taking the reference frame as a reference, M image frames are captured forward and / or backward from the original image data, wherein the selected image frames include the reference frame and the M image frames, and M is an integer greater than 1.

5. The method according to claim 1, characterized in that The front ISP is provided with an ASIC chip and a MIPI switching bus network; the number of the image sensors is N, and the electronic device further comprises a selection module, the N first ends of the selection module are connected to the N image sensors in a one-to-one correspondence, the second end of the selection module is connected to the MIPI switching bus network, and the MIPI switching bus network is connected to the application processor; The MIPI switching bus network is also communicatively connected with the ASIC chip; Before performing the first image processing on the original image data by the front ISP using the AI ​​image processing algorithm, the method further includes: According to the shooting scene, controlling the selection module to select at least one of the N image sensors as a target image sensor; Collecting raw image data through the target image sensor; In a case where it is determined to turn on the ASIC chip according to the shooting scene and the shooting parameters, the original image data is transmitted to the ASIC chip through the MIPI switching bus network; The performing first image processing on the original image data by using the AI ​​image processing algorithm through the front ISP includes: The ASIC chip uses an AI image processing algorithm to perform a first image processing on the original image data.

6. The method according to claim 5, characterized in that The method further comprises: When it is determined not to turn on the ASIC chip according to the shooting scene and the shooting parameters, the original image data is transparently transmitted to the application processor through the MIPI switching bus network, and the original image data is processed by the application processor.

7. The method according to claim 5, characterized in that In the case where it is determined according to the shooting scene and the shooting parameters to turn on the ASIC chip, transmitting the original image data to the ASIC chip through the MIPI switching bus network includes: When the ASIC chip is not currently turned on and it is determined that the ASIC chip needs to be turned on according to the shooting scene and shooting parameters, the target image sensor is kept in an activated state, and the original image data is transmitted to the ASIC chip through the MIPI switching bus network.

8. An electronic device, characterized in that: include: N image sensors, a selection module, a pre-image signal processor ISP and an application processor, wherein the image sensor is connected to the pre-ISP, and the pre-ISP is also connected to the application processor; The front ISP is provided with an ASIC chip and a MIPI switching bus network; The N first ends of the selection module are connected to the N image sensors in a one-to-one correspondence, the second end of the selection module is connected to the MIPI switching bus network, and the MIPI switching bus network is connected to the application processor; The MIPI switching bus network is also communicatively connected with the ASIC chip; The MIPI switching bus network is used to determine whether to transmit the original image data to the ASIC chip for processing according to the shooting scene and shooting parameters.

9. A photographing device, characterized in that: The electronic device includes an image sensor, a pre-image signal processor ISP and an application processor, wherein the image sensor is connected to the pre-ISP, and the pre-ISP is also connected to the application processor; wherein: The image sensor is used to collect raw image data when receiving a shooting instruction; The front ISP is used to perform first image processing on the original image data using an AI image processing algorithm to obtain first image data; The application processor is used to perform second image processing on the first image data and output target image data.

10. The photographing device according to claim 9, characterized in that: The front ISP is provided with an ASIC chip and a MIPI switching bus network; the number of the image sensors is N, and the electronic device further comprises a selection module, the N first ends of the selection module are connected to the N image sensors in a one-to-one correspondence, the second end of the selection module is connected to the MIPI switching bus network, and the MIPI switching bus network is connected to the application processor; The MIPI switching bus network is also communicatively connected with the ASIC chip; The application processor is further configured to control the selection module to select at least one of the N image sensors as a target image sensor according to a shooting scene; The target image sensor is used to collect raw image data; The MIPI switching bus network is used to transmit the original image data to the ASIC chip when it is determined to turn on the ASIC chip according to the shooting scene and shooting parameters; The ASIC chip is used to perform a first image processing on the original image data using an AI image processing algorithm.

11. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the shooting method according to any one of claims 1 to 7 are implemented.

12. A readable storage medium, characterized in that: The readable storage medium stores a program or an instruction, and when the program or the instruction is executed by the processor, the steps of the shooting method according to any one of claims 1 to 7 are implemented.

Images