Photographing method and apparatus, electronic device and medium

Abstract

The present application relates to the technical field of communications, and discloses a photographing method and apparatus, an electronic device and a medium. The electronic device comprises an image sensor, a pre-image signal processor (ISP) and an application processor, wherein the image sensor is connected to the pre-ISP, and the pre-ISP is connected to the application processor. The method comprises: when receiving a photographing instruction, acquiring original image data by means of the image sensor; performing first image processing on the original image data by means of the pre-ISP by using an AI image processing algorithm, so as to obtain first image data; and performing second image processing on the first image data by means of the application processor, so as to output target image data.

Description

Photographing method and device, electronic device and medium

[0001] Cross-reference to Related Applications

[0002] The present application claims priority to Chinese Patent Application No. 202411344888.0, filed on September 25, 2024, the contents of which are incorporated herein by reference in its entirety. TECHNICAL FIELD

[0003] The present application belongs to the technical field of communication, and specifically relates to a photographing method, device, electronic device and medium. BACKGROUND

[0004] With the popularity of electronic devices such as mobile phones and the convenience of electronic devices in photographing, users are increasingly dependent on using electronic devices for daily photographing, and the demand for photographing functions is also increasing. The photographing effect and experience of the electronic device are closely related to the processing capability of its image system, therefore, how to improve the processing capability of the image system becomes particularly important.

[0005] Currently, electronic devices are mainly developed based on application processors (APs), and the image system needs to rely on artificial intelligence (AI) image processing algorithms to improve the image processing effect after photographing. However, due to the limitations of the computing power of the AP platform itself and the system power consumption, some AI image processing algorithms with high computing power requirements cannot be deployed, which ultimately affects the overall processing effect of the image. SUMMARY

[0006] The purpose of the embodiments of the present application is to provide a photographing method, device, electronic device and medium, which can solve the problem of poor overall processing effect of the existing photographing processing scheme based on the application processor (AP).

[0007] In a first aspect, the embodiments of the present application provide a photographing method, executed by an electronic device, the electronic device comprising an image sensor, a front image signal processor (ISP) and an application processor, the image sensor being connected with the front ISP, and the front ISP being connected with the application processor; the method comprising:

[0008] In the case of receiving a photographing instruction, acquiring original image data through the image sensor;

[0009] Performing first image processing on the original image data by the front ISP using an artificial intelligence (AI) image processing algorithm to obtain first image data;

[0010] The application processor performs second image processing on the first image data to output target image data.

[0011] In a second aspect, an electronic device is provided, which includes N image sensors, a selection module, a front-end image signal processor (ISP), and an application processor. The image sensors are connected to the front-end ISP, and the front-end ISP is connected to the application processor.

[0012] An application specific integrated circuit (ASIC) chip and a mobile industry processor interface (MIPI) bus network are arranged on the front-end ISP. N first input ends of the selection module are connected to the N image sensors one by one, a second end of the selection module is connected to the MIPI bus network, the MIPI bus network is connected to the application processor, and the MIPI bus network is further connected to the ASIC chip.

[0013] The MIPI bus network is configured to determine whether to transmit the raw image data to the ASIC chip for processing according to a shooting scene and shooting parameters.

[0014] In a third aspect, a shooting device is provided, which is arranged in an electronic device and includes an image sensor, a front-end image signal processor (ISP), and an application processor. The image sensor is connected to the front-end ISP, and the front-end ISP is further connected to the application processor.

[0015] The image sensor is configured to collect raw image data when a shooting instruction is received.

[0016] The front-end ISP is configured to perform first image processing on the raw image data by using an AI image processing algorithm to obtain first image data.

[0017] The application processor is configured to perform second image processing on the first image data to output target image data.

[0018] In a fourth aspect, an electronic device is provided, which includes a processor and a memory. The memory stores programs or instructions that can be run on the processor. When the programs or instructions are executed by the processor, the steps of the shooting method according to the first aspect are implemented.

[0019] In a fifth aspect, an embodiment of the present application provides a readable storage medium, the readable storage medium storing a program or instructions, the program or instructions being executed by a processor to implement the steps of the photographing method according to the first aspect.

[0020] In a sixth aspect, an embodiment of the present application provides a chip, the chip including a processor and a communication interface, the communication interface being coupled to the processor, and the processor being configured to run a program or instructions to implement the photographing method according to the first aspect.

[0021] In a seventh aspect, an embodiment of the present application provides a computer program product, the program product being stored in a storage medium, and the program product being executed by at least one processor to implement the photographing method according to the first aspect.

[0022] In an embodiment of the present application, an electronic device includes an image sensor, a front image signal processor (ISP) and an application processor, the image sensor being connected to the front ISP, and the front ISP being further connected to the application processor; in a case where a photographing instruction is received, raw image data is collected by the image sensor; the raw image data is subjected to first image processing by the front ISP using an AI image processing algorithm to obtain first image data; and the first image data is subjected to second image processing by the application processor to output target image data. In this way, some AI image processing algorithms are deployed by using an externally connected front ISP, so that the preliminary processing of the photographed image data can be accelerated by the front ISP, and further image processing can be performed on the image data that has been preliminarily processed by the ISP by the application processor, thereby effectively improving the image photographing effect and the image processing performance. BRIEF DESCRIPTION OF DRAWINGS

[0023] FIG. 1 is a structural schematic diagram of a multimedia processing system of an electronic device according to an embodiment of the present application;

[0024] FIG. 2 is a flowchart of a photographing method according to an embodiment of the present application;

[0025] FIG. 3 is a flowchart of a photographing method according to another embodiment of the present application;

[0026] FIG. 4 is a framework structural diagram of a high-performance special-purpose multimedia chip of an electronic device according to an embodiment of the present application;

[0027] FIG. 5 is a framework structural diagram of a multimedia chip of an electronic device according to an embodiment of the present application, the multimedia chip being provided with a multi-layer MIPI switching bus network;

[0028] FIG. 6 is a structural diagram of an electronic device according to an embodiment of the present application;

[0029] FIG. 7 is a hardware structural diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0030] The technical solutions in the embodiments of the present application will be clearly described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art are within the protection scope of the present application.

[0031] The terms "first", "second", and the like in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than that illustrated or described herein, and the objects distinguished by "first", "second", and the like are generally of a kind and are not limited in number, for example, the first object can be one or more. In addition, "and / or" in the specification and claims indicates at least one of the connected objects, and the character " / ", generally indicates that the front and rear associated objects are in an "or" relationship.

[0032] To make the embodiments of the present application clearer, the related technical knowledge involved in the embodiments of the present application will be further described in detail below, and the purposes and improvements of the embodiments of the present application will be briefly introduced, as follows:

[0033] AP: Application Processor, application processor;

[0034] ISP: Image Signal Processor, image signal processing unit;

[0035] PRE_ISP: front ISP processor, which processes image data output by a camera, i.e. a sensor, in front of an AP multimedia processing unit;

[0036] ORB: Oriented FAST and Rotated BRIEF, a commonly used image feature detection algorithm, which is divided into two parts, feature point extraction and feature point description;

[0037] NPU: Neural Processing Unit, neural processing unit.

[0038] Currently, multimedia systems, especially video / image shooting functions, are very important to electronic devices such as smart phones, tablet computers, vehicle display screens, etc. The quality of the shooting function, i.e. the shooting effect and experience, is closely related to the multimedia system capability of the electronic device. In particular, the current AI algorithm requires increasing computing power, and therefore, currently, improving the processing capability of the multimedia system is the key.

[0039] Current electronic devices are developed based on an application processor (AP): the current AI processing algorithm is developing rapidly, which greatly improves the shooting / video processing effect of the mobile phone, especially in image detail processing. The AI algorithm has a more intelligent processing result than the traditional computer vision (CV) algorithm, and the processing result is more in line with human processing habits and requirements. The AI processing algorithm puts higher requirements on the computing power and performance of the mobile phone processor. Currently, due to the limitations of the computing power and system power consumption of the AP platform, some image processing AI algorithms with high computing power requirements cannot be deployed, resulting in a decline in the overall image processing effect and a low image success rate. In addition, due to the limitations of the shared chip memory and bus resources of the multi-processing unit of the AP platform, in the user overload scenario, the AI algorithm runs for a long time on the AP platform, and the overall shooting delay is large, which affects the user experience.

[0040] To solve the technical problems existing in the prior art, the embodiment of the present application proposes a system scheme of externally hanging a high-performance front-end dedicated ISP chip at the application processor (AP) end. As shown in FIG. 1, in the system architecture, this scheme adopts a front-end (PRE) architecture, that is, the externally hung multimedia chip, i.e. the front-end PRE ISP chip, is hung at the front side of the AP, and is between the image sensor (Sensor) and the AP, and processes the image data in the raw (RAW) domain. Specifically, the shooting AI processing algorithm, image preprocessing algorithm (such as image frame selection and ORB fast registration preprocessing algorithm) and the like can be superimposed on the front-end ISP chip to improve the shooting effect. On the system interface, the embodiment of the present application can adopt a flexible multi-layer Mobile Industry Processor Interface (MIPI) exchange bus network to realize the working mode switching and timing control of the externally hung AI acceleration chip in different scenarios. The embodiment of the present application also implements the shooting processing enhancement algorithm in the multimedia chip using a dedicated ASIC circuit to reduce the processing delay and improve the system performance.

[0041] The shooting method and the multimedia chip architecture of the electronic device provided by the embodiment of the present application will be described in detail below in combination with the drawings and specific embodiments and application scenarios.

[0042] Please refer to FIG. 1 and FIG. 2, FIG. 1 is a structural schematic diagram of a multimedia processing system of an electronic device provided in an embodiment of the present application, and FIG. 2 is a flowchart of a photographing method provided in an embodiment of the present application, the method being executed by an electronic device, as shown in FIG. 1, the electronic device comprises an image sensor 101, a front image signal processor (ISP) 102 and an application processor (AP) 103, the image sensor 101 is connected with the front ISP 102, and the front ISP 102 is further connected with the AP 103; as shown in FIG. 2, the method comprises the following steps:

[0043] Step 201: In a case where a photographing instruction is received, raw image data is collected by the image sensor 101.

[0044] In the embodiment of the present application, the image sensor 101 can be understood as a camera enabled during photographing, the front ISP 102 is an image processing chip externally connected, and the AP 103 is an AP chip.

[0045] Specifically, when the photographing instruction is received, the camera, i.e., the image sensor 101, enters a working state to collect image data, and specifically, a plurality of frames of image data can be collected, so as to be further processed by the front ISP 102, such as frame selection, registration and even image synthesis, to finally form image data with enhanced photographing effect. The photographing instruction can be a photographing instruction or a video recording instruction. It should be noted that the image data collected by the image sensor 101 is raw image data, i.e., RAW domain data without any image processing.

[0046] In addition, the raw image data collected by the image sensor 101 can be transmitted to the front ISP 102 through a related data interface, such as a MIPI interface.

[0047] Step 202: The raw image data is subjected to first image processing by the front ISP 102 using an artificial intelligence (AI) image processing algorithm to obtain first image data.

[0048] In the embodiment of the present application, in order to accelerate real-time processing of the photographing image data, a part of the AI image processing algorithm can be deployed on the front ISP 102, so as to reduce the burden of the AP 103, improve the image processing capability of the photographing image data and improve the processing efficiency.

[0049] Specifically, in this step, after receiving the raw image data transmitted by the image sensor 101, the front-end ISP 102 can perform first image processing on the raw image data by using the AI image processing algorithm deployed thereon, that is, the preliminary image data, to obtain the first image data. The first image processing can be, for example, RAW domain noise reduction, dynamic range enhancement such as HDR processing, image synthesis, etc.

[0050] Step 203: performing second image processing on the first image data by the application processor 103, and outputting target image data.

[0051] Specifically, additional image processing algorithms such as image synthesis, image quality enhancement, filters, etc. AI image processing algorithms can be superimposed on the AP platform, that is, the application processor 103. In specific applications, corresponding AI image processing algorithms can be used in combination with actual shooting scenes to process the captured image data.

[0052] In this step, after receiving the preliminary processed image data output by the front-end ISP 102, the application processor 103 can apply corresponding AI image processing algorithms to further process the processed image data, such as image quality enhancement, image synthesis, special filters, etc.

[0053] After the second image processing by the application processor 103, the target image data after the final processing is output, and the target image data can be transmitted to the display screen 104 for display, so that the user can preview the shooting effect after image processing.

[0054] In some embodiments, before the step 202, the method further includes:

[0055] The front-end ISP 102 performs frame selection processing on the raw image data by using a preset frame selection strategy, and performs registration processing on the selected image frames by using an image feature detection algorithm;

[0056] The step 202 includes:

[0057] The front-end ISP 102 performs first image processing on the registered image data to obtain the first image data. The first image processing includes at least one of noise reduction, dynamic range enhancement, demosaicing, and image synthesis.

[0058] The step 203 includes:

[0059] Step 203: performing second image processing on the first image data by the application processor 103, and outputting target image data.

[0060] In some embodiments, pre-processing algorithms such as frame selection and registration can also be superimposed on the front-end ISP 102, and some AI image processing algorithms other than the pre-processing algorithms such as frame selection and registration, such as one or more algorithms of AI noise reduction, high dynamic range imaging (HDR), AI demosaicing, AI image synthesis, etc. can also be superimposed on the front-end ISP 102. Accordingly, additional image processing algorithms such as AI image processing algorithms of image quality enhancement, blurring, brightening, etc. can be superimposed on the AP platform, i.e. the application processor 103.

[0061] Therefore, the registered image data can also be subjected to first image processing by the front-end ISP 102, i.e. first-step image enhancement processing such as noise reduction, HDR, demosaicing, and image synthesis, etc. to obtain first image data. The first image data is then transmitted to the application processor 103 for second image processing, i.e. second-step image enhancement processing such as image synthesis, image quality enhancement, blurring, brightening, etc. to obtain final target image data.

[0062] Through this embodiment, the overall computing power and processing performance of the system can be improved by superimposing image pre-processing and image AI enhancement processing algorithms on the front-end ISP 102, thereby improving the image processing effect and shooting efficiency in shooting.

[0063] Optionally, as shown in FIG. 4, the front-end ISP 102 includes an application specific integrated circuit (ASIC) chip 110; the ASIC chip 110 includes a shooting pre-processing unit 111 and a shooting acceleration processing unit 112, the image sensor 101 is connected to the shooting pre-processing unit 111, the shooting pre-processing unit 111 is connected to the shooting acceleration processing unit 112, and the shooting acceleration processing unit 112 is connected to the application processor 103.

[0064] The first image processing of the original image data by the front-end ISP 102 using AI image processing algorithms to obtain first image data includes:

[0065] The shooting pre-processing unit 111 performs frame selection processing on the original image data using the preset frame selection strategy, and performs registration processing on the selected image frames using an image feature detection algorithm.

[0066] The shooting acceleration processing unit 112 performs image enhancement processing on the registered image data to obtain the first image data, wherein the image enhancement processing includes at least one of noise reduction, dynamic range enhancement, demosaicing, and image synthesis.

[0067] In some embodiments, as shown in FIG. 4, the front-end ISP 102 can be implemented by a dedicated ASIC chip 110, and the ASIC chip 110 is integrated with a shooting preprocessing unit 111 and a shooting acceleration processing unit 112. The shooting preprocessing unit 111 is implemented by a dedicated ASIC acceleration hardware circuit, and the shooting acceleration processing unit 112 is implemented by a dedicated NPU acceleration unit, that is, both of them are implemented by a dedicated hardware circuit, which can accelerate the processing of the shooting image. The shooting preprocessing unit 111 can also be called a shooting acceleration preprocessing unit, and the shooting acceleration processing unit 112 can also be called a dedicated AI accelerator.

[0068] Specifically, the shooting preprocessing unit 111 can be used to implement the frame selection processing of the original image data according to a preset frame selection strategy, and the registration processing of the selected image frames can be implemented by using an image feature detection algorithm such as ORB. The shooting acceleration processing unit 112 can be used to implement the image enhancement processing of the registered image data, such as AI noise reduction, AI dynamic range enhancement, AI demosaicing, AI image synthesis, and the like. The processed data is output to the application processor 103 for a second image enhancement processing, such as a quality enhancement processing, to obtain the final target image data.

[0069] The preset frame selection strategy provides a strategy for how to select a reference frame and how to capture a frame, so that the original image data can be processed according to the preset frame selection strategy to select a plurality of image frames from the plurality of original image frames captured by the camera for subsequent processing, such as selecting the 1st, 3rd, and 5th frames according to the preset frame selection strategy.

[0070] The above-mentioned registration processing of the selected image frames by using an algorithm such as ORB can be the image registration processing of the selected image frames by using an image feature detection algorithm such as ORB algorithm. Specifically, the selected image frames can be aligned in spatial position so that the same features in these images are as close as possible to the same position.

[0071] It should be noted that the above-mentioned frame selection and registration processing can be collectively referred to as preprocessing. When implemented, the frame selection and registration preprocessing algorithms can be superimposed on the shooting preprocessing unit 111 of the front-end ISP 102, so that the original image data captured by the image sensor 101 can be preprocessed by the shooting preprocessing unit 111, thereby reducing the image processing pressure of the application processor 103, that is, the AP platform.

[0072] In addition, the image data preprocessed by the front-end ISP 102 can be transmitted to the application processor 103 through a related data interface such as a MIPI interface for further image enhancement processing.

[0073] It should be noted that in some embodiments, as shown in FIG. 4, the front-end ISP 102 further includes a storage unit 113, which can be an integrated high-performance on-chip dedicated dynamic random access memory (DRAM) storage unit, or a high-performance double data rate synchronous dynamic random access memory (DDR) storage unit.

[0074] The photograph preprocessing unit 111 is connected with the storage unit 113, and the storage unit 113 is also connected with the photograph acceleration processing unit 112.

[0075] Specifically, the image data processed by the photograph preprocessing unit 111 can be written into the storage unit 113 for caching, and the photograph acceleration processing unit 112 accesses the storage unit 113 when needed to read out the image data from the storage unit 113 for further processing.

[0076] In this way, the storage-computing integrated architecture based on the front-end ISP chip can be realized, and by integrating the high-performance on-chip dedicated storage unit on the dedicated ASIC chip 110, the delay of the AI accelerator reading and writing access to the memory can be reduced, and the AI processing performance can be improved.

[0077] Optionally, the preset frame selection strategy includes:

[0078] determining a reference frame in the original image data;

[0079] grabbing M image frames forward and / or backward from the reference frame in the original image data, wherein the selected image frames include the reference frame and the M image frames, and M is an integer greater than 1.

[0080] Specifically, the preset frame selection strategy can include a set number of grabbed frames, a preset reference frame selection method, and a frame grabbing strategy based on the reference frame to grab image frames forward and / or backward. For example, the preset frame selection strategy stipulates that the first frame is the reference frame, a total of 3 frames are grabbed, and 3 frames are grabbed forward from the reference frame. Thus, the reference frame in the original image data can be determined according to the preset frame selection strategy, and a number of frames can be grabbed forward / backward from the reference frame according to the frame grabbing strategy. It should be noted that when the frames are grabbed forward, the front-end ISP chip can continuously grab the image data output by the camera and store the historical collected image data, and when the frames are selected, a number of frames can be grabbed from the historical collected image data based on the current reference frame.

[0081] It should be noted that the frame grabbing strategy based on the reference frame to grab the image frame forward and / or backward can be specifically as follows. In some embodiments, the frame grabbing strategy based on the reference frame to grab the image frame forward, for example, 3 frames forward from the reference frame, can be adopted. In other embodiments, the frame grabbing strategy based on the reference frame to grab the image frame backward, for example, 3 frames backward from the reference frame, can be adopted. In still other embodiments, the frame grabbing strategy based on the reference frame to grab the image frame forward and backward, for example, 3 frames forward and 3 frames backward from the reference frame, can be adopted.

[0082] Through the embodiment, the preset frame selection strategy can be adopted to realize the frame selection processing, and the effect of the image enhancement processing based on the selected frame can be ensured.

[0083] The first embodiment of the present application will be described below in combination with the embodiment shown in FIG. 3.

[0084] The embodiment proposes a system scheme in which a high-performance front-end ISP multimedia photographing AI acceleration chip is externally connected to an AP. In the system architecture, the scheme adopts a front-end processing architecture, that is, the multimedia chip is externally connected before the AP, and AI processing is performed on image data in the RAW domain. As shown in FIG. 3, the scheme specifically includes the following steps.

[0085] 301. The camera acquires image / video data according to the configured frame rate, and outputs the image / video data to the PRE_ISP externally connected chip through a high-performance MIPI interface.

[0086] 302. After the dedicated front-end ISP multimedia chip receives the image RAW data, frame selection processing is performed according to a preset frame selection strategy (including the number of frames to be grabbed and based on the reference frame to grab the image frame forward / backward).

[0087] 303. For the selected data frame, image registration processing is performed by using an ORB algorithm.

[0088] 304. The processing result is output to a dedicated AI acceleration hardware unit for processing, such as RAW domain noise reduction, HDR processing, image synthesis, and the like.

[0089] 305. After the dedicated front-end ISP multimedia chip completes the post-processing of the image, the image data and feature image are returned to the AP through different MIPIs.

[0090] 306. After the AP platform receives the image data output by the dedicated front-end ISP multimedia chip, corresponding image processing algorithms, such as a quality enhancement algorithm, are superimposed, and the image data is written into a camera storage unit.

[0091] The embodiment improves the image shooting effect and performance of the electronic device, in particular a mobile phone, by externally connecting a high-performance special shooting AI acceleration chip, i.e., a front ISP chip, and deploying a high-performance AI shooting acceleration algorithm.

[0092] Optionally, before the raw image data is subjected to the frame selection processing by the shooting preprocessing unit 111 according to the preset frame selection strategy, the method further includes:

[0093] The raw image data is subjected to pixel point correction processing by the shooting preprocessing unit 111, wherein the pixel point correction includes at least one of defective pixel correction and black level correction.

[0094] The raw image data is subjected to frame selection processing by the shooting preprocessing unit 111 according to the preset frame selection strategy, including:

[0095] The raw image data subjected to the pixel point correction is subjected to frame selection processing by the shooting preprocessing unit 111 according to the preset frame selection strategy.

[0096] In some embodiments, the raw image data collected by the camera can also be subjected to pixel point correction processing by the shooting preprocessing unit 111, such as defective pixel correction and black level correction, to obtain complete raw image data without defective pixels, and then the corrected image data is subjected to frame selection, registration and other processing.

[0097] Specifically, defective pixel correction (DPC) mainly processes pixel defects caused by manufacturing process problems. These defective pixels have a large difference from the surrounding pixels, and may appear as white spots in a completely dark environment or black spots in a high-light environment. Defective pixels are divided into static defective pixels and dynamic defective pixels, the former always remains constant, and the latter changes under the influence of exposure or temperature conditions. The defective pixel correction process includes two steps of detection and correction. Detection is usually performed by comparing the difference between the center pixel and the adjacent pixels, and if the difference exceeds a preset threshold, it is determined as a defective pixel. Correction is to repair these defective pixels by interpolation replacement and other methods.

[0098] Black level correction (BLC) processes the distortion of dark part information of an image caused by insufficient sensor accuracy or circuit heat radiation. In the case where the lens module is completely blocked and no light is incident, the sensor can still read non-zero voltage data, which does not represent the actual image information, but is caused by the inaccuracy of the sensor or the electronic excitation of the circuit heat radiation. Black level correction corrects these inaccurate dark part data to ensure the accuracy of the dark part of the image, thereby improving the image quality.

[0099] The following describes the second application embodiment with reference to the high-performance special-purpose multimedia chip framework shown in FIG. 4.

[0100] The embodiment proposes a scheme for implementing image AI processing by using hardware circuits in a special-purpose multimedia chip, reducing AI algorithm processing delay and power consumption, and the main process is as follows:

[0101] 401. The camera sensor transmits image RAW domain data to the special-purpose front-end ISP multimedia chip through the MIPI interface. The first and second interfaces in FIG. 4 are both MIPI interfaces.

[0102] 402. The special-purpose front-end ISP multimedia chip performs preprocessing on the image by using a special-purpose ASIC acceleration hardware circuit, including a front-end basic ISP hardware circuit unit, to complete bad pixel correction, black level correction, etc.

[0103] 403. The special-purpose front-end ISP multimedia chip performs AI noise reduction, AI demosaicking, AI dynamic range enhancement, etc. on the image by using a special-purpose NPU acceleration unit.

[0104] 404. The special-purpose front-end ISP multimedia chip reduces the delay of AI accelerator read-write access to the memory by integrating a high-performance on-chip special-purpose DRAM storage unit, thereby improving AI processing performance.

[0105] 405. The special-purpose multimedia chip transmits the processed data back to the AP through the MIPI interface. The third and fourth interfaces in FIG. 4 are both MIPI interfaces.

[0106] In the embodiment, by using a special-purpose ASIC circuit, an AI accelerator, and a DDR memory, a large-power AI model can be deployed on a mobile phone or other electronic device processor, effectively improving the energy efficiency ratio and delay performance of multimedia processing of the electronic device, especially the mobile phone.

[0107] Optionally, as shown in FIG. 5, the front-end ISP 102 is provided with an ASIC chip 110 and a MIPI exchange bus network 120; the number of image sensors 101 is N, and the electronic device further includes a selection module 105, N first ends of the selection module 105 are connected to the N image sensors 101 one by one, a second end of the selection module 105 is connected to the MIPI exchange bus network 120, the MIPI exchange bus network 120 is connected to the application processor 103, and the MIPI exchange bus network 120 is further in communication connection with the ASIC chip 110.

[0108] Before the original image data is subjected to first image processing by the front-end ISP 102 using an AI image processing algorithm, the method further includes:

[0109] According to the shooting scene, the control selection module 105 selects at least one of the N image sensors 101 as a target image sensor;

[0110] The target image sensor collects raw image data;

[0111] In the case of determining to turn on the ASIC chip 110 according to the shooting scene and the shooting parameters, the raw image data is transmitted to the ASIC chip 110 through the MIPI exchange bus network 120;

[0112] The first image processing of the raw image data by the front-end ISP 102 using an AI image processing algorithm includes:

[0113] The first image processing of the raw image data by the ASIC chip 110 using an AI image processing algorithm.

[0114] That is, in some embodiments, a flexible multi-layer MIPI exchange bus network is proposed, which realizes the working mode switching and timing control of the externally hung AI acceleration chip in different scenes, so as to flexibly realize the function switching in different scenes to make the optimal combination of the overall performance, power consumption and effect of the camera. The main system block diagram is shown in FIG. 5.

[0115] The electronic device can include multiple image sensors 101, i.e., equipped with multiple cameras Sensor, such as Sensor0, Sensor1, Sensor2 and Sensor3, which can be front cameras, rear cameras, infrared cameras, rear ultra-wide cameras, etc. The multiple Sensors are controlled by the selection module 105 to determine whether to be enabled, specifically, the target Sensor to be enabled can be flexibly selected according to the actual use scene requirements. For example, two of the Sensors can be flexibly selected as the input of the photographing AI acceleration externally hung chip, such as the rear main camera and the rear ultra-wide camera.

[0116] The MIPI exchange bus network 120 can support physical layer bypass and dynamic switching functions, i.e., can support direct transmission of raw image data to the application processor 103, and at the same time support switching of raw image data between the ASIC chip 110 and the application processor 103. The camera system can select whether to turn on the photographing AI acceleration externally hung chip according to the shooting scene, the focal length of the enabled camera, and other shooting parameters, i.e., whether to turn on the ASIC chip 110 to process the raw image data. In the case of selecting to turn on, the MIPI exchange bus network 120 delivers the raw image data to the ASIC chip 110 for processing, such as frame selection, registration, etc. of the raw image data by the ASIC chip 110.

[0117] The MIPI switching bus network 120 also supports data link layer bypass and dynamic switching functions. The AI acceleration external chip can continuously capture image data output by the sensor and perform forward frame capturing processing based on a reference frame to improve the shooting effect, and output the image data to the AP for preview.

[0118] The MIPI switching bus network 120 also supports using a dedicated MIPI channel and supports custom data frame packaging. The output ghost image and meta information, i.e., intermediate processing result information, are sent to the AP for processing, which can ensure high transmission efficiency.

[0119] Optionally, the ASIC chip 110 is in a sleep state without being turned on.

[0120] That is, in some embodiments, the ASIC chip 110 can be in a sleep state without being turned on, i.e., without turning on the AI acceleration external chip. Specifically, the shooting preprocessing unit 111 and the shooting acceleration processing unit 112 of the external ASIC chip 110 are in a power-off state, thereby achieving the purpose of reducing system power consumption.

[0121] Optionally, the method further comprises:

[0122] In a case where it is determined according to the shooting scene and the shooting parameter that the ASIC chip 110 is not to be turned on, the original image data is transmitted to the application processor 103 through the MIPI switching bus network 120, and the original image data is processed by the application processor 103.

[0123] In a case where it is determined according to the shooting scene, the focal length of the enabled camera, and other shooting parameters that the ASIC chip 110 is not to be turned on, i.e., in a case where the shooting function of the external chip is not turned on, the MIPI switching bus network 120 can directly output the image data output by the sensor to the application processor 103, i.e., the AP, through the physical layer bypass interface, and the image processing is completed by the AP.

[0124] In this way, the flexible multi-layer MIPI switching bus network can realize the working mode switching and timing control of the external AI acceleration chip, and further realize the flexible function switching in different scenes, and realize the comprehensive optimization of the overall performance, power consumption, and effect of the camera.

[0125] Optionally, in a case where it is determined according to the shooting scene and the shooting parameter that the ASIC chip 110 is to be turned on, the original image data is transmitted to the ASIC chip 110 through the MIPI switching bus network 120, comprising:

[0126] In the case that the ASIC chip 110 is not currently started, and it is determined according to the shooting scene and the shooting parameter that the ASIC chip 110 needs to be started, the target image sensor is kept in the starting state, and the raw image data is transmitted to the ASIC chip 110 through the MIPI exchange bus network 120.

[0127] That is, the external chip system can support seamless switching. When the next frame of image needs to start the photographing AI acceleration external chip, the camera does not need to be closed, the target image sensor can be kept in the starting state, and seamless switching to the AI acceleration external chip for picture processing is realized.

[0128] The following describes the third embodiment of the application in combination with the multimedia special-purpose chip system framework shown in FIG. 5, which is provided with a multi-layer MIPI exchange bus network.

[0129] This embodiment proposes a flexible multi-layer MIPI exchange bus network, realizes the working mode switching and timing control of the external AI acceleration chip in different scenes, and flexibly realizes the function switching in different scenes, so that the overall performance, power consumption and effect of the camera are optimally combined. The main points are as follows:

[0130] 1. The system hardware circuit design can flexibly select two Sensors as the inputs of the photographing AI acceleration external chip according to the use scene requirements of the specific product, such as selecting the rear main camera and the rear ultra-wide camera.

[0131] 2. The internal MIPI exchange network of the external chip supports the physical layer bypass and dynamic switching functions. The camera system can select whether to start the photographing AI acceleration external chip according to the scene, focal length and the like. When the chip photographing function is not started, the image data output by the Sensor is output to the AP through the physical layer bypass interface and the image processing is completed by the AP. The system supports seamless switching, that is, when the next frame of image needs to start the photographing AI acceleration external chip, the camera does not need to be closed and seamlessly switches to the AI acceleration external chip for picture processing. At the same time, when the AI acceleration external chip is not started, the image preprocessing and AI acceleration processing units of the external chip are in the power-off state, so that the system power consumption is minimized.

[0132] 3. The internal MIPI exchange network of the external chip supports the data link layer bypass and dynamic switching functions. The AI acceleration external chip can continuously capture the image data output by the Sensor and perform forward frame grabbing processing based on the reference frame to improve the photographing effect, and output the image data to the AP for preview.

[0133] 4. The internal MIPI exchange network of the external chip supports using a dedicated MIPI channel and supports custom data frame packaging, and sends the output ghost image and meta information to the AP for processing.

[0134] In this embodiment, on the system interface, a flexible multi-layer MIPI exchange bus network is adopted to realize the working mode switching and timing control of the externally hung AI acceleration chip in different scenarios, so as to realize the function switching in different scenarios and make the optimal combination of the overall performance, power consumption and effect of the camera.

[0135] It should be noted that in actual application, more hardware accelerators, DSPs and other special processor units can be stacked on the basis of the PRE architecture to improve the system processing flexibility.

[0136] The photographing method in the embodiment of the application is executed by an electronic device, the electronic device comprising an image sensor, a front image signal processor (ISP) and an application processor, the image sensor being connected with the front ISP, and the front ISP being further connected with the application processor; in the case that a photographing instruction is received, raw image data is collected by the image sensor; the raw image data is subjected to first image processing by the front ISP using an AI image processing algorithm to obtain first image data; and the first image data is subjected to second image processing by the application processor to output target image data. In this way, by deploying some AI image processing algorithms using the externally hung front ISP, the preliminary processing of the photographed image data can be accelerated by the front ISP, and further image processing of the image data preliminarily processed by the ISP can be performed by the application processor, thereby effectively improving the image photographing effect and image processing performance.

[0137] The embodiment of the application mainly processes photographing acceleration, that is, by using an AI photographing processing algorithm, a plurality of images are combined into one image to realize photographing effect enhancement processing; meanwhile, in the image preprocessing module, based on a reference frame, continuous images can be captured forward or backward, and the captured images are subjected to registration processing using an ORB algorithm to reduce the influence of camera shaking; on the system interface, a flexible multi-layer MIPI exchange bus network is adopted to realize the working mode switching and timing control of the externally hung AI acceleration chip in different scenarios.

[0138] Please refer to FIG. 5, which is a framework structure diagram of an electronic device in which a multi-layer MIPI exchange bus network is arranged according to an embodiment of the application. As shown in FIG. 5, the electronic device comprises N image sensors 101, a selection module 105, a front image signal processor (ISP) 102 and an application processor 103, the image sensors 101 being connected with the front ISP 102, and the front ISP 102 being further connected with the application processor 103.

[0139] The ASIC chip 110 and the MIPI switch bus network 120 are arranged on the front ISP 102; N first ends of the selection module 105 are connected with N image sensors 101 one by one, a second end of the selection module 105 is connected with the MIPI switch bus network 120, and the MIPI switch bus network 120 is connected with the application processor 103; the MIPI switch bus network 120 is also in communication connection with the ASIC chip 110;

[0140] The MIPI switch bus network 120 is used for determining whether to transmit the raw image data to the ASIC chip 110 for processing according to a shooting scene and shooting parameters.

[0141] This embodiment is a structure embodiment of an electronic device corresponding to the foregoing method embodiment, and specific introduction can be made by referring to the related description in the foregoing embodiment. The electronic device can achieve the same technical effects as the foregoing embodiment. To avoid repetition, details are not described herein.

[0142] The shooting method provided in the embodiment can be executed by a shooting device. In the embodiment, the shooting method is executed by a shooting device, and the shooting device provided in the embodiment is described.

[0143] Referring to FIG. 1, FIG. 1 is a structural schematic diagram of a shooting device provided in the embodiment, which is arranged in an electronic device and includes an image sensor 101, a front ISP 102, and an application processor 103. The image sensor 101 is connected with the front ISP 102, and the front ISP 102 is further connected with the application processor 103. Wherein,

[0144] The image sensor 101 is configured to collect raw image data when receiving a shooting instruction.

[0145] The front ISP 102 is configured to perform first image processing on the raw image data by using an artificial intelligence (AI) image processing algorithm to obtain first image data.

[0146] The application processor 103 is configured to perform image enhancement processing on the first image data to output target image data.

[0147] Optionally, as shown in FIG. 4, the front ISP 102 includes an application specific integrated circuit (ASIC) chip 110. The ASIC chip 110 includes a shooting preprocessing unit 111 and a shooting acceleration processing unit 112. The image sensor 101 is connected with the shooting preprocessing unit 111, the shooting preprocessing unit 111 is connected with the shooting acceleration processing unit 112, and the shooting acceleration processing unit 112 is connected with the application processor 103.

[0148] The photographing preprocessing unit 111 is configured to perform frame selection processing on the original image data according to a preset frame selection strategy, and perform registration processing on the selected image frames by using an image feature detection algorithm.

[0149] The photographing acceleration processing unit 112 is configured to perform image enhancement processing on the registered image data to obtain first image data, wherein the image enhancement processing includes at least one of noise reduction, dynamic range enhancement, demosaicing, and image synthesis.

[0150] Optionally, the photographing preprocessing unit 111 is further configured to perform pixel point correction processing on the original image data, wherein the pixel point correction includes at least one of bad point correction and black level correction.

[0151] Optionally, the front-end ISP 102 is configured to:

[0152] determine a reference frame in the original image data;

[0153] capture M image frames forward and / or backward from the original image data based on the reference frame, wherein the selected image frames include the reference frame and the M image frames, and M is an integer greater than 1.

[0154] Optionally, as shown in FIG. 5, the front-end ISP 102 is provided with an ASIC chip 110 and a MIPI switch bus network 120; the number of image sensors 101 is N, and the photographing device further includes a selection module 105, the N first ends of the selection module 105 are connected to the N image sensors 101 one by one, the second end of the selection module 105 is connected to the MIPI switch bus network 120, the MIPI switch bus network 120 is connected to the application processor 103; and the MIPI switch bus network 120 is further in communication connection with the ASIC chip 110.

[0155] The photographing device further includes:

[0156] The application processor 103 is configured to control the selection module 105 to select at least one of the N image sensors as a target image sensor according to a photographing scene.

[0157] The target image sensor is configured to collect original image data.

[0158] The MIPI switch bus network 120 is configured to, in a case where it is determined to start the ASIC chip 110 according to the photographing scene and photographing parameters, transmit the original image data to the ASIC chip 110.

[0159] The ASIC chip 110 is configured to perform first image processing on the original image data by using an AI image processing algorithm.

[0160] Optionally, the photographing device further comprises:

[0161] The MIPI switching bus network 120 is further configured to, in a case where it is determined according to the photographing scene and the photographing parameter that the ASIC chip 110 is not to be started, transmit the raw image data to the application processor 103 in a transparent manner.

[0162] The application processor 103 is configured to process the raw image data.

[0163] Optionally, in a case where the ASIC chip 110 is not to be started, the ASIC chip 110 is in a dormant state.

[0164] Optionally, in a case where the ASIC chip is not currently started and it is determined according to the photographing scene and the photographing parameter that the ASIC chip is to be started, the target image sensor remains in an activated state.

[0165] A photographing device is arranged in an electronic device and comprises an image sensor, a front-end image signal processor (ISP) and an application processor. The image sensor is connected to the front-end ISP, and the front-end ISP is further connected to the application processor. The image sensor is configured to, in a case where a photographing instruction is received, collect raw image data. The front-end ISP is configured to perform first image processing on the raw image data by using an AI image processing algorithm to obtain first image data. The application processor is configured to perform second image processing on the first image data to output target image data. In this way, some AI image processing algorithms are deployed by using an externally connected front-end ISP, so that the preliminary processing of the photographing image data can be accelerated by the front-end ISP, and the image data that has been preliminarily processed by the ISP can be further processed by the application processor, thereby effectively improving the image photographing effect and the image processing performance.

[0166] The photographing apparatus in the embodiments of the present applicationapplicationbe an electronic device, or a component in an electronic device, such as an integrated circuit or a chip. The electronic deviceapplicationbe a terminal, or another device other than a terminal. For example, the electronic deviceapplicationbe a mobile phone, a tablet computer, a notebook computer, a palm computer, a vehicle-mounted electronic device, a Mobile Internet Device (MID), an augmented reality (AR) / virtual reality (VR) device, a robot, a wearable device, an ultra-mobile personal computer (UMPC), a netbook, or a personal digital assistant (PDA), andapplicationbe a server, a Network Attached Storage (NAS), a personal computer (PC), a television (TV), a teller machine, or a self-service machine, and the like. The embodiments of the present application do not make a specific limitation.

[0167] The photographing apparatus in the embodiments of the present applicationapplicationbe a device having an operating system. The operating systemapplicationbe an Android operating system, an iOS operating system, or another possible operating system, and the embodiments of the present application do not make a specific limitation.

[0168] The photographing apparatus provided in the embodiments of the present applicationapplicationimplement each process of the method embodiments of FIG. 2 or FIG. 3, and achieve the same technical effects. To avoid repetition, the details are not described herein.

[0169] Optionally, as shown in FIG. 6, the embodiments of the present application further provide an electronic device 600, which includes a processor 601 and a memory 602. The memory 602 stores programs or instructions that can run on the processor 601. When the programs or instructions are executed by the processor 601, each step of the photographing method embodiments described above is implemented, and the same technical effects are achieved. To avoid repetition, the details are not described herein.

[0170] It should be noted that the electronic device in the embodiments of the present applicationapplicationinclude the mobile electronic device and the non-mobile electronic device described above.

[0171] FIG. 7 is a schematic diagram of a hardware structure of an electronic device according to an embodiment of the present application.

[0172] The electronic device 100 includes, but is not limited to, a radio frequency unit 701, a network module 702, an audio output unit 703, an input unit 704, a sensor 705, a display unit 706, a user input unit 707, an interface unit 708, a memory 709, and a processor 710, and the like. The electronic device further includes an image sensor, a front image signal processor (ISP), and an application processor, wherein the image sensor is connected to the front ISP, and the front ISP is connected to the application processor.

[0173] Those skilled in the art can understand that the electronic device 700 can further include a power supply (such as a battery) for supplying power to each component, and the power supply can be logically connected to the processor 710 through a power management system, so as to realize functions such as management of charging, discharging, and power consumption management through the power management system. The electronic device structure shown in FIG. 7 does not constitute a limitation on the electronic device, and the electronic device can include more or fewer components than those shown, or combine certain components, or different component arrangements, which will not be described here.

[0174] The image sensor is configured to acquire raw image data when a shooting instruction is received.

[0175] The front ISP is configured to perform first image processing on the raw image data by using an artificial intelligence (AI) image processing algorithm to obtain first image data.

[0176] The application processor is configured to perform second image processing on the first image data to output target image data.

[0177] Optionally, the front ISP includes an application specific integrated circuit (ASIC) chip, wherein the ASIC chip includes a shooting pre-processing unit and a shooting acceleration processing unit, the image sensor is connected to the shooting pre-processing unit, the shooting pre-processing unit is connected to the shooting acceleration processing unit, and the shooting acceleration processing unit is connected to the application processor.

[0178] The shooting pre-processing unit is configured to perform first image processing on the raw image data by using an AI image processing algorithm to obtain first image data.

[0179] The shooting acceleration processing unit is further configured to perform image enhancement processing on the registered image data to obtain the first image data, wherein the image enhancement processing includes at least one of noise reduction, dynamic range enhancement, demosaicing, and image synthesis.

[0180] Optionally, the shooting pre-processing unit is further configured to perform pixel point correction processing on the raw image data, wherein the pixel point correction includes at least one of bad point correction and black level correction.

[0181] The photograph preprocessing unit is configured to perform frame selection processing on the corrected original image data according to the preset frame selection strategy.

[0182] Optionally, the front-end ISP is further configured to:

[0183] determine a reference frame in the original image data;

[0184] capture M image frames from the original image data forward and / or backward with the reference frame as a reference, wherein the selected image frames include the reference frame and the M image frames, and M is an integer greater than 1.

[0185] Optionally, the front-end ISP is provided with an ASIC chip and a MIPI switch bus network; the number of the image sensors is N, and the electronic device further includes a selection module, N first ends of the selection module are connected to the N image sensors one by one, a second end of the selection module is connected to the MIPI switch bus network, the MIPI switch bus network is connected to the application processor, and the MIPI switch bus network is further connected to the ASIC chip in communication.

[0186] The processor 710 is configured to control the selection module to select at least one of the N image sensors as a target image sensor according to a photographing scene.

[0187] The target image sensor is configured to collect original image data.

[0188] The MIPI switch bus network is configured to transmit the original image data to the ASIC chip in a case where it is determined to start the ASIC chip according to the photographing scene and photographing parameters.

[0189] The ASIC chip is configured to perform first image processing on the original image data by using an AI image processing algorithm.

[0190] Optionally, the MIPI switch bus network is further configured to transmit the original image data to the application processor in a case where it is determined not to start the ASIC chip according to the photographing scene and photographing parameters, and perform processing on the original image data by using the application processor.

[0191] Optionally, in the case where the ASIC chip is not started, the ASIC chip is in a dormant state.

[0192] Optionally, the processor 710 is further configured to keep the target image sensor in an activated state in a case where the ASIC chip is not currently started and it is determined to start the ASIC chip according to the photographing scene and photographing parameters.

[0193] The MIPI exchange bus network is used to transmit the original image data to the ASIC chip.

[0194] It should be understood that in the embodiments of the present application, the input unit 704 can include a graphics processor (Graphics Processing Unit, GPU) 7041 and a microphone 7042, and the graphics processor 7041 processes image data of a still picture or a video obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The display unit 706 can include a display panel 7061, which can be configured in the form of a liquid crystal display, an organic light-emitting diode, etc. The user input unit 707 includes at least one of a touch panel 7071 and other input devices 7072. The touch panel 7071 is also called a touch screen. The touch panel 7071 can include two parts of a touch detection device and a touch controller. The other input devices 7072 can include, but are not limited to, a physical keyboard, function keys (such as volume control keys, on-off keys, etc.), a trackball, a mouse, a joystick, and the like, which will not be described here.

[0195] The memory 709 can be used to store software programs and various data. The memory 709 can mainly include a first storage area storing programs or instructions and a second storage area storing data, wherein the first storage area can store an operating system, application programs or instructions required by at least one function (such as a sound playing function, an image playing function, etc.), and the like. In addition, the memory 709 can include a volatile memory or a non-volatile memory, or the memory 709 can include both a volatile memory and a non-volatile memory. The non-volatile memory can be a Read-Only Memory (ROM), a Programmable ROM (PROM), an Erasable PROM (EPROM), an Electrically EPROM (EEPROM), or a flash memory. The volatile memory can be a Random Access Memory (RAM), a Static RAM (SRAM), a Dynamic RAM (DRAM), a Synchronous DRAM (SDRAM), a Double Data Rate SDRAM (DDR SDRAM), an Enhanced SDRAM (ESDRAM), a Synch link DRAM (SLDRAM), and a Direct Rambus RAM (DRRAM). The memory 709 in the embodiments of the present application includes but is not limited to these and any other suitable types of memory.

[0196] The processor 710 can include one or more processing units; optionally, the processor 710 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and an application program, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It can be understood that the above-mentioned modem processor can also not be integrated into the processor 710.

[0197] The embodiments of the present application also provide a readable storage medium, the readable storage medium stores programs or instructions, the programs or instructions are executed by a processor to realize various processes of the above-mentioned photographing method embodiments, and the same technical effects can be achieved. To avoid repetition, details are not described here.

[0198] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes a computer readable storage medium, such as a computer readable only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0199] The embodiment of the present application further provides a chip, which comprises a processor and a communication interface, the communication interface is coupled with the processor, the processor is used for running programs or instructions to realize the processes of the above-mentioned photographing method embodiments and achieve the same technical effects. To avoid repetition, details are not described herein.

[0200] It should be understood that the chip mentioned in the embodiment of the present application can also be referred to as a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0201] The embodiment of the present application provides a computer program product, which is stored in a storage medium, and is executed by at least one processor to realize the processes of the above-mentioned photographing method embodiments and achieve the same technical effects. To avoid repetition, details are not described herein.

[0202] It should be noted that in this document, the term "comprising" or "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the method and device in the embodiment of the present application is not limited to the order of performing the functions as shown or discussed, but can also include performing the functions in a substantially simultaneous manner or in a reverse order, for example, the described method can be performed in an order different from the described order, and various steps can also be added, omitted or combined. In addition, the features described with reference to some examples can be combined in other examples.

[0203] Through the above description of the embodiments, those skilled in the art can clearly understand that the above-mentioned example methods can be realized by means of software and a necessary general hardware platform, and of course, can also be realized by hardware, but in many cases, the former is a better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of a computer software product in essence or in the form of a part that contributes to the prior art, which is stored in a storage medium (such as a ROM / RAM, a magnetic disk, or an optical disk) and includes a plurality of instructions for causing a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present application.

[0204] The embodiments of the present application are described above in combination with the drawings, but the present application is not limited to the above-mentioned specific embodiments, and the above-mentioned specific embodiments are only illustrative and not restrictive. Those skilled in the art can make many forms under the inspiration of the present application without departing from the scope of the present application and the scope protected by the claims.

Claims

1. A photographing method, executed by an electronic device, the electronic device comprising an image sensor, a front image signal processor (ISP) and an application processor, the image sensor being connected with the front ISP, and the front ISP being further connected with the application processor; the method comprising: collecting raw image data by the image sensor upon receiving a photographing instruction; performing first image processing on the raw image data by the front ISP using an artificial intelligence (AI) image processing algorithm to obtain first image data; and performing second image processing on the first image data by the application processor to output target image data. The front ISP comprises an application specific integrated circuit (ASIC) chip; the ASIC chip comprises a photographing pre-processing unit and a photographing acceleration processing unit, the image sensor is connected with the photographing pre-processing unit, the photographing pre-processing unit is connected with the photographing acceleration processing unit, and the photographing acceleration processing unit is connected with the application processor. The first image processing on the raw image data by the front ISP using an AI image processing algorithm to obtain first image data comprises: performing frame selection processing on the raw image data by the photographing pre-processing unit using a preset frame selection strategy, and performing registration processing on the selected image frames using an image feature detection algorithm; 2. The method of claim 1, wherein, performing image enhancement processing on the registered image data by the photographing acceleration processing unit to obtain the first image data, wherein the image enhancement processing comprises at least one of noise reduction, dynamic range enhancement, demosaicing and image synthesis. Before the frame selection processing on the raw image data by the photographing pre-processing unit using a preset frame selection strategy, the method further comprises: performing pixel point correction processing on the raw image data by the photographing pre-processing unit, wherein the pixel point correction comprises at least one of bad point correction and black level correction; The frame selection processing on the raw image data by the photographing pre-processing unit using a preset frame selection strategy comprises:

3. The method of claim 2, wherein, performing frame selection processing on the corrected raw image data by the photographing pre-processing unit using the preset frame selection strategy. The frame selection processing on the raw image data using a preset frame selection strategy comprises: determining a reference frame in the raw image data; grabbing M image frames forward and / or backward from the raw image data based on the reference frame, wherein the selected image frames comprise the reference frame and the M image frames, and M is an integer greater than 1.

4. The method of claim 2, wherein, The front ISP is provided with an ASIC chip and a MIPI switch bus network; the number of image sensors is N, the electronic device further comprises a selection module, N first ends of the selection module are connected with N image sensors one by one, a second end of the selection module is connected with the MIPI switch bus network, and the MIPI switch bus network is connected with the application processor. The MIPI switch bus network is further in communication connection with the ASIC chip. ​ 5. The method of claim 1, wherein, ​ ​ Before the first image processing of the raw image data by the pre-ISP using an AI image processing algorithm, the method further comprises: controlling the selection module to select at least one of the N image sensors as a target image sensor according to a shooting scene; collecting raw image data by the target image sensor; transmitting the raw image data to the ASIC chip through the MIPI bus network in a case where it is determined to turn on the ASIC chip according to the shooting scene and shooting parameters; the first image processing of the raw image data by the pre-ISP using an AI image processing algorithm, comprising: the first image processing of the raw image data by the ASIC chip using an AI image processing algorithm.

6. The method of claim 5, wherein, The method further comprises: transmitting the raw image data to the application processor through the MIPI bus network in a case where it is determined not to turn on the ASIC chip according to the shooting scene and shooting parameters, and processing the raw image data by the application processor.

7. The method of claim 5, wherein, the transmitting of the raw image data to the ASIC chip through the MIPI bus network in a case where it is determined to turn on the ASIC chip according to the shooting scene and shooting parameters, comprising: keeping the target image sensor in an activated state and transmitting the raw image data to the ASIC chip through the MIPI bus network in a case where the ASIC chip is not currently turned on and it is determined to turn on the ASIC chip according to the shooting scene and shooting parameters.

8. An electronic device comprising: N image sensors, a selection module, a pre-image signal processor (ISP) and an application processor, the image sensors are connected with the pre-ISP, and the pre-ISP is further connected with the application processor; an ASIC chip and a MIPI bus network are arranged on the pre-ISP; N first ends of the selection module are connected with the N image sensors one by one, a second end of the selection module is connected with the MIPI bus network, and the MIPI bus network is connected with the application processor; the MIPI bus network is further connected in communication with the ASIC chip; the MIPI bus network is used to determine whether to transmit raw image data to the ASIC chip for processing according to a shooting scene and shooting parameters.

9. A shooting device arranged in an electronic device, comprising an image sensor, a pre-image signal processor (ISP) and an application processor, the image sensor is connected with the pre-ISP, and the pre-ISP is further connected with the application processor; wherein the image sensor is used to collect raw image data in a case where a shooting instruction is received; the pre-ISP is used to perform first image processing on the raw image data using an AI image processing algorithm to obtain first image data; the application processor is used to perform second image processing on the first image data to output target image data.

10. The photographing apparatus according to claim 9, wherein An ASIC chip and a MIPI switch bus network are arranged on the front-end ISP; the number of image sensors is N, and the electronic device further comprises a selection module, N first ends of the selection module are connected to the N image sensors one by one, a second end of the selection module is connected to the MIPI switch bus network, and the MIPI switch bus network is connected to the application processor; The MIPI switch bus network is further connected in communication with the ASIC chip; The application processor is further configured to control the selection module to select at least one of the N image sensors as a target image sensor according to a shooting scene; The target image sensor is configured to collect raw image data; The MIPI switch bus network is configured to transmit the raw image data to the ASIC chip when it is determined to turn on the ASIC chip according to the shooting scene and shooting parameters; The ASIC chip is configured to perform first image processing on the raw image data by using an AI image processing algorithm.

11. An electronic device, comprising a processor and a memory, the memory storing programs or instructions executable on the processor, and the programs or instructions, when executed by the processor, implement the steps of the shooting method according to any one of claims 1 to 7.

12. A readable storage medium, the readable storage medium storing programs or instructions, and the programs or instructions, when executed by a processor, implement the steps of the shooting method according to any one of claims 1 to 7.

Images