Multimedia processing chip, electronic device and dynamic image processing method

Abstract

The embodiment of the present application provides a multimedia processing chip, an electronic device and a dynamic image processing method. The multimedia processing chip includes an image signal processor and a neural network processor, the image signal processor is used to count the state information of the image data; the neural network processor is used to perform neural network algorithm processing on the image data; the multimedia processing chip is used to pre-process the image data at least through the neural network processor, and send the state information and the pre-processed image data to the application processing chip. The embodiment of the present application can improve the image processing capability.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a multimedia processing chip, an electronic device and a dynamic image processing method. Background Art

[0002] Various multimedia devices that can perform video shooting and photo taking functions (such as digital cameras, smart phones, tablet computers, etc.) generally have image sensors for acquiring images, multimedia processing chips that can perform image processing, and application processors (APs). Image sensors can be connected to multimedia processing chips via MIPI (Mobile Industry Processor Interface) lines, and multimedia processing chips can be connected to APs via MIPI lines.

[0003] Image sensors may include complementary metal-oxide-semiconductor (CMOS) image sensors, charge coupled device (CCD) image sensors, etc. Multimedia processing chips generally use image signal processors (ISPs) to process images acquired by image sensors. After the multimedia processing chip completes the image processing, it obtains the processing result and transmits the processing result to the AP. However, the multimedia processing chip in the related art has limited image processing capabilities. Summary of the invention

[0004] The embodiments of the present application provide a multimedia processing chip, an electronic device, and a dynamic image processing method, which can improve the image processing capability of the multimedia processing chip.

[0005] The embodiment of the present application discloses a multimedia processing chip, which includes:

[0006] An image signal processor, used for collecting statistics of status information of image data; and

[0007] A neural network processor for performing neural network algorithm processing on image data;

[0008] The multimedia processing chip is used to pre-process the image data at least through the neural network processor, and send the status information and the pre-processed image data to the application processing chip.

[0009] The present application discloses an electronic device, which includes:

[0010] A multimedia processing chip, which is the multimedia processing chip as described above; and

[0011] The application processing chip is used to obtain the pre-processing result and statistical status information from the multimedia processing chip, and the application processing chip performs post-processing on the pre-processing result based on the status information.

[0012] The present application embodiment discloses a dynamic image processing method, which includes:

[0013] Acquire dynamic image data;

[0014] According to the dynamic image data, statistics of state information of the dynamic image data are collected through a multimedia processing chip, and the dynamic image data is pre-processed;

[0015] Sending the status information counted by the multimedia processing chip and the pre-processed dynamic image data to the application processing chip;

[0016] The pre-processed dynamic image data is post-processed by the application processing chip based on the status information.

[0017] In the embodiment of the present application, before the application processing chip processes the image data such as dynamic image data, it can first collect state information from the image data such as dynamic image data. When the application processing chip processes the image data such as dynamic image data, it can process the image data based on the state information to improve the processing capability of the image data. At the same time, in the embodiment of the present application, the multimedia processing chip first processes the image data, and then the application processing chip further processes the image, which can save the power consumption of the application processing chip. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for describing the embodiments are briefly introduced below.

[0019] Figure 1 A first structural schematic diagram of an image processing device provided in an embodiment of the present application.

[0020] Figure 2 A second structural schematic diagram of the image processing device provided in an embodiment of the present application.

[0021] Figure 3 for Figure 1 A schematic diagram of a first application scenario of the image processing device is shown.

[0022] Figure 4 for Figure 1 A schematic diagram of a second application scenario of the image processing device is shown.

[0023] Figure 5 Schematic diagram of a method for processing a video image by an image processing device provided in an embodiment of the present application

[0024] Figure 6 A schematic diagram of the first structure of the multimedia processing chip provided in an embodiment of the present application.

[0025] Figure 7 A second structural schematic diagram of the multimedia processing chip provided in an embodiment of the present application.

[0026] Figure 8 A third structural schematic diagram of the multimedia processing chip provided in an embodiment of the present application.

[0027] Fig. 9 A fourth structural schematic diagram of the multimedia processing chip provided in an embodiment of the present application.

[0028] Fig.10 A schematic diagram of a first data flow for processing image data by a multimedia processing chip provided in an embodiment of the present application.

[0029] Fig.11 A schematic diagram of a first method for processing image data by a multimedia processing chip provided in an embodiment of the present application.

[0030] Fig.12 A schematic diagram of a second data flow for processing image data by the multimedia processing chip provided in an embodiment of the present application.

[0031] Fig.13 A schematic diagram of a second method for processing image data by the multimedia processing chip provided in an embodiment of the present application.

[0032] Fig.14 A third data flow diagram for processing image data by the multimedia processing chip provided in an embodiment of the present application.

[0033] Fig.15 Schematic diagram of a third method for processing image data by a multimedia processing chip provided in an embodiment of the present application

[0034] Fig.16 A fourth data flow diagram for processing image data by the multimedia processing chip provided in an embodiment of the present application.

[0035] Fig.17 A schematic diagram of a fourth method for processing image data by the multimedia processing chip provided in an embodiment of the present application.

[0036] Fig.18 A fifth structural schematic diagram of the multimedia processing chip provided in an embodiment of the present application.

[0037] Fig.19 A flowchart of an offline static image processing method provided in an embodiment of the present application.

[0038] Fig. 20 This is a flow chart of a method for editing RAW images using a multimedia processing chip according to an embodiment of the present application.

[0039] Fig.21 A flowchart of an offline dynamic image processing method provided in an embodiment of the present application.

[0040] Fig. 22 This is a flow chart of a method for processing image data for video playback using a multimedia processing chip according to an embodiment of the present application.

[0041] Fig.23 A sixth structural schematic diagram of the multimedia processing chip provided in an embodiment of the present application.

[0042] Fig.24 A seventh structural schematic diagram of the multimedia processing chip provided in an embodiment of the present application.

[0043] Fig.25 This is a schematic diagram of the eighth structure of the multimedia processing chip provided in the embodiment of the present application.

[0044] Fig.26 A schematic diagram of the first structure of an electronic device provided in an embodiment of the present application.

[0045] Fig. 27 A second structural schematic diagram of an electronic device provided in an embodiment of the present application.

[0046] Fig.28 A third structural schematic diagram of the electronic device provided in an embodiment of the present application.

[0047] Fig.29 A schematic diagram of a process for processing image data by an image signal processor in an application processing chip provided in an embodiment of the present application.

[0048] Fig.30 A comparison diagram between the embodiment of the present application and the related art.

[0049] Fig.31 A comparison diagram between the embodiment of the present application and the related art.

[0050] Fig.32 A first structural schematic diagram of a circuit board provided in an embodiment of the present application.

[0051] Fig.33 A second structural schematic diagram of a circuit board provided in an embodiment of the present application. DETAILED DESCRIPTION

[0052] The embodiment of the present application provides a multimedia processing chip, an electronic device and a dynamic image processing method. The multimedia processing chip can be integrated into a circuit board such as a mainboard to be applied to an electronic device to process images and improve image quality.

[0053] See also Figure 1 The image processing device 110 can process the acquired data such as RAW data so that other image processors can further process the image data to improve the image quality.

[0054] The image processing device 110 can process static image data, such as static image data acquired by a user in a camera mode, or dynamic image data, such as dynamic image data acquired by a user in a preview mode or a video recording mode.

[0055] It is understandable that both static image data and dynamic image data can be processed by the processor of the platform side (System-on-a-Chip, SoC chip). The platform side can be understood as an application processing chip, and the processor of the platform side can be understood as an image signal processor (Image Signal Processing, ISP) and an application processor (AP, Application Processor). However, the platform side often has limited processing capabilities for image data. As users have higher and higher requirements for image quality, processing image data only through the platform side often cannot meet user needs.

[0056] In order to improve the image quality, it can be understood as the quality of the image when it is displayed. Some embodiments of the present application may provide an image preprocessor (pre-ISP) such as a neural network processor (Neural-network Processing Unit, NPU) to pre-process the image first, and transmit the pre-processing result to the platform end. The platform end uses the processing result of the pre-ISP as input data and performs post-processing. Thereby, the image quality can be improved.

[0057] In the actual research and development process, the present application found that for static image data, the pre-ISP first pre-processes the static image data, and the pre-processing operation generally does not destroy the state information of the static image data. After the pre-ISP pre-processes the static image data, it can be directly transmitted to the platform end, and the platform end can directly post-process the static image data processed by the pre-ISP.

[0058] The status information may be understood as information required by the platform to post-process the image data, that is, the platform may post-process the pre-processing result of the image data based on the status information.

[0059] Among them, the status information may include automatic white balance (AWB) status information, automatic exposure (AE) status information and automatic focus (AF) status information, which can be referred to as 3A status information. The status information can also be understood as status data. It should be noted that the status information is not limited to this. For example, the status information also includes lens shading correction (LSC) status information. Automatic white balance status information can be understood as status information required for white balance processing, automatic exposure status information can be understood as status information required for exposure, automatic focus status information can be understood as status information required for focusing, and lens shading correction status information can be understood as status information required for lens shading correction.

[0060] However, for dynamic image data such as video image data, pre-processing by a pre-ISP such as a neural network processor will destroy the state information of the dynamic image data, such as image color, image brightness, data required for focusing, etc. Even if the pre-ISP transmits the result of its pre-processing of the dynamic image data to the platform side, the pre-processing result obtained after the pre-ISP such as a neural network processor processes the dynamic image data destroys the state information, resulting in the platform side being unable to perform post-processing based on the pre-processing by the pre-ISP.

[0061] Based on this, some embodiments of the present application may use an image processing device such as Figure 1 The statistical module 112 in the image processing device 110 shown performs statistics on the dynamic image data to obtain status information from the dynamic image data. After the pre-ISP, such as a neural network processor, pre-processes the dynamic image data, the platform can post-process the pre-processing result based on the status information obtained by the statistical module 112 of the image processing device 110 to improve the dynamic image quality.

[0062] However, it is understandable that for dynamic image data, whether it is playing a video or recording a video, if the video is stuck, it will have a greater impact on the user. In order to maintain the continuity of the video image as much as possible, or to reduce or even eliminate the problem of video image stuck, some embodiments of the present application optimize the dynamic image during the processing of the dynamic image to reduce or even eliminate the problem of playback stuck.

[0063] Based on this, some embodiments of the present application may use an image processing device such as Figure 1 The optimization module 114 in the image processing device 110 shown optimizes the dynamic image data to solve the problems such as bad pixels in the dynamic image data. The optimized data is then transmitted to the pre-ISP such as a neural network processor, which can speed up the convergence of the neural network processor to increase the time taken by the neural network processor to process a frame of image, thereby ensuring that the neural network processor can process a frame of dynamic image data within a preset time period. The preset time period is, for example, 33 nanoseconds.

[0064] The optimization processing performed by the optimization module 114 on the dynamic image data may include at least one of bad pixel correction (BPC), linearization processing, and black level correction (BLC). The algorithm performed by the optimization module 114 on the dynamic image data may include at least one of a black level correction algorithm, a bad pixel compensation algorithm, and a linearization processing algorithm. The optimization module 114 may perform black level correction on the dynamic image data by executing the black level correction algorithm, the optimization module 114 may perform bad pixel compensation on the dynamic image data by executing the bad pixel compensation algorithm, and the optimization module 114 may perform linearization processing on the dynamic image data by executing the linearization processing algorithm.

[0065] It should be noted that the optimization processing performed by the optimization module 114 on the dynamic image data is not limited to this. For example, the optimization processing performed by the optimization module 114 on the dynamic image data may also include at least one of an image cropping (Crop) processing and an image reduction (Bayerscaler) processing. The algorithm performed by the optimization module 114 on the dynamic image data may include at least one of an image cropping algorithm and an image reduction algorithm. The optimization module 114 may perform the image cropping algorithm to crop the dynamic image, and the optimization module 114 may perform the image reduction algorithm to reduce the dynamic image.

[0066] In some embodiments of the present application, several different optimization modules may be used to respectively execute different algorithms to achieve different optimization results. The optimization module may also be divided into several optimization submodules to respectively execute different algorithms to achieve different optimization results.

[0067] See also Figure 2, the optimization module 114 of the image processing device 110 may include multiple optimization submodules, and the optimization submodules may be defined as optimization units. For example, the optimization module 114 includes a first optimization unit 1142 and a second optimization unit 1144. The first optimization unit 1142 may perform bad pixel compensation on the dynamic image data, and the second optimization unit 1144 may perform linearization processing on the dynamic image data. This ensures that the data optimized by the optimization module 114 accelerates the convergence speed of the pre-ISP, such as the neural network processor, and further ensures that the pre-ISP, such as the neural network processor, can complete the processing of a frame of image within a preset time period to solve the problem of playback freeze.

[0068] It is understandable that the optimization units of the optimization module 114 are not limited to the first optimization unit 1142 and the second optimization unit 1144. For example, the optimization module 114 may also include a third optimization unit, which may perform black level correction on the dynamic image data. The optimization module 114 may also include a fourth optimization unit that may perform image cropping processing on the dynamic image data, and the optimization module 114 may also include a fifth optimization unit that may perform image reduction processing on the dynamic image data.

[0069] It should be noted that the number and functions of the optimization units of the optimization module 114 are not limited to this, and the above are only some examples of the optimization units of the optimization module in some embodiments of the present application. The functional sub-modules that can accelerate the convergence speed of pre-ISP such as neural network processors in processing dynamic image data after the optimization module 114 optimizes the dynamic image data are all within the protection scope of the present application.

[0070] It should also be noted that the optimization processing of dynamic image data by the optimization module 114 may not be for accelerating the convergence speed of the pre-ISP such as the neural network processor in processing dynamic image data. The optimization processing of dynamic image data by the optimization module 114 may be designed according to actual needs.

[0071] The image processing device 110 provided in the embodiment of the present application can also perform statistics on the static image data to obtain the status information. The image processing device 110 provided in the embodiment of the present application can also perform optimization processing on the static image data to improve the static image quality.

[0072] The above is the definition of the embodiment of the present application from the perspective of the statistical module 112 and the optimization module 114. In order to further illustrate the flow of the image processing device 110 defined in the embodiment of the present application during data processing, the image processing device 110 of the embodiment of the present application is combined with other circuits for illustration.

[0073] See also Figure 3The image processing device 110 can be connected to one or more cameras 120 to obtain image data such as dynamic image data collected by the camera 120 from the camera 120. It can also be understood that the image processing device 110 is connected to the camera 120 and can receive dynamic image data sent by the camera 120 to the image processing device 110. The dynamic image data can be divided into two paths, one path can be transmitted to the statistical module 112, and the other path can be transmitted to the optimization module 114.

[0074] It is understood that the electrical connection between the two devices defined in the embodiment of the present application can be understood as the two devices being connected by a signal such as a wire to achieve signal transmission. Of course, it can also be understood as the two devices being connected together, such as being welded together through a welding point.

[0075] After receiving the dynamic image data, the statistical module 112 can count some information based on the dynamic image data, which can be defined as status information, such as 3A status information, etc. When the statistical module 112 completes the statistical data, that is, when the statistical module 112 counts the status information, the status information can be directly sent to the first image processor 130. The first image processor 130 can be understood as a processor on the platform side such as an ISP and an AP.

[0076] After receiving the dynamic image data, the optimization module 114 may perform one or more optimization processes on the dynamic image data, such as bad pixel compensation, linearization, etc. After the optimization module 114 completes the optimization process on the dynamic image data, the optimized dynamic image data may be transmitted to the second image processor 140. The second image processor 140 may be understood as a pre-ISP, such as a neural network processor.

[0077] It should be noted that in the actual production process, the first image processor 130, the second image processor 14 and the image processing device 110 need to be completed separately, which will increase the cost in the production stage. In the signal transmission and processing stage, some signals will be transmitted from one device to another, which will increase the time and power consumption.

[0078] Based on this, in order to save cost, time and power consumption, some other embodiments of the present application may integrate the second image processor 140, the statistics module 112 and the optimization module 114 into one device.

[0079] See also Figure 4 , Figure 4 and Figure 3The difference is that the second image processor 140, the statistical module 112 and the optimization module 114 are integrated in one device, such as the image processing device 110. Therefore, in terms of structure, the statistical module 112, the optimization module 114 and the second processor 140 can be integrated together, which can save costs, speed up the rate of mutual data transmission, and save time and power consumption.

[0080] In summary, some embodiments of the present application can not only count some status information through the image processing device 110, but also optimize the dynamic image data. Thereby, the convergence speed of per-ISP, such as the neural network processor, can be accelerated to ensure that it completes the processing of a frame of image within a preset time period, and it can also ensure that the first image processor 130 can perform post-processing on the basis of pre-ISP pre-processing based on the status information counted by the statistical module 112. Thereby improving the quality of dynamic images. After processing the dynamic image data in RAW format, the first image processor 130 can convert the format of the dynamic image data, such as converting the RAW format image data into YUV format image data. The first image processor 130 can also process the YUV format image data, such as RGBToYUV. The first image processor 130 can display the processed image data on the display screen, and store it in the memory.

[0081] In order to further illustrate the flow of the image processing device 110 in the process of processing data as defined in the embodiment of the present application, the following description is made from the perspective of the method of processing data by the image processing device 110.

[0082] See also Figure 5 , the dynamic image processing method includes:

[0083] 1001, the image processing device 110 obtains dynamic image data. The image processing device 110 may obtain dynamic image data from the camera 120. The dynamic image data may be RAW data.

[0084] 1002, the statistical module 112 of the image processing device 110 collects statistical status information from the dynamic image data. The status information may include 3A status information.

[0085] 1003, the image processing device 110 sends the status information counted by the statistical module to the first image processor 130. The first image processor 130 can be understood as the AP and ISP of the platform, and can perform image processing based on the status information. For example, if the status information includes 3A status information, the first image processor 130 can perform 3A processing based on the 3A status information. 3A processing can be understood as processing based on the 3A status information.

[0086] 1004, the optimization module 114 of the image processing device 110 performs optimization processing on the dynamic image data. The optimization processing may include at least one of bad pixel compensation, linearization processing and black level correction of the dynamic image data.

[0087] 1005, the image processing device 110 sends the optimized dynamic image data to the second image processor 140. The second image processor 140 can be understood as a neural network processor. The optimized dynamic image data is given to the second image processor 140, so that the second image processor 140 can process a frame of dynamic image data within a preset time period, or in other words, it can speed up its convergence speed for dynamic image data. In this way, the second image processor 140 can transmit the processed dynamic image data to the first image processor 130 in real time to solve the problem of playback freeze.

[0088] The neural network processor can perform neural network algorithm processing on the dynamic image data after optimization processing, and can transmit the processed image data to the first image processor 130. The first image processor 130 can perform post-processing such as 3A processing on the dynamic image data after the neural network algorithm processing based on the status information.

[0089] It should be noted that when the second image processor, such as a neural network processor, performs some algorithms on the dynamic image data, the bit width of the data will be greater than the bit width of the image data processed by the first image processor. Based on this, after receiving the result of the second image processor, such as a neural network processor, processing the dynamic image data, the embodiment of the present application can adjust the bit width of the processing result of the second image processor through the optimization module 114, so that the data after the bit width adjustment conforms to the bit width of the data processed by the first image processor. Then, the dynamic image data after the bit width adjustment is sent to the first image processor, so that the first image processor can further process the data after the bit width adjustment based on the reference data.

[0090] The following is a detailed explanation from the perspective of the integration of the statistical module, optimization module and neural network processor.

[0091] See also Figure 6 The multimedia processing chip, such as the multimedia processing chip 200, can process the acquired image data, such as RAW data, to improve the image quality. It should be noted that the multimedia processing chip 200 can transmit its processing results to the application processing chip so that the application processing chip can post-process the image data for display or storage. The image data can also be understood as image information.

[0092] RAW data retains more details than other image data such as YUV data.

[0093] The multimedia processing chip 200 may include a neural network processor (Neural-network Processing Unit, NPU) 220, which may enhance the image data acquired by the multimedia processing chip 200. The neural network processor 220 may run an artificial intelligence training network to process an image algorithm to enhance the image data. The neural network processor 220 processes image data with high efficiency and significantly improves image quality.

[0094] In some embodiments of the present application, the neural network processor 220 may be a dedicated processor for processing images, which may be referred to as a dedicated processor. Hardening may be performed during hardware configuration processes such as circuit layout and programming, thereby ensuring the stability of the neural network processor 220 in the process of processing image data, and reducing the power consumption and time required for the neural network processor 220 to process image data. It is understandable that when the neural network processor 220 is a dedicated processor, its function is to process image data, and it cannot process other data such as text information. It should be noted that in some other embodiments, the neural network processor 220 may also process other information such as text information.

[0095] The neural network processor 220 may process the image data by reading the data blocks in a row manner and processing the data blocks in a row manner. For example, the neural network processor 220 may read the data blocks in a multi-row manner and process the data blocks in a multi-row manner. It is understandable that a frame of image may have a multi-row data block, that is, the neural network processor 220 may process a portion of a frame of image such as Frames are processed, where n is a positive integer, such as 2, 4, 5, etc. When the neural network processor 220 has not completely processed a frame of image, the neural network processor 220 can have a built-in cache to store the data of multiple rows of data blocks processed by the neural network processor 220 in the process of processing a frame of image. After the neural network processor 220 completes processing a frame of image, the neural network processor 220 can write the processed data to a memory such as the memory 230 of the multimedia processing chip 200. The memory 230 can be built into the multimedia processing chip 200 or external. A storage controller can be used to realize data transmission.

[0096] The neural network processor 220 can process RAW data. It can be understood that the information of RAW data is relatively complete. Compared with processing YUV data, the neural network processor 220 can improve the image quality in more details by processing RAW data.

[0097] It should be noted that the neural network processor 220 can complete the processing in the data stream according to the preset time. The preset time is, for example, 30fps = 33ms (milliseconds). In other words, the preset time for the neural network processor 220 to process a frame of image is 33ms, so that the neural network processor 220 can realize real-time data transmission on the basis of fast processing of image data.

[0098] It is understandable that some neural network processors process images by loading a frame of image from a memory storing image data and performing corresponding algorithm processing on the frame of image. During the processing, the temporary data calculated by the convolution layer of the neural network processor often needs to be saved in the memory. It can be seen that, compared with some neural network processors, the neural network processor 220 defined in some embodiments of the present application is a dedicated neural network processor, which can speed up the processing of image data and ensure that the processing of a frame of image is completed within a preset time.

[0099] The neural network processor 220 can process dynamic image data, such as dynamic image data acquired by the user in video recording mode. The neural network processor 220 may include algorithms for processing dynamic image data, such as night scene algorithms, HDR algorithms, blur algorithms, noise reduction algorithms, super-resolution algorithms, etc. Among them, the dynamic image data may include image data of recorded videos, image data of video playback, and data of preview images. In the embodiment of the present application, dynamic image data may be understood as video image data.

[0100] The neural network processor 220 may also process static image data, such as static image data acquired by the user in the camera mode. The neural network processor 220 may include algorithms for processing static image data, such as HDR algorithms, night scene algorithms, blur algorithms, noise reduction algorithms, super-resolution algorithms, semantic segmentation algorithms, etc. It should be noted that the static image data may also include images displayed when the photo album application is opened.

[0101] The neural network processor 220 defined in the embodiment of the present application can process both dynamic image data and static image data, so that the multimedia processing chip 200 can be applied to different scenarios, such as photo taking scenarios and video recording scenarios. It should be noted that the neural network processor 220 defined in the embodiment of the present application can also only process dynamic image data without processing static image data. The following is an example of the neural network processor 220 processing dynamic image data.

[0102] It should be noted that after the multimedia processing chip 200 obtains image data such as dynamic image data, if the neural network processor 220 directly processes the dynamic image data, the neural network processor 220 processes the dynamic image data according to the preset algorithm to obtain the processing result. However, often after the neural network processor 220 processes the dynamic image data through the preset algorithm, the dynamic image data will be distorted. If the multimedia processing chip 200 sends the distorted data processed by the neural network processor 220 to the application processing chip, it will cause the application processing chip to have incorrect status information of the dynamic image data, such as the status information required for autofocus, which will lead to focus failure and the camera being unable to focus. Among them, the dynamic image data can be understood as data received by the multimedia processing chip 200 but not processed. For example, the data sent by the image sensor to the multimedia processing chip 200 is defined as the initial dynamic image data.

[0103] Based on this, in some embodiments of the present application, a statistical module can be integrated in the multimedia processing chip 200, and the statistical module is used to count the data required by the application processing chip for image data processing, or the statistical module is used to count the state information required by the application processing chip for image data processing. After the statistical module completes the statistics of the data required by the application processing chip, the statistical module can send the statistical data to the application processing chip to ensure that the application processing chip can successfully complete image data processing such as 3A processing.

[0104] Please continue reading Figure 6 In some embodiments of the present application, the statistical module can be integrated into an image signal processor (Image Signal Processing, ISP) 210, or the multimedia processing chip 200 also includes an image signal processor 210, and the image signal processor 210 includes a statistical module 212. After the multimedia processing chip 200 obtains the initial dynamic image data, it can be preferentially transmitted to the image signal processor 210, and the statistical module 212 of the image signal processor 210 performs statistics on the initial dynamic image data to calculate the state information required by the application processing chip, such as 3A state information. This ensures that the application processing chip performs post-processing on the processing results sent by the multimedia processing chip 200 to the application processing chip based on the state information counted by the statistical module 212.

[0105] It is understandable that the status information counted by the statistics module 212 of the image signal processor 210 is not limited to the 3A status information, for example, the statistics module 212 of the image signal processor 210 counts status information such as lens shading correction status information.

[0106] It should also be noted that after the multimedia processing chip 200 obtains the dynamic image data, if the neural network processor 220 directly processes the dynamic image data, the neural network processor 220 processes the dynamic image data according to the preset algorithm to obtain the processing result. However, dynamic image data often have problems such as bad pixels. The neural network processor 220 directly processes the dynamic image data through the preset algorithm, which slows down the convergence speed of the neural network processor 220, thereby reducing the time required for the neural network processor 220 to process a frame of image, making it difficult to achieve the purpose of quickly processing image data and effectively improving image quality.

[0107] Based on this, in some embodiments of the present application, an optimization module can be integrated in the image signal processor 210, and the optimization module can perform a first preprocessing on the dynamic image data, such as bad pixel compensation, to obtain a first processing result. Then, the neural network processor 220 performs a second preprocessing on the first preprocessing result, which can not only solve the problem of bad pixels in the image, but also improve the convergence speed of the neural network algorithm of the neural network processor 220, and ensure that the neural network processor 220 can complete the processing of a frame of image within a preset time, thereby achieving the purpose of fast and real-time image processing.

[0108] See also Figure 7 The image signal processor 210 further includes an optimization module 214. The optimization module 214 can perform bad pixel compensation on the dynamic image data. The optimization module 214 can execute a bad pixel compensation algorithm to achieve bad pixel compensation on the dynamic image data. The optimization module 214 can perform linearization processing on the dynamic image data. The optimization module 214 can execute a linearization processing algorithm to achieve linearization processing on the dynamic image data. The optimization module 214 can perform black level correction on the dynamic image data. The optimization module 214 can execute a black level correction algorithm to achieve black level correction on the dynamic image data.

[0109] It is understandable that the first pre-processing of the dynamic image by the optimization module 214 of the image signal processor 210 is not limited to this. For example, the optimization module 214 performs image cropping processing on the initial image data, and the optimization module 214 can execute an image cropping algorithm to achieve cropping of the dynamic image data. For another example, the optimization module 214 performs image reduction processing on the dynamic image data, and the optimization module 214 can execute an image reduction algorithm to achieve reduction of the dynamic image data.

[0110] It should also be noted that after the neural network processor 220 processes the image data, the multimedia processing chip 200 can directly send the data processed by the neural network processor 220 to the application processing chip. However, in some cases, the data processed by the neural network processor 220 often differs in bit width from the bit width of the data processed by the application processing chip. For example, the bit width of the dynamic image data processed by the neural network processor 220 using the video HDR (High-Dynamic Range) algorithm is 20 bits, while the bit width of the data to be processed by the application processing chip is 14 bits. Therefore, the bit width of the image data processed by the neural network processor 220 exceeds the bit width of the data to be processed by the application processing chip. Therefore, it is necessary to perform a bit width adjustment operation on the data processed by the neural network processor 220 so that the bit width of the data transmitted from the multimedia processing chip 200 to the application processing chip is the same.

[0111] Based on this, in some embodiments of the present application, after the neural network processor 220 of the multimedia processing chip 200 processes the dynamic image data, the optimization module 214 of the image signal processor 210 can first perform a bit width adjustment process (tonemapping), so that the bit width of the data adjusted by the optimization module 214 is the same as the bit width of the data to be processed by the application processing chip. This ensures that after the multimedia processing chip 200 transmits the data processed by the dynamic image data to the application processing chip, the application processing chip can perform post-processing on the data to improve the image quality.

[0112] In some embodiments of the present application, several different optimization modules may be used to respectively execute different algorithms to achieve different optimization results. The optimization module 214 may also be divided into several optimization submodules to respectively execute different algorithms to achieve different optimization results. For example, a submodule of the optimization module 214 may perform bad pixel compensation on dynamic image data, a submodule of the optimization module 214 may perform linearization processing on dynamic image data, a submodule of the optimization module 214 may perform black level correction on dynamic image data, a submodule of the optimization module 214 may perform image cropping processing on dynamic image data, a submodule of the optimization module 214 may perform image reduction processing on dynamic image data, and a submodule of the optimization module 214 may perform bit width adjustment processing on image data.

[0113] It should be noted that the optimization module 214 may have one or more of the above submodules, and the optimization module 214 may perform one or more of the above operations, thereby ensuring that the data transmitted from the multimedia processing chip 200 to the application processing chip can be further processed by the application processing chip. Of course, it can also ensure that the neural network processor 220 can accelerate convergence to achieve the purpose of improving image quality. It is understandable that the optimization module 214 may also have other submodules, which will not be illustrated one by one here.

[0114] Please continue reading Figure 6 and Figure 7 The multimedia processing chip 200 may include a first interface 201 and a second interface 202. Both the first interface 201 and the second interface 202 may be a mobile industry processor interface (MIPI). The first interface 201 may receive image data such as RAW data, such as the first interface 201 may receive RAW data acquired from a camera. The image data such as RAW data received by the first interface 201 may be image data, that is, the image data received by the first interface 201 is image data that has not been processed, and specifically, the original image data may be understood as image data that has not been processed by an image processor. After receiving image data such as original image data, the first interface 201 may transmit the image data to the image signal processor 210.

[0115] The second interface 202 may receive the result of image data processing by the image signal processor 210, and the second interface 202 may also receive the result of image data processing by the neural network processor 220. The second interface 202 may be connected to the application processing chip to transmit the image data such as dynamic image data received by the second interface 202 to the application processing chip.

[0116] The first interface 201 and the second interface 202 can be connected through the image signal processor 210. The data received by the first interface 201 can be divided into at least two paths for transmission, such as one path of data is transmitted to the statistical module 212 of the image signal processor 210, and the other path of data is stored in the memory 230. Or the other path of data is processed by the optimization module 214. The second interface 202 can transmit the data counted by the statistical module 212, and the second interface 202 can also transmit the data processed by the optimization module 214.

[0117] Please continue reading Figure 6 and Figure 7, the memory 230 stores various data and instructions of the multimedia processing chip 200. For example, the memory 230 can store original image data, the memory 230 can store data processed by the optimization module 214, the memory 230 can store data processed by the neural network processor 220, and the memory 230 can also store the operating system of the multimedia processing chip 200. The number of memories 230 can be one, two, three, or even more. The type of memory 230 can be a static memory or a dynamic memory, such as DDR (Double Data Rate SDRAM). The memory 230 can be built-in or external. For example, in the packaging process, the image signal processor 210, the neural network processor 220 and other devices are first packaged, and then packaged with the memory 230.

[0118] The data transmission of the multimedia processing chip 200 may be implemented by one or more memory access controllers.

[0119] See also Figure 8 The multimedia processing chip 200 may further include a storage access controller 250, which may be a direct memory access controller (DMA), which has high efficiency in moving data and can move large data. The direct memory access controller 250 may move data from one address space to another address space. For example, the direct memory access controller 250 may move data stored in the memory 230 to the neural network processor 220.

[0120] The direct storage access controller 250 may include an AHB (Advanced High performance Bus) direct storage access controller, or may include an AXI (Advanced eXtensible Interface) direct storage access controller.

[0121] Please continue reading Figures 6 to 8 , various components of the multimedia processing chip 200 can be connected by the system bus 240. For example, the image signal processor 210 is connected to the system bus 240, the neural network processor 220 is connected to the system bus 240, the memory 230 is connected to the system bus 240, and the storage access controller 250 is connected to the system bus 240.

[0122] The multimedia processing chip 200 may be controlled by a processor to implement the operation of the multimedia processing chip 200 system.

[0123] See also Fig. 9The multimedia processing chip 200 may further include a central processing unit (CPU) 240, which is used to control the operation of the system of the multimedia processing chip 200, such as peripheral parameter configuration, control interrupt response, etc.

[0124] In order to further illustrate the processing of image data, especially dynamic image data, by the multimedia processing chip provided in the embodiment of the present application, Figures 5 to 8 , the following describes the data flow and method of processing data from the perspective of a multimedia processing chip.

[0125] See also Fig.10 and Fig.11 The method for processing data by the multimedia processing chip 200 includes:

[0126] 2011 , the first interface 201 of the multimedia processing chip 200 receives original data, such as dynamic image data.

[0127] 2012, the raw data is transmitted to the statistical module 212 of the image signal processor 210, and the statistical module 212 performs statistical processing on the raw data received to obtain statistical status information. It should be noted that the raw data can also be stored in the memory 230 first, and then the statistical module 212 performs statistical processing on the raw data stored in the memory 230 to obtain statistical status information.

[0128] 2013, the data collected by the statistics module 212 is transmitted through the second interface 202, such as to the application processing chip. It should be noted that the data collected by the statistics module 212, such as the status information, can also be stored in the memory 230 first and then transmitted through the second interface 202.

[0129] 2014, the original data is stored in the memory 230 through another path.

[0130] In 2015, the original data stored in the memory 230 is sent to the neural network processor 220, and is processed by the neural network processor 220. Alternatively, the neural network processor 220 obtains the original data from the memory 230, and processes the original data, such as using a neural network algorithm.

[0131] 2016, storing the processed data of the neural network processor 220 in the memory 230. Here, the result of the data processing by the neural network processor 220 can be defined as a preprocessing result.

[0132] 2017, the data processed by the neural network processor 220 is transmitted through the second interface 202, such as to the application processing chip.

[0133] The above is the first way for the multimedia processing chip 200 to perform data processing in the embodiment of the present application. The application processing chip can further process the processing results of the neural network processor 220 based on the status information to improve image quality, such as improving the quality of video playback.

[0134] See also Fig.12 and Fig.13 The method for processing data by the multimedia processing chip 200 includes:

[0135] 2021, the first interface 201 of the multimedia processing chip 200 receives original data, such as dynamic image data.

[0136] 2022, the raw data is transmitted to the statistical module 212 of the image signal processor 210, and the statistical module 212 performs statistical processing on the raw data received to obtain statistical status information. It should be noted that the raw data can also be stored in the memory 230 first, and then the statistical module 212 performs statistical processing on the raw data stored in the memory 230 to obtain statistical status information.

[0137] 2023, the data collected by the statistics module 212 is transmitted through the second interface 202, such as to the application processing chip. It should be noted that the state information collected by the statistics module 212 can also be stored in the memory 230 first, and then transmitted through the second interface 202.

[0138] 2024, the original data is transmitted to the optimization module 214 through another path, and the optimization module 214 performs optimization processing, such as bad pixel compensation, linearization processing, etc.

[0139] 2025, the data processed by the optimization module 214 is sent to the neural network processor 220, and the neural network processor 220 processes it. It should be noted that the data processed by the optimization module 214 can be sent to the memory 230 first, and then the data stored in the memory 230 and processed by the optimization module 214 is transmitted to the neural network processor 220, and the neural network processor 220 processes the data processed by the optimization module 214.

[0140] 2026, storing the processed data of the neural network processor 220 in the memory 230. Here, the result of the data processing by the neural network processor 220 can be defined as a preprocessing result.

[0141] 2027, the data processed by the neural network processor 220 is transmitted through the second interface 203, such as to the application processing chip.

[0142] The above is the second way of data processing by the multimedia processing chip 200 of the embodiment of the present application. The multimedia processing chip 200 can transmit the original data to the statistical module 212 for data statistics and the optimization module 214 for optimization processing in different channels. The optimized data can be processed by the neural network processor 220 again. The data and status information processed by the neural network processor 220 are transmitted to the application processing chip, which can not only ensure that the application processing chip further processes the processing results of the neural network processor 220 based on the status information to improve the image quality, such as improving the quality of video playback. It can also speed up the convergence speed of the neural network processor 220 to improve the smoothness of video playback.

[0143] It should also be noted that after the optimization module 214 of the multimedia processing chip 200 performs optimization processing on the data, such as bad pixel compensation and linearization processing, the application processing chip does not need to perform corresponding processing on the image data it receives. For example, if the optimization module 214 performs bad pixel compensation, linearization processing and black level correction on the image data, the application processing chip does not need to perform bad pixel compensation, linearization processing and black level correction on the image data it receives, thereby reducing the power consumption of the application processing chip.

[0144] See also Fig.14 and Fig.15 The method for processing data by the multimedia processing chip 200 includes:

[0145] 2031 , the first interface 201 of the multimedia processing chip 200 receives original data, such as dynamic image data.

[0146] At 2032, the raw data is transmitted to the statistical module 212 of the image signal processor 210, and the statistical module 212 performs statistical processing on the raw data received to obtain statistical data such as status information. It should be noted that the raw data can also be stored in the memory 230 first, and then the statistical module 212 performs statistical processing on the raw data stored in the memory 230 to obtain statistical status information.

[0147] 2033, the data collected by the statistics module 212 is transmitted through the second interface 202, such as to the application processing chip. It should be noted that the state information collected by the statistics module 212 can also be stored in the memory 230 first, and then transmitted through the second interface 202.

[0148] 2034, store the original data into the memory 230 through another path.

[0149] 2035, the original data stored in the memory 230 is sent to the neural network processor 220, and the neural network processor 220 processes it. Alternatively, the neural network processor 220 obtains the original data from the memory 230, and processes the original data, such as neural network algorithm processing.

[0150] 2036, the processed data of the neural network processor 220 is transmitted to the optimization module 214, and the optimization module 214 performs bit width adjustment processing on the data processed by the neural network processor 220, so that the adjusted bit width is the same as the bit width of the data required to be processed by the application processing chip. The result of the data processing by the optimization module 214 can be defined as the preprocessing result. It should be noted that the data processed by the neural network processor 220 can be sent to the memory 230 first, and then the data stored in the memory 230 and processed by the neural network processor 220 can be transmitted to the optimization module 214, and the optimization module 214 can perform bit width adjustment processing on the data processed by the neural network processor 220.

[0151] 2037 , the data processed by the optimization module 214 after the bit width adjustment is transmitted through the second interface 203 , such as to the application processing chip.

[0152] The above is the third way for the multimedia processing chip 200 to process data in the embodiment of the present application, which can ensure that the application processing chip further processes the data after bit width adjustment based on the status information to improve image quality, such as improving video playback quality.

[0153] See also Fig.16 and Fig.17 The method for processing data by the multimedia processing chip 200 includes:

[0154] 2041 , the first interface 201 of the multimedia processing chip 200 receives original data, such as dynamic image data.

[0155] At 2042, the raw data is transmitted to the statistical module 212 of the image signal processor 210, and the statistical module 212 performs statistical processing on the raw data received to obtain statistical data such as status information. It should be noted that the raw data can also be stored in the memory 230 first, and then the statistical module 212 performs statistical processing on the raw data stored in the memory 230 to obtain statistical status information.

[0156] 2043, transmit the data collected by the statistics module 212 through the second interface 202, such as to the application processing chip. It should be noted that the state information collected by the statistics module 212 can also be stored in the memory 230 first, and then transmitted through the second interface 202.

[0157] 2044, the original data is transmitted to the optimization module 214 through another path, and the optimization module 214 performs the first optimization processing, such as bad pixel compensation, linearization processing, black level correction, etc.

[0158] 2045, the data after the first optimization processing by the optimization module 214 is sent to the neural network processor 220, and the neural network processor 220 processes it. It should be noted that the data processed by the optimization module 214 can be sent to the memory 230 first, and then the data stored in the memory 230 and optimized by the optimization module 214 for the first time is transmitted to the neural network processor 220, and the neural network processor 220 processes the data after the first optimization processing by the optimization module 214.

[0159] 2046, the data processed by the neural network processor 220 is transmitted to the optimization module 214, and the optimization module 214 performs a second optimization process on the data processed by the neural network processor 220. The result of the second optimization process performed by the optimization module 214 can be defined as a preprocessing result. It should be noted that the data processed by the neural network processor 220 can be first stored in the memory 230, and then the data stored in the memory 230 and processed by the neural network processor 220 is transmitted to the optimization module 214, and the optimization module 214 performs a second optimization process on the data processed by the neural network processor 220. The second optimization process can include adjusting the bit width of the data so that the adjusted bit width is the same as the bit width of the data required to be processed by the application processing chip.

[0160] 2047, transmitting the data after the second optimization processing by the optimization module 214 through the second interface 203, such as transmitting it to the application processing chip.

[0161] The above is the fourth way of data processing by the multimedia processing chip 200 of the embodiment of the present application, which can ensure that the application processing chip further processes the data after the bit width adjustment based on the state information to improve the image quality, such as improving the quality of video playback. It can also speed up the convergence speed of the neural network processor 220 to improve the smoothness of video playback.

[0162] Regarding the above four data processing methods of the multimedia processing chip 200 in the embodiment of the present application, it is also necessary to explain that when the multimedia processing chip 200 receives the image data, the main control processor 260 can determine whether the image data has problems such as bad pixels. If so, the optimization module 214 can be started to perform bad pixel optimization processing on the image data first. If not, the neural network processor 220 can directly process it. After the neural network processor 220 completes the data processing, the main control processor 260 can determine whether the bit width of the data processed by the neural network processor 220 is the same as the preset bit width. If they are the same, the data processed by the neural network processor 220 can be directly transmitted to the application processing chip. If they are not the same, the optimization processing of bit width adjustment can be performed through the optimization module 214, so that the bit width of the data after the bit width adjustment by the optimization module 214 is the same as the preset bit width. It can be understood that the preset bit width can be understood as the bit width required for the application processing chip to process the data.

[0163] It should be noted that the connection method between the multimedia processing chip 200 and other devices such as the application processing chip in the embodiment of the present application is not limited to this. For example, the multimedia processing chip 200 may also include a third interface connected to the application processing chip.

[0164] See also Fig.18 The multimedia processing chip 200 may further include a third interface 203, which may be referred to as an interconnect bus interface, such as a high-speed interconnect bus interface (Peripheral Component Interconnect Express, PCIE) 203, which may also be referred to as a high-speed peripheral component interconnect interface, an external device interconnect bus interface, which is an interface of a high-speed serial computer expansion bus standard. It should be noted that the third interface 203 may also be a low-speed interconnect bus interface.

[0165] The third interface 203 is connected to the system bus 240, and the third interface 203 can realize data transmission with other devices through the system bus 240. For example, the third interface 203 can receive the result of image data processing by the image signal processor 210, and the third interface 203 can also receive the result of image data processing by the neural network processor 220. The third interface 203 can also be connected to the application processing chip to transmit the data processed by the multimedia processing chip 200 to the application processing chip.

[0166] The third interface 203 can transmit image data offline. For example, the third interface 203 can transmit static image data offline, and the third interface 203 can also transmit dynamic image data offline. The multimedia processing chip 200 of the embodiment of the present application can not only process the image data collected by the camera, but also process static image data and / or dynamic image data offline to enhance the image quality and achieve the quality of video playback.

[0167] See also Fig.19 , the offline static image processing method includes:

[0168] 3011, receiving an album viewing instruction. The album viewing instruction may be received by an application processor of an electronic device to which the multimedia chip 200 is applied, such as a smart phone.

[0169] 3012, determine whether to enter the picture enhancement mode according to the album viewing instruction. Whether to enter the picture enhancement mode can be determined by the application processor of the electronic device to which the multimedia chip 200 is applied, such as a smart phone. For example, after the user enters the album interface, the album interface displays two virtual controls, "enhanced mode" and "normal mode". When the user touches the "enhanced mode" virtual control, the application processor determines to enter the enhanced mode, and executes step 3013. When the user touches the "normal mode" virtual control, the application processor determines not to enter the enhanced mode, and executes 3016. It should be noted that the method of determining whether to enter the picture enhancement mode is not limited to this, it is only for example.

[0170] The picture enhancement mode can be understood as a mode for improving picture quality, that is, a mode in which the multimedia processing chip 200 defined in the embodiment of the present application processes picture data. The normal mode can be understood as a mode in which picture data is not processed by the multimedia processing chip 200 defined in the present application.

[0171] 3013, sending the image data to be displayed to the multimedia processing chip 200. An application processor of an electronic device such as a smart phone to which the multimedia chip 200 is applied may issue an instruction to send the image data to be displayed to the multimedia processing chip 200.

[0172] 3014, the multimedia processing chip 200 performs enhancement processing on the image data to be displayed to improve the image quality.

[0173] 3015, displaying the picture enhanced by the multimedia processing chip 200. The picture enhanced by the multimedia processing chip 200 may be displayed on a display screen of an electronic device to which the multimedia chip 200 is applied, such as a smart phone.

[0174] 3016, display the picture. The picture can be directly displayed on the display screen of the electronic device to which the multimedia chip 200 is applied, such as a smart phone, without being enhanced by the multimedia processing chip 200.

[0175] For example, when a user uses an electronic device such as a smart phone to open an album, the application processing chip can transmit the image data of the album photos to the multimedia processing chip 200 through the third interface 203, and the multimedia processing chip 200 can perform RAW image editing on the image data. After the multimedia processing chip 200 completes the processing of the RAW image data, it transmits the processed data through the third interface 203 to be displayed on the display screen of the electronic device. Thus, the multimedia processing chip 200 of some embodiments of the present application can realize the processing of the RAW images of the gallery.

[0176] It should be noted that when a user uses an electronic device such as a smart phone to capture an image, the image can be stored as RAW data so that the multimedia processing chip 200 can perform RAW image editing on the photos in the album or the photo library.

[0177] See also Fig. 20 The method of editing and processing RAW images using a multimedia processing chip includes:

[0178] 3021, an application processor of the application processing chip receives a first instruction to open an album;

[0179] 3022 , the application processor of the application processing chip transmits the RAW image data of the photos to be processed in the album to the multimedia processing chip 200 through the third interface 203 according to the first instruction.

[0180] 3023, the multimedia processing chip 200 performs RAW image editing processing on the RAW image data.

[0181] 3024 , the multimedia processing chip 200 transmits the RAW image data after performing RAW image editing processing to the external memory through the third interface 203 .

[0182] It should be noted that after the multimedia processing chip 200 performs RAW image editing processing on the RAW image data, it can be transmitted to an external memory, such as a memory used by the electronic device to store photos in an album, through the third interface 203. Then, the photos processed by the multimedia processing chip 200 on the RAW image data can be displayed on the display screen of the electronic device. It can be understood that the external memory can be understood as a memory outside the multimedia processing chip.

[0183] See also Fig.21, the offline dynamic image processing method includes:

[0184] 4011, receiving a play instruction; the play instruction may be received by an application processor of an electronic device to which the multimedia chip 200 is applied, such as a smart phone.

[0185] 4012, determine whether to enter the video enhancement mode according to the playback instruction. The application processor of the electronic device to which the multimedia chip 200 is applied, such as a smart phone, can determine whether to enter the video enhancement mode. For example, after the user enters the video interface, the video interface displays two virtual controls, "enhanced mode" and "normal mode". When the user touches the "enhanced mode" virtual control, the application processor determines to enter the enhancement mode, and executes step 4013. When the user touches the "normal mode" virtual control, the application processor determines not to enter the enhancement mode, and executes 4016. It should be noted that the method for determining whether to enter the video enhancement mode is not limited to this, and it is only for example.

[0186] The video enhancement mode can be understood as a mode for improving video playback quality, that is, a mode in which the multimedia processing chip 200 defined in the embodiment of the present application processes video playback data. The normal mode can be understood as a mode in which video playback data is not processed by the multimedia processing chip 200 defined in the present application.

[0187] 4013, sending the video data to be played to the multimedia processing chip according to the play instruction. The instruction may be issued by an application processor of an electronic device such as a smart phone to which the multimedia chip 200 is applied, so as to send the video data to be played to the multimedia processing chip 200.

[0188] 4014, the multimedia processing chip 200 performs enhancement processing on the video data to be played, so as to improve the video playing quality.

[0189] 4015, play the video enhanced by the multimedia processing chip 200. The video data enhanced by the multimedia processing chip 200 can be played by the display screen of an electronic device such as a smart phone to which the multimedia chip 200 is applied.

[0190] 4016 , play the video. The video can be directly played by the display screen of the electronic device to which the multimedia chip 200 is applied, such as a smart phone, without being enhanced by the multimedia processing chip 200 .

[0191] For example, when a user uses an electronic device such as a smart phone to play a video, or when the electronic device is in a video playing mode, the application processing chip can transmit the image data of the played video to the multimedia processing chip 200 through the third interface 203, and the multimedia processing chip 200 can process the image data, such as processing the image data through the neural network processor 220. The resolution of video playback can be improved, and the problem of particles appearing during video playback can be solved. When the multimedia processing chip 200 completes the processing of the image data, the multimedia processing chip 200 transmits the processed data through the third interface 203 to play it through the display screen of the electronic device. Thus, the multimedia processing chip 200 of some embodiments of the present application can realize the processing of video playback.

[0192] See also Fig. 22 The method of processing the image data of video playback using a multimedia processing chip includes:

[0193] 4021, an application processor of the application processing chip receives a second instruction for video playback;

[0194] 4022 , the application processor of the application processing chip transmits the image data in the video playback process to the multimedia processing chip 200 through the third interface 203 according to the second instruction.

[0195] 4023, the multimedia processing chip 200 performs enhancement processing on the image data during the video playback process through the neural network processor 220, such as SR (Super Resolution) processing, to improve the resolution of the video playback and solve the problem of particles appearing during the video playback process.

[0196] 4024 , the multimedia processing chip 200 transmits the processed image data during the video playback process to the external memory through the third interface 203 .

[0197] It should be noted that after the multimedia processing chip 200 processes the image data of the video playback, it can be transmitted to an external memory, such as a memory used by the electronic device to store videos, through the third interface 203. Then, the video processed by the multimedia processing chip 200 on the image data can be displayed on the display screen of the electronic device. It can be understood that the external memory can be understood as a memory outside the multimedia processing chip.

[0198] Among them, the multimedia processing chip 200 can be understood to process the image data in two ways: one is that the image signal processor 210 performs statistics on the image data to calculate the status information. The other is that all or part of the image processors in the multimedia processing chip 200, such as the image signal processor 210 and the neural network processor 220, preprocess the image data. The preprocessing can be understood as firstly performing a first preprocessing such as optimization processing on the image data by the image signal processor 210, and then performing a second preprocessing such as neural network algorithm processing on the image data after the first preprocessing by the neural network processor, and then performing a third preprocessing such as bit width adjustment processing on the image data after the second preprocessing by the image signal processor 210. It should be noted that another way of preprocessing the image data by all or part of the image processors in the multimedia processing chip 200 at least includes performing a neural network algorithm processing on the image data by the neural network processor 220, and the image signal processor 210 can first optimize the image data before the neural network processor 220 processes it. After the neural network processor 220 processes it, the image signal processor 210 can also perform bit width adjustment processing on the image data.

[0199] It is understandable that when the multimedia processing chip 200 of the embodiment of the present application processes offline pictures or offline videos, the third interface 203 can be used to transmit data, which will not occupy the second interface 202. The second interface 202 can transmit real-time data.

[0200] It should be noted that the module for processing image data in the multimedia processing chip 200 of the embodiment of the present application is not limited thereto. The multimedia processing chip 200 may also include other processing modules to process image data, such as the multimedia processing chip 200 may also include a digital signal processor.

[0201] See also Fig.23 The multimedia processing chip 200 may further include a digital signal processor (Digital Signal Processing) 270, which may be used to assist the image signal processor 210 and the neural network processor 220. However, the digital signal processor 270 may also process image data with a smaller amount of calculation.

[0202] The digital signal processor 270 uses some general algorithms to process the image data, such as the digital signal processor 270 can use an image quality detection algorithm to select a frame of image from multiple frames of image. It should be noted that in some cases, the neural network processor 220 cannot support some algorithms. For example, for an ultra-wide-angle camera, if deformity correction processing is required, the neural network processor 220 may not be able to implement it, and the digital signal processor 270 can be used for processing.

[0203] It can be seen that the digital signal processor 270 of the embodiment of the present application is mainly used to process some image data with a small amount of data, and the neural network processor 220 is mainly used to process some image data with a large amount of data. For example, the digital signal processor 270 can be used to process static images, and the neural network processor 220 is used to process dynamic images such as video images. For another example, the digital signal processor 270 is used to process image data in the camera mode, and the neural network processor 220 is used to process image data in the video recording mode, the video playback mode and the preview image mode. Therefore, the embodiment of the present application adopts a combination of the digital signal processor 270 and the neural network processor 220 to achieve better and more comprehensive image processing optimization, so that the quality of the image data processed by the multimedia processing chip 200 is better and the display effect is better.

[0204] In some embodiments, in the photo shooting mode, the multimedia processing chip 200 can transmit the image data in the photo shooting mode through the third interface 203. In the video recording mode, the multimedia processing chip 200 can transmit the image data in the video recording mode through the second interface 202. In the preview image mode, the multimedia processing chip 200 can transmit the image data in the preview image mode through the second interface 202. In the video playback mode, the multimedia processing chip 200 can transmit the image data of the video playback through the third interface 203. In the album display photo mode, the multimedia processing chip 200 can transmit the image data of the displayed photo through the third interface 203.

[0205] The third interface 203 can transmit image data in real time or offline, and can also transmit data such as configuration parameters. The third interface 203 has high efficiency in transmitting data. Based on this, the embodiment of the present application can assign different data to the second interface 202 and the third interface 203 for transmission. To improve the transmission efficiency of data. The main control processor 260 can determine which type of image data the image data received by the multimedia processing chip 200 is, or the main control processor 260 can determine in which mode the image data received by the multimedia processing chip 200 is the image data obtained. When the multimedia processing chip 200 receives the image data, the main control processor 260 can determine in which mode the image data is obtained according to the image data. When the main control processor 260 determines that the image data received by the multimedia processing chip 200 is image data in the video recording mode and the preview image mode, the main control processor 260 can control the neural network processor 220 to process the image data. When the main control processor 260 determines that the image data received by the multimedia processing chip 200 is image data in the photographing mode, the main control processor 260 may control the digital signal processor 270 to process the image data.

[0206] It should be noted that the image data in the photo shooting mode can also be transmitted through the second interface 202 .

[0207] See also Fig.24 , Fig.24 The multimedia processing chip 200 shown in FIG. Fig.23 The difference of the multimedia processing chip 200 shown is: Fig.24 The multimedia processing chip 200 shown is not provided with a third interface. The data processed by the multimedia processing chip 200 for static images can also be transmitted through the second interface 202. For example, the second interface 202 has multiple paths. When the multimedia processing chip 200 processes dynamic images, it can be directly transmitted through one or more paths of the second interface 202. That is, each path of the second interface 202 is preferentially configured to the data processed by the multimedia processing chip 200 for dynamic images. When the multimedia processing chip 200 processes static images, the main control processor 260 can first determine whether there are idle paths in each path of the second interface 202, that is, whether there are paths that are not transmitting dynamic image data. If one or more of the multiple paths of the second interface 202 are in an idle state, the data processed by the multimedia processing chip 200 for static images can be transmitted through one or more paths in an idle state.

[0208] It should be noted that when all the paths of the second interface 202 are not in an idle state, the static image data can be transmitted through the idle path until at least one path of the second interface 202 is in an idle state. Of course, other embodiments of the present application can also use other methods to transmit the static image data without determining whether to transmit the static image data based on the path state of the second interface 202.

[0209] See also Fig.25 The first interface 201 and the second interface 202 of the multimedia processing chip 200 can also be directly connected, so that the first interface 201 can directly transmit some image data such as static image data received to the second interface 202 without processing the image data through the image signal processor 210 and / or the neural network processor 220.

[0210] In some embodiments, when the multimedia processing chip 200 receives image data of a recorded video, the image data may be transmitted to the image signal processor 210 for processing through the first interface 201. When the multimedia processing chip 200 receives data of a preview image, the image data may be directly transmitted to the second interface 202 through the first interface 201. When the multimedia processing chip 200 receives a photographed image in a photographing mode, the image data may be directly transmitted to the second interface 202 through the first interface 201.

[0211] In some other embodiments, when the multimedia processing chip 200 receives a preview image in the preview mode, the image data may be transmitted to the image signal processor 210 for processing through the first interface 201, thereby solving the problem of picture consistency.

[0212] In order to further illustrate the data interaction between the multimedia processing chip and other devices provided in the embodiment of the present application, the following description is made from the perspective of the application of the multimedia processing chip. It can be understood that the multimedia processing chip 200 can be applied to an electronic device such as a smart phone, a tablet computer, etc.

[0213] See also Fig.26 , the electronic device 20 may include an image sensor 600 , a multimedia processing chip 200 , and an application processing chip 400 .

[0214] Among them, the camera 600 can collect image data. The camera 600 can be a front camera or a rear camera. The camera 600 may include an image sensor and a lens, and the image sensor may be a complementary metal oxide semiconductor (CMOS) image sensor, a charge coupled device (CCD) image sensor, etc. The camera 600 may be electrically connected to the multimedia processing chip 200, such as the camera 600 is electrically connected to the first interface 201 of the multimedia processing chip 200. The camera 600 may collect raw image data such as RAW image data, and transmit it to the multimedia processing chip 200 through the first interface 201, so as to be processed by the image processor inside the multimedia processing chip 200, such as the image signal processor 210 and the neural network processor 220.

[0215] The multimedia processing chip 200 is any of the above multimedia processing chips 200 , which will not be described in detail herein.

[0216] The application processing chip 400 can control various functions of the electronic device 20. For example, the application processing chip 400 can control the camera 600 of the electronic device 20 to collect images, and the application processing chip 400 can also control the multimedia processing chip 200 to process the images collected by the camera 600. The application processing chip 400 can also process image data.

[0217] The image data collected by the camera 600 can be transmitted to the interface of the multimedia processing chip 200. The multimedia processing chip 200 can pre-process the image data, and the application processing chip 400 can post-process the image data. The processing of the image data between the multimedia processing chip 200 and the application processing chip 400 can be differentiated or the same.

[0218] The following is a detailed description of the application of one of the multimedia processing chips to the electronic device 20.

[0219] See also Fig. 27 The application processing chip 400 of the electronic device 20 may include an application processor 410 , an image signal processor 420 , a memory 430 , a system bus 440 and a fourth interface 401 .

[0220] The application processor 410 may serve as a control center of the electronic device 20. It may also execute some algorithms to process image data.

[0221] The memory 430 may store various data such as image data, system data, etc. The memory 430 may be built in the application processing chip 410 or may be external to the application processor 410 .

[0222] The fourth interface 401 may be a mobile industry processor interface. The fourth interface 401 is electrically connected to the second interface 202 and may receive data processed by the multimedia processing chip 200 .

[0223] The image signal processor 420 may process the image data.

[0224] In some embodiments of the present application, the application processor 410 and the image signal processor 420 may jointly post-process the pre-processed image data. It is understandable that the application processor 410 and the image signal processor 420 may also jointly process the image data collected by the camera 600.

[0225] In some embodiments, the result of image data processing by the image signal processor 210 can be transmitted to the memory 430 of the application processing chip 400 through the connection between the second interface 202 and the fourth interface 401. The result of image data processing by the neural network processor 220 can be transmitted to the memory 430 of the application processing chip 400 through the connection between the second interface 202 and the fourth interface 401. The result of image data processing by the digital signal processor 270 can be transmitted to the memory 430 of the application processing chip 400 through the connection between the second interface 202 and the fourth interface 401.

[0226] Combination Fig.25 It should be noted that, in some cases, the data received by the multimedia processing chip 200, such as image data in the photo taking mode, can be directly transmitted from the first interface 201 to the second interface 202, and then transmitted to the memory 430 through the connection between the second interface 202 and the fourth interface 401.

[0227] It should be noted that the manner in which the multimedia processing chip 200 transmits data to the application processing chip 400 is not limited thereto.

[0228] See also Fig.28 The multimedia processing chip 200 in the electronic device 20 can refer to Fig.23 The application processing chip 400 may further include a fifth interface 402, which may be referred to as an interconnect bus interface. For example, the fifth interface 402 is a high-speed interconnect bus interface, which may also be referred to as a high-speed peripheral component interconnect interface, an external device interconnect bus interface, which is an interface of a high-speed serial computer expansion bus standard. It should be noted that the fifth interface 402 may also be a low-speed interconnect bus interface.

[0229] The fifth interface 402 is connected to the third interface 203. In some embodiments, the fifth interface 402 and the third interface 203 are of the same type, such as the fifth interface 402 and the third interface 203 are both high-speed interconnect bus interfaces. The multimedia processing chip 200 can transfer some image data such as static image data and preview image data to the memory 430 through the connection between the third interface 203 and the fifth interface 402. Of course, the multimedia processing chip 200 can also transfer some data such as album photo data and video playback data to the memory 430 through the connection between the third interface 203 and the fifth interface 402.

[0230] In some embodiments, when the multimedia processing chip 200 transmits the processed image data to the application processing chip 400, the application processing chip 400 post-processes the data processed by the multimedia processing chip 200, and stores and displays the processed data on a display screen.

[0231] The following is a description from the perspective of the data processing process.

[0232] The multimedia processing chip 200 obtains dynamic image data such as image data recorded by a video. The multimedia processing chip 200 counts the state information of the dynamic image data according to the dynamic image data obtained, and pre-processes the dynamic image data through the multimedia processing chip 200. After the multimedia processing chip 200 counts the state information and pre-processes the dynamic image data, the multimedia processing chip 200 can send the counted state information and the pre-processed dynamic image data to the application processing chip 400. The application processing chip 400 post-processes the pre-processed dynamic image data based on the state information. Thereby, the image quality can be improved.

[0233] The application processor 410 receives a third instruction for starting the camera 600 .

[0234] The application processor 410 starts the camera 600 based on the third instruction for starting the camera 600. The application processor 410 may configure the camera 600 to start the camera 600.

[0235] The camera 600 collects image data and transmits the image data to the first interface 201 of the multimedia processing chip 200 .

[0236] The statistical module 212 of the image signal processor 210 performs statistical processing on the image data to obtain statistical status information, and transmits the statistical status information to the fourth interface 401 through the second interface 202 .

[0237] The optimization module 214 of the image signal processor 210 performs optimization processing on the image data, such as linearization processing, bad pixel compensation, black level correction, etc., and transmits the optimized data to the neural network processor 220. It is understandable that the optimization module 214 can directly transmit the optimized data to the neural network processor 210, or store it in the memory 230, and the neural network processor 220 obtains it from the memory 230.

[0238] The neural network processor 210 processes the data optimized by the optimization module 214, such as neural network algorithm processing, and transmits the processed data to the memory 230.

[0239] The multimedia processing chip 200 transmits the processed data to the fourth interface 401 through the second interface 202 .

[0240] The application processor 410 and the image signal processor 420 perform post-processing, such as 3A processing, on the data processed by the neural network processor 220 based on the state information.

[0241] It is understandable that if the image data is processed by the application processing chip alone, the image signal processor of the application processing chip needs to perform statistics on the status information of the image data, and the application processor of the application processing chip executes some algorithms to calculate some parameters such as focus parameters, exposure parameters, white balance parameters, lens shading correction parameters, etc. based on the status information. Based on the calculated parameters, the application processor can configure the camera, and the image signal processor can perform correction processing on the image data. The entire processing process is executed by the application processing chip, resulting in higher power consumption of the application processing chip. The application processing chip often needs to control various other functions, which may affect the performance of the application processing chip during the entire image processing process.

[0242] In the embodiment of the present application, part of the image data is processed by the multimedia processing chip 400, and the other part is processed by the application processing chip 200, so that the image quality can be improved on the basis of saving the power consumption of the application processing chip 200. Among them, after the image signal processor 210 counts the status information, it can send the counted status information to the application processing chip 400, and the application processor 410 executes some algorithms to calculate some parameters such as focus parameters, exposure parameters, white balance parameters, lens shading correction parameters, etc. based on the status information. Based on the calculated parameters, the application processor 410 can configure the camera 600, and the image signal processor 420 can correct the image data. It should be noted that after the image signal processor 210 counts the status information, it can also perform calculations without executing some algorithms through the application processor 410, such as the main control processor 260 executing some algorithms, calculating some parameters based on the status information, and then transmitting the parameters to the application processing chip 400, and the application processor 410 can configure the camera 600, and the image signal processor 420 can correct the image data. In addition, after the image signal processor 210 calculates the status information, the image signal processor 420 may execute some algorithms, calculate some parameters based on the status information, and the application processor 410 may configure the camera 600 based on the parameters, and the image signal processor 420 may perform correction processing on the image data. It is understandable that the algorithms executed by the application processor 410 and the main control processor 260 may be updated, while the algorithms executed by the image signal processor 420 often cannot be updated. In actual application, the application processor 410 or the main control processor 260 may be preferentially selected to execute the relevant algorithms to calculate the status information.

[0243] The following are specific examples for different status information.

[0244] The status information may include auto-focus status information, and the application processor 410 may execute relevant algorithms, calculate focus parameters based on the auto-focus status information, and configure the focus parameters to the camera 600. The camera 600 may focus based on the focus parameters. The main control processor 260 may also execute relevant algorithms, calculate focus parameters based on the auto-focus status information, and then configure the focus parameters to the camera 600, or send them to the application processing chip 400, and the application processor 410 configures the focus parameters to the camera 600. Of course, it is also possible for the image signal processor 420 to execute relevant algorithms to calculate the focus parameters. Among them, the auto-focus status information may include one or more of phase focus status information, contrast focus status information, laser focus status information, and TOF (Time of Flight) focus status information.

[0245] The auto focus state information may include contrast focus state information, and the image signal processor 210 may process the image data, such as dynamic image data, using a preset algorithm to calculate the contrast focus state information. The application processor 410 may execute a related algorithm, calculate a contrast focus parameter based on the contrast focus state information, and configure the contrast focus parameter to the camera 600. The camera 600 may focus based on the contrast focus parameter. The main control processor 260 may also execute a related algorithm, calculate a contrast focus parameter based on the contrast focus state information, and then configure the contrast focus parameter to the camera 600, or send the contrast focus parameter to the application processing chip 400, and the application processor 410 may configure the contrast focus parameter to the camera 600. Of course, it is also possible for the image signal processor 420 to execute a related algorithm to calculate the contrast focus parameter.

[0246] The autofocus state information may also include phase focus state information, and the image data such as dynamic image data may be extracted by the image sensor 210, such as marking and distinguishing the image data, to extract the phase focus state information. The application processor 410 may execute a related algorithm, calculate a phase focus parameter based on the phase focus state information, and configure the phase focus parameter to the camera 600. The camera 600 may focus based on the phase focus parameter. The main control processor 260 may also execute a related algorithm, calculate a phase focus parameter based on the phase focus state information, and then configure the phase focus parameter to the camera 600, or send it to the application processing chip 400, and the application processor 410 configures the phase focus parameter to the camera 600. Of course, it is also possible for the image signal processor 420 to execute a related algorithm to calculate the phase focus parameter.

[0247] The autofocus status information may also include laser focus status information, and the image signal processor 210 may process the image data, such as dynamic image data, using a preset algorithm to calculate the laser focus status information. The application processor 410 may execute relevant algorithms, calculate laser focus parameters based on the laser focus status information, and configure the laser focus parameters to the camera 600. The camera 600 may focus based on the laser focus parameters. The main control processor 260 may also execute relevant algorithms, calculate laser focus parameters based on the laser focus status information, and then configure the laser focus parameters to the camera 600, or send them to the application processing chip 400, and the application processor 410 may configure the laser focus parameters to the camera 600. Of course, it is also possible for the image signal processor 420 to execute relevant algorithms to calculate the laser focus parameters.

[0248] The auto focus state information may also include TOF focus state information, and the image signal processor 210 may perform a preset algorithm processing on the image data such as dynamic image data to calculate the TOF focus state information. The application processor 410 may execute a related algorithm, calculate the TOF focus parameter based on the TOF focus state information, and configure the TOF focus parameter to the camera 600. The camera 600 may focus based on the TOF focus parameter. The main control processor 260 may also execute a related algorithm, calculate the TOF focus parameter based on the TOF focus state information, and then configure the TOF focus parameter to the camera 600, or send it to the application processing chip 400, and the application processor 410 configures the TOF focus parameter to the camera 600. Of course, it is also possible for the image signal processor 420 to execute a related algorithm to calculate the TOF focus parameter.

[0249] The status information may also include automatic white balance status information. The application processor 410 may execute relevant algorithms to calculate white balance parameters based on the automatic white balance status information. The image signal processor 420 may perform white balance processing, or image correction processing, on the image data pre-processed by the multimedia processing chip 200 based on the white balance parameters. The main control processor 260 may also execute relevant algorithms to calculate white balance parameters based on the automatic white balance status information, and then send the white balance parameters to the application processing chip 400. The image signal processor 420 may perform white balance processing on the image data pre-processed by the multimedia processing chip 200 based on the white balance parameters. Of course, it is also possible for the image signal processor 420 to execute relevant algorithms to calculate white balance parameters.

[0250] The status information may also include automatic exposure status information. The application processor 410 may execute relevant algorithms, calculate exposure parameters based on the automatic exposure status information, and configure the exposure parameters to the camera 600. The camera 600 may perform exposure based on the exposure parameters. The main control processor 260 may also execute relevant algorithms, calculate exposure parameters based on the automatic exposure status information, and then configure the exposure parameters to the camera 600, or send the information to the application processing chip 400, and the application processor 410 may configure the exposure parameters to the camera 600. Of course, it is also possible for the image signal processor 420 to execute relevant algorithms to calculate the exposure parameters. It should be noted that when the exposure parameters need to be compensated, the image signal processor 420 may perform compensation processing on the exposure parameters, and then the application processor 410 may configure the compensated exposure parameters to the camera 600, and the camera 600 may perform exposure based on the compensated exposure parameters.

[0251] The status information also includes lens shading correction status information. The application processor 410 can execute relevant algorithms to calculate lens shading correction parameters based on the lens shading correction status information, and the image signal processor 420 can perform lens shading correction on the image data pre-processed by the multimedia processing chip 200 based on the lens shading correction parameters. The main control processor 260 can also execute relevant algorithms to calculate lens shading correction parameters based on the lens shading correction status information, and then send the lens shading correction parameters to the application processing chip 400, and the image signal processor 420 performs white balance processing on the image data pre-processed by the multimedia processing chip 200 based on the white balance parameters. Of course, it is also possible for the image signal processor 420 to execute relevant algorithms to calculate the lens shading correction parameters.

[0252] The image signal processor 420 counting the state information of the image data can be understood as: calculating some state information by an algorithm and / or extracting some state information by extraction.

[0253] The image signal processor 420 does not need to process the image data processed by the optimization module 214. For example, the optimization module 214 performs bad pixel compensation linearization processing and black level correction on the image data, and the image signal processor 420 does not need to perform bad pixel compensation, linearization processing, and black level correction. In the embodiment of the present application, the multimedia processing chip 200 and the application processing chip 400 perform differential processing on the image data. Thus, the power consumption of the application processing chip 400 can be saved. For example, the optimization module 214 performs bad pixel compensation linearization processing and black level correction on the image data, and the image signal processor 420 does not need to perform bad pixel compensation, linearization processing, and black level correction. However, the application processing chip 400 and the multimedia processing chip 200 can also perform part of the same processing on the image data. For example, the multimedia processing chip 200 performs noise reduction processing on the dynamic image data, and the application processing chip 400 also performs noise reduction processing on the dynamic image data. For example, the multimedia processing chip 200 performs statistical processing on the dynamic image data, and the application processing chip 400 also performs statistical processing on the dynamic image data.

[0254] The image signal processor 420 sends the processed data to the display screen and the memory 430 to display and store the image.

[0255] It should be noted that if the image is a dynamic image, it can be encoded by an encoder before storage, and then stored after the encoding is completed. If the image is a static image, it can be compressed in the memory, such as JPEG compression, and then stored after compression.

[0256] It should also be noted that the image data processed by the multimedia processing chip 200 may be RAW image data, and the application processing chip 400 may process the RAW image data, such as 3A processing, or convert the RAW format into a YUV format to process the image in the YUV format, such as the image signal processor 420 performing RGBToYUV processing on the image in the YUV format.

[0257] Before the image signal processor 210 transmits the data processed by the neural network processor 220 to the fourth interface 401 through the second interface 202, the main control processor 260 may first determine whether the bit width of the data processed by the neural network processor 220 is the same as the bit width of the data to be processed by the application processing chip 400. If they are the same, the image signal processor 210 transmits the data processed by the neural network processor 220 to the fourth interface 401 through the second interface 202. If they are not the same, the optimization module 214 of the image signal processor 210 performs bit width adjustment processing on the data processed by the neural network processor 220, so that the bit width of the adjusted data is the same as the bit width of the data to be processed by the application processing chip 400. This ensures that the application processing chip 400 can normally process the data transmitted by the multimedia processing chip 200.

[0258] It should also be noted that, in some other embodiments, when the multimedia processing chip 200 processes the image data, the original image may be directly processed by the neural network processor 220 instead of being optimized by the optimization module 214 .

[0259] The method of processing image data by the multimedia processing chip 200 in the embodiment of the present application can be referred to in Figures 10 to 17 , I will not go into details here.

[0260] The following describes the process of processing image data by the application processing chip 400.

[0261] See also Fig.29 , the method for processing image data by the application processing chip 400 includes:

[0262] 5011 , the fourth interface 401 of the application processing chip 400 receives status information of statistics performed on dynamic image data by the statistics module 212 .

[0263] 5012, the fourth interface 401 of the application processing chip 400 receives the result of the neural network processor 220 performing the neural network algorithm processing on the dynamic image data.

[0264] It should be noted that before the neural network processor 220 performs neural network algorithm processing on the dynamic image data, the optimization module 214 may perform optimization processing on the dynamic image data.

[0265] 5013, the application processing chip 400 performs secondary processing on the result of the processing of the dynamic image data by the neural network processor 220 based on the state information.

[0266] It should be noted that before the application processing chip 400 performs processing, the optimization module 214 can perform bit width adjustment processing on the data processed by the neural network processor 220.

[0267] To further illustrate that the multimedia processing chip 200 pre-processes the image data and then the application processing chip 400 post-processes the image data to improve the image quality, please refer to the following. Fig.30 and Fig.31 . Fig.30 The first figure shows a frame of image displayed by the multimedia processing chip 200 and the application processing chip 400 of the embodiment of the present application jointly processing the image, which includes the HDR algorithm processing of the image data by the neural network processor 220 of the embodiment of the present application. Fig.30 The second figure shows a frame of image displayed by processing the image only by the application processing chip. From the comparison between the first and second figures, it can be seen that the two frames of images are different in many aspects and multiple areas. For example, the brightness around the character in the second figure is too bright, and the objects close to the character are displayed too clearly, such as the clarity of the objects in the second area B is greater than the clarity of the second area A, resulting in the character not being prominent enough. And the details of the surroundings of the second figure, such as the first area B, are not as good as those of the first area A.

[0268] in, Fig.31 The third figure shows a frame of image displayed by the multimedia processing chip 200 and the application processing chip 400 of the embodiment of the present application jointly processing the image, which includes the video night scene algorithm processing of the image signal by the neural network processor 220 of the embodiment of the present application. Fig.31 The fourth figure shows a frame of image displayed by processing the image only by the application processing chip. From the comparison between the third and fourth figures, it can be seen that there are differences in multiple areas between the two frames of image. For example, the third area A of the third figure is clearer than the third area B of the fourth figure. For another example, the fourth area A of the third figure shows more details than the fourth area B of the fourth figure.

[0269] It can be understood that the camera 600, multimedia processing chip 200 and application processing chip 400 defined in the embodiment of the present application can be installed together, such as the camera 600, multimedia processing chip 200 and application processing chip 400 are installed on a circuit board.

[0270] See also Fig.32The circuit board 22 is mounted with an image sensor 600, a multimedia processing chip 200 and an application processing chip 400. The camera 600, the multimedia processing chip 200 and the application processing chip 400 are all connected through signal lines to achieve signal transmission.

[0271] It is understandable that the circuit board 22 may also be installed with other components, which are not listed one by one here.

[0272] See also Fig.33 The camera 600 may also be installed on a circuit board other than the multimedia processing chip 200 and the application processing chip 400. For example, the camera 600 may be installed on a circuit board alone, the multimedia processing chip 200 and the application processing chip 400 may be installed on a circuit board 22, and the camera 600 is connected to the multimedia processing chip 200 via a signal line.

[0273] The multimedia processing chip, electronic device and dynamic image processing method provided by the embodiments of the present application are described in detail above. Specific examples are used herein to illustrate the principles and implementation methods of the present application, and the description of the above embodiments is only used to help understand the present application. At the same time, for those skilled in the art, according to the ideas of the present application, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as limiting the present application.

Claims

1. A multimedia processing chip, It is characterized in that The multimedia processing chip comprises: An image signal processor, used for collecting statistics of state information of the image data and performing a first preprocessing on the image data; and A neural network processor, used for performing a second preprocessing on the image data after the first preprocessing by using a neural network algorithm; Among them, when the image data is dynamic image data, the multimedia processing chip is used to perform a first preprocessing on the dynamic image data at least through the image signal processor and a second preprocessing on the dynamic image data after the first preprocessing by the neural network processor, and send the status information and the second preprocessed dynamic image data to the application processing chip, the first preprocessing includes at least one of bad pixel compensation, linearization processing, and black level correction for the dynamic image data, and the first preprocessing is used to accelerate the convergence speed of the neural network processor.

2. The multimedia processing chip according to claim 1, It is characterized in that The image signal processor is also used to perform a third preprocessing on the image data after the second preprocessing. The third preprocessing of the image data by the image signal processor includes bit width adjustment on the image data so that the bit width of the image data after the bit width adjustment is the same as the bit width of the image data processed by the application processing chip.

3. The multimedia processing chip according to claim 2, It is characterized in that The first pre-processing of the image data by the image signal processor further includes image cropping processing and / or image reduction processing.

4. The multimedia processing chip according to claim 1, It is characterized in that The image signal processor is also used to perform bit width adjustment processing on the image data processed by the neural network algorithm, so that the bit width of the image data after the bit width adjustment is the same as the bit width of the image data processed by the application processing chip.

5. The multimedia processing chip according to any one of claims 1 to 4, It is characterized in that The neural network algorithm used by the neural network processor to process the dynamic image data includes at least one of a night scene algorithm, an HDR algorithm, a blur algorithm, a noise reduction algorithm, a super-resolution algorithm, and a semantic segmentation algorithm.

6. The multimedia processing chip according to claim 5, It is characterized in that The multimedia processing chip is used to process the dynamic image data in real time, and transmit the processed dynamic image data to the application processing chip in real time.

7. The multimedia processing chip according to any one of claims 1 to 6, It is characterized in that The image data includes static image data, the multimedia processing chip is used to process the static image data, and the neural network processor is used to process the static image data. The neural network algorithm includes at least one of a night scene algorithm, an HDR algorithm, a blur algorithm, a noise reduction algorithm, a super-resolution algorithm, and a semantic segmentation algorithm.

8. The multimedia processing chip according to any one of claims 1 to 6, It is characterized in that The multimedia processing chip is also used for off-line processing of static image data and / or dynamic image data.

9. The multimedia processing chip according to any one of claims 1 to 6, It is characterized in that The image data is RAW image data, and the multimedia processing chip is used to process the RAW image data.

10. The multimedia processing chip according to any one of claims 1 to 6, It is characterized in that The status information includes at least one of automatic exposure status information, automatic white balance status information and automatic focus status information.

11. The multimedia processing chip according to claim 10, It is characterized in that The status information also includes lens shading correction status information.

12. An electronic device, It is characterized in that include: A multimedia processing chip, which is a multimedia processing chip as claimed in any one of claims 1 to 11; and The application processing chip is used to obtain the preprocessing result and statistical status information from the multimedia processing chip, and the application processing chip performs post-processing on the preprocessing result based on the status information.

13. The electronic device according to claim 12, It is characterized in that The state information includes at least one of auto focus state information, auto white balance state information and auto exposure state information, and the application processing chip is used for: Calculating focus parameters based on the automatic focus state information, and configuring the focus parameters to a camera of the electronic device; Calculating white balance parameters based on the automatic white balance state information, and performing white balance processing on the preprocessing result based on the white balance parameters; An exposure parameter is calculated based on the automatic exposure state information, and the exposure parameter is configured to the camera of the electronic device, or the exposure parameter is compensated and then configured to the camera of the electronic device.

14. The electronic device according to claim 13, It is characterized in that The automatic focus state information includes phase focus state information and contrast focus state information, and the image signal processor of the multimedia processing chip is used for: Processing the image data using a preset algorithm to obtain contrast focus state information; Extracting phase focus state information from the image data; The application processing chip is also used for: Calculating contrast focus parameters based on the contrast focus state information, and configuring the contrast focus parameters to a camera of the electronic device; A phase focus parameter is calculated based on the phase focus state information, and the phase focus parameter is configured to a camera of the electronic device.

15. The electronic device according to claim 13, It is characterized in that The state information also includes lens shading correction state information, and the application processing chip is further used for: A lens shading correction parameter is calculated based on the lens shading correction state information, and lens shading correction is performed on the preprocessing result based on the lens shading correction parameter.

16. A dynamic image processing method, It is characterized in that The method comprises: Acquire dynamic image data; According to the dynamic image data, the image signal processor of the multimedia processing chip collects statistics on the state information of the dynamic image data and performs a first preprocessing on the dynamic image data, and the neural network processor of the multimedia processing chip performs a second preprocessing on the dynamic image data after the first preprocessing, wherein the first preprocessing includes at least one of bad pixel compensation, linearization processing, and black level correction for the dynamic image data, and the first preprocessing is used to accelerate the convergence speed of the neural network processor; Sending the status information counted by the multimedia processing chip and the dynamic image data after the second preprocessing to the application processing chip; The dynamic image data after the second pre-processing is post-processed by the application processing chip based on the state information.

17. The dynamic image processing method according to claim 16, It is characterized in that The first preprocessing of the dynamic image data by the multimedia processing chip includes: Optimizing the dynamic image data; The optimized dynamic image data is processed by a neural network algorithm.

18. The dynamic image processing method according to claim 16 or 17, It is characterized in that The state information includes at least one of auto focus state information, auto white balance state information, and auto exposure state information, and the post-processing of the pre-processed dynamic image data by the application processing chip based on the state information includes: Calculating focus parameters based on the auto-focus state information, and configuring the focus parameters to the camera; Calculating white balance parameters based on the automatic white balance state information, and performing white balance processing on the preprocessing result based on the white balance parameters; An exposure parameter is calculated based on the automatic exposure state information, and the exposure parameter is configured to the camera, or the exposure parameter is compensated and then configured to the camera.

Images