Image processing chips, methods, application processing chips and electronic devices

Abstract

The present application provides an image processing chip, a method, an application processing chip and an electronic device. The image processing chip includes a data packaging unit. N frames of RAW images are obtained through the data packaging unit, and the obtained N frames of RAW images are reorganized according to the image processing requirement information of the application processing chip to obtain M data blocks, and the M data blocks are encapsulated into L data packets, and then the encapsulated L data packets are transmitted to the application processing chip. Among them, N and M are positive integers greater than 1, and L is a positive integer less than N. In this way, by encapsulating N frames of RAW images into L data packets less than N for transmission, at most L data transmission channels of the application processing chip need to be occupied. Compared with the related art that occupies N data transmission channels to realize the transmission of N frames of RAW images, the present application can reduce the occupation of the data transmission channels of the application processing chip by image data, thereby reducing the impact of image data on other types of data transmission and processing.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image processing chip, method, application processing chip and electronic equipment. Background Art

[0002] At present, users usually use electronic devices with cameras (such as digital cameras, smart phones, etc.) to capture images, so as to record what is happening around them and what they see anytime and anywhere. In related technologies, in order to improve the quality of captured images, electronic devices usually perform specific image processing on the captured original images. For example, multiple frames of images can be captured, and high-dynamic synthesis processing can be performed on the multiple frames of images to obtain high-dynamic synthetic images, or multiple frames of images can be subjected to multi-frame synthesis noise reduction processing to obtain noise-reduced synthetic images. When it is necessary to process multiple frames of images, the image sensor in the camera needs to occupy multiple data transmission channels of the application processing chip to transmit the captured multiple frames of images to the application processing chip for image processing. However, the number of data transmission channels of the application processing chip is limited. When the data transmission channels are occupied by too much image data, it will affect the transmission and processing of other types of data. Summary of the invention

[0003] The present application provides an image processing chip, an application processing chip, an electronic device and an image processing method, which can reduce the occupation of image data on the data transmission channel of the application processing chip, thereby reducing the impact of image data on other types of data transmission and processing.

[0004] The present application discloses an image processing chip, comprising:

[0005] A data packaging unit is used to obtain N frames of RAW images, where N is a positive integer greater than 1; and according to the image processing requirement information of the application processing chip, reorganize the image data in the obtained N frames of RAW images into M data blocks, where M is a positive integer greater than 1; and encapsulate the M data blocks into L data packets, where L is a positive integer less than N;

[0006] The image processing chip is used to transmit L data packets to the application processing chip.

[0007] The present application also discloses an application processing chip, which is used to transmit image processing requirement information to an image processing chip, and obtain L data packets corresponding to the image processing requirement information from the image processing chip, including:

[0008] An image processing unit, used for generating image processing requirement information;

[0009] A data unpacking unit is used to split L data packets into M data blocks and perform parallel parsing to obtain image data, wherein the L data packets are obtained by reorganizing image data in N frames of RAW images into M data blocks and then encapsulating them, where N and M are positive integers greater than 1, and L is a positive integer less than N;

[0010] The image processing unit is also used to perform image processing on the image data obtained through analysis.

[0011] The present application discloses an electronic device, including an image processing chip and an application processing chip, wherein:

[0012] The application processing chip is used to generate image processing requirement information and transmit the image processing requirement information to the image processing chip;

[0013] The image processing chip is used to obtain N frames of RAW images, where N is a positive integer greater than 1; and reorganize the image data in the N frames of RAW images into M data blocks according to the image processing requirement information, where M is a positive integer greater than 1; and encapsulate the M data blocks into L data packets and transmit them to the application processing chip, where L is a positive integer less than N;

[0014] The application processing chip is also used to split the L data packets into M data blocks and then parse them in parallel to obtain image data, and perform image processing on the parsed image data.

[0015] The present application discloses an image processing method, which is applied to an image processing chip, comprising:

[0016] Get N frames of RAW images, where N is a positive integer greater than 1;

[0017] According to the image processing requirement information of the application processing chip, the image data in the N frames of RAW images are reorganized into M data blocks, where M is a positive integer greater than 1;

[0018] Encapsulate M data blocks into L data packets, where L is a positive integer less than N;

[0019] The L data packets are transmitted to the application processing chip.

[0020] The present application discloses an image processing method, which is applied to an application processing chip, comprising:

[0021] Generate image processing requirement information, and transmit the image processing requirement information to the image processing chip;

[0022] Obtain L data packets corresponding to image processing requirement information from the image processing chip;

[0023] The L data packets are split into M data blocks and then analyzed in parallel to obtain image data. The L data packets are obtained by reorganizing the image data in N frames of RAW images into M data blocks and then encapsulating them. N and M are positive integers greater than 1, and L is a positive integer less than N.

[0024] Perform image processing on the image data obtained by analysis.

[0025] When the image processing chip provided by the present application is applied to electronic devices, N frames of RAW images are encapsulated into L data packets less than N for transmission through the image processing chip, which requires at most L data transmission channels of the application processing chip. Compared with the related art that occupies N data transmission channels to realize the transmission of N frames of RAW images, the present application can reduce the occupation of image data by the data transmission channels of the application processing chip, thereby reducing the impact of image data on other types of data transmission and processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for describing the embodiments are briefly introduced below.

[0027] Figure 1 A schematic diagram of the structure of an image processing chip provided in an embodiment of the present application.

[0028] Figure 2 This is an example diagram of a data packing unit in an embodiment of the present application reorganizing three RAW image frames into two data blocks.

[0029] Figure 3 This is an example diagram of a data packing unit in an embodiment of the present application recombining two RAW image frames into two data blocks.

[0030] Figure 4 A schematic diagram comparing the packet transmission method provided in the embodiment of the present application and related technologies.

[0031] Figure 5 This is an example diagram of an application in an embodiment of the present application where a processing chip is used to unpack, parse, and perform image processing on a data packet transmitted by an image processing chip.

[0032] Figure 6 This is an example diagram of an area of ​​interest of a shooting scene in an embodiment of the present application.

[0033] Figure 7 This is an example diagram of a data packing unit in an embodiment of the present application reorganizing image data into data blocks.

[0034] Figure 8 Schematic diagram of dividing a row of image data in a RAW image into blocks in an embodiment of the present application.

[0035] Fig. 9 This is an example diagram of encapsulating reassembled data blocks into data packets in an embodiment of the present application.

[0036] Fig.10 A schematic diagram of the structure of the application processing chip provided in an embodiment of the present application.

[0037] Fig.11 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0038] Fig.12 A flowchart of an image processing method according to an embodiment of the present application is shown.

[0039] Fig.13 Another flowchart of the image processing method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0040] It should be noted that the principles of the present application are illustrated by implementing them in an appropriate computing environment. The following description is based on the illustrated specific embodiments of the present application and should not be considered as limiting other specific embodiments of the present application that are not described in detail herein.

[0041] Relational terms such as first and second involved in the following embodiments of the present application are only used to distinguish one object or operation from another object or operation, and are not used to limit the actual sequential relationship between these objects or operations.

[0042] Please refer to Figure 1 , Figure 1 The schematic diagram of the structure of the image processing chip 100 provided in the embodiment of the present application is as follows: Figure 1 As shown, the image processing chip 100 includes a first data interface unit 110 , a data packaging unit 120 and a second data interface unit 130 .

[0043] It should be noted that the image processing chip 100 provided in the present application can be configured in an electronic device having an image sensor and an application processing chip (which can be integrated with a central processing unit, a modem, and an image signal processor, etc.), and is used to reorganize the image data in N (a positive integer greater than 1) frames of RAW images from the image sensor into M (a positive integer greater than 1) data blocks according to the image processing demand information of the application processing chip, and encapsulate these M data blocks into L (a positive integer less than N) data packets and transmit them to the application processing chip, so that the application processing chip can perform image processing on demand. In this way, by encapsulating N frames of RAW images into L data packets less than N, the present application needs to occupy at most L data transmission channels to realize the transmission of N frames of RAW images. Compared with the related art that needs to occupy N data transmission channels to realize the transmission of N frames of RAW images, the present application can reduce the number of data transmission channels required to realize the transmission of N frames of RAW images. Among them, the entity type of the electronic device can be a mobile electronic device such as a smart phone, a tablet computer, a PDA, a laptop computer, or a fixed electronic device such as a desktop computer and a TV, and the present application does not make specific restrictions on this.

[0044] In this embodiment, the first data interface unit 110 is used to provide a data transmission channel between the image sensor and the image processing chip 100 to realize data transmission between the image sensor and the image processing chip 100. Here, there is no specific restriction on the entity type of the first data interface unit 110 in the embodiment of the present application, including but not limited to the mobile industry processor interface (Mobile Industry Processor Interface, MIPI), PCI-E interface, etc. For example, the entity type of the first data interface unit 110 in the present application is a mobile industry processor interface. The image processing chip 100 can obtain N frames of unprocessed original RAW images collected by the image sensor from the image sensor through the first data interface unit 110, where N is a positive integer greater than 1.

[0045] In an optional embodiment, the image processing chip 100 may further include an image preprocessing unit, which is configured to perform image preprocessing on N frames of RAW images acquired by the image processing chip 100 from the image sensor to obtain N frames of preprocessed RAW images.

[0046] It should be noted that the image preprocessing performed by the image preprocessing unit can be understood as image processing performed before the application processing chip performs image processing, which is distinguished from the image processing performed by the application processing chip. It can be any type of image processing and can be specifically configured by technical personnel in this field according to actual needs.

[0047] For example, the image preprocessing unit may include two image processing subunits, namely, a first image processing subunit for performing traditional image processing, and a second image processing subunit for performing image processing based on artificial intelligence. The image processing required by the first image processing subunit (including but not limited to bad pixel correction processing, time domain noise reduction processing, 3D noise reduction processing, linearization processing, black level correction processing and other image processing methods) can be configured according to actual needs, and the image processing required by the second image processing subunit (including but not limited to super-resolution processing based on artificial intelligence, noise reduction processing and other image processing methods) can be configured.

[0048] It should be noted that the image processing chip 100 can perform image preprocessing on the N frames of RAW images from the image sensor through the image preprocessing unit as needed, or not perform image preprocessing on the N frames of RAW images from the image sensor.

[0049] The data packaging unit 120 is configured to acquire N RAW image frames, and reorganize the acquired N RAW image frames into M data blocks according to the image processing requirement information of the application processing chip, and encapsulate the M data blocks into L data packets.

[0050] The image processing requirement information is used to describe the image processing requirements of the application processing chip, including but not limited to what type of image processing to perform and how to perform the image processing.

[0051] In this embodiment, the N frames of RAW images acquired by the data packaging unit 120 may be N frames of unprocessed RAW images directly acquired by the image processing chip 100 from the image sensor, or may be N frames of RAW images preprocessed by the image preprocessing unit. For the acquired N frames of RAW images, the data packaging unit 120 reorganizes the image data in the N frames of RAW images according to the image processing requirement information of the application processing chip, and correspondingly obtains M data blocks that match the image processing requirements of the application processing chip, and the image data in each data block can be used independently in the image processing of the application processing chip. For the M data blocks obtained by the reorganization, the data packaging unit 120 further encapsulates the M data blocks into L data packets according to the configured data encapsulation strategy. The configuration of the data encapsulation strategy is not specifically limited here, and can be configured by those skilled in the art according to actual needs, wherein, with the constraint that L is less than N, L can be dynamically taken as any positive integer, for example, it can be taken as 1.

[0052] For example, image processing requirements information can describe:

[0053] The application processing chip needs to use three RAW images of the same shooting scene for high dynamic range synthesis processing, and needs to first perform high dynamic range synthesis processing on the region of interest (ROI) of the shooting scene, and then perform high dynamic range synthesis processing on the non-interest region of the shooting scene.

[0054] Accordingly, the image sensor can collect three RAW images with different exposure parameters of the aforementioned shooting scene, namely, a long exposure RAW image, a medium exposure RAW image, and a short exposure RAW image. Accordingly, the image processing chip 100 obtains the long exposure RAW image, the medium exposure RAW image, and the short exposure RAW image from the image sensor through the first data interface unit 110.

[0055] Please refer to Figure 2 According to the above image processing requirement information, the data packing unit 120 reorganizes the image data of the region of interest in the long-exposure RAW image, the medium-exposure RAW image, and the short-exposure RAW image into one data block (i.e., data block 1), and reorganizes the image data of the non-interest region in the long-exposure RAW image, the medium-exposure RAW image, and the short-exposure RAW image into another data block (i.e., data block 2). In this way, the image data in the three frames of RAW images are reorganized into two data blocks, and each data block includes all the image data required for the application processing chip to perform high dynamic range synthesis processing.

[0056] For example, image processing requirement information can describe:

[0057] The application processing chip needs to use two RAW images of the same shooting scene to perform synthetic noise reduction processing, and needs to perform synthetic noise reduction processing to different degrees on the focus area and non-focus area of ​​the shooting scene.

[0058] Accordingly, the image sensor can collect two RAW images of the aforementioned shooting scene, namely RAW image 1 and RAW image 2. Accordingly, the image processing chip 100 obtains the RAW image 1 and RAW image 2 from the image sensor through the first data interface unit 110.

[0059] Please refer to Figure 3 According to the above image processing requirement information, the data packing unit 120 reorganizes the image data of the focused areas in RAW image 1 and RAW image 2 into one data block (i.e., data block 3), and reorganizes the image data of the non-focused areas in RAW image 1 and RAW image 2 into another data block (i.e., data block 4). In this way, the image data in the two frames of RAW images are reorganized into two data blocks, and each data block includes all the image data required for the application processing chip to perform synthetic noise reduction processing.

[0060] The second data interface unit 130 is used to provide a data transmission channel between the application processing chip and the image processing chip 100 to realize data transmission between the application processing chip and the image processing chip 100. There is no specific restriction on the entity type of the second data interface unit 130, including but not limited to the mobile industry processor interface (Mobile Industry Processor Interface, MIPI), PCI-E interface, etc. For example, in the embodiment of the present application, the entity type of the second data interface unit 130 is a mobile industry processor interface.

[0061] In this embodiment, the image processing chip 100 transmits the L data packets obtained by the data packaging unit 120 to the application processing chip through the second data interface unit 130, so that the application processing chip performs image processing. It can be understood that since N frames of RAW images are packaged into L data packets, only L data transmission channels of the application processing chip are required for transmission. For example, please refer to Figure 4 (wherein the arrows represent data transmission channels), when the image sensor directly transmits N frames of RAW images to the application processing chip, it is necessary to occupy N data transmission channels of the application processing chip to realize the transmission of N frames of RAW images. When the image processing chip 100 provided by the present application is used to encapsulate the N frames of RAW images collected by the image sensor into L data packets and transmit them to the application processing chip, if the N frames of RAW images are encapsulated into 1 data packet by the data packaging unit 120, only 1 data transmission channel of the application processing chip is needed.

[0062] In addition, it should be noted that in the embodiment of the present application, the data packaging unit 120 does not directly encapsulate N frames of RAW images into data packets, but reorganizes the image data in the N frames of RAW images into M data blocks, and these M data blocks are matched with the image processing required by the application processing chip, and then encapsulates these M data blocks into L data packets. Accordingly, after receiving the data packet transmitted by the image processing chip 100 through the second data interface unit 130, the application processing chip can split the data packet into M data blocks and perform parallel parsing, and perform image processing on the image data obtained by parsing the M data blocks.

[0063] For example, please refer to Figure 2 and Figure 5 On the one hand, the image processing chip 100 is Figure 2 In the reorganization method shown, the three RAW image frames are reorganized into two data blocks through the data packing unit 120, and the two data blocks are packaged into one data packet, and the packaged one data packet is transmitted to the application processing chip through the second data interface unit 130.

[0064] On the other hand, after receiving the data packet, the application processing chip splits the data packet into two data blocks, namely, data block 1 and data block 2, and then parses data block 1 and data block 2 in parallel, and obtains the image data of the area of ​​interest in the long-exposure RAW image from data block 1 ( Figure 5 ), image data of the region of interest in the medium-exposure RAW image ( Figure 5 ) and the image data of the region of interest in the short-exposure RAW image ( Figure 5 The image data of the non-interested area in the long-exposure RAW image, the medium-exposure RAW image and the short-exposure RAW image are obtained by parsing the data block 2. After that, the application processing chip performs high dynamic range synthesis processing on the image data obtained by parsing the data block 1 to obtain the image data of the area of ​​interest in the high dynamic range ( Figure 5 The image data obtained by parsing the data block 2 is processed by high dynamic range synthesis to obtain the image data of the non-interest region of high dynamic range. Finally, the application processing chip splices the image data of the interest region of high dynamic range and the image data of the non-interest region of high dynamic range to obtain the high dynamic range synthesized image.

[0065] As can be seen from the above, the image processing chip 100 provided by the present application includes a data packaging unit 120, through which N frames of RAW images are obtained, and according to the image processing requirement information of the application processing chip, the obtained N frames of RAW images are reorganized to obtain M data blocks, and the M data blocks are encapsulated into L data packets and transmitted to the application processing chip. Among them, N and M are positive integers greater than 1, and L is a positive integer less than N. In this way, by encapsulating N frames of RAW images into L data packets less than N, the image processing chip 100 needs to occupy at most L data transmission channels of the application processing chip to realize the transmission of N frames of RAW images. Compared with the related art that needs to occupy N data transmission channels to realize the transmission of N frames of RAW images, the present application can reduce the occupation of image data on the data transmission channels of the application processing chip, thereby reducing the impact of image data on other types of data transmission and processing. In addition, the reorganized M data blocks can be parsed in parallel by the application processing chip, which can also reduce the time required for parsing.

[0066] In an optional embodiment, the data packaging unit 120 is configured to assign priorities to image data at different positions in each RAW image according to image processing requirement information; and reorganize image data of the same priority in N frames of RAW images into data blocks to obtain M data blocks.

[0067] In this embodiment, in addition to describing what type of image processing the application processing chip performs and how to perform image processing, the image processing requirement information may also describe the importance information defined by the application processing chip for different areas in the shooting scene of the image sensor. Accordingly, when the data packaging unit 120 allocates priorities, it may allocate them according to the importance of the image data. Among them, when the image processing requirement information describes that the application processing chip needs to use image data at the same position in different RAW images for image processing, in order to match the image processing requirements of the application processing chip, the image data at the same position in different RAW images will be assigned the same priority by the data packaging unit 120.

[0068] For example, please refer to Figure 6 For a shooting scene, the application processing chip identifies two regions of interest, namely region of interest 1 ( Figure 6 ) and region of interest 2 ( Figure 6 ROI2 in the figure), and define that the importance of region of interest 1 is greater than that of region of interest 2, and the importance of region of interest 2 is greater than that of non-region of interest (i.e. Figure 6 The importance of the area (except the area of ​​interest 1 and the area of ​​interest 2) in the image.

[0069] Image processing requirements information can describe:

[0070] The application processing chip needs to perform high dynamic range synthesis processing using 3 frames of RAW images of the same shooting scene, the importance of region of interest 1 in the shooting scene is greater than that of region of interest 2, the importance of region of interest 2 is greater than that of non-regions of interest, and it is necessary to perform a high degree of high dynamic range synthesis processing on region of interest 1 in the shooting scene, perform a medium degree of high dynamic range synthesis processing on region of interest 2 in the shooting scene, and perform a low degree of high dynamic range synthesis processing on regions of non-interest in the shooting scene.

[0071] Accordingly, the image sensor can collect three RAW images with different exposure parameters of the aforementioned shooting scene, namely, a long exposure RAW image, a medium exposure RAW image, and a short exposure RAW image. Accordingly, the image processing chip 100 obtains the long exposure RAW image, the medium exposure RAW image, and the short exposure RAW image from the image sensor through the first data interface unit 110. The image preprocessing unit performs image preprocessing on the obtained long exposure RAW image, the medium exposure RAW image, and the short exposure RAW image.

[0072] The data packaging unit 120 obtains the pre-processed long-exposure RAW image, medium-exposure RAW image and short-exposure RAW image, and according to the above image processing requirement information, assigns a priority of "high" to the image data corresponding to the area of ​​interest 1 in the pre-processed long-exposure RAW image, medium-exposure RAW image and short-exposure RAW image, assigns a priority of "medium" to the image data corresponding to the area of ​​interest 2 in the long-exposure RAW image, medium-exposure RAW image and short-exposure RAW image, and assigns a priority of "low" to the image data corresponding to the non-area of ​​interest in the long-exposure RAW image, medium-exposure RAW image and short-exposure RAW image. Then, the data packaging unit 120 reorganizes the image data with a priority of "high" in the long-exposure RAW image, the medium-exposure RAW image and the short-exposure RAW image into a data block, reorganizes the image data with a priority of "medium" in the long-exposure RAW image, the medium-exposure RAW image and the short-exposure RAW image into a data block, and reorganizes the image data with a priority of "low" in the long-exposure RAW image, the medium-exposure RAW image and the short-exposure RAW image into a data block, thereby obtaining three data blocks in total, which respectively include the image data corresponding to the area of ​​interest 1 in the long-exposure RAW image, the medium-exposure RAW image and the short-exposure RAW image, the image data corresponding to the area of ​​interest 2 in the long-exposure RAW image, the medium-exposure RAW image and the short-exposure RAW image, and the image data corresponding to the area of ​​non-interest in the long-exposure RAW image, the medium-exposure RAW image and the short-exposure RAW image.

[0073] It should be noted that in other embodiments, the data packaging unit 120 may also allocate priorities based on other attribute information other than the importance of the image data. For example, the data packaging unit 120 may also allocate priorities based on the exposure level of the image data.

[0074] In an optional embodiment, the data block includes multiple data rows, the data rows include a row header portion and a row body portion, and the data packaging unit 120 is configured to write image data of the same priority in N frames of RAW images into the row body portions of different data rows of the same data block in sequence; and obtain first description information of the image data written in the row body portion of each data row, and write the first description information corresponding to each data row into the row header portion of each data row.

[0075] It should be noted that the first description information at least includes description information for describing how the data packaging unit reorganizes the image data. For example, the first description information may include the source RAW image information of the image data written in the row body part of the data row (used to describe which RAW image the image data comes from), the position information in the source RAW image, the data size information and the priority information, etc.

[0076] When reorganizing the image data, the data packing unit 120 may alternately write the image data of the same priority in the N frames of RAW images into the row bodies of different data rows of the same data block. That is, in the same data block, the image data written into the row bodies of all data rows have the same priority, but the image data written into the row bodies of two adjacent data rows come from different RAW images.

[0077] For example, according to Figure 6 The priority allocation method corresponding to the shown region of interest (please refer to the relevant description in the above embodiment for details) is explained by taking the reorganization of image data with "high" priority in long-exposure RAW images, medium-exposure RAW images and short-exposure RAW images as an example.

[0078] Please refer to Figure 7, where the value of O depends on the data size of the image data to be written. The data packaging unit 120 identifies whether there is image data with a "high" priority in the first row of image data in the short-exposure RAW image. If so, the image data with a "high" priority is written into the row body portion of data row 1. If not, the data packaging unit 120 continues to identify whether there is image data with a "high" priority in the next row of image data until it is identified that the image data with a "high" priority is written into the row body portion of data row 1. Then, the data packaging unit 120 identifies whether there is image data with a "high" priority in the first row of image data in the medium-exposure RAW image. If so, the image data with a "high" priority is written into the row body portion of data row 2. If not, the data packing unit 120 continues to identify whether there is image data with a "high" priority in the next row of image data, until it is identified that the image data with a "high" priority is written into the row body portion of data row 2; then, the data packing unit 120 identifies whether there is image data with a "high" priority in the first row of image data in the long-exposure RAW image, and if so, writes the image data with a "high" priority therein into the row body portion of data row 3; if not, the data packing unit 120 continues to identify whether there is image data with a "high" priority in the next row of image data, until it is identified that the image data with a "high" priority is written into the row body portion of data row 3. In this way, the data packing unit 120 writes the image data with a "high" priority in the short-exposure RAW image, the medium-exposure RAW image, and the long-exposure RAW image into the row body portions of different data rows of the same data block in an alternating manner, row by row, in the order of the short-exposure RAW image, the medium-exposure RAW image, and the long-exposure RAW image. In addition, for each data row, the data packaging unit 120 also obtains the first description information of the image data written in the row body of each data row, and writes the first description information corresponding to each data row into the row header of each data row. For example, the first description information obtained at this time may include: source RAW image information (here used to describe that the written image data comes from a short-exposure RAW image, a medium-exposure RAW image, or a long-exposure RAW image), position information in the source RAW image (used to describe the number of rows in the entire source RAW image where the written image data is located), data size information, priority information (used to describe whether the priority of the written image data is high, medium, or low), marking information of whether it belongs to the region of interest, the number of blocks (describing how many blocks of a row of image data in the source RAW image are included in the written image data), the starting horizontal coordinate of the region of interest, and the ending horizontal coordinate of the region of interest, etc.

[0079] Please continue to refer to Figure 8, the number of blocks is explained by taking the reorganization of a row of image data in a long exposure RAW image as an example. The row of image data can be divided into 5 blocks, namely block 1, block 2, block 3, block 4 and block 5, wherein block 1, block 3 and block 5 belong to the non-interested region, block 2 belongs to the interested region 1, and block 4 belongs to the interested region 2. Accordingly, block 1, block 3 and block 5 will be reorganized into the same data row of the same data block by the data packaging unit 120, and for the data row, the number of blocks in its first description information is 3; block 2 will be reorganized into a data row of a data block by the data packaging unit 120, and for the data row, the number of blocks in its first description information is 1; block 2 will be reorganized into a data row of a data block by the data packaging unit 120, and for the data row, the number of blocks in its first description information is 1.

[0080] In an optional embodiment, the data packet includes a header part and a body part, when M is greater than or equal to N, L is equal to 1, and the data packaging unit 120 is configured to write the M data blocks into the body part of the same data packet in sequence according to the priorities corresponding to the M data blocks, and obtain the second description information of the M data blocks, and write the second description information into the header part of the same data packet to obtain one data packet; or,

[0081] When M is less than N, L is equal to M, and the data packaging unit 120 is configured to write the M data blocks into the body of M data packets respectively, and obtain the third description information of the data blocks written in the body of each data packet, and write the third description information corresponding to each data packet into the header of each data packet, so as to obtain M data packets.

[0082] The second description information includes the number of data blocks written in the packet body (i.e., the value of M), the priority information of each data block, the position information (used to describe the position of the data block in the entire data packet), and at least one of the data size information; the third description information includes the sequence number information of the data blocks written in the packet body (used to describe the number of the written data block among the M data blocks), the priority information, and at least one of the data size information.

[0083] For example, please refer to Fig. 9, the data packing unit 120 reorganizes the N frames of RAW images acquired by the first data interface unit 110 into M (3) data blocks, namely data block 1, data block 2 and data block 3, and assigns a priority of "high" to data block 1, a priority of "medium" to data block 2, and a priority of "low" to data block 3. Correspondingly, since M is equal to N at this time, the data packing unit 120 writes data block 1, data block 2 and data block 3 into the body of the same data packet in sequence, and then the data packing unit 120 obtains the second description information of the three data blocks and writes it into the header, which includes the number information of the written data blocks "3", the position information, priority information and data size information of data block 1, the position information, priority information and data size information of data block 2, and the position information, priority information and data size information of data block 3.

[0084] In an optional embodiment, the image processing chip 100 is configured to send the header portion of the data packet to the application processing chip; and send the M data blocks in the body portion of the data packet to the application processing chip in sequence according to the priorities corresponding to the M data blocks in the body portion of the data packet.

[0085] For example, the second data interface unit 130 first sends the header part of the data packet to the application processing chip, and then sends the M data blocks in the packet body to the application processing chip in descending order according to the priority of the M data blocks written in the packet body.

[0086] In this embodiment, the application processing chip may perform parallel parsing and image processing on the M data blocks after receiving a complete data packet, or may perform parsing and image processing on the received data blocks after receiving one data block each time.

[0087] In an optional embodiment, the image processing chip 100 is configured to send M data packets to the application processing chip in parallel.

[0088] For example, assuming that the data packaging unit 120 encapsulates three data packets, the image processing chip 100 sends the three encapsulated data packets in parallel to the application processing chip through the three data transmission channels provided by the second data interface unit 130, so that the application processing chip performs image processing.

[0089] In an optional embodiment, the data packaging unit 120 is configured to acquire N frames of RAW images when the number of idle data transmission channels of the application processing chip is less than N, and reorganize the image data in the N frames of RAW images into M data blocks and then encapsulate them into L data packets according to the image processing requirement information of the application processing chip.

[0090] In this embodiment, the image processing chip 100 does not always package and transmit the N frames of images from the image sensor. The data packaging unit 120 identifies the number of idle data transmission channels of the application processing chip in real time, and when the number of idle data transmission channels of the application processing chip is less than N, obtains N frames of RAW images, and reorganizes the image data in the N frames of RAW images into M data blocks and then encapsulates them into L data packets according to the image processing requirement information of the application processing chip. Please refer to the relevant description in the above embodiment for details, which will not be repeated here.

[0091] In this embodiment, the second data interface unit 130 is configured to transmit the data packet to the application processing chip through an idle data transmission channel of the application processing chip.

[0092] For example, when the application processing chip has only one idle data transmission channel, the image processing chip 100 may transmit the data packet to the application processing chip through the only data transmission channel provided by the second data interface unit 130;

[0093] For another example, when the application processing chip has more than one idle data transmission channel, the image processing chip 100 may transmit the data packet to the application processing chip through any idle data transmission channel provided by the second data interface unit 130, and may also split the data packet into the same number of sub-data packets according to the number of idle data transmission channels, and balance the split sub-data packets to each idle data transmission channel for transmission to the application processing chip.

[0094] In other embodiments, when the number of idle data channels of the application processing chip is greater than or equal to N, the image processing chip 100 is configured to transmit N frames of RAW images to the application processing chip in parallel.

[0095] Please refer to Fig.10 The present application also provides an application processing chip 200, such as Fig.10 As shown, the application processing chip 200 includes a third data interface unit 210 , a data unpacking unit 220 and an image processing unit 230 .

[0096] It should be noted that the application processing chip 200 provided in the present application can be configured in an electronic device having an image sensor and an image processing chip provided in the present application to implement image processing. The physical type of the electronic device can be a mobile electronic device such as a smart phone, a tablet computer, a PDA, a laptop computer, or a fixed electronic device such as a desktop computer or a television, and the present application does not impose any specific restrictions on this.

[0097] In this embodiment, the third data interface unit 210 is used to provide a data transmission channel between the application processing chip 200 and the image processing chip to realize data transmission between the application processing chip 200 and the image processing chip. There is no specific restriction on the entity type of the third data interface unit 210, including but not limited to the mobile industry processor interface (Mobile Industry Processor Interface, MIPI), PCI-E interface, etc. For example, in the embodiment of the present application, the entity type of the third data interface unit 210 is a mobile industry processor interface.

[0098] The image processing unit 230 is configured to generate image processing requirement information, where the image processing requirement information is used to describe the image processing requirements of the image processing unit 230 , including but not limited to what type of image processing to perform and how to perform the image processing.

[0099] The application processing chip 200 is configured to transmit the image processing requirement information of the image processing unit 230 to the image processing chip through the third data interface unit 210, so that the image processing chip can reorganize the image data in the acquired N frames of RAW images into M data blocks according to the image processing requirement information and then encapsulate them into L data packets, where N and M are positive integers greater than 1, and L is a positive integer less than N. For specific instructions, please refer to the relevant description in the above image processing chip embodiment, which will not be repeated here. In addition, the application processing chip 200 is also configured to obtain L data packets corresponding to the image processing requirement information from the image processing chip. Among them, the application processing chip 200 can receive L data packets sent by the image processing chip through the third data interface unit 210.

[0100] The data unpacking unit 220 is configured to split the L data packets into M data blocks and then perform parallel parsing to obtain image data. After the application processing chip 200 obtains L data packets corresponding to the image processing requirement information, the data unpacking unit 220 reversely parses the L data packets into M data blocks, and performs parallel parsing on the M data blocks to obtain the corresponding image data.

[0101] The image processing unit 230 is configured to perform image processing on the image data obtained by parsing the M data blocks.

[0102] For example, image processing requirements information can describe:

[0103] The image processing unit 230 needs to perform high dynamic range synthesis processing using three RAW images of the same shooting scene, and needs to first perform high dynamic range synthesis processing on the region of interest (ROI) of the shooting scene, and then perform high dynamic range synthesis processing on the non-interest region of the shooting scene.

[0104] Accordingly, the image sensor can collect 3 frames of RAW images with different exposure parameters of the aforementioned shooting scene, which are long exposure RAW images, medium exposure RAW images and short exposure RAW images. Accordingly, the image processing chip obtains long exposure RAW images, medium exposure RAW images and short exposure RAW images from the image sensor. And according to the above image processing requirement information, the image data of the area of ​​interest in the long exposure RAW images, medium exposure RAW images and short exposure RAW images are reorganized into one data block, and the image data of the non-interested area in the long exposure RAW images, medium exposure RAW images and short exposure RAW images are reorganized into another data block. In this way, the image data in the 3 frames of RAW images are reorganized into 2 data blocks, and each data block includes all the image data required for the image processing unit 230 to perform high dynamic range synthesis processing. Afterwards, the image processing chip packages the reorganized 2 data blocks into 2 data packets and transmits them to the application processing chip 200. On the other hand, the application processing chip 200 will receive the 2 data packets transmitted by the image processing chip through the third data interface unit 210.

[0105] Afterwards, the data unpacking unit 220 splits the two data packets into two data blocks, and then parses the two data blocks in parallel, obtains image data of the area of ​​interest in the long-exposure RAW image, the medium-exposure RAW image, and the short-exposure RAW image from one of the data blocks, and obtains image data of the area of ​​non-interest in the long-exposure RAW image, the medium-exposure RAW image, and the short-exposure RAW image from the other data block.

[0106] Afterwards, the image processing unit 230 performs high dynamic range synthesis processing on the image data of the region of interest in the long-exposure RAW image, the medium-exposure RAW image, and the short-exposure RAW image to obtain image data of the region of interest with a high dynamic range, and performs high dynamic range synthesis processing on the image data of the non-interest region in the long-exposure RAW image, the medium-exposure RAW image, and the short-exposure RAW image to obtain image data of the non-interest region with a high dynamic range. Finally, the image processing unit 230 splices the image data of the region of interest with the image data of the non-interest region with a high dynamic range to obtain a high dynamic range synthesized image.

[0107] In an optional embodiment, the data unpacking unit 220 is configured to call M idle parsing threads from a preset thread pool to parse the M data blocks in parallel.

[0108] The thread pool includes multiple different types of threads, which are pre-built.

[0109] For example, assuming that the image processing chip reorganizes the acquired 3 frames of RAW images into 2 data blocks, the data unpacking unit 220 can call 2 idle parsing threads from the preset thread pool to parse the 2 data blocks in parallel.

[0110] In an optional embodiment, the data unpacking unit 220 is further configured to call the idle parsing threads in the thread pool to parse the M data blocks in sequence according to the priority corresponding to each data block when the number of idle parsing threads in the thread pool is less than M.

[0111] In this embodiment, when there are not enough idle parsing threads to perform parallel parsing on the split data blocks, the data unpacking unit 220 will parse the data blocks in sequence according to their priorities.

[0112] For example, when there is only one idle parsing thread, the data unpacking unit 220 may call the only idle parsing thread to parse the M data blocks obtained by splitting the data packet in order from high to low priority;

[0113] For another example, when there are K idle parsing threads (K is a positive integer greater than 1 and less than M), the data unpacking unit 220 can call K idle parsing threads in order of priority from high to low to perform parallel parsing on the K data blocks with the highest priority first; if the number of remaining unparsed data blocks is greater than K, the K idle parsing threads can be called again to perform parallel parsing on the remaining K data blocks with the highest priority until the number of remaining unparsed data blocks is less than K; at this time, according to the number of remaining unparsed data blocks, a corresponding number of idle parsing threads can be called to parse the remaining unparsed data blocks, where if there is 1 unparsed data block left, 1 idle parsing thread is called to parse the remaining 1 unparsed data block, and if there are multiple unparsed data blocks left, a corresponding number of idle parsing threads are called to perform parallel parsing on the remaining unparsed data blocks.

[0114] In an optional embodiment, the image processing unit 230 is further configured to perform image processing on the image data obtained by parsing each time the data unpacking unit 220 completes parsing of a data block among the M data blocks.

[0115] It should be noted that different data blocks have different data sizes. When the data unpacking unit 220 parses the data blocks, some data blocks will be parsed first and some data blocks will be parsed later. Since the image data in each data block can be used for image processing independently, in order to improve the image processing efficiency, the image processing unit 230 can perform image processing on the parsed image data each time the data unpacking unit 220 completes parsing of one data block among the M data blocks.

[0116] Please refer to Fig.11 The present application also provides an electronic device 10, comprising an image processing chip 100 and an application processing chip 200, wherein:

[0117] The application processing chip 200 is configured to generate image processing requirement information and transmit the image processing requirement information to the image processing chip 100;

[0118] The image processing chip 100 is configured to acquire N frames of RAW images, where N is a positive integer greater than 1; and reorganize the image data in the N frames of RAW images into M data blocks according to the image processing requirement information, where M is a positive integer greater than 1; and encapsulate the M data blocks into L data packets and transmit them to the application processing chip 200;

[0119] The application processing chip 200 is further configured to split the L data packets into M data blocks and then perform parallel parsing to obtain image data, and perform image processing on the parsed image data.

[0120] Please refer to the above related embodiments for specific details, which will not be repeated here.

[0121] Please refer to Fig.12 The present application also provides an image processing method, which is applied to an image processing chip and includes:

[0122] In 410 , N frames of RAW images are acquired, where N is a positive integer greater than 1.

[0123] In 420 , according to the image processing requirement information of the application processing chip, the image data in the N frames of RAW images are reorganized into M data blocks, where M is a positive integer greater than 1.

[0124] In 430 , the M data blocks are encapsulated into L data packets, where L is a positive integer less than N.

[0125] In 440 , the L data packets are transmitted to the application processing chip.

[0126] In an optional embodiment, priorities are assigned to image data at different positions in each RAW image according to image processing requirement information of an application processing chip; and image data of the same priority in N frames of RAW images are reorganized into data blocks to obtain M data blocks.

[0127] In an optional embodiment, the data block includes multiple data rows, and the data row includes a row header portion and a row body portion. Image data of the same priority in N frames of RAW images are sequentially written into the row body portions of different data rows of the same data block; first description information of the image data written in the row body portion of each data row is obtained, and the first description information corresponding to each data row is written into the row header portion of each data row.

[0128] In an optional embodiment, the first description information includes at least one of source RAW image information of the image data written in the row body portion of the data row, position information in the source RAW image, data size information, and priority information.

[0129] In an optional embodiment, the data packet includes a header part and a body part. When M is greater than or equal to N, L is equal to 1, and the M data blocks are written into the body part in sequence according to the priorities corresponding to the M data blocks; the second description information of the M data blocks is obtained, and the second description information is written into the header part.

[0130] In an optional embodiment, the packet header portion is sent to the application processing chip; and the M data blocks in the packet body portion are sent to the application processing chip in sequence according to the priorities corresponding to the M data blocks.

[0131] In an optional embodiment, when M is less than N, L is equal to M, the M data blocks are respectively written into the body of M data packets, and the third description information of the data blocks written in the body of each data packet is obtained, and the third description information corresponding to each data packet is written into the header of each data packet.

[0132] In an optional embodiment, the second description information includes at least one of the number information of data blocks written in the package body, priority information of each data block, location information and data size information; and / or, the third description information includes at least one of the sequence number information, priority information and data size information of the data blocks written in the package body.

[0133] It should be noted that the image processing method provided in the embodiment of the present application and the above image processing chip belong to the same concept. For detailed description, please refer to the relevant description of the image processing chip in the above embodiment, which will not be repeated here.

[0134] Please refer to Fig.13 The present application also provides an image processing method, which is applied to an application processing chip, including:

[0135] In 510 , image processing requirement information is generated and transmitted to the image processing chip.

[0136] In 520, L data packets corresponding to image processing requirement information are obtained from the image processing chip.

[0137] In 530, L data packets are split into M data blocks and then parsed in parallel to obtain image data. The L data packets are obtained by reorganizing image data in N frames of RAW images into M data blocks and then encapsulating them. N and M are positive integers greater than 1, and L is a positive integer less than N.

[0138] In 540, image processing is performed on the analyzed image data.

[0139] In an optional embodiment, M idle parsing threads are called from a preset thread pool to parse the M data blocks in parallel.

[0140] In an optional embodiment, when the number of idle parsing threads in the thread pool is less than M, the idle parsing threads in the thread pool are called to parse the M data blocks in sequence according to the priority corresponding to each data block.

[0141] In an optional embodiment, each time when parsing of a data block among the M data blocks is completed, image processing is performed on the image data obtained by parsing.

[0142] It should be noted that the image processing method provided in the embodiment of the present application and the application processing chip provided above belong to the same concept. For detailed description, please refer to the relevant description of the application processing chip in the above embodiment, which will not be repeated here.

[0143] The above describes in detail the image processing chip, application processing chip, electronic device and image processing method provided by the embodiments of the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the present application. At the same time, for those skilled in the art, according to the ideas of the present application, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as limiting the present application.

Claims

1. An image processing chip, characterized in that: The image processing chip is used to obtain N frames of RAW images from an image sensor through N transmission channels, where N is a positive integer greater than 1, and includes: a data packaging unit, configured to reorganize the image data used for the same image processing of the application processing chip in the N frames of RAW images into data blocks according to the image processing requirement information of the application processing chip, to obtain M data blocks, where M is a positive integer greater than 1; and encapsulate the M data blocks into L data packets, where L is a positive integer less than N; The image processing chip is used to transmit the L data packets to the application processing chip through at most L transmission channels.

2. The image processing chip according to claim 1, characterized in that: The data packaging unit is used to assign priorities to image data in each RAW image used for different image processing by the application processing chip according to the image processing requirement information; and to reorganize image data of the same priority in the N frames of RAW images into data blocks to obtain the M data blocks.

3. The image processing chip according to claim 2, characterized in that: The data block includes a plurality of data rows, each of which includes a row header portion and a row body portion, and the data packaging unit is used to sequentially write image data of the same priority in the N frames of RAW images into row body portions of different data rows in the same data block; And obtaining the first description information of the image data written in the row body part of each data row, and writing the first description information corresponding to each data row into the row header part of each data row.

4. The image processing chip according to claim 3, characterized in that: The first description information includes at least one of source RAW image information of the image data written in the row body portion of the data row, position information in the source RAW image, data size information, and priority information.

5. The image processing chip according to claim 2, characterized in that: The data packet includes a header part and a body part. When M is greater than or equal to N, L is equal to 1. The data packaging unit is used to write the M data blocks into the body part in sequence according to the priorities corresponding to the M data blocks; and obtain second description information of the M data blocks, and write the second description information into the header part.

6. The image processing chip according to claim 5, characterized in that: The image processing chip is used to send the packet header part to the application processing chip; and send the M data blocks in the packet body part to the application processing chip in sequence according to the priorities corresponding to the M data blocks.

7. The image processing chip according to claim 5, characterized in that: When M is less than N, L is equal to M, and the data packaging unit is used to write the M data blocks into the packet body of M data packets respectively, and obtain the third description information of the data blocks written in the packet body of each data packet, and write the third description information corresponding to each data packet into the header part of each data packet.

8. The image processing chip according to claim 7, characterized in that: The second description information includes at least one of the number of data blocks written in the packet body, priority information, location information and data size information of each data block; and / or The third description information includes at least one of sequence number information, priority information and data size information of the data block written in the packet body.

9. An application processing chip, characterized in that: The application processing chip is used to transmit image processing requirement information to the image processing chip, and obtain L data packets corresponding to the image processing requirement information from the image processing chip, including: An image processing unit, used to generate the image processing requirement information; a data unpacking unit, configured to split the L data packets into M data blocks and perform parallel parsing to obtain image data, wherein the L data packets are obtained by the image processing chip encapsulating the M data blocks and transmitting them through at most L transmission channels, and each data block is obtained by the image processing chip reassembling image data used for the same image processing of the application processing chip from N frames of RAW images acquired from the image sensor through N transmission channels, wherein N and M are positive integers greater than 1, and L is a positive integer less than N; The image processing unit is also used to perform image processing on the image data obtained by analysis.

10. The application processing chip according to claim 9, characterized in that: The data unpacking unit is used to call M idle parsing threads from a preset thread pool to parse the M data blocks in parallel.

11. The application processing chip according to claim 10, characterized in that: The data unpacking unit is further used to call the idle parsing threads in the thread pool to parse the M data blocks in sequence according to the priority corresponding to each data block when the number of idle parsing threads in the thread pool is less than M.

12. The application processing chip according to claim 11, characterized in that: The image processing unit is further configured to perform image processing on the image data obtained by parsing each time the data unpacking unit completes parsing of a data block among the M data blocks.

13. An electronic device, characterized in that: It includes image sensor, image processing chip and application processing chip, among which: The application processing chip is used to generate image processing requirement information and transmit the image processing requirement information to the image processing chip; The image processing chip is used to obtain N frames of RAW images from an image sensor through N transmission channels, where N is a positive integer greater than 1; and reorganize the image data in the N frames of RAW images for the same image processing of the application processing chip into data blocks according to the image processing requirement information to obtain M data blocks, where M is a positive integer greater than 1; and encapsulate the M data blocks into L data packets and transmit them to the application processing chip through at most L transmission channels, where L is a positive integer less than N; The application processing chip is further used for splitting the L data packets into the M data blocks and then performing parallel parsing to obtain image data, and performing image processing on the parsed image data.

14. An image processing method, applied to an image processing chip, characterized in that: include: Acquire N frames of RAW images from the image sensor through N transmission channels, where N is a positive integer greater than 1; According to the image processing requirement information of the application processing chip, the image data in the N frames of RAW images used for the same image processing of the application processing chip are reorganized into data blocks to obtain M data blocks, where M is a positive integer greater than 1; Encapsulating the M data blocks into L data packets, where L is a positive integer less than N; The L data packets are transmitted to the application processing chip through at most L transmission channels.

15. An image processing method, applied to an application processing chip, characterized in that: include: Generate image processing requirement information, and transmit the image processing requirement information to the image processing chip; Acquire L data packets corresponding to the image processing requirement information from the image processing chip; The L data packets are split into M data blocks and then analyzed in parallel to obtain image data, wherein the L data packets are obtained by the image processing chip encapsulating the M data blocks and transmitting them through at most L transmission channels, and each data block is obtained by the image processing chip recombining image data used for the same image processing of the application processing chip from N frames of RAW images acquired from the image sensor through N transmission channels, where N and M are positive integers greater than 1, and L is a positive integer less than N; Perform image processing on the image data obtained by analysis.

Images