Image processing circuit and method, electronic equipment and chip

Abstract

The invention discloses an image processing circuit and method, electronic equipment and a chip, and belongs to the technical field of image processing. The image processing circuit comprises a main control chip and an image processing chip; a first mobile industry processor interface MIPI of the image processing chip is connected with a second MIPI of the main control chip to form a first path; the image processing chip is used for acquiring a preview image acquired by the image sensor and metadata of the preview image; transmitting the preview image and the metadata to the main control chip through at least one virtual channel in the first channel; and the main control chip is used for carrying out image processing on the preview image according to the metadata and outputting a shot image.

Description

Technical Field

[0001] The present application belongs to the field of image processing technology, and specifically relates to an image processing circuit, a circuit method, an electronic device and a chip. Background Art

[0002] In order to improve the image processing capability of electronic devices, external image processing chips are usually customized as an auxiliary to the image signal processor (ISP).

[0003] After the RAW image output by the image sensor is processed by the external image processing chip, metadata of the corresponding image frame is usually added, such as image depth data, image motion vector (MV) data, etc. These metadata need to be aligned with the preview image frame and synchronously output to the application processor (AP) for further processing by the AP.

[0004] Due to the interface limitation between AP and System on Chip (SoC), preview image frames can only be transmitted between external image processing chip and AP using Mobile Industry Processor Interface (MIPI). Usually, preview image is transmitted to AP through one MIPI interface, while metadata information is transmitted to AP through another MIPI interface or Secure Digital Input Output (SDIO) interface. In this scheme, because preview image frame and corresponding metadata are transmitted using different interfaces, the timestamps of preview image frame and corresponding metadata will be out of sync. Summary of the invention

[0005] The purpose of the embodiments of the present application is to provide a data transmission method, circuit, electronic device and chip, wherein the image processing chip can use a single MIPI interface to perform synchronous transmission of preview images and metadata, thereby avoiding the problem of asynchronous timestamps of preview image frames and metadata, and saving the hardware interface and transmission power consumption used to transmit metadata.

[0006] In a first aspect, an embodiment of the present application provides an image processing circuit, comprising a main control chip and an image processing chip, wherein a first mobile industry processor interface MIPI of the image processing chip is connected to a second MIPI of the main control chip to form a first path; the image processing chip is used to obtain a preview image captured by an image sensor and metadata of the preview image; the preview image and the metadata are transmitted to the main control chip through at least one virtual channel in the first path; the main control chip is used to perform image processing on the preview image according to the metadata and output a captured image.

[0007] In a second aspect, an embodiment of the present application provides an image processing device, comprising the image processing circuit as described in the first aspect.

[0008] In a third aspect, an embodiment of the present application provides a data transmission method, which is performed by the image processing device according to the second aspect, and the method includes:

[0009] The image processing chip acquires a preview image captured by the image sensor and metadata of the preview image;

[0010] The image processing chip transmits the preview image and the metadata to the main control chip through at least one virtual channel in the first path;

[0011] The main control chip performs image processing on the preview image and the metadata according to the metadata to generate a captured image.

[0012] In a fourth aspect, an embodiment of the present application provides an image processing chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the steps performed by the image processing chip in the image processing method described in the third aspect.

[0013] In the fifth aspect, an embodiment of the present application provides a main control chip, wherein the image processing chip includes a processor and a communication interface, the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the steps performed by the main control chip in the image processing method described in the third aspect.

[0014] In a sixth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the third aspect are implemented.

[0015] In a seventh aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the method described in the third aspect.

[0016] In an embodiment of the present application, the image processing circuit includes a main control chip and an image processing chip, wherein the first mobile industry processor interface MIPI of the image processing chip is connected to the second MIPI of the main control chip to form a first path; the image processing chip is used to obtain a preview image collected by an image sensor and metadata of the preview image; the preview image and the metadata are transmitted to the main control chip through at least one virtual channel in the first path; the main control chip is used to perform image processing on the preview image according to the metadata and output a captured image. In this way, the image processing chip can realize the transmission of preview image frames and metadata using a single MIPI interface, avoiding the problem of asynchronous timestamps of preview image frames and metadata, and saving the hardware interface and transmission power consumption for transmitting metadata. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 is a schematic diagram of an image processing circuit in some embodiments of the present application;

[0018] Figure 2 is a schematic diagram of the data frame format in some embodiments of the present application;

[0019] Figure 3 is a schematic diagram of the data frame format in some embodiments of the present application;

[0020] Figure 4 is a schematic diagram of the data frame format in some embodiments of the present application;

[0021] Figure 5 is a schematic diagram of the data frame format in some embodiments of the present application;

[0022] Figure 6 is a schematic diagram of the data frame format in some embodiments of the present application;

[0023] Figure 7 is a schematic diagram of the data frame format in some embodiments of the present application;

[0024] Figure 8 is a flowchart of an image processing method in some embodiments of the present application;

[0025] Fig. 9 is a schematic diagram of the hardware structure of an image processing device in some embodiments of the present application;

[0026] Fig.10 It is a schematic diagram of the hardware structure of the image processing device in some embodiments of the present application. DETAILED DESCRIPTION

[0027] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0028] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0029] For example, in application scenarios such as video or image shooting, the preview image captured by the image sensor will generate metadata after being processed by the image processing chip. Usually, the preview image frame and the corresponding metadata are transmitted to the application processor (i.e., the main control chip) using different MIPI interfaces, which will cause the preview image frame and metadata to be unable to be transmitted synchronously. In view of this application scenario, the present application proposes an image processing circuit and method for realizing the synchronous transmission of preview images and metadata.

[0030] The image processing circuit provided in the embodiment of the present application is described in detail below through specific embodiments and application scenarios in conjunction with the accompanying drawings.

[0031] like Figure 1 As shown, an embodiment of the present application provides an image processing circuit, including a main control chip and an image processing chip, wherein a first mobile industry processor interface MIPI of the image processing chip is connected to a second MIPI of the main control chip to form a first path;

[0032] The image processing chip is used to obtain a preview image captured by an image sensor and metadata of the preview image; and transmit the preview image and the metadata to the main control chip through at least one virtual channel in the first path;

[0033] The main control chip is used to perform image processing on the preview image according to the metadata and output a captured image.

[0034] It should be noted that the image processing chip is a chip that is external to the main control chip; the main control chip can be used as a central control unit in an electronic device to implement various control logics, for example, the main control chip can be an application processor (AP) or a system on chip (SOC). The main control chip can be used to perform conventional image processing on image data.

[0035] Exemplarily, in an image taking scenario, a user opens a camera application installed in an electronic device and enters a shooting preview interface. When the user triggers a photo taking operation through a photo taking control, the camera application, driven by the image processing chip, sends a shooting instruction and related shooting parameters to the image processing chip. The image processing chip controls the image sensor to complete the frame grabbing action and reads a preview image frame from the memory of the image sensor. The preview image frame is an original image in RAW format, i.e., an unprocessed image. Furthermore, metadata of the preview image frame is obtained through the image processing chip.

[0036] Exemplarily, in a video recording scenario, a user opens a camera application installed in an electronic device and enters a shooting preview interface. When the user triggers a video recording operation through a video recording control, the camera application, driven by the image processing chip, sends a video recording instruction and related shooting parameters to the image processing chip. The image processing chip controls the image sensor to complete a frame capture action and reads a preview image frame sequence from the memory of the image sensor. The preview image frame sequence is a series of original images in RAW format. Furthermore, metadata of each preview image is obtained through the image processing chip. Finally, the image processing chip transmits each preview image and corresponding metadata to the main control chip through at least one virtual channel in the first path.

[0037] Optionally, the metadata includes at least one of the following: embedded data, depth map, motion vector of the image, etc.; wherein the embedded data is additional information related to the image; the depth map is used to indicate the distance between the object in the scene and the camera; the motion vector data is used to describe the movement direction and speed of the object in the image.

[0038] For example, Figure 1 MIPI TX in is the first MIPI of image processing chip. Figure 3The MIPI RX in the main control chip is the second MIPI, the first MIPI is connected with the second MIPI to form a first channel, the first channel includes multiple virtual channels (VC), the image processing chip obtains the preview image frame collected by the image sensor and obtains the metadata of the preview image, and then sends the preview image and metadata to the main control chip through one of the virtual channels VC0, and uses the data type DT supported by the virtual channel to distinguish and identify the preview image frame and different types of metadata. Figure 1 In the process, the data type DT0 in the virtual channel VC0 is used to carry the preview image, the data type DT1 in the virtual channel VC0 is used to carry the embedded data, the data type DT2 in the virtual channel VC0 is used to carry the depth map, and the data type DT3 in the virtual channel VC0 is used to carry the motion vector data, so as to realize the transmission of the preview image and metadata to the main control chip through the data of four different DTs of the same VC.

[0039] In a specific implementation, the processing pipeline of the image signal processor (ISP) on the main control chip side may include: using the image front end (IFE) to perform denoising, color filtering, exposure control and other processing on the preview image according to the metadata to obtain the first image data; using the image processing engine (IPE) to perform color correction, sharpening, noise reduction and image enhancement and other processing on the first image data according to the metadata to obtain the second image data; using the codec to compress and encode the second image data to obtain the captured image.

[0040] In the above embodiment, the image processing chip forms at least one virtual channel in the first path through the first MIPI and the second MIPI connection, and sends the preview image and the metadata of the preview image to the main control chip. In this way, the image processing chip can use a single MIPI interface to perform the transmission of the preview image and metadata, that is, the preview image and metadata are transmitted simultaneously on the same physical link. Since there is a unified clock reference in one MIPI path, the synchronous transmission of the preview image and the metadata can be guaranteed, and the problem of the timestamp asynchronization of the preview image and the metadata can be avoided. Moreover, since the image processing chip and the main control chip only use one MIPI, the hardware interface and transmission for transmitting metadata are saved.

[0041] In some optional embodiments, the image processing chip is specifically used for:

[0042] Transmitting the preview image to the main control chip through a first virtual channel;

[0043] During a vertical blank period of the preview image, the metadata is transmitted to the main control chip through the first virtual channel; wherein the vertical blank period is a blank time period between the preview image and an adjacent image frame.

[0044] For example, Figure 2 In the example, the first virtual channel is VC0. Assuming that VC0 supports four data types DT, and metadata includes three types of embedded data, depth map, and motion vector of the image, DT0 can be used to carry the preview image, DT1 can be used to carry the embedded data, DT2 can be used to carry the depth map, and DT3 can be used to carry the motion vector data. Since there is a vertical blank period (VBlank) between two adjacent image frames, DT0 can be used to carry the preview image frame and place the preview image at the front of the entire data frame, while the embedded data, depth map, and motion vector data are placed as DT1\DT2\DT3 in the vertical blank period after the preview image frame for transmission, forming a structure such as Figure 2 The image data is shown in Figure 1. These metadata are located in the Vblank signal on the MIPI physical signal. Figure 2 The frame interval in is the blank time interval in the vertical blank period excluding the period occupied by metadata. In this way, it is possible to use different data types supported by the same virtual channel to achieve synchronous transmission of preview images and metadata, and to achieve simultaneous transmission of preview images and metadata on the same physical link. Since a MIPI channel has the same clock reference, it is possible to ensure synchronous transmission of preview images and metadata.

[0045] in, Figure 2 FS indicates the start position of the frame, and FE indicates the end position of the frame.

[0046] It should be noted that in the high dynamic range (HDR) mode of the image sensor, in extreme cases, multiple frames of images taken with different exposure times in the same scene will overlap or have very small gaps in the vertical blanking period, and the vertical blanking period cannot accommodate the transmission of metadata. To solve this problem, the present application provides the following embodiments to achieve the transmission of metadata through the horizontal blanking period; wherein the horizontal blanking period refers to the blanking period between two adjacent pixel rows.

[0047] It should be noted that the length and definition of the horizontal blanking period may vary in different video signal standards. The main function of the horizontal blanking period is to allow the display device to make necessary electrical adjustments and preparations before scanning the next pixel row to ensure the stability and continuity of the image. The length of the horizontal blanking period is usually fixed and depends on the resolution and refresh rate of the display device. Higher resolutions usually result in longer horizontal blanking periods.

[0048] In some optional embodiments, the image processing chip is specifically used for:

[0049] Transmitting the preview image to the main control chip through a first virtual channel;

[0050] During at least one horizontal blank period of the preview image, transmitting the metadata to the main control chip through the first virtual channel;

[0051] The horizontal blank period is a blank period between two adjacent pixel rows in the preview image.

[0052] For example, Figure 3 In the example, the first virtual channel is VC0. Assuming that VC0 supports four data types DT, and metadata includes three types of data, namely, embedded data, depth map, and motion vector of the image, DT0 can be used to carry the preview image, DT1 can be used to carry the embedded data, DT2 can be used to carry the depth map, and DT3 can be used to carry the motion vector data. Since the preview image frame includes multiple pixel rows, and there is a horizontal blank period (HBlank) between two adjacent pixel rows, the horizontal blank period of the preview image can be used to transmit metadata, forming a Figure 3 A frame of image data is shown, Figure 3 DT1, DT2 and DT3 are respectively placed after at least one pixel row of DT0, so that the vertical blanking period of the preview image frame can be kept unchanged. In this way, when the vertical blanking period is very small or there is no vertical blanking period, the metadata can still be embedded in the horizontal blanking period of the preview image for transmission, and the preview image and metadata can be synchronously transmitted using different data types of the same virtual channel.

[0053] It should be noted that a preview image is composed of multiple pixel rows, including multiple horizontal blank periods. That is, each pixel row includes a blank period as a horizontal blank period. It is understandable that Figure 3 The line interval in is the blank time interval in the horizontal blank period excluding the period occupied by metadata.

[0054] In some optional embodiments, the header information of the data packet transmitting the preview image carries a first identifier and a second identifier, the first identifier is a channel identifier of the first virtual channel, and the second identifier is used to indicate a data type of the preview image;

[0055] The metadata includes at least one kind of metadata, and the header information of the data packet transmitting each kind of metadata carries the first identifier and the third identifier; the third identifier is used to indicate the data type corresponding to the metadata, and different kinds of metadata correspond to different data types.

[0056] It should be noted that the first identifier and the second identifier are specifically identifiers. The channel identifier of the virtual channel, that is, the identification (ID) of the virtual channel, is used to distinguish different data streams, and the data type is used to indicate the data content or format of the transmitted data so that the receiving end can correctly parse and process the data.

[0057] In specific implementation, the image processing chip and the main control chip pre-agree on the virtual channel and the corresponding data type DT for transmitting the preview image and metadata. By carrying the channel identifier and data type identifier of the virtual channel in the header information of the data packet, the main control chip can identify and route the data based on the header information of the data packet after receiving the data packet corresponding to the preview image and metadata. For example, if the data packet comes from VC0, it is passed to the cache for processing VC0 data.

[0058] In some optional embodiments, the image processing chip is specifically used for:

[0059] Transmitting the preview image to the main control chip through a first virtual channel in the first path, and transmitting the metadata to the main control chip through a second virtual channel in the first path;

[0060] The second virtual channel includes at least one virtual channel except the first virtual channel.

[0061] Exemplarily, the first virtual channel is VC0, and the second virtual channel includes: VC1, VC2 and VC3, wherein DT0 in VC0 is used to carry the preview image frame, DT0 in VC1 is used to carry the embedded data, DT0 in VC2 is used to carry the depth map, and DT0 in VC3 is used to carry the motion vector data, forming Figure 4 In this way, it is possible to use different virtual channels in a MIPI channel to transmit preview images and different types of metadata, that is, to transmit preview images and metadata simultaneously on the same physical link. Since a MIPI channel has the same clock reference, the synchronous transmission of preview images and metadata can be guaranteed.

[0062] Exemplarily, the first virtual channel is VC0, the second virtual channel is VC1, DT0 in VC0 is used to carry the preview image frame, DT0 in VC1 is used to carry the embedded data, DT1 in VC1 is used to carry the depth map, and DT2 in VC1 is used to carry the motion vector data, forming the following Figure 5In this way, preview images and different types of metadata can be transmitted using different virtual channels in a MIPI channel, and preview images and metadata can be transmitted simultaneously on the same physical link. Since a MIPI channel has the same clock reference, the synchronous transmission of preview images and metadata can be guaranteed.

[0063] in, Figure 4 and Figure 5 The frame interval in is the blank time interval in the vertical blank period excluding the period occupied by metadata.

[0064] In some optional embodiments, the image processing chip is specifically used for:

[0065] Transmitting the first sub-data in the metadata and the preview image to the main control chip through the first virtual channel in the first path, and transmitting the second sub-data in the metadata to the main control chip through the second virtual channel in the first path;

[0066] The second virtual channel includes at least one virtual channel except the first virtual channel; and the metadata consists of the first sub-data and the second sub-data.

[0067] Exemplarily, assuming that the metadata includes three types of data, namely, embedded data, depth map, and motion vector data, one or two of the metadata may be used as the first sub-data, and the remaining metadata may be used as the second sub-data. For example, the embedded data may be used as the first sub-data, and the depth map and motion vector data may be used as the second sub-data; for example, the embedded data and motion vector data may be used as the first sub-data, and the depth map may be used as the second sub-data, etc.

[0068] Exemplarily, the first virtual channel is VC0, the second virtual channel is VC1, DT0 in VC0 is used to carry the preview image, DT1 in VC0 is used to carry the embedded data, DT0 in VC1 is used to carry the depth map, and DT1 in VC1 is used to carry the motion vector data to form a frame of data. In this way, it is possible to use different virtual channels in the first channel to transmit preview images and different types of metadata, and to simultaneously transmit preview images and metadata on the same physical link. Since the same MIPI channel has the same clock reference, it is possible to ensure the synchronous transmission of preview image frames and metadata.

[0069] In some optional embodiments, the image processing chip is specifically used for:

[0070] In a vertical blank period or at least one horizontal blank period of the preview image, transmitting the first sub-data to the main control chip through the first virtual channel;

[0071] The horizontal blank period is a blank period between two adjacent pixel rows in the preview image; and the vertical blank period is a blank period between the preview image and an adjacent image frame.

[0072] Exemplarily, the first virtual channel is VC0, the second virtual channel is VC1, DT0 in VC0 is used to carry the preview image, DT1 in VC0 is used to carry the embedded data, DT0 in VC1 is used to carry the depth map, DT1 in VC1 is used to carry the motion vector data, and the horizontal blank period after the pixel row of DT0 in VC0 is used to transmit the embedded data carried by DT1 in VC0, and the vertical blank period after the preview image is used to transmit the depth map and motion vector data carried by DT0 and DT1 in VC1, so as to form the following: Figure 6 In this way, the preview image and different types of metadata can be transmitted by using different virtual channels in the first channel, and the preview image and metadata can be transmitted simultaneously on the same physical link. Since the same MIPI channel has the same clock reference, the synchronous transmission of the preview image and metadata can be guaranteed.

[0073] in, Figure 6 The frame interval is the blank time interval in the vertical blank period excluding the period occupied by metadata, and the line interval is the blank time interval in the horizontal blank period excluding the period occupied by metadata.

[0074] Exemplarily, the first virtual channel is VC0, the second virtual channel is VC1, DT0 in VC0 is used to carry the preview image, DT1 in VC0 is used to carry the embedded data, DT0 in VC1 is used to carry the depth map, and DT1 in VC1 is used to carry the motion vector data, forming the following Figure 7 In this way, the preview image and different types of metadata can be transmitted using different virtual channels in the first channel, that is, the preview image and metadata are transmitted simultaneously on the same physical link. Since the same MIPI channel has a unified clock reference, the synchronous transmission of the preview image and metadata can be guaranteed. Figure 7 The frame interval in is the blank time interval in the vertical blank period excluding the period occupied by metadata.

[0075] In some optional embodiments, the header information of the data packet transmitting the preview image carries a first identifier and a second identifier, the first identifier is a channel identifier of the first virtual channel, and the second identifier is used to indicate a data type of the preview image;

[0076] The first sub-data includes at least one kind of metadata, and the header information of the data packet transmitting the metadata carries the first identifier and the third identifier; the third identifier is used to indicate the data type corresponding to the metadata, and different types of metadata correspond to different data types;

[0077] The second sub-data includes at least one kind of metadata, and the header information of the data packet transmitting the metadata carries the third identifier and the fourth identifier; the fourth identifier is the channel identifier of the second virtual channel.

[0078] In specific implementation, the image processing chip and the main control chip pre-agree on the virtual channel and the corresponding data type DT for transmitting the preview image and metadata. By carrying the channel identifier and data type identifier of the virtual channel in the header information of the data packet, the main control chip can identify and route the data based on the header information of the data packet after receiving the data packet corresponding to the preview image and metadata. For example, according to the pre-agreement, if the data packet comes from VC0 and VC1, it means that the data packets of VC0 and VC1 are associated with the same preview image, and the data packets of VC0 and VC1 should be routed to the same cache queue to ensure that the data packets of VC0 and VC1 can be processed together. If the data packet is successfully routed to the cache queue, the main control chip can start image processing, data parsing, image generation and other processing operations.

[0079] For example, Figure 7 As shown, the first virtual channel is VC0, the second virtual channel is VC1, DT0 in VC0 is used to carry preview images, DT1 in VC0 is used to carry embedded data, DT0 in VC1 is used to carry depth maps, and DT1 in VC1 is used to carry motion vector data. The preview image and embedded data transmitted in VC0 are distinguished by DT0 and DT1, and the depth map and motion vector data transmitted in VC1 are distinguished by DT0 and DT1. In this way, it is possible to use different virtual channels in the first channel to transmit preview images and different types of metadata, ensuring the synchronous transmission of preview image frames and metadata.

[0080] It should be noted that the image processing method provided in the embodiments of the present application can be executed by electronic devices such as mobile phones, tablet computers, laptop computers, PDAs, and vehicle-mounted electronic devices. In some embodiments of the present application, the image processing method provided in the embodiments of the present application is described by taking the electronic device as the execution subject to execute the image processing method as an example.

[0081] The image processing method provided in the embodiments of the present application can be applied to shooting scenarios, wherein one specific application scenario is taking photos, and another specific application scenario is recording videos.

[0082] See also Figure 8The present application provides an image processing method, which is performed by an image processing device. The method includes the following steps:

[0083] Step 101: The image processing chip acquires a preview image captured by an image sensor and metadata of the preview image;

[0084] Exemplarily, in a photo-taking scenario, the user opens a camera application installed in an electronic device and enters a shooting preview interface. When the user triggers a photo-taking operation through a photo-taking control, the camera application, driven by the image processing chip, sends the shooting instruction and related shooting parameters to the image processing chip. The image processing chip controls the image sensor to complete the frame grabbing action and reads a preview image frame from the memory of the image sensor. The preview image frame is an original image in RAW format, i.e., a raw image. Furthermore, metadata of the preview image frame is obtained through the image processing chip.

[0085] Exemplarily, in a video recording scenario, a user opens a camera application installed in an electronic device and enters a shooting preview interface. When the user triggers a video recording operation through a video recording control, the camera application, driven by the image processing chip, sends the video recording instruction and related shooting parameters to the image processing chip. The image processing chip controls the image sensor to complete the frame capture action and reads a preview image frame sequence from the memory of the image sensor. The preview image frame sequence is a series of original images in RAW format. Furthermore, metadata of each preview image frame is obtained through the image processing chip.

[0086] Optionally, the metadata includes at least one of the following: embedded data, depth map, motion vector of the image, etc.; wherein the embedded data is additional information related to the image; the depth map is used to indicate the distance between the object in the scene and the camera; the motion vector data is used to describe the movement direction and speed of the object in the image.

[0087] Step 102: the image processing chip transmits the preview image and the metadata to the main control chip through at least one virtual channel in the first path;

[0088] For example, Figure 1 MIPI TX in is the first MIPI of image processing chip. Figure 3 The MIPI RX in the main control chip is the second MIPI, the first MIPI is connected with the second MIPI to form a first channel, the first channel includes multiple virtual channels (VC), the image processing chip obtains the preview image frame collected by the image sensor and obtains the metadata of the preview image frame, and then sends the preview image and metadata to the main control chip through one of the virtual channels VC0, and uses the data type DT supported by the virtual channel to distinguish and identify the preview image frame and different types of metadata. Figure 1 In the process, DT0 in the virtual channel VC0 is used to carry the preview image, DT1 in the virtual channel VC0 is used to carry the embedded data, DT2 in the virtual channel VC0 is used to carry the depth map, and DT3 in the virtual channel VC0 is used to carry the motion vector data, so that the data of four different DTs of the same VC are packaged into an image frame and transmitted to the main control chip.

[0089] Step 103: The main control chip performs image processing on the preview image according to the metadata to generate a captured image.

[0090] In a specific implementation, the processing pipeline of the image signal processor (ISP) on the main control chip side may include: using the image front end (IFE) to perform denoising, color filtering, exposure control and other processing on the preview image according to the metadata to obtain the first image data; using the image processing engine (IPE) to perform color correction, sharpening, noise reduction and image enhancement and other processing on the first image data according to the metadata to obtain the second image data; using the codec to compress and encode the second image data to obtain the captured image.

[0091] In the above embodiment, the image processing chip forms at least one virtual channel in the first path through the first MIPI and the second MIPI connection, and sends the preview image and the metadata of the preview image to the main control chip. In this way, the image processing chip can use a single MIPI interface to perform the transmission of the preview image and the metadata, avoiding the problem of the timestamp asynchrony between the preview image and the metadata.

[0092] In some optional embodiments, the image processing chip transmits the preview image and the metadata to the main control chip through at least one virtual channel in the first path, including:

[0093] The image processing chip transmits the preview image to the main control chip through the first virtual channel;

[0094] The image processing chip transmits the metadata to the main control chip through a first virtual channel in at least one horizontal blank period of the preview image; wherein the horizontal blank period is a blank time period between two adjacent pixel rows in the preview image.

[0095] It should be noted that the explanation of this embodiment can be found in the embodiment of the image processing circuit. Figure 2 The examples introduced here can achieve the same technical effect and will not be described again here to avoid repetition.

[0096] In some optional embodiments, the image processing chip transmits the preview image and the metadata to the main control chip through at least one virtual channel in the first path, including:

[0097] The image processing chip transmits the preview image to the main control chip through the first virtual channel;

[0098] The image processing chip transmits the metadata to the main control chip through the first virtual channel during a vertical blank period of the preview image; wherein the vertical blank period is a blank time period between the preview image and an adjacent image frame.

[0099] It should be noted that the explanation of this embodiment can be found in the embodiment of the image processing circuit. Figure 3 The examples introduced here can achieve the same technical effect and will not be described again here to avoid repetition.

[0100] In some optional embodiments, the header information of the data packet transmitting the preview image carries a first identifier and a second identifier, the first identifier is a channel identifier of the first virtual channel, and the second identifier is used to indicate the data type corresponding to the preview image;

[0101] The metadata includes at least one kind of metadata, and the header information of the data packet transmitting the metadata carries the first identifier and the third identifier; the third identifier is used to indicate the data type corresponding to the metadata, and different types of metadata correspond to different data types.

[0102] In specific implementation, the image processing chip and the main control chip pre-agree on the virtual channel and the corresponding data type DT for transmitting the preview image and metadata. By carrying the channel identifier and data type identifier of the virtual channel in the header information of the data packet, the main control chip can identify and route the data based on the header information of the data packet after receiving the data packet corresponding to the preview image and metadata. For example, if the data packet comes from VC0, it is passed to the cache for processing VC0 data.

[0103] In some optional embodiments, the image processing chip transmits the preview image and the metadata to the main control chip through at least one virtual channel in the first path, including:

[0104] The image processing chip transmits the preview image to the main control chip through a first virtual channel in the first path, and transmits the metadata to the main control chip through a second virtual channel in the first path; wherein the second virtual channel includes at least one virtual channel other than the first virtual channel.

[0105] It should be noted that the explanation of this embodiment can be found in the embodiment of the image processing circuit. Figure 4 and Figure 5 The examples introduced here can achieve the same technical effect and will not be described again here to avoid repetition.

[0106] In some optional embodiments, the image processing chip transmits the preview image and the metadata to the main control chip through at least one virtual channel in the first path, including:

[0107] The image processing chip transmits the first sub-data in the metadata and the preview image to the main control chip through the first virtual channel in the first path, and transmits the second sub-data in the metadata to the main control chip through the second virtual channel in the first path; wherein the second virtual channel includes at least one virtual channel except the first virtual channel; and the metadata consists of the first sub-data and the second sub-data.

[0108] Exemplarily, assuming that the metadata includes three types of data, namely, embedded data, depth map, and motion vector data, one or two of the metadata may be used as the first sub-data, and the remaining metadata may be used as the second sub-data. For example, the embedded data may be used as the first sub-data, and the depth map and motion vector data may be used as the second sub-data; for example, the embedded data and motion vector data may be used as the first sub-data, and the depth map may be used as the second sub-data, etc.

[0109] Exemplarily, the first virtual channel is VC0, the second virtual channel is VC1, DT0 in VC0 is used to carry the preview image, DT1 in VC0 is used to carry the embedded data, DT0 in VC1 is used to carry the depth map, and DT1 in VC1 is used to carry the motion vector data to form a frame of data. In this way, it is possible to use different virtual channels in the first channel to transmit preview images and different types of metadata, ensuring the synchronous transmission of preview image frames and metadata.

[0110] It should be noted that, in a photo-taking scenario, the preview image is an original image completely captured by the image sensor; in a video-recording scenario, the preview image is an original image in a series of original images completely captured by the image sensor.

[0111] In some optional embodiments, the image processing chip transmits the preview image and the first part of the metadata to the main control chip through the first virtual channel in the first path, including:

[0112] The image processing chip transmits the first sub-data to the main control chip through the first virtual channel during a vertical blank period or at least one horizontal blank period of the preview image;

[0113] The horizontal blank period is a blank period between two adjacent pixel rows in the preview image; and the vertical blank period is a blank period between the preview image and an adjacent image frame.

[0114] It should be noted that the explanation of this embodiment can be found in the embodiment of the image processing circuit. Figure 6 and Figure 7 The examples introduced here can achieve the same technical effect and will not be described again here to avoid repetition.

[0115] In some optional embodiments, the header information of the data packet transmitting the preview image carries a first identifier and a second identifier, the first identifier is a channel identifier of the first virtual channel, and the second identifier is used to indicate a data type of the preview image;

[0116] The first sub-data includes at least one kind of metadata, and the header information of the data packet transmitting the metadata carries the first identifier and the third identifier; the third identifier is used to indicate the data type corresponding to the metadata, and different types of metadata correspond to different data types;

[0117] The second sub-data includes at least one kind of metadata, and the header information of the data packet transmitting the metadata carries the third identifier and the fourth identifier; the fourth identifier is the channel identifier of the second virtual channel.

[0118] In specific implementation, the image processing chip and the main control chip pre-agree on the virtual channel and the corresponding data type DT for transmitting the preview image and metadata. By carrying the channel identifier and data type identifier of the virtual channel in the header information of the data packet, the main control chip can identify and route the data based on the header information of the data packet after receiving the data packet corresponding to the preview image and metadata. For example, according to the pre-agreement, if the data packet comes from VC0 and VC1, it means that the data packets of VC0 and VC1 are associated with the same preview image, and the data packets of VC0 and VC1 should be routed to the same cache queue to ensure that the data packets of VC0 and VC1 can be processed together. If the data packet is successfully routed to the cache queue, the main control chip can start image processing, data parsing, image generation and other processing operations.

[0119] In some optional embodiments, the present application also provides an image processing device, including the image processing circuit as described above.

[0120] like Fig. 9As shown, the image processing device 900 includes the image processing circuit, image sensor 903 and memory 902 as described above, the image processing circuit includes a main control chip 901 and an image processing chip 904, and the main control chip 901 includes a processor 9011; the memory 902 stores programs or instructions that can be run on the processor 9011. When the program or instruction is executed by the processor 9011, the various steps of the above-mentioned data transmission method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0121] It should be noted that the image processing apparatus in the embodiment of the present application may include a mobile electronic device and a non-mobile electronic device.

[0122] Fig.10 A schematic diagram of the hardware structure of an image processing device for implementing an embodiment of the present application.

[0123] The electronic device 1000 includes but is not limited to: a radio frequency unit 1001, a network module 1002, an audio output unit 1003, an input unit 1004, a sensor 1005, a display unit 1006, a user input unit 1007, an interface unit 1008, a memory 1009, and components such as a main control chip 1010 and an image processing chip 1011.

[0124] Those skilled in the art will appreciate that the image processing device 1000 may also include a power source (such as a battery) for supplying power to each component, and the power source may be logically connected to the main control chip 1010 through a power management system, thereby implementing functions such as charging, discharging, and power consumption management through the power management system. Fig.10 The structure shown in the figure does not constitute a limitation on the image processing device. The image processing device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0125] The image processing chip 1011 is used for the image processing chip to obtain the preview image and metadata of the preview image collected by the image sensor; and transmit the preview image and the metadata to the main control chip through at least one virtual channel in the first path;

[0126] The main control chip 1010 is used to perform image processing on the preview image according to the metadata to generate a captured image.

[0127] Optionally, the image processing chip 1011 is specifically used to transmit the preview image to the main control chip through a first virtual channel; in at least one horizontal blank period of the preview image, the metadata is transmitted to the main control chip through the first virtual channel; wherein the horizontal blank period is a blank time period between two adjacent pixel rows in the preview image.

[0128] Optionally, the image processing chip 1011 is specifically used to transmit the preview image to the main control chip through a first virtual channel; in a vertical blank period corresponding to the preview image, the metadata is transmitted to the main control chip through the first virtual channel; wherein the vertical blank period is a blank time period between the preview image and an adjacent image frame.

[0129] Optionally, the header information of the data packet transmitting the preview image carries a first identifier and a second identifier, the first identifier is a channel identifier of the first virtual channel, and the second identifier is used to indicate a data type corresponding to the preview image;

[0130] The metadata includes at least one kind of metadata, and the header information of the data packet transmitting the metadata carries the first identifier and the third identifier; the third identifier is used to indicate the data type corresponding to the metadata, and different types of metadata correspond to different data types.

[0131] Optionally, the image processing chip 1011 is specifically used to transmit the preview image to the main control chip through the first virtual channel in the first path, and to transmit the metadata to the main control chip through the second virtual channel in the first path; wherein the second virtual channel includes at least one virtual channel other than the first virtual channel.

[0132] Optionally, the image processing chip 1011 is specifically used to transmit the first sub-data in the metadata and the preview image to the main control chip through the first virtual channel in the first path, and to transmit the second sub-data in the metadata to the main control chip through the second virtual channel in the first path; wherein the second virtual channel includes at least one virtual channel other than the first virtual channel; and the metadata consists of the first sub-data and the second sub-data.

[0133] Optionally, the image processing chip 1011 is specifically used to transmit the first sub-data to the main control chip through the first virtual channel in a vertical blank period or at least one horizontal blank period of the preview image; wherein the horizontal blank period is a blank time period between two adjacent pixel rows in the preview image; and the vertical blank period is a blank time period between the preview image and an adjacent image frame.

[0134] Optionally, the header information of the data packet transmitting the preview image carries a first identifier and a second identifier, the first identifier is a channel identifier of the first virtual channel, and the second identifier is used to indicate a data type of the preview image;

[0135] The first sub-data includes at least one kind of metadata, and the header information of the data packet transmitting the metadata carries the first identifier and the third identifier; the third identifier is used to indicate the data type corresponding to the metadata, and different types of metadata correspond to different data types;

[0136] The second sub-data includes at least one kind of metadata, and the header information of the data packet transmitting the metadata carries the third identifier and the fourth identifier; the fourth identifier is the channel identifier of the second virtual channel.

[0137] It should be understood that in the embodiment of the present application, the input unit 1004 may include a graphics processor (Graphics Processing Unit, GPU) 10041 and a microphone 10042, and the graphics processor 10041 processes the image data of the static picture or video obtained by the image capture device (such as a camera) in the video capture mode or the image capture mode. The display unit 1006 may include a display panel 10061, and the display panel 10061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 1007 includes a touch panel 10071 and at least one of other input devices 10072. The touch panel 10071 is also called a touch screen. The touch panel 10071 may include two parts: a touch detection device and a touch controller. Other input devices 10072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0138] The memory 1009 can be used to store software programs and various data. The memory 1009 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 1009 may include a volatile memory or a non-volatile memory, or the memory 1009 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 1009 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0139] The main control chip 1010 may include one or more processing units; optionally, the main control chip 1010 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to the operating system, user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the above-mentioned modem processor may not be integrated into the main control chip 1010.

[0140] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the above-mentioned image processing method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0141] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0142] An embodiment of the present application further provides an image processing chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned image processing method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0143] An embodiment of the present application further provides a main control chip, which includes a processor and a communication interface, the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned image processing method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0144] It should be understood that the main control chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0145] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned image processing method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0146] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0147] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0148] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. An image processing circuit, characterized in that: It includes a main control chip and an image processing chip, wherein a first mobile industry processor interface MIPI of the image processing chip is connected with a second MIPI of the main control chip to form a first path; The image processing chip is used to obtain a preview image captured by an image sensor and metadata of the preview image; and transmit the preview image and the metadata to the main control chip through at least one virtual channel in the first path; The main control chip is used to perform image processing on the preview image according to the metadata and output a captured image.

2. The image processing circuit according to claim 1, characterized in that: The image processing chip is specifically used for: Transmitting the preview image to the main control chip through a first virtual channel; During at least one horizontal blank period of the preview image, transmitting the metadata to the main control chip through the first virtual channel; The horizontal blank period is a blank period between two adjacent pixel rows in the preview image.

3. The image processing circuit according to claim 1, characterized in that: The image processing chip is specifically used for: Transmitting the preview image to the main control chip through a first virtual channel; During a vertical blank period of the preview image, the metadata is transmitted to the main control chip through the first virtual channel; wherein the vertical blank period is a blank time period between the preview image and an adjacent image frame.

4. The image processing circuit according to claim 2 or 3, characterized in that: The header information of the data packet transmitting the preview image carries a first identifier and a second identifier, the first identifier is a channel identifier of the first virtual channel, and the second identifier is used to indicate a data type of the preview image; The metadata includes at least one kind of metadata, and the header information of the data packet transmitting the metadata carries the first identifier and the third identifier; the third identifier is used to indicate the data type corresponding to the metadata, and different types of metadata correspond to different data types.

5. The image processing circuit according to claim 1, characterized in that: The image processing chip is specifically used for: Transmitting the preview image to the main control chip through a first virtual channel in the first path, and transmitting the metadata to the main control chip through a second virtual channel in the first path; The second virtual channel includes at least one virtual channel except the first virtual channel.

6. The image processing circuit according to claim 1, characterized in that: The image processing chip is specifically used for: Transmitting the first sub-data in the metadata and the preview image to the main control chip through the first virtual channel in the first path, and transmitting the second sub-data in the metadata to the main control chip through the second virtual channel in the first path; The second virtual channel includes at least one virtual channel except the first virtual channel; and the metadata consists of the first sub-data and the second sub-data.

7. The image processing circuit according to claim 6, characterized in that: The image processing chip is specifically used for: In a vertical blank period or at least one horizontal blank period of the preview image, transmitting the first sub-data to the main control chip through the first virtual channel; The horizontal blank period is a blank period between two adjacent pixel rows in the preview image; and the vertical blank period is a blank period between the preview image and an adjacent image frame.

8. The image processing circuit according to claim 6, characterized in that: The header information of the data packet transmitting the preview image carries a first identifier and a second identifier, the first identifier is a channel identifier of the first virtual channel, and the second identifier is used to indicate a data type of the preview image; The first sub-data includes at least one kind of metadata, and the header information of the data packet transmitting the metadata carries the first identifier and the third identifier; the third identifier is used to indicate the data type corresponding to the metadata, and different types of metadata correspond to different data types; The second sub-data includes at least one kind of metadata, and the header information of the data packet transmitting the metadata carries the third identifier and the fourth identifier; the fourth identifier is the channel identifier of the second virtual channel.

9. An image processing device, characterized in that: The invention comprises the image processing circuit as claimed in any one of claims 1 to 8.

10. An image processing method, executed by the image processing device according to claim 9, characterized in that: The method comprises: The image processing chip acquires a preview image captured by the image sensor and metadata of the preview image; The image processing chip transmits the preview image and the metadata to the main control chip through at least one virtual channel in the first path; The main control chip performs image processing on the preview image according to the metadata to generate a captured image.

11. The image processing method according to claim 10, characterized in that: The image processing chip transmits the preview image and the metadata to the main control chip through at least one virtual channel in the first path, including: The image processing chip transmits the preview image to the main control chip through the first virtual channel; The image processing chip transmits the metadata to the main control chip through the first virtual channel in at least one horizontal blank period of the preview image; wherein the horizontal blank period is a blank time period between two adjacent pixel rows in the preview image.

12. The image processing method according to claim 10, characterized in that: The image processing chip transmits the preview image and the metadata to the main control chip through at least one virtual channel in the first path, including: The image processing chip transmits the preview image to the main control chip through the first virtual channel; The image processing chip transmits the metadata to the main control chip through the first virtual channel during a vertical blank period of the preview image; wherein the vertical blank period is a blank time period between the preview image and an adjacent image frame.

13. The image processing method according to claim 11 or 12, characterized in that: The header information of the data packet transmitting the preview image carries a first identifier and a second identifier, the first identifier is a channel identifier of the first virtual channel, and the second identifier is used to indicate a data type corresponding to the preview image; The metadata includes at least one kind of metadata, and the header information of the data packet transmitting the metadata carries the first identifier and the third identifier; the third identifier is used to indicate the data type corresponding to the metadata, and different types of metadata correspond to different data types.

14. The image processing method according to claim 10, characterized in that: The image processing chip transmits the preview image and the metadata to the main control chip through at least one virtual channel in the first path, including: The image processing chip transmits the preview image to the main control chip through a first virtual channel in the first path, and transmits the metadata to the main control chip through a second virtual channel in the first path; wherein the second virtual channel includes at least one virtual channel other than the first virtual channel.

15. The image processing method according to claim 10, characterized in that: The image processing chip transmits the preview image and the metadata to the main control chip through at least one virtual channel in the first path, including: The image processing chip transmits the first sub-data in the metadata and the preview image to the main control chip through the first virtual channel in the first path, and transmits the second sub-data in the metadata to the main control chip through the second virtual channel in the first path; wherein the second virtual channel includes at least one virtual channel other than the first virtual channel; and the metadata consists of the first sub-data and the second sub-data.

16. The image processing method according to claim 15, characterized in that: The image processing chip transmits the preview image and the first part of the metadata to the main control chip through the first virtual channel in the first path, including: The image processing chip transmits the first sub-data to the main control chip through the first virtual channel during a vertical blank period or at least one horizontal blank period of the preview image; The horizontal blank period is a blank period between two adjacent pixel rows in the preview image; and the vertical blank period is a blank period between the preview image and an adjacent image frame.

17. The image processing method according to claim 15, characterized in that: The header information of the data packet transmitting the preview image carries a first identifier and a second identifier, the first identifier is a channel identifier of the first virtual channel, and the second identifier is used to indicate a data type of the preview image; The first sub-data includes at least one kind of metadata, and the header information of the data packet transmitting the metadata carries the first identifier and the third identifier; the third identifier is used to indicate the data type corresponding to the metadata, and different types of metadata correspond to different data types; The second sub-data includes at least one kind of metadata, and the header information of the data packet transmitting the metadata carries the third identifier and the fourth identifier; the fourth identifier is the channel identifier of the second virtual channel.

18. An image processing chip, characterized in that: The image processing chip includes a processor and a communication interface, the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the steps performed by the image processing chip in the image processing method according to any one of claims 10 to 17.

19. A main control chip, characterized in that: The main control chip includes a processor and a communication interface, the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the steps performed by the main control chip in the image processing method according to any one of claims 10 to 17.

Images