Transmitting device, receiving device and transmission system
The system addresses the lack of demosaic processing in ROI image transmission by incorporating control units for demosaic information acquisition and transmission, ensuring efficient and constrained ROI image processing.
Patent Information
- Application Number
- JP2021527476
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2019-06-28
- Filing Date
- 2020-05-19
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2040-05-19
AI Technical Summary
Existing technologies do not consider demosaic processing when transmitting a region of interest (ROI) cut out from a captured image, leading to potential transmission system failures and inefficiencies.
A transmitting device and receiving device system that includes control units for demosaic information acquisition and transmission, using color information of the ROI edge and edge combination information to perform demosaic processing on ROI image data.
Enables efficient transmission and processing of ROI images even under various constraints, enhancing transmission efficiency and speed by including demosaic information in the transmission data.
Smart Images

Figure 0007584412000001 
Figure 0007584412000002 
Figure 0007584412000003
Abstract
Description
[Technical field]
[0001] The present disclosure relates to a transmitting device, a receiving device, and a transmission system. [Background technology]
[0002] In recent years, there has been an increase in applications that require the transmission of large amounts of data. This puts a heavy load on the transmission system, and in the worst case scenario, the transmission system may crash, making it impossible to transmit data.
[0003] In order to avoid a breakdown in the transmission system, it is known that, for example, instead of transmitting all captured images, the object being photographed is identified and only a portion of the image cut out from the identified object is transmitted (e.g., Patent Documents 1 to 4). [Prior art documents] [Patent documents]
[0004] [Patent Document 1] JP 2016-201756 A [Patent Document 2] JP 2014-39219 A [Patent Document 3] JP 2013-164834 A [Patent Document 4] JP 2012-209831 A Summary of the Invention [Problem to be solved by the invention]
[0005] There is no consideration whatsoever about demosaic processing in the case of transmitting a part of a region of interest (ROI (Region Of Interest)) cut out from a captured image. An object of the present disclosure is to realize demosaic processing of a part of a region of interest (ROI) cut out from a captured image. [Means for solving the problem]
[0006] A transmission device according to an embodiment of the present disclosure includes a control unit that controls acquisition of demosaic information used for demosaic processing of image data of a region of interest (ROI), and a transmission unit that transmits image data of the ROI as payload data and transmits ROI information as embedded data. The control unit acquires, as the demosaic information, color information of the end portion of the image data of the ROI and information indicating which combination of odd-numbered rows and even-numbered rows and odd-numbered columns and even-numbered columns the end portion is. .
[0007] A receiving device according to an embodiment of the present disclosure includes a receiving unit that receives a transmission signal including image data of a region of interest (ROI) in payload data and ROI information in embedded data, a control unit that controls extraction of demosaic information used for demosaic processing of image data of the ROI from the transmission signal received by the receiving unit, and a processing unit that performs demosaic processing of the image data of the ROI using the demosaic information extracted by the control unit. The control unit extracts, as the demosaic information, color information of the end portion of the image data of the ROI and information indicating which combination of odd-numbered rows and even-numbered rows and odd-numbered columns and even-numbered columns the end portion is. .
[0008] A transmission system according to an embodiment of the present disclosure includes a transmitting device having a control unit that acquires demosaic information used for demosaic processing of image data of a region of interest (ROI), and a transmitting unit that transmits the image data as payload data and transmits ROI information as embedded data; a receiving unit that receives a transmission signal that includes image data of the ROI in the payload data and includes the ROI information in the embedded data; a control unit that controls extraction of demosaic information used for demosaic processing of image data of the ROI from the transmission signal received by the receiving unit; and a processing unit that performs demosaic processing of the image data of the ROI using the demosaic information extracted by the control unit. The transmitting device transmits, as the demosaic information, color information of the end of the image data of the ROI and information indicating which combination of odd-numbered rows and even-numbered rows and odd-numbered columns the end is made of from the transmitting unit, and the receiving device receives, as the demosaic information, the color information and information indicating which combination of odd-numbered rows and even-numbered rows and odd-numbered columns the end is made of from the receiving unit. . [Brief description of the drawings]
[0009] [Figure 1] FIG. 1 is a diagram illustrating an example of a schematic configuration of a video transmission system. [Diagram 2]2 is a diagram illustrating an example of a schematic configuration of the video transmission device of FIG. 1. [Diagram 3] 11 is a diagram illustrating an example of a procedure for generating transmission data when a captured image includes two ROIs. FIG. [Figure 4] FIG. 13 is a diagram illustrating an example of the configuration of a packet header. [Diagram 5] FIG. 2 is a diagram illustrating an example of a configuration of transmission data. [Figure 6] FIG. 2 is a diagram illustrating an example of a configuration of transmission data. [Figure 7] FIG. 13 is a diagram illustrating an example of the configuration of payload data of a long packet. [Figure 8] 2 is a diagram illustrating an example of a schematic configuration of the video receiving device of FIG. 1. [Figure 9] 11 is a diagram illustrating an example of a procedure for generating two ROI images included in a captured image when two images are included in transmission data. FIG. [Figure 10] FIG. 10 is a diagram illustrating a schematic area in which an object identified in a captured image is located. [Figure 11] FIG. 13 is a diagram showing an example of a ROI region set for a specified object. [Figure 12] 13 is a diagram showing an example of the configuration of transmission data in which position information of an ROI image is included in payload data of a long packet. FIG. [Figure 13] 4 is a diagram illustrating an example of a color arrangement of an imaging element provided in an imaging region of an imaging section. FIG. [Figure 14A] 1 is a diagram showing a schematic arrangement example A of a color arrangement of imaging elements provided in an imaging region of an imaging section. FIG. [Figure 14B] 14B is a diagram showing an arrangement pattern of a color arrangement of an imaging element cut out from the imaging region shown in FIG. 14A. FIG. [Figure 15A] 13 is a diagram showing a schematic arrangement example B of a color arrangement of imaging elements provided in an imaging region of the imaging section. FIG. [Figure 15B] 15B is a diagram showing an arrangement pattern of a color arrangement of an imaging element cut out from the imaging region shown in FIG. 15A. FIG. [Figure 16A] 13 is a diagram showing a schematic arrangement example C of the color arrangement of the imaging elements provided in the imaging region of the imaging section. FIG. [Figure 16B] 16B is a diagram showing an arrangement pattern of a color arrangement of an imaging element cut out from the imaging region shown in FIG. 16A. FIG. [Figure 17A] 13 is a diagram showing a schematic arrangement example D of a color arrangement of imaging elements provided in an imaging region of the imaging section. FIG. [Figure 17B] 17B is a diagram showing an arrangement pattern of a color arrangement of an imaging element cut out from the imaging region shown in FIG. 17A. FIG. [Figure 18] FIG. 1 is a diagram illustrating a demosaic process. [Figure 19] 1 is a block diagram showing a schematic configuration of a transmitting device, a receiving device, and a transmission system according to a first embodiment. [Figure 20] 5 is a flowchart showing an example of the flow of demosaic processing in the transmitting device, the receiving device, and the transmission system according to the first embodiment. [Figure 21] FIG. 2 is a diagram showing an example of a timing chart of demosaic processing in the transmitting device, the receiving device, and the transmission system according to the first embodiment. [Figure 22] FIG. 13 is a block diagram showing a schematic configuration of a transmitting device, a receiving device, and a transmission system according to a modified example of the first embodiment. [Diagram 23] 13 is a flowchart showing an example of the flow of demosaic processing in the transmitting device, the receiving device, and the transmission system according to the second embodiment. [Figure 24] FIG. 11 is a diagram showing an example of a timing chart of demosaic processing in a transmitting device, a receiving device, and a transmission system according to the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0010] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. The following description is a specific example of the present disclosure, and the present disclosure is not limited to the following embodiment.
[0011] Hereinafter, modes for carrying out the technology according to the present disclosure (hereinafter, referred to as "embodiments") will be described in the following order. 1. Premise technology 1 of this disclosure (technology for transmitting a region of interest (ROI) that is a part (rectangular in shape) cut out from a captured image) 2. Premise Technology 2 of the Present Disclosure (Technology for transmitting a portion (non-rectangular shape) of a region of interest (ROI) cut out from a captured image) 3. Principle of Demosaicing in the Embodiments of the Present Disclosure 4. Transmitter, receiver, and transmission system according to the first embodiment of the present disclosure 5. Transmitter, receiver, and transmission system according to a modification of the first embodiment of the present disclosure 6. Transmitter, receiver, and transmission system according to the second embodiment of the present disclosure
[0012] 1. Prerequisite technology 1 of this disclosure: [composition] In recent years, mobile devices such as smartphones and camera devices handle increasing amounts of image data, and there is a demand for faster data transmission within devices or between different devices with lower power consumption. To meet these demands, the standardization of high-speed interface standards such as the C-PHY standard and the D-PHY standard, established by the MIPI Alliance, is underway as connection interfaces for mobile devices and camera devices. The C-PHY standard and the D-PHY standard are interface standards for the physical layer (PHY) of communication protocols. In addition, there are DSI for displays of mobile devices and CSI for camera devices as upper protocol layers of the C-PHY standard and the D-PHY standard.
[0013] A video transmission system 1 according to the prerequisite technology of the present disclosure is a system capable of transmitting and receiving signals in accordance with various standards, for example, the MIPI CSI-2 standard, the MIPI CSI-3 standard, or the MIPI DSI standard. FIG. 1 shows an overview of the video transmission system 1 according to the prerequisite technology of the present disclosure. The video transmission system 1 is applied to the transmission of a data signal, a clock signal, and a control signal, and includes a video transmitting device 100 and a video receiving device 200. The video transmission system 1 includes a data lane DL for transmitting a data signal such as image data across the video transmitting device 100 and the video receiving device 200, a clock lane CL for transmitting a clock signal, and a camera control interface CCI for transmitting a control signal. FIG. 1 shows an example in which one data lane DL is provided, but multiple data lanes DL may be provided. The camera control interface CCI includes an I 2 It is a bidirectional control interface that is compatible with the C (Inter-Integrated Circuit) standard.
[0014] The video transmitting device 100 is a device that transmits signals in accordance with the MIPI CSI-2 standard, the MIPI CSI-3 standard, or the MIPI DSI standard. The video transmitting device 100 has a CSI transmitter 100A and a CCI slave 100B. The video receiving device 200 has a CSI receiver 200A and a CCI master 200B. In the clock lane CL, the CSI transmitter 100A and the CSI receiver 200A are connected by a clock signal line. In the data lane DL, the CSI transmitter 100A and the CSI receiver 200A are connected by a clock signal line. In the camera control interface CCI, the CCI slave 100B and the CCI master 200B are connected by a control signal line.
[0015] The CSI transmitter 100A is, for example, a differential signal transmission circuit that generates a differential clock signal as a clock signal and outputs it to a clock signal line. The CSI transmitter 100A can be configured to transmit not only differential signals but also single-ended or three-phase signals. The CSI transmitter 100A is also a differential signal transmission circuit that generates a differential data signal as a data signal and outputs it to a data signal line. The CSI receiver 200A is a differential signal receiving circuit that receives a differential clock signal as a clock signal via a clock signal line and performs a predetermined process on the received differential clock signal. The CSI receiver 200A is also a differential signal receiving circuit that receives a differential data signal as a data signal via a data signal line and performs a predetermined process on the received differential data signal.
[0016] (Video transmitting device 100) Fig. 2 shows an example of the configuration of video transmitting device 100. Video transmitting device 100 corresponds to a specific example of CSI transmitter 100A. Video transmitting device 100 includes, for example, an imaging unit 110, image processing units 120 and 130, and a transmission unit 140. Video transmitting device 100 transmits transmission data 147A generated by performing a predetermined process on captured image 111 obtained by imaging unit 110 to video receiving device 200 via a data lane DL. Fig. 3 shows an example of a procedure for generating transmission data 147A.
[0017] The imaging unit 110 converts an optical image signal obtained through, for example, an optical lens into image data. The imaging unit 110 includes, for example, a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor. The imaging unit 110 has an analog-digital conversion circuit and converts analog image data into digital image data. The data format after conversion may be a YCbCr format in which the color of each pixel is expressed by a luminance component Y and color difference components Cb and Cr, or may be an RGB format. The imaging unit 110 outputs a captured image 111 (digital image data) obtained by imaging to the image processing unit 120.
[0018] The image processing unit 120 is a circuit that performs a predetermined process on the captured image 111 input from the imaging unit 110. In the base technology 1, a case will be described in which the image processing unit 120 performs a predetermined process on the captured image 111 input from the imaging unit 110 when a control signal instructing the cutting out of the ROI is input from the video receiving device 200 via the camera control interface CCI. However, the base technology 1 can also be applied to a case in which the video transmitting device 100, that is, the transmitting side, instructs the coordinates of the cutout of the ROI. In this case, the transmitting side is configured to receive information such as a "person" or "object" to be acquired in the ROI sent from the receiving side, and determine and instruct the coordinates of the cutout. As a result, the image processing unit 120 generates various data (120A, 120B, 120C) and outputs them to the transmitting unit 140. The image processing unit 130 is a circuit that performs a predetermined process on the captured image 111 input from the imaging unit 110. When a control signal instructing output of a normal image is input from the video receiving device 200 via the camera control interface CCI, the image processing unit 130 performs a predetermined process on the captured image 111 input from the imaging unit 110. As a result, the image processing unit 130 generates image data 130A and outputs it to the transmission unit 140.
[0019] The image processing unit 130 includes, for example, an encoding unit 131. The encoding unit 131 encodes the captured image 111 to generate compressed image data 130A. The image processing unit 130 compresses the captured image 111 in a compression format conforming to the JPEG (Joint Photographic Experts Group) standard as the format of the compressed image data 130A, for example.
[0020] The image processing unit 120 includes, for example, an ROI extraction unit 121, an ROI analysis unit 122, an overlap detection unit 123, a priority setting unit 124, an encoding unit 125, and an image processing control unit 126.
[0021] The ROI cutout unit 121 identifies one or more objects of the subject to be photographed included in the captured image 111 input from the imaging unit 110, and sets a region of interest ROI for each identified object. The region of interest ROI is, for example, a rectangular region including the identified object. The ROI cutout unit 121 cuts out an image of each region of interest ROI (for example, the ROI image 112 in FIG. 3) from the captured image 111. The ROI cutout unit 121 further assigns a region number as an identifier to each set region of interest ROI. For example, when two regions of interest ROI are set in the captured image 111, the ROI cutout unit 121 assigns region number 1 to one region of interest ROI (for example, the region of interest ROI1 in FIG. 3) and assigns region number 2 to the other region of interest ROI (for example, the region of interest ROI2 in FIG. 3). The ROI cutout unit 121 stores the assigned identifier (region number) in a storage unit, for example. The ROI cutout unit 121 stores, for example, each ROI image 112 cut out from the captured image 111 in the storage unit. The ROI cutout unit 121 further stores, for example, an identifier (region number) given to each region of interest ROI in the storage unit in association with the ROI image 112.
[0022] The ROI analysis unit 122 derives position information 113 of the region of interest ROI in the captured image 111 for each region of interest ROI. The position information 113 is composed of, for example, the upper left end coordinates (Xa, Ya) of the region of interest ROI, the length of the region of interest ROI in the X-axis direction, and the length of the region of interest ROI in the Y-axis direction. The length of the region of interest ROI in the X-axis direction is, for example, the physical region length XLa of the region of interest ROI in the X-axis direction. The length of the region of interest ROI in the Y-axis direction is, for example, the physical region length YLa of the region of interest ROI in the Y-axis direction. The physical region length refers to the physical length (data length) of the region of interest ROI. The position information 113 may include coordinates of a position different from the upper left end of the region of interest ROI. The ROI analysis unit 122 stores the derived position information 113 in, for example, a storage unit. The ROI analysis unit 122 stores the derived position information 113 in, for example, a storage unit in association with an identifier (region number) assigned to the region of interest ROI.
[0023] The ROI analysis unit 122 may further derive, for example, an output region length XLc in the X-axis direction of the region of interest ROI and an output region length YLc in the Y-axis direction of the region of interest ROI as the position information 113 for each region of interest ROI. The output region length is, for example, the physical length (data length) of the region of interest ROI after a resolution change is performed on the region of interest ROI by thinning processing, pixel addition, etc. The ROI analysis unit 122 may derive, for example, sensing information, exposure information, gain information, AD (Analog-Digital) word length, image format, etc., in addition to the position information 113 for each region of interest ROI, and store them in the storage unit.
[0024] The sensing information refers to the calculation contents for an object included in the region of interest ROI, supplementary information for subsequent signal processing of the ROI image 112, etc. The exposure information refers to the exposure time of the region of interest ROI. The gain information refers to the gain information of the region of interest ROI. The AD word length refers to the word length of data per pixel that has been AD converted in the region of interest ROI. The image format refers to the format of the image of the region of interest ROI. The ROI analysis unit 122 may, for example, derive the number of regions of interest ROI (number of ROIs) included in the captured image 111 and store it in the memory unit.
[0025] When a plurality of objects of the photographing target are specified in the captured image 111, the overlap detection unit 123 detects an overlap region (ROO (Region Of Overlap)) where two or more regions of interest ROI overlap each other based on the position information 113 of the plurality of regions of interest ROI in the captured image 111. That is, the overlap detection unit 123 derives the position information 114 of the overlap region ROO in the captured image 111 for each overlap region ROO. For example, the overlap detection unit 123 stores the derived position information 114 in the storage unit. For example, the overlap detection unit 123 stores the derived position information 114 in the storage unit in association with the overlap region ROO. The overlap region ROO is, for example, a rectangular region having the same size as or smaller than the smallest region of interest ROI among two or more regions of interest ROI that overlap each other. The position information 114 is composed of, for example, the coordinates (Xb, Yb) of the upper left end of the overlapping region ROO, the length of the overlapping region ROO in the X-axis direction, and the length of the overlapping region ROO in the Y-axis direction. The length of the overlapping region ROO in the X-axis direction is, for example, the physical region length XLb. The length of the overlapping region ROO in the Y-axis direction is, for example, the physical region length YLb. The position information 114 may include coordinates of a position different from the upper left end of the region of interest ROI.
[0026] The priority setting unit 124 assigns a priority 115 to each region of interest ROI in the captured image 111. For example, the priority setting unit 124 stores the assigned priority 115 in the storage unit. For example, the priority setting unit 124 associates the assigned priority 115 with the region of interest ROI and stores it in the storage unit. The priority setting unit 124 may assign the priority 115 to each region of interest ROI separately from the region number assigned to each region of interest ROI, or may use the region number assigned to each region of interest ROI instead of the priority 115. For example, the priority setting unit 124 may associate the priority 115 with the region of interest ROI and store it in the storage unit, or may associate the region number assigned to each region of interest ROI with the region of interest ROI and store it in the storage unit.
[0027] The priority 115 is an identifier of each region of interest ROI, and is discrimination information that can discriminate which of the multiple regions of interest ROI in the captured image 111 has had an overlapping region ROO omitted. For example, the priority setting unit 124 assigns 1 as the priority 115 to one of two regions of interest ROIs each including an overlapping region ROO, and assigns 2 as the priority 115 to the other region of interest ROI. In this case, when creating a transmission image 116 described later, the overlapping region ROO is omitted from the region of interest ROI having the larger priority 115 value. Note that the priority setting unit 124 may assign the same number as the region number assigned to each region of interest ROI as the priority 115 to the region of interest ROI. For example, the priority setting unit 124 associates the priority 115 assigned to each region of interest ROI with the ROI image 112 and stores it in the storage unit.
[0028] The encoding unit 125 encodes each transmission image 116 to generate compressed image data 120A. For example, the encoding unit 125 compresses each transmission image 116 using a compression format conforming to the JPEG standard as the format of the compressed image data 120A. The encoding unit 125 generates each transmission image 116 before performing the compression process. The encoding unit 125 generates a plurality of transmission images 116 in which the images 118 are omitted from the plurality of ROI images 112 obtained from the captured image 111 so that the images 118 of the overlapping regions ROO are not included in the plurality of ROI images 112 obtained from the captured image 111 in a duplicated manner.
[0029] The encoding unit 125 determines which of the multiple ROI images 112 the image 118 is to be omitted from, based on the priority 115 given to each region of interest ROI, for example. The encoding unit 125 may determine which of the multiple ROI images 112 the image 118 is to be omitted from, by using, for example, the region number given to each region of interest ROI as the priority 115. The encoding unit 125 determines the ROI image 112 specified as above from which the image 118 is omitted as the transmission image 116 (for example, the transmission image 116a2 in FIG. 3). For the ROI image 112 that does not include the overlapping region ROO or the ROI image 112 from which the image 118 is not to be omitted by the above determination, the encoding unit 125 determines the ROI image 112 itself as the transmission image 116 (for example, the transmission image 116a1 in FIG. 3).
[0030] The image processing control unit 126 generates the ROI information 120B and the frame information 120C and transmits them to the transmission unit 140. The ROI information 120B includes, for example, each piece of position information 113. The ROI information 120B further includes, for example, at least one of the data type of each region of interest ROI, the number of regions of interest ROI included in the captured image 111, the region number (or priority 115) of each region of interest ROI, the data length of each region of interest ROI, and the image format of each region of interest ROI. The frame information 120C includes, for example, the number of a virtual channel assigned to each frame, the data type of each region of interest ROI, the payload length for each line, and the like. The data type includes, for example, YUV data, RGB data, or RAW data. The data type further includes, for example, data in ROI format or data in normal format. The payload length is, for example, the number of pixels included in the payload of a long packet, for example, the number of pixels for each region of interest ROI. Here, the payload refers to the main data (application data) transmitted between the video transmitting device 100 and the video receiving device 200. The long packet refers to a packet placed between the packet header PH and the packet footer PF.
[0031] The transmitting unit 140 is a circuit that generates and transmits transmission data 147A based on various data (120A, 120B, 120C, 130A) input from the image processing units 120 and 130. The transmitting unit 140 transmits ROI information 120B for each region of interest ROI in the captured image 111 as embedded data (EmbeddedData). When a control signal instructing to cut out an ROI is input from the video receiving device 200 via the camera control interface CCI, the transmitting unit 140 further transmits image data of each region of interest ROI (compressed image data 120A) as payload data (PayloadData) of a long packet. At this time, the transmitting unit 140 transmits the image data of each region of interest ROI (compressed image data 120A) via a common virtual channel. The transmitting unit 140 also transmits image data (compressed image data 120A) of each region of interest ROI by an image data frame, and transmits ROI information 120B for each region of interest ROI by the header of the image data frame. The transmitting unit 140 also transmits normal image data (compressed image data 130A) by payload data of a long packet when a control signal instructing output of a normal image is input from the video receiving device 200 via the camera control interface CCI.
[0032] The transmission unit 140 has, for example, a LINK control unit 141, an ECC generation unit 142, a PH generation unit 143, an EBD buffer 144, an ROI data buffer 145, a normal image data buffer 146, and a synthesis unit 147. The LINK control unit 141, the ECC generation unit 142, the PH generation unit 143, the EBD buffer 144, and the ROI data buffer 145 output to the synthesis unit 147 when a control signal instructing cutting out an ROI is input from the video receiving device 200 via the camera control interface CCI. The normal image data buffer 146 outputs to the synthesis unit 147 when a control signal instructing output of a normal image is input from the video receiving device 200 via the camera control interface CCI.
[0033] The ROI data buffer 145 may also serve as the normal image data buffer 146. In this case, the transmission unit 140 may have a selector between the output terminals of the ROI data buffer 145 and the input terminal of the synthesis unit 147, which selects either the output of the ROI data buffer 145 or the output of the ROI data buffer 145.
[0034] The LINK control unit 141 outputs, for example, the frame information 120C to the ECC generation unit 142 and the PH generation unit 143 for each line. The ECC generation unit 142 generates an error correction code for one line based on data of the frame information 120C (for example, the number of the virtual channel, the data type of each region of interest ROI, the payload length for each line, etc.). The ECC generation unit 142 outputs, for example, the generated error correction code to the PH generation unit 143. The PH generation unit 143 generates a packet header PH for each line using, for example, the frame information 120C and the error correction code generated by the ECC generation unit 142. At this time, the packet header PH is, for example, a packet header of payload data of a long packet as shown in FIG. 4. This packet header PH includes, for example, DI, WC, and ECC. WC is an area for indicating the end of the packet to the video receiving device 200 by the number of words. WC includes, for example, the payload length, and includes, for example, the number of pixels for each region of interest ROI. The ECC is an area for storing values for correcting bit errors. The ECC includes an error correction code. The DI is an area for storing a data identifier. The DI includes a VC (Virtual Channel) number and DataType (data type of each region of interest ROI). The VC (Virtual Channel) is a concept introduced for packet flow control, and is a mechanism for supporting multiple independent data streams sharing the same link. The PH generation unit 143 outputs the generated packet header PH to the synthesis unit 147.
[0035] The EBD buffer 144 temporarily stores the ROI information 120B and outputs the ROI information 120B as embedded data to the synthesis unit 147 at a predetermined timing. The embedded data refers to additional information that can be embedded in the header or footer of an image data frame (see FIG. 5 described later). The embedded data includes, for example, the ROI information 120B.
[0036] The ROI data buffer 145 temporarily stores the compressed image data 120A, and outputs the compressed image data 120A to the synthesis unit 147 as payload data of a long packet at a predetermined timing. When a control signal instructing to cut out an ROI is input from the video receiving device 200 via the camera control interface CCI, the ROI data buffer 145 outputs the compressed image data 120A to the synthesis unit 147 as payload data of a long packet. The normal image data buffer 146 temporarily stores the compressed image data 130A, and outputs the compressed image data 130A to the synthesis unit 147 as payload data of a long packet at a predetermined timing. When a control signal instructing to output a normal image is input from the video receiving device 200 via the camera control interface CCI, the normal image data buffer 146 outputs the compressed image data 130A to the synthesis unit 147 as payload data of a long packet.
[0037] When a control signal instructing output of a normal image is input from the video receiving device 200 via the camera control interface CCI, the synthesis unit 147 generates transmission data 147A based on the input data (compressed image data 130A). The synthesis unit 147 outputs the generated transmission data 147A to the video receiving device 200 via the data lane DL. On the other hand, when a control signal instructing cut-out of an ROI is input from the video receiving device 200 via the camera control interface CCI, the synthesis unit 147 generates transmission data 147A based on the input various data (packet header PH, ROI information 120B, and compressed image data 120A). The synthesis unit 147 outputs the generated transmission data 147A to the video receiving device 200 via the data lane DL. That is, the synthesis unit 147 includes DataType (data type of each region of interest ROI) in the packet header PH of the payload data of the long packet and sends it out. Moreover, the synthesis unit 147 outputs the image data (compressed image data 120A) of each region of interest ROI through a common virtual channel.
[0038] The transmission data 147A is configured by an image data frame as shown in FIG. 5, for example. The image data frame usually has a header area, a packet area, and a footer area. In FIG. 5, the footer area is omitted for convenience. The frame header area R1 of the transmission data 147A includes embedded data. At this time, the embedded data includes ROI information 120B. In FIG. 5, the packet area R2 of the transmission data 147A includes payload data of a long packet for each line, and further includes a packet header PH and a packet footer PF at a position where the payload data of the long packet is sandwiched. Furthermore, a low power mode LP is included at a position where the packet header PH and the packet footer PF are sandwiched.
[0039] At this time, the packet header PH includes, for example, DI, WC, and ECC. The WC includes, for example, a payload length, and includes, for example, the number of pixels for each region of interest ROI. The ECC includes an error correction code. The DI includes VC (a virtual channel number) and DataType (a data type for each region of interest ROI). In this embodiment, the VC of each line is assigned a common virtual channel number. In addition, in FIG. 5, the packet area R2 of the transmission data 147A includes compressed image data 147B. The compressed image data 147B is composed of one compressed image data 120A or a plurality of compressed image data 120A. Here, in FIG. 5, the packet group close to the packet header PH includes, for example, the compressed image data 120A (120A1) of the transmission image 116a1 in FIG. 3, and the packet group away from the packet header PH includes, for example, the compressed image data 120A (120A2) of the transmission image 116a2 in FIG. 3. These two compressed image data 120A1 and 120A2 constitute the compressed image data 147B. The payload data of the long packet for each line contains pixel data for one line in the compressed image data 147B.
[0040] Fig. 6 shows an example of the configuration of the transmission data 147A. The transmission data 147A includes, for example, a frame header area R1 and a packet area R2. Note that Fig. 6 shows a detailed example of the contents of the frame header area R1. Also, Fig. 6 omits the low power mode LP.
[0041] The frame header region R1 includes, for example, a frame number F1 as an identifier of the transmission data 147A. The frame header region R1 includes information about the compressed image data 147B included in the packet region R2. The frame header region R1 includes, for example, the number of compressed image data 120A included in the compressed image data 147B (the number of ROIs) and information about the ROI image 112 corresponding to each compressed image data 120A included in the compressed image data 147B (ROI information 120B).
[0042] For example, in the packet region R2 of the transmission data 147A, the synthesis unit 147 arranges the compressed image data 147B by dividing it into pixel rows of the compressed image data 120A. Therefore, the packet region R2 of the transmission data 147A does not include overlapping compressed image data corresponding to the image 118 in the overlapping region ROO. Also, for example, the synthesis unit 147 omits pixel rows of the captured image 111 that do not correspond to each transmission image 116 in the packet region R2 of the transmission data 147A. Therefore, the packet region R2 of the transmission data 147A does not include pixel rows of the captured image 111 that do not correspond to each transmission image 116. Note that in the packet region R2 of FIG. 6, the area surrounded by a dashed line corresponds to the compressed image data of the image 118 in the overlapping region ROO.
[0043] The boundary between the packet group closer to the packet header PH (e.g., 1(n) in FIG. 6) and the packet group farther from the packet header PH (e.g., 2(1) in FIG. 6) is specified by the physical region length XLa1 of the ROI image 112 corresponding to the compressed image data of the packet group closer to the packet header PH (e.g., 1(n) in FIG. 6). In the compressed image data corresponding to the image 118 of the overlapping region ROO included in the packet group closer to the packet header PH (e.g., 1(n) in FIG. 6), the starting position of the packet is specified by the physical region length XLa2 of the ROI image 112 corresponding to the packet group farther from the packet header PH (e.g., 2(1) in FIG. 6).
[0044] For example, when generating payload data of a long packet for each line in the packet region R2 of the transmission data 147A, the synthesis unit 147 may include, in the payload data of the long packet, the ROI information 120B, for example, as shown in FIG. 7, in addition to pixel data for one line in the compressed image data 147B. That is, the synthesis unit 147 may include the ROI information 120B in the payload data of the long packet and transmit it. In this case, the ROI information 120B includes at least one of the number of regions of interest ROI (number of ROIs) included in the captured image 111, the region number (or priority 115) of each region of interest ROI, the data length of each region of interest ROI, and the image format of each region of interest ROI, for example, as shown in FIG. 7(A) to FIG. 7(K). It is preferable that the ROI information 120B is arranged at the end of the payload data of the long packet on the packet header PH side (i.e., at the beginning of the payload data of the long packet).
[0045] (Video receiving device 200) Next, the video receiving device 200 will be described. FIG. 8 shows an example of the configuration of the video receiving device 200. FIG. 9 shows an example of a procedure for generating an ROI image 223A in the video receiving device 200. The video receiving device 200 is a device that receives a signal in a standard common to the video transmitting device 100 (for example, the MIPI CSI-2 standard, the MIPI CSI-3 standard, or the MIPI CSI-3 standard). The video receiving device 200 has, for example, a receiving unit 210 and an information processing unit 220. The receiving unit 210 is a circuit that receives the transmission data 147A output from the video transmitting device 100 via the data lane DL, performs a predetermined process on the received transmission data 147A, and generates various data (214A, 215A, 215B) and outputs the data to the information processing unit 220. The information processing unit 220 is a circuit that generates an ROI image 223A based on various data (214A, 215A) received from the receiving unit 210, and generates a normal image 224A based on data (215B) received from the receiving unit 210.
[0046] The receiving unit 210 includes, for example, a header separation unit 211, a header interpretation unit 212, a payload separation unit 213, an EBD interpretation unit 214, and an ROI data separation unit 215.
[0047] The header separation unit 211 receives the transmission data 147A from the video transmission device 100 via the data lane DL. That is, the header separation unit 211 receives the transmission data 147A including the ROI information 120B for each region of interest ROI in the captured image 111 in the embedded data and including the image data (compressed image data 120A) of each region of interest ROI in the payload data of the long packet. The header separation unit 211 separates the received transmission data 147A into a frame header region R1 and a packet region R2. The header interpretation unit 212 specifies the position of the payload data of the long packet included in the packet region R2 based on the data (specifically, the embedded data) included in the frame header region R1. The payload separation unit 213 separates the payload data of the long packet included in the packet region R2 from the packet region R2 based on the position of the payload data of the long packet specified by the header interpretation unit 212.
[0048] The EBD interpretation unit 214 outputs the embedded data as EBD data 214A to the information processing unit 220. The EBD interpretation unit 214 further determines, from the data type included in the embedded data, whether the image data included in the payload data of the long packet is compressed image data 120A of the ROI image data 116 or compressed image data 130A of normal image data. The EBD interpretation unit 214 outputs the determination result to the ROI data separation unit 215.
[0049] When the image data included in the payload data of the long packet is compressed image data 120A of the ROI image data 116, the ROI data separation unit 215 outputs the payload data of the long packet as payload data 215A to the information processing unit 220 (specifically, the ROI decoding unit 222). When the image data included in the payload data is compressed image data 130A of normal image data, the ROI data separation unit 215 outputs the payload data of the long packet as payload data 215B to the information processing unit 220 (specifically, the normal image decoding unit 224). When the payload data of the long packet includes ROI information 120B, the payload data 215A includes the ROI information 120B and one line of pixel data of the compressed image data 147B.
[0050] The information processing unit 220 extracts the ROI information 120B from the embedded data included in the EBD data 214A. The information processing unit 220 extracts an image of each region of interest ROI (ROI image 112) in the captured image 111 from the payload data of the long packet included in the transmission data 147A received by the receiving unit 210 based on the ROI information 120B extracted by the information extraction unit 221. The information processing unit 220 has, for example, the information extraction unit 221, an ROI decoding unit 222, an ROI image generation unit 223, and a normal image decoding unit 224.
[0051] The normal image decoding unit 224 decodes the payload data 215B to generate a normal image 224A. The ROI decoding unit 222 decodes the compressed image data 147B included in the payload data 215A to generate image data 222A. This image data 222A is composed of one or more transmission images 116.
[0052] The information extraction unit 221 extracts the ROI information 120B from the embedded data included in the EBD data 214A. For example, the information extraction unit 221 extracts the number of attention regions ROI included in the captured image 111, the area number (or priority 115) of each attention region ROI, the data length of each attention region ROI, and the image format of each attention region ROI from the embedded data included in the EBD data 214A. That is, the transmission data 147A includes the area number (or priority 115) of the attention region ROI corresponding to each transmission image 116 as discrimination information capable of discriminating which of the multiple transmission images 116 obtained from the transmission data 147A has omitted the image 118 of the overlapping region ROO.
[0053] Based on the ROI information 120B obtained by the information extraction section 221, the ROI image generation section 223 detects an overlapping region ROO where two or more regions of interest ROI overlap each other.
[0054] The information extraction unit 221 extracts, for example, from the embedded data included in the EBD data 214A, the coordinates (for example, the upper left coordinates (Xa1, Ya1)), length (for example, the physical region length XLa1, YLa1), and region number 1 (or priority 115 (=1)) of the region of interest ROI corresponding to the ROI image 112a1. The information extraction unit 221 further extracts, for example, from the embedded data included in the EBD data 214A, the coordinates (for example, the upper left coordinates (Xa2, Ya2)), length (for example, the physical region length XLa2, YLa2), and region number 2 (or priority 115 (=2)) of the region of interest ROI corresponding to the ROI image 112a2.
[0055] At this time, the ROI image generating unit 223 derives the position information 114 of the overlapping region ROO based on the extracted information (hereinafter referred to as "extracted information 221A"). The ROI image generating unit 223 derives, for example, the coordinates (e.g., the upper left corner coordinate (Xb1, Yb1)) and length (e.g., the physical region length XLb1, YLb1) of the overlapping region ROO as the position information 114 of the overlapping region ROO.
[0056] The ROI image generating unit 223 may obtain the ROI information 120B from the payload data 215A instead of obtaining the ROI information 120B from the embedded data included in the EBD data 214A. In this case, the ROI image generating unit 223 may detect an overlapping region ROO where two or more regions of interest ROI overlap each other based on the ROI information 120B included in the payload data 215A. The ROI image generating unit 223 may also extract the extraction information 221A from the ROI information 120B included in the payload data 215A, and may derive the position information 114 of the overlapping region ROO based on the extraction information 221A extracted in this way.
[0057] The ROI image generating unit 223 further generates images of the regions of interest ROI (ROI images 112a1, 112a2) in the captured image 111 based on the image data 222A, the extraction information 221A, and the position information 114 of the overlapping region ROO. The ROI image generating unit 223 outputs the generated images as ROI image 223A.
[0058] [procedure] Next, an example of a procedure for data transmission in the video transmission system 1 will be described with reference to FIGS.
[0059] First, the imaging unit 110 outputs a captured image 111 (digital image data) obtained by imaging to the image processing unit 120. The ROI cutout unit 121 specifies two regions of interest ROI1 and ROI2 included in the captured image 111 input from the imaging unit 110. The ROI cutout unit 121 cuts out images of the regions of interest ROI1 and ROI2 (ROI images 112a1 and 112a2) from the captured image 111. The ROI cutout unit 121 assigns region number 1 as an identifier to the region of interest ROI1, and assigns region number 2 as an identifier to the region of interest ROI2.
[0060] The ROI analysis unit 122 derives position information 113 of the region of interest ROI in the captured image 111 for each region of interest ROI. Based on the region of interest ROI1, the ROI analysis unit 122 derives the top left coordinate (Xa1, Ya1) of the region of interest ROI1, the length of the region of interest ROI1 in the X-axis direction (XLa1), and the length of the region of interest ROI1 in the Y-axis direction (YLa1). Based on the region of interest ROI2, the ROI analysis unit 122 derives the top left coordinate (Xa2, Ya2) of the region of interest ROI2, the length of the region of interest ROI2 in the X-axis direction (XLa2), and the length of the region of interest ROI2 in the Y-axis direction (YLa2).
[0061] The overlap detection unit 123 detects an overlapping region ROO where the two regions of interest ROI1, ROI2 overlap with each other, based on position information 113 of the two regions of interest ROI1, ROI2 in the captured image 111. That is, the overlap detection unit 123 derives position information 114 of the overlapping region ROO in the captured image 111. The overlap detection unit 123 derives the upper left corner coordinates (Xb1, Yb1) of the overlapping region ROO, the length in the X-axis direction of the overlapping region ROO (XLb1), and the length in the Y-axis direction of the overlapping region ROO (YLb1) as the position information 114 of the overlapping region ROO in the captured image 111.
[0062] Of the two regions of interest ROI1 and ROI2, the priority setting unit 124 assigns a priority 115 of 1 to the region of interest ROI1 and assigns a priority 115 of 2 to the other region of interest ROI2.
[0063] The encoding unit 125 generates two transmitted images 116a1, 116a2 by omitting the image 118 from the two ROI images 112a1, 112a2 obtained from the captured image 111 so that the image 118 of the overlap region ROO is not overlapped in the two regions of interest ROI1, ROI2.
[0064] The encoding unit 125 determines which of the two ROI images 112a1 and 112a2 the image 118 is to be omitted from, based on the region numbers (or priority 115) of the two regions of interest ROI1 and ROI2. The encoding unit 125 omits the image 118 from the ROI image 112a2 corresponding to the region of interest ROI2, which has the larger region number (or priority 115), of the two regions of interest ROI1 and ROI2, thereby generating a transmission image 116a2. For the ROI image 112a1 corresponding to the region of interest ROI1, which has the smaller region number (or priority 115) of the two regions of interest ROI1 and ROI2, the encoding unit 125 sets the ROI image 112a1 itself as the transmission image 116a1.
[0065] The image processing control unit 126 generates ROI information 120B and frame information 120C and transmits them to the transmission unit 140. The transmission unit 140 generates transmission data 147A based on various data (120A, 120B, 120C, 130A) input from the image processing units 120 and 130. The transmission unit 140 sends the generated transmission data 147A to the video receiving device 200 via the data lane DL.
[0066] The receiving unit 210 receives the transmission data 147A output from the video transmitting device 100 via the data lane DL. The receiving unit 210 performs a predetermined process on the received transmission data 147A to generate EBD data 214A and payload data 215A, and outputs them to the information processing unit 220.
[0067] The information extraction unit 221 extracts the ROI information 120B from the embedded data included in the EBD data 214A. The information extraction unit 221 extracts the coordinates (e.g., upper left coordinates (Xa1, Ya1)), length (e.g., physical region length XLa1, YLa1), and region number 1 (or priority 115 (=1)) of the region of interest ROI corresponding to the ROI image 112a1 from the embedded data included in the EBD data 214A. The information extraction unit 221 further extracts the coordinates (e.g., upper left coordinates (Xa2, Ya2)), length (e.g., physical region length XLa2, YLa2), and region number 2 (or priority 115 (=2)) of the region of interest ROI corresponding to the ROI image 112a2. The ROI decoding unit 222 decodes the compressed image data 147B included in the payload data 215A to generate image data 222A.
[0068] The ROI image generating unit 223 derives the position information 114 of the overlapping region ROO based on the extracted information (extraction information 221A). The ROI image generating unit 223 derives, for example, the coordinates (e.g., upper left corner coordinate (Xb1, Yb1)) and length (e.g., physical region length XLb1, YLb1) of the overlapping region ROO as the position information 114 of the overlapping region ROO. The ROI image generating unit 223 further generates images of the regions of interest ROI (ROI images 112a1, 112a2) in the captured image 111 based on the image data 222A, the extraction information 221A, and the position information 114 of the overlapping region ROO.
[0069] [effect] Next, effects of the video transmission system 1 according to the present embodiment will be described.
[0070] In recent years, there has been an increase in applications that require the transmission of large amounts of data. This puts a heavy load on the transmission system, and in the worst case scenario, the transmission system may crash, making it impossible to transmit data.
[0071] Conventionally, in order to avoid a breakdown in the transmission system, for example, instead of transmitting all captured images, the object being photographed is identified and only a portion of the image extracted from the identified object is transmitted.
[0072] By the way, MIPI CSI-2 is sometimes used as a method for transmitting ROI from an image sensor to an application processor. However, when trying to transmit ROI using this method, various constraints make it difficult to transmit the ROI.
[0073] On the other hand, in this embodiment, ROI information 120B for each region of interest ROI in captured image 111 is sent as embedded data, and image data of each region of interest ROI is sent as payload data of a long packet. This allows the device (video receiving device 200) that receives transmission data 147A sent from video transmitting device 100 to easily extract image data (ROI image 112) of each region of interest ROI from transmission data 147A. As a result, the region of interest ROI can be transmitted even under various constraints.
[0074] In this embodiment, image data (compressed image data 120A) of each region of interest ROI is sent through a common virtual channel. This allows multiple ROI images 112 to be sent in the same packet, making it unnecessary to enter LP mode while sending multiple ROI images 112, and thus achieving high transmission efficiency.
[0075] In this embodiment, the data type of each region of interest ROI is included in the packet header PH of the payload data of the long packet and sent out. This allows the data type of each region of interest ROI to be obtained simply by accessing the packet header PH of the payload data of the long packet without having to access the embedded data. This allows the processing speed in the video receiving device 200 to be increased, resulting in high transmission efficiency.
[0076] In addition, in this embodiment, when the ROI information 120B is included in the payload data of a long packet and sent, the ROI information 120B can be obtained by only accessing the payload data of the long packet without accessing the embedded data. This can increase the processing speed in the video receiving device 200, thereby achieving high transmission efficiency.
[0077] In this embodiment, ROI information 120B for each region of interest ROI is extracted from embedded data included in the transmission data 147A, and an image of each region of interest ROI (ROI image 112) is extracted from payload data of a long packet included in the transmission data 147A based on the extracted ROI information 120B. This makes it possible to easily extract an image of each region of interest ROI (ROI image 112) from the transmission data 147A. As a result, it is possible to transmit the region of interest ROI even under various constraints.
[0078] 2. Prerequisite technology 2 of this disclosure: A technique for transmitting a region of interest (ROI) of a part (having a non-rectangular shape) cut out from a captured image will be described with reference to Figs. 1 to 9 and Figs. 10 to 12. That is, a technique for transmitting and receiving an image of an object of a capture target having a shape other than a square (rectangular) shape will be described. Fig. 10 is a diagram showing a schematic area in which an object identified in a captured image 111 is located. Note that, for ease of understanding, Fig. 10 shows a schematic area in which a captured image 111 captured in an imaging area composed of imaging elements of 15 rows x 23 columns. Fig. 11 is a diagram showing an example of an ROI area set for an identified object.
[0079] In the base technology 2, as in the base technology 1, a case will be described in which a control signal instructing to cut out an ROI is input from the video receiving device 200 to the video transmitting device 100 via the camera control interface CCI, and a predetermined process is performed on the captured image 111 input from the imaging unit 110. However, the base technology 2 can also be applied to a case in which the video transmitting device 100, i.e., the transmitting side, instructs the coordinates of the ROI cut-out. In this case, the transmitting side is configured to receive information such as "person" or "object" to be acquired in the ROI sent from the receiving side, and determine and instruct the coordinates of the cut-out.
[0080] A control signal instructing to cut out the ROI is input from the video receiving device 200 via the camera control interface CCI. As a result, as shown in FIG. 10, the ROI cutout unit 121 specifies four objects 1 to 4 of the shooting target included in the captured image 111. For example, the object 1 has a rectangular shape occupying a part of the upper left region in the captured image 111. For example, the object 2 occupies a part of the right side region of the object 1 in the captured image 111, and has a shape in which both upper corners and a part of the bottom side are missing. For example, the object 3 occupies a part of the lower region of the object 2 in the captured image 111, and has a rectangular shape in which four corners are missing. For example, the object 4 occupies a part of the lower region of the object 3 in the captured image 111, and has a rectangular shape in which both upper corners are missing. The objects 3 and 4 partially overlap each other.
[0081] As shown in FIG. 11, the ROI cutout unit 121 (see FIG. 2) sets the smallest rectangles including the identified objects 1 to 4 as regions of interest ROI1 to ROI4. The ROI cutout unit 121 sets a region of interest ROI1 for object 1 and cuts out an ROI image 112a1. The ROI cutout unit 121 also sets a region of interest ROI2 for object 2 and cuts out an ROI image 112a2. The ROI cutout unit 121 also sets a region of interest ROI3 for object 3 and cuts out an ROI image 112a3. The ROI cutout unit 121 also sets a region of interest ROI4 for object 4 and cuts out an ROI image 112a4.
[0082] The ROI cutout unit 121 associates the region of interest ROI1 with the region number "1" assigned to the region of interest ROI1 and stores them in the storage unit. The ROI cutout unit 121 associates the region of interest ROI2 with the region number "2" assigned to the region of interest ROI2 and stores them in the storage unit. The ROI cutout unit 121 associates the region of interest ROI3 with the region number "3" assigned to the region of interest ROI3 and stores them in the storage unit. The ROI cutout unit 121 associates the region of interest ROI4 with the region number "4" assigned to the region of interest ROI4 and stores them in the storage unit.
[0083] The ROI analysis unit 122 (see FIG. 2) derives position information of each of the regions of interest ROI1 to ROI4. The ROI analysis unit 122 derives, for example, a physical region length XLa1 in the X-axis direction and a physical region length YLa1 in the Y-axis direction as the position information of the region of interest ROI1. The ROI analysis unit 122 derives, for example, a physical region length XLa2 in the X-axis direction and a physical region length YLa2 in the Y-axis direction as the position information of the region of interest ROI2. The ROI analysis unit 122 derives, for example, a physical region length XLa3 in the X-axis direction and a physical region length YLa3 in the Y-axis direction as the position information of the region of interest ROI3. The ROI analysis unit 122 derives, for example, a physical region length XLa4 in the X-axis direction and a physical region length YLa4 in the Y-axis direction as the position information of the region of interest ROI4. The ROI analysis unit 122 may further derive, for each region of interest ROI, as the position information 113, for example, an output region length XLc of the region of interest ROI in the X-axis direction and an output region length YLc of the region of interest ROI in the Y-axis direction.
[0084] The ROI analysis unit 122 derives the length of each of the regions of interest ROI in the X-axis direction and the Y-axis direction, thereby deriving the size and total data amount of each of the regions of interest ROI1 to ROI4 as information for the subsequent stage. This allows the video receiving device 200, which corresponds to the subsequent stage, to secure a memory area.
[0085] The ROI analysis unit 122 is configured to derive position information of the ROI images 112a1-112a4, instead of position information of the region of interest ROI, when the shapes of the object to be photographed and the region of interest do not match. The ROI analysis unit 122 derives the left end coordinates (xn,yn) of each row and the physical region length XLn in the X-axis direction as position information of the ROI images 112a1-112a4. When the ROI images are separated, such as the second row of the ROI image 112a2, the ROI analysis unit 122 derives position information for each of the separated parts. The ROI analysis unit 122 associates the region numbers of the regions of interest ROI1-ROI4 with the position information of the ROI images 112a1-112a4 and stores them in the storage unit.
[0086] In addition, the ROI analysis unit 122 may derive, for example, sensing information, exposure information, gain information, AD word length, image format, etc. in addition to position information for each of the regions of interest ROI1 to ROI4, and store them in the storage unit in association with the region number.
[0087] When the object to be photographed is rectangular, the overlap detection unit 123 (see FIG. 2) derives an overlap region between ROI images, not an overlap region between attention regions, as an overlap region. As shown in FIG. 11, the overlap detection unit 123 derives an overlap region ROO as an overlap region between the ROI image 112a3 and the ROI image 123a4. The overlap detection unit 123 stores the derived overlap region ROO in the storage unit in association with the position information of the attention regions ROI3 and ROI4.
[0088] The priority setting unit 124 (see FIG. 2) assigns a priority of "1" to the region of interest ROI1, and stores the priority 1 in association with the region of interest ROI1 in the storage unit. The priority setting unit 124 assigns a priority of "2" lower than the priority "1" to the region of interest ROI2, and stores the priority 2 in association with the region of interest ROI2 in the storage unit. The priority setting unit 124 assigns a priority of "3" lower than the priority "2" to the region of interest ROI3, and stores the priority 3 in association with the region of interest ROI3 in the storage unit. The priority setting unit 124 assigns a priority of "4" lower than the priority "3" to the region of interest ROI4, and stores the priority 4 in association with the region of interest ROI4 in the storage unit.
[0089] The encoding unit 125 (see FIG. 2) generates a transmission image for each of the ROI images 112a1 to 112a4. Since the region of interest ROI4 has a lower priority than the region of interest ROI3, the encoding unit 125 generates a transmission image by omitting the overlapping region ROO from the ROI image 112a4.
[0090] The image processing control unit 126 (see FIG. 2) generates ROI information and frame information, and transmits them to the transmission unit 140 (see FIG. 2). The ROI information includes, for example, position information of each of the ROI images 112a1-112a4. The ROI information also includes information similar to that in the case where the object to be photographed is rectangular (for example, the data type of each of the regions of interest ROI1-ROI4, the number of regions of interest ROI1-ROI4 included in the captured image 111, region numbers and priorities of the regions of interest ROI1-ROI4, etc.). The frame information includes, for example, the data type of the regions of interest ROI1-ROI4, etc., similar to that in the case where the object to be photographed is rectangular (for example, the data type of the regions of interest ROI1-ROI4, etc.).
[0091] The LINK control unit 141 provided in the transmission unit 140 (see FIG. 2) outputs the frame information and ROI information input from the image processing control unit 126 to the ECC generation unit 142 and the PH generation unit 143 (see FIG. 2 for both). For example, the ECC generation unit 142 generates an error correction code for one line based on data of the line in the frame information (for example, the virtual channel number, the data type of each of the regions of interest ROI1 to ROI4, the payload length for each line, etc.). For example, the ECC generation unit 142 outputs the generated error correction code to the PH generation unit 143. For example, the PH generation unit 143 generates a packet header PH (see FIG. 4) for each line using the frame information and the error correction code generated by the ECC generation unit 142.
[0092] The EBD buffer 144 (see FIG. 2) temporarily stores the ROI information, and outputs the ROI information as embedded data to the synthesis unit 147 (see FIG. 2) at a predetermined timing.
[0093] The ROI data buffer 145 (see FIG. 2) temporarily stores the compressed image data input from the encoding unit 125, and outputs the compressed image data 120A to the synthesis unit 147 as payload data of a long packet when, for example, a control signal instructing extraction of an ROI is input from the video receiving device 200 via the camera control interface CCI.
[0094] When a control signal instructing to extract an ROI is input from the video receiving device 200 via the camera control interface CCI, the synthesis unit 147 generates transmission data 147A based on the various input data (packet header PH, ROI information, and compressed image data input from the encoding unit 125 via the ROI data buffer 145). The synthesis unit 147 outputs the generated transmission data 147A to the video receiving device 200 via the data lane DL. That is, the synthesis unit 147 includes the data type of each of the regions of interest ROI1 to ROI4 in the packet header PH of the payload data of the long packet and transmits it. Furthermore, the synthesis unit 147 transmits the image data (compressed image data) of each of the regions of interest ROI1 to ROI4 via a common virtual channel.
[0095] If the object to be photographed is not rectangular, the position information of the ROI images 112a1-112a4 is included in the packet header PH or the payload data of the long packet. The position information of the ROI images 112a1-112a4 is included in the packet header PH by the PH generation unit 143. On the other hand, the position information of the ROI images 112a1-112a4 is included in the payload data of the long packet by the synthesis unit 147.
[0096] Fig. 12 is a diagram showing a configuration example of transmission data 147A in which position information of ROI images 112a1-112a4 is included in payload data of a long packet. As shown in Fig. 12, the transmission data 147A includes, for example, a frame header area R1 and a packet area R2. Note that Fig. 12 shows an example of the contents of the frame header area R1 in detail. Also, Fig. 12 omits the low power mode LP.
[0097] The frame header region R1 includes, for example, a frame number F1 as an identifier of the transmission data 147A. The frame header region R1 includes information about the compressed image data included in the packet region R2. The frame header region R1 includes, for example, the number of compressed image data (number of ROIs) and information (ROI information) about each of the ROI images 112a1 to 112a4 corresponding to each compressed image data. The ROI information includes a region number, a physical region length, a rectangular output region size, a priority, exposure information, gain information, an AD word length, and an image format. The physical region length is the maximum length of the ROI image, and the rectangular output region size is the size of the region of interest ROI.
[0098] "Info" shown in FIG. 12 indicates area information stored in the payload of a long packet. The position information of the ROI images 112a1 to 112a4 is stored in, for example, "info". The position information of the ROI images 112a1 to 112a4 is stored in the beginning of the payload of a long packet. When the physical area length in the X-axis direction of each of the consecutive pixel rows constituting an ROI image is the same and the corresponding pixel rows do not include ROI images with different area numbers, the payload of a long packet including image data of the second and subsequent pixel rows among the corresponding pixel rows may not store the area information "info". In this example, the ROI image 112a1 has the same physical area length in the X-axis direction in the consecutive first to fourth pixel rows among all pixel rows, and the corresponding first to fourth pixel rows do not include ROI images with different area numbers. Therefore, the payload of each long packet including image data of the second to fourth pixel rows, which correspond to the second and subsequent rows among the first to fourth consecutive pixel rows constituting the ROI image 112a1, does not store the region information "info". In addition, in this example, the ROI image 112a4 has the same physical region length in the X-axis direction in the second and third consecutive pixel rows among all pixel rows, and the second and third pixel rows do not include ROI images with different region numbers. Therefore, the payload of a long packet including image data of the third pixel row, which corresponds to the second and subsequent rows among the second and third consecutive pixel rows constituting the ROI image 112a4, does not store the region information "info". Note that even if the physical region length in the X-axis direction is the same and the pixel rows do not include ROI images with different region numbers, the region information "info" may be stored in the payload of each row.
[0099] For example, the synthesis unit 147 arranges the compressed image data generated by compressing each of the ROI images 112a1 to 112a4 in the packet region R2 of the transmission data 147A by dividing them into pixel rows. "1" shown in FIG. 12 indicates the compressed image data of the ROI image 112a1 stored in the payload of the long packet. "2" shown in FIG. 12 indicates the compressed image data of the ROI image 112a2 stored in the payload of the long packet. "3" shown in FIG. 12 indicates the compressed image data of the ROI image 112a3. "4" shown in FIG. 12 indicates the compressed image data of the ROI image 112a4 stored in the payload of the long packet. Note that in FIG. 12, each compressed image data is shown separated for ease of understanding, but the data stored in the payload of the long packet is not separated. The packet region R2 of the transmission data 147A does not include the compressed image data 112b corresponding to the image of the overlap region ROO in a duplicated manner. Furthermore, for example, in the packet region R2 of the transmission data 147A, the synthesis unit 147 omits pixel rows that do not correspond to any of the transmission images in the captured image 111. Therefore, the packet region R2 of the transmission data 147A does not include pixel rows that do not correspond to any of the transmission images in the captured image 111.
[0100] Next, the operation of the video receiving device 200 when it receives the transmission data 147A will be described. The header separation unit 211 (see FIG. 8 for both) provided in the receiving unit 210 receives the transmission data 147A from the video transmitting device 100 via the data lane DL. That is, the header separation unit 211 receives the transmission data 147A including ROI information for the regions of interest ROI1 to ROI4 in the captured image 111 in embedded data and including image data (compressed image data) of the regions of interest ROI1 to ROI4 in payload data of a long packet. The header separation unit 211 separates the received transmission data 147A into a frame header region R1 and a packet region R2.
[0101] The header interpretation unit 212 (see FIG. 8) identifies the position of the payload data of the long packet included in the packet area R2 based on the data (specifically, embedded data) included in the frame header area R1.
[0102] Based on the position of the payload data of the long packet identified by the header interpretation unit 212, the payload separation unit 213 (see FIG. 8) separates the payload data of the long packet contained in the packet area R2 from the packet area R2.
[0103] The EBD interpretation unit 214 outputs the embedded data as EBD data to the information processing unit 220 (see FIG. 8). The EBD interpretation unit 214 further determines, from the data type included in the embedded data, whether the image data included in the payload data of the long packet is compressed image data of the ROI image data 116 or compressed image data of normal image data. The EBD interpretation unit 214 outputs the determination result to the ROI data separation unit 215 (see FIG. 8).
[0104] When image data included in the payload data of a long packet is input as ROI image data, the ROI data separation unit 215 outputs the payload data of the long packet as payload data to the information processing unit 220 (specifically, the ROI decoding unit 222 (see FIG. 8)). The payload data of the long packet including the ROI information includes the ROI information and pixel data for one line of the compressed image data.
[0105] The information extraction unit 221 (see FIG. 8) provided in the information processing unit 220 extracts the number of regions of interest ROI1-ROI4 (four in this example) included in the captured image 111, the region numbers 1-4 and priorities 1-4 of the regions of interest ROI1-ROI4, the data lengths of each of the regions of interest ROI1-ROI4, and the image formats of each of the regions of interest ROI1-ROI4, from the embedded data included in the EBD data input from the EBD interpretation unit 214. Furthermore, the information extraction unit 221 extracts position information of the ROI images 112a1-112a4 from the embedded data.
[0106] The ROI decoding unit 222 decodes the compressed image data 147B included in the payload data, extracts position information of the ROI images 112a1 to 112a4, and generates image data (composed of the transmitted image). For example, when payload data corresponding to the sixth pixel row is input, the ROI decoding unit 222 extracts one piece of position information of the ROI image 112a1 and two pieces of position information of the ROI image 112a2 from the payload data, and generates image data (transmitted images) of the ROI images 112a1 and 112b1 corresponding to the sixth pixel row.
[0107] When payload data corresponding to, for example, the 10th pixel row is input, the ROI decoding unit 222 extracts one piece of position information of the ROI image 112a3 and one piece of position information of the ROI image 112a4 from the payload data, and generates image data (transmitted images) of the ROI images 112a3 and 112b4, respectively.
[0108] The ROI image generation unit 223 (see FIG. 8) generates ROI images 112a1-112a4 of the regions of interest ROI1-ROI4 in the captured image 111 based on the ROI information obtained by the information extraction unit 221, the position information of the ROI images 112a1-112a4 extracted by the ROI decoding unit 222, and the transmitted image generated by the ROI decoding unit 222. For example, when the ROI image generation unit 223 receives one piece of position information of ROI image 112a1 and two pieces of position information of ROI image 112a2 extracted from payload data corresponding to the sixth pixel row, as well as their transmission images, it generates an ROI image 112a1 of five pixels extending in the X-axis direction, an ROI image 112a2 of four pixels extending in the X-axis direction at a position five pixels away from the ROI image 112a1, and an ROI image 112a2 of two pixels extending in the X-axis direction at a position two pixels away from the ROI image 112a2 (see FIG. 10).
[0109] Furthermore, the ROI image generating unit 223 detects an overlapping area ROO where the regions of interest ROI3 and ROI4 overlap, based on the ROI information obtained by the information extracting unit 221. The ROI image generating unit 223 generates an ROI image 112a3 of four pixels extending in the X-axis direction and an ROI image 112a4 of three pixels extending in the X-axis direction with one pixel overlapping the ROI image 112a3, based on the detected overlapping area ROO, the position information of the ROI images 112a3 and 112a4 extracted from the payload data corresponding to the tenth pixel row, and the transmitted image (see FIG. 10).
[0110] The ROI image generating unit 223 outputs the generated image as an ROI image to a downstream device (not shown).
[0111] In this way, the video transmitting device 100 and the video receiving device 200 can transmit and receive an ROI image even if the object to be photographed has a shape other than a rectangle.
[0112] 3. Principle of demosaicing in the embodiment of the present disclosure: Next, the principle of demosaic processing in the embodiment of the present disclosure will be described with reference to FIGS.
[0113] FIG. 13 is a diagram showing an example of a color arrangement of an imaging element provided in an imaging region of an imaging unit. FIG. 14A is a diagram showing an arrangement example A of a color arrangement of an imaging element provided in an imaging region of an imaging unit. FIG. 14B is a diagram showing arrangement patterns PA1 to PA4 of a color arrangement of an imaging element cut out from the imaging region shown in FIG. 14A. FIG. 15A is a diagram showing an arrangement example B of a color arrangement of an imaging element provided in an imaging region of an imaging unit. FIG. 15B is a diagram showing arrangement patterns PB1 to PB4 of a color arrangement of an imaging element cut out from the imaging region shown in FIG. 15A. FIG. 16A is a diagram showing an arrangement example C of a color arrangement of an imaging element provided in an imaging region of an imaging unit. FIG. 16B is a diagram showing arrangement patterns PC1 to PC4 of a color arrangement of an imaging element cut out from the imaging region shown in FIG. 16A. FIG. 17A is a diagram showing an arrangement example D of a color arrangement of an imaging element provided in an imaging region of an imaging unit. Fig. 17B is a diagram showing arrangement patterns PD1 to PD4 of color arrangements of the imaging element cut out from the imaging region shown in Fig. 17A. Fig. 18 is a diagram showing a schematic diagram of demosaic processing.
[0114] 13, the imaging region IR in this example has a red pixel (hereinafter abbreviated as "R pixel") located at the upper left edge. The imaging region IR also has an R pixel at the left edge, odd-numbered rows in which R pixels and green pixels (hereinafter abbreviated as "G pixel") are alternately arranged, and an G pixel at the left edge, even-numbered rows in which G pixels and blue pixels (hereinafter abbreviated as "B pixel") are alternately arranged.
[0115] 13, the color arrangement of pixels provided in the imaging region IR of the imaging unit is fixed. Therefore, in normal demosaic processing in which demosaic processing is performed on the entire imaging region IR, the video receiving device can perform the demosaic processing as long as it has information on the color arrangement of the entire imaging region IR.
[0116] However, in the ROI, the range and size to be cut out as the region of interest are unknown. Therefore, as shown in FIG. 13, the upper left end of the region of interest ROI-α is a G pixel, and the upper left end of the region of interest ROI-β is a B pixel. Therefore, even if the video receiving device has information on the color arrangement of the entire imaging area IR, the upper left end of each of the regions of interest ROI-α and ROI-β is an R pixel and demosaic processing is performed. Therefore, the image after demosaic is different from the original image. In this way, demosaic processing cannot be performed on the ROI whose position and size are arbitrarily selected in the imaging area IR.
[0117] Therefore, in this embodiment, the video transmitting device is configured to obtain demosaic information, which is information used for demosaic processing of image data of the ROI, and transmit it to the video receiving device. The video receiving device is configured to perform demosaic processing using the demosaic information transmitted from the video transmitting device.
[0118] As shown in FIG. 14A, the color array example A of the image sensor has the same color array as the imaging area IR shown in FIG. 13. As shown in FIG. 14B, there are four array patterns PA1 to PA4 as color arrays cut out from the array example A. As shown in FIG. 14B, the upper left end in the array pattern PA1 corresponds to the R pixels in the odd rows (H) and odd columns (V). In addition, the color array of the entire imaging area in the array example A is known (see FIG. 14A). Therefore, if the size of the area to be cut out is 3 rows and 3 columns, as shown in FIG. 14B, the array pattern PA1 is determined to be "R pixel, G pixel, R pixel" in the first row, "G pixel, B pixel, G pixel" in the second row, and "R pixel, G pixel, R pixel" in the third row.
[0119] As shown in Fig. 14B, the upper left corner in array pattern PA2 corresponds to the G pixels in the even rows (H) and odd columns (V). Furthermore, the color array of the entire imaging area in array example A is known (see Fig. 14A). Therefore, if the size of the area to be cut out is 3 rows and 3 columns, as shown in Fig. 14B, it is determined that in array pattern PA2, the first row will be "G pixels, R pixels, G pixels," the second row will be "B pixels, G pixels, B pixels," and the third row will be "G pixels, R pixels, G pixels."
[0120] As shown in Fig. 14B, the upper left corner in array pattern PA3 corresponds to the G pixels in odd rows (H) and even columns (V). Furthermore, the color array of the entire imaging area in array example A is known (see Fig. 14A). Therefore, if the size of the area to be cut out is 3 rows and 3 columns, as shown in Fig. 14B, it is determined that in array pattern PA3, the first row will be "G pixel, B pixel, G pixel," the second row will be "R pixel, G pixel, R pixel," and the third row will be "G pixel, B pixel, G pixel."
[0121] As shown in Fig. 14B, the upper left corner in array pattern PA4 corresponds to the B pixel in the even row (H) and even column (V). Furthermore, the color array of the entire imaging area in array example A is known (see Fig. 14A). Therefore, if the size of the area to be cut out is 3 rows and 3 columns, as shown in Fig. 14B, it is determined that in array pattern PA4, the first row will be "B pixel, G pixel, B pixel," the second row will be "G pixel, R pixel, G pixel," and the third row will be "B pixel, G pixel, B pixel."
[0122] 15A, the color array example B of the image sensor has a G pixel arranged at the upper left corner. Also, the left corner of the array example B is a G pixel, and the odd rows have G and R pixels arranged alternately, and the left corner is a B pixel, and the even rows have B and G pixels arranged alternately.
[0123] As shown in FIG. 15B, there are four array patterns PB1 to PB4 as color arrays cut out from array example B. As shown in FIG. 15B, the upper left end in array pattern PB1 corresponds to the G pixels in the odd rows (H) and odd columns (V). Also, the color array of the entire imaging area in array example B is known (see FIG. 15A). Therefore, if the size of the area to be cut out is 3 rows and 3 columns, as shown in FIG. 15B, it is determined that array pattern PB1 will have "G pixel, R pixel, G pixel" in the first row, "B pixel, G pixel, B pixel" in the second row, and "G pixel, R pixel, G pixel" in the third row.
[0124] As shown in Fig. 15B, the upper left corner in array pattern PB2 corresponds to the R pixels in the even rows (H) and odd columns (V). Furthermore, the color array of the entire imaging area in array example B is known (see Fig. 15A). For this reason, if the size of the area to be cut out is 3 rows and 3 columns, as shown in Fig. 15B, it is determined that in array pattern PB2, the first row will be "R pixel, G pixel, R pixel," the second row will be "G pixel, B pixel, G pixel," and the third row will be "R pixel, G pixel, R pixel."
[0125] As shown in Fig. 15B, the upper left corner in array pattern PB3 corresponds to the B pixel in the odd row (H) and even column (V). Furthermore, the color array of the entire imaging area in array example B is known (see Fig. 15A). Therefore, if the size of the area to be cut out is 3 rows and 3 columns, as shown in Fig. 15B, it is determined that in array pattern PB3, the first row will be "B pixel, G pixel, B pixel," the second row will be "G pixel, R pixel, G pixel," and the third row will be "B pixel, G pixel, B pixel."
[0126] As shown in Fig. 15B, the upper left corner in array pattern PB4 corresponds to the G pixel in the even row (H) and even column (V). Furthermore, the color array of the entire imaging area in array example B is known (see Fig. 15A). Therefore, if the size of the cut-out area is 3 rows and 3 columns, as shown in Fig. 15B, it is determined that in array pattern PB4, the first row will be "G pixel, B pixel, G pixel," the second row will be "R pixel, G pixel, R pixel," and the third row will be "G pixel, B pixel, G pixel."
[0127] 16A, the color array example C of the image sensor has a B pixel arranged at the upper left end. Also, the left end of the array example C has a B pixel, odd rows in which B pixels and G pixels are arranged alternately, and a G pixel, even rows in which G pixels and R pixels are arranged alternately.
[0128] As shown in FIG. 16B, there are four array patterns PC1 to PC4 as color arrays cut out from array example C. As shown in FIG. 16B, the upper left end in array pattern PC1 corresponds to the B pixel in the odd row (H) and odd column (V). Also, the color array of the entire imaging area in array example C is known (see FIG. 16A). Therefore, if the size of the area to be cut out is 3 rows and 3 columns, as shown in FIG. 16B, it is determined that array pattern PC1 will have "B pixel, G pixel, B pixel" in the first row, "G pixel, R pixel, G pixel" in the second row, and "B pixel, G pixel, B pixel" in the third row.
[0129] As shown in Fig. 16B, the upper left corner in array pattern PC2 corresponds to the G pixels in the even rows (H) and odd columns (V). Furthermore, the color array of the entire imaging area in array example C is known (see Fig. 16A). Therefore, if the size of the area to be cut out is 3 rows and 3 columns, as shown in Fig. 16B, it is determined that the first row of array pattern PC2 will be "G pixel, B pixel, G pixel," the second row will be "R pixel, G pixel, R pixel," and the third row will be "G pixel, B pixel, G pixel."
[0130] As shown in Fig. 16B, the upper left corner in array pattern PC3 corresponds to the G pixels in odd rows (H) and even columns (V). Furthermore, the color array of the entire imaging area in array example C is known (see Fig. 16A). Therefore, if the size of the area to be cut out is 3 rows and 3 columns, as shown in Fig. 16B, it is determined that the first row of array pattern PC3 will be "G pixel, R pixel, G pixel," the second row will be "B pixel, G pixel, B pixel," and the third row will be "G pixel, R pixel, G pixel."
[0131] As shown in Fig. 16B, the upper left corner in array pattern PC4 corresponds to the R pixel in the even row (H) and even column (V). Furthermore, the color array of the entire imaging area in array example C is known (see Fig. 16A). For this reason, if the size of the area to be cut out is 3 rows and 3 columns, as shown in Fig. 16B, it is determined that the first row of array pattern PC4 will be "R pixel, G pixel, R pixel," the second row will be "G pixel, B pixel, G pixel," and the third row will be "R pixel, G pixel, R pixel."
[0132] As shown in Fig. 17A, the color array example D of the imaging element has an R pixel arranged at the upper left end. Also, the left end of the array example D has an R pixel, and odd rows in which R pixels and G pixels are arranged alternately, and an even row in which the left end is a white pixel (hereinafter abbreviated as "W pixel"), and W pixels and B pixels are arranged alternately. The W pixel is, for example, a pixel that does not have a color element (color filter) on a photoelectric conversion element (e.g., a photodiode).
[0133] As shown in FIG. 17B, there are four array patterns PD1 to PD4 as color arrays cut out from array example D. As shown in FIG. 17B, the upper left end in array pattern PD1 corresponds to the R pixels in odd rows (H) and odd columns (V). Also, the color array of the entire imaging area in array example D is known (see FIG. 17A). Therefore, if the size of the area to be cut out is 3 rows and 3 columns, as shown in FIG. 17B, it is determined that the first row of array pattern PD1 will be "R pixel, G pixel, R pixel," the second row will be "W pixel, B pixel, W pixel," and the third row will be "R pixel, G pixel, R pixel."
[0134] As shown in Fig. 17B, the upper left corner in array pattern PD2 corresponds to the G pixels in the even rows (H) and odd columns (V). Furthermore, the color array of the entire imaging area in array example D is known (see Fig. 17A). Therefore, if the size of the area to be cut out is 3 rows and 3 columns, as shown in Fig. 17B, it is determined that the first row of array pattern PD2 will be "G pixel, R pixel, G pixel," the second row will be "B pixel, W pixel, B pixel," and the third row will be "G pixel, R pixel, G pixel."
[0135] As shown in Fig. 17B, the upper left corner in array pattern PD3 corresponds to the W pixel in the odd row (H) and even column (V). Also, the color array of the entire imaging area in array example D is known (see Fig. 17A). Therefore, if the size of the cut-out area is 3 rows and 3 columns, as shown in Fig. 17B, it is determined that the first row of array pattern PD3 will be "W pixel, B pixel, W pixel," the second row will be "R pixel, G pixel, R pixel," and the third row will be "W pixel, B pixel, W pixel."
[0136] As shown in Fig. 17B, the upper left corner in array pattern PD4 corresponds to the B pixel in an even row (H) and even column (V). Also, the color array of the entire imaging area in array example D is known (see Fig. 17A). Therefore, if the size of the area to be cut out is 3 rows and 3 columns, as shown in Fig. 17B, it is determined that the first row of array pattern PD4 will be "B pixel, W pixel, B pixel," the second row will be "G pixel, R pixel, G pixel," and the third row will be "B pixel, W pixel, B pixel."
[0137] In this way, when the color arrangement of the entire imaging area is known, the color arrangement of the pixels included in the region of interest ROI can be determined by obtaining information on the color of the pixel at the upper left corner, information on whether the row and column at the upper left corner are odd or even, and information on the size of the cut-out area. Therefore, by the video transmitting device transmitting such information on the color arrangement of the pixels for each region of interest ROI to the video receiving device as demosaic information, the video receiving device can perform demosaic processing on the region of interest ROI.
[0138] For example, assume that the video transmitting device and the video receiving device each store information in a storage unit that the imaging unit has an imaging area having a color array of the array example A, and information on the upper left end of the array pattern PA1 and information on the size of the attention region ROI-γ are transmitted from the video transmitting device to the video receiving device as demosaic information. Based on the demosaic information, the video receiving device can determine that the color array of the attention region ROI-γ is the array shown on the left side of FIG. 18. Based on the determined color array of the attention region ROI-γ, the video receiving device performs demosaic processing to convert the RGB values on the mosaic into RGB signals for each pixel. For pixels that do not exist before mosaic, the video receiving device generates the pixel by processing such as interpolation from surrounding pixels of the same color. The video receiving device performs pixel interpolation processing based on the color array in the attention region ROI-γ, and can generate red pixel data, green pixel data, and blue pixel data after demosaic processing, as shown to the right of the thick arrow in FIG. 18.
[0139] 4. First embodiment of the present disclosure: Next, a transmitting device, a receiving device, and a transmission system according to a first embodiment of the present disclosure will be described with reference to Fig. 19 to Fig. 22. First, a schematic configuration of a transmitting device, a receiving device, and a transmission system according to this embodiment will be described with reference to Fig. 19. Fig. 19 is a block diagram showing a schematic configuration of a video transmitting device 3, a video receiving device 4, and a video transmission system 10 according to this embodiment.
[0140] As shown in FIG. 19, a video transmission system 10 according to the present embodiment includes a video transmission device (an example of a transmission device) 3 that functions as an image sensor, and a video reception device (an example of a reception device) 4 that functions as an image processing (Image Signal Processor: ISP). In the video transmission system (an example of a transmission system) 10, the video transmission device 3 is configured to transmit a signal conforming to the MIPI (Mobile Industry Processor Interface) D-PHY standard, the MIPI C-PHY standard, or the MIPI CSI (Camera Serial Interface)-2 standard from a transmission unit 322. In addition, in the video transmission system 10, the video reception device 4 is configured to receive a signal conforming to the MIPI D-PHY standard, the MIPI C-PHY standard, or the MIPI CSI-2 standard at a reception unit 412. In addition, the video transmission system 10 may be configured to transmit and receive signals conforming to the MIPI CSI-3 standard or the MIPI DSI standard between the video transmission device 3 and the video reception device 4, similar to the video transmission systems 1 according to the above-mentioned base technologies 1 and 2.
[0141] The video transmitting device 3 included in the video transmission system 10 is configured to perform the same functions as the video transmitting device 100 according to the above-mentioned base technologies 1 and 2. That is, the video transmitting device 3 is configured to be able to execute the same process as the video transmitting device 100 on the captured image input from the imaging unit 31 when a control signal instructing to cut out an ROI is input from the video receiving device 4. Also, the video transmitting device 3 is configured to be able to execute the same process as the video transmitting device 100 on the captured image input from the imaging unit 31 when a control signal instructing to output a normal image is input from the video receiving device 4. Furthermore, the video transmitting device 3 is configured to obtain demosaic information used in the above-mentioned demosaic process and to transmit the demosaic information to the video receiving device 4.
[0142] The video receiving device 4 is configured to perform functions equivalent to those of the video receiving device 200 according to the above-mentioned base technologies 1 and 2. In other words, the video receiving device 4 is configured to be able to execute the same processing as the video receiving device 200 according to the above-mentioned base technologies 1 and 2 on the transmission data transmitted from the video transmitting device 3. Furthermore, the video receiving device 4 is configured to be able to execute demosaic processing using the demosaic information transmitted from the video transmitting device 3.
[0143] Therefore, in FIG. 19, the video transmitting device 3 and the video receiving device 4 are illustrated mainly with respect to the configuration related to the demosaic processing.
[0144] As shown in FIG. 19, the video transmission device 3 includes an imaging unit 31 that captures an image of an object. The imaging unit 31 includes a photoelectric conversion unit 311 that converts incident light into an electric signal. The photoelectric conversion unit 311 is configured, for example, by a CCD image sensor or a CMOS image sensor. The imaging unit 31 also includes a signal conversion unit 312 that converts an analog electric signal input from the photoelectric conversion unit 311 into digital image data. The signal conversion unit 312 is configured to perform a signal amplification (AGC) process of the analog electric signal input from the photoelectric conversion unit 311, and an analog-to-digital conversion (ADC) process of converting the amplified electric signal into a digital signal. The imaging unit 31 includes an amplifier 313 that applies a digital gain to the image data input from the signal conversion unit 312. The amplifier 313 is configured to output the image data to which the digital gain has been applied to the transmission unit 322.
[0145] The video transmission device 3 includes a control unit 32 that controls the imaging unit 31 and predetermined signal processing. The control unit 32 includes a sensor CPU 321 and a transmission unit 322. The sensor CPU 321 is configured to perform the same function as the image processing units 120 and 130 (see FIG. 2). The transmission unit 322 is configured to perform the same function as the transmission unit 140 (see FIG. 2). In the control unit 32, the sensor CPU 321 may be replaced with the image processing units 120 and 130, and the transmission unit 322 may be replaced with the transmission unit 140.
[0146] The sensor CPU 321 has an exposure control unit 321a that controls the exposure conditions of the photoelectric conversion unit 311. The sensor CPU 321 also has a conversion region control unit (an example of a control unit) 321b that controls acquisition of demosaic information that is information used in demosaic processing of image data of the ROI. The sensor CPU 321 and the control unit 32 that have the conversion region control unit 321b each correspond to an example of a control unit that controls acquisition of demosaic information that is information used in demosaic processing of image data of the region of interest ROI.
[0147] The conversion area control unit 321b is configured to acquire demosaic information of the region of interest ROI. When a plurality of regions of interest ROI are set, the conversion area control unit 321b is configured to acquire demosaic information for each region of interest ROI. The conversion area control unit 321b is configured to acquire the color arrangement of the image data of the region of interest ROI or the color information of the end of the image data of the region of interest ROI as demosaic information. More specifically, the conversion area control unit 321b acquires, as the information of the color arrangement, color information of the end of the region of interest ROI and information indicating which combination of odd and even rows and odd and even columns the end is. In this embodiment, the conversion area control unit 321b is configured to acquire color information of the upper left end as the end of the region of interest ROI, but as long as the information can specify the color arrangement of the region of interest ROI, it may be any of the four corners or information of other locations. The conversion region control unit 321b acquires color information of the pixel at the upper left corner of the region of interest ROI, for example, and even or odd row and column information at the upper left corner, and outputs them to the transmission unit 322.
[0148] In addition, the conversion area control unit 321b is configured to send information on the entire color arrangement of the imaging area of the imaging unit 31 to the video receiving device 4 when the first demosaic information is sent after the video transmitting device 3 and the video receiving device 4 are started up, or each time demosaic processing information is sent.
[0149] The sensor CPU 321, like the ROI cropping unit 121 (see FIG. 2), sets the smallest rectangle including the object to be cropped as the region of interest ROI, even if the object is not rectangular. Also, like the ROI analysis unit 122 (see FIG. 2), the sensor CPU 321 is configured to derive position information of the region of interest ROI (upper left end, length in the X-axis direction, length in the Y-axis direction) and send it to the video receiving device 4.
[0150] The video receiving device 4 can determine the overall color arrangement of the region of interest ROI based on information on the overall size of the region of interest ROI, demosaic information on the upper left corner of the region of interest ROI, and information on the color arrangement of the entire imaging area. This enables the video receiving device 4 to interpolate pixels that do not exist before mosaic from surrounding pixels of the same color in the demosaic process, even if the shape of the object to be cut out is not rectangular.
[0151] The transmitting unit 322 generates transmission data (see FIGS. 6 and 12) including the coordinates (cutout coordinates), size, and demosaic information of the region of interest ROI input from the sensor CPU 321, as well as image data input from the imaging unit 31, and transmits the data to the video receiving device 4. The demosaic information is included in the ROI information and transmitted from the transmitting unit 322. Since the ROI information is included in the embedded data, the demosaic information is included in the embedded data and transmitted from the transmitting unit 322.
[0152] In this way, the video transmitting device 3 transmits the demosaic information included in the ROI information from the transmitting unit 322. In other words, the video transmitting device 3 transmits the color arrangement of the image data of the region of interest ROI or color information of the end of the image data of the region of interest ROI as demosaic information from the transmitting unit 322. In other words, the video transmitting device transmits color information of the end of the region of interest ROI (the upper left end in this embodiment) and information indicating which combination of odd and even rows and odd and even columns the end is as demosaic information from the transmitting unit 322.
[0153] 19, the video transmitting device 3 includes a transmitting unit 322 that transmits image data of a region of interest ROI as payload data of a long packet and transmits ROI information as embedded data. The transmitting unit 322 includes demosaic information as one piece of ROI information in the embedded data and transmits the data to the video receiving device 4. The transmitting unit 322 is configured to transmit transmission data including demosaic information and the like in accordance with the MIPI D-PHY standard, the MIPI C-PHY standard, or the MIPI CSI-2 standard.
[0154] As shown in FIG. 19, the video receiving device 4 includes a control unit 41 that controls a predetermined signal processing using transmission data transmitted from the video transmitting device 3. The control unit 41 includes a CamCPU 411, a receiving unit 412, a Raw processing unit 413, and an embedded data acquisition unit 414. The CamCPU 411 is configured to perform the same functions as the information processing unit 220 (see FIG. 8) except for the information extraction unit 221 and the ROI image generation unit 223 (see FIG. 8). The video receiving device 4 is configured such that the Raw processing unit 413 performs the same functions as the ROI image generation unit 223. The reception 412 is configured to perform the same functions as the receiving unit 210 (see FIG. 8) except for the EBD interpretation unit 214 (see FIG. 8). The video receiving device 4 is configured such that the embedded data acquisition unit 414 performs the same functions as the EBD interpretation unit 214 and the information extraction unit 221. In addition, in the control unit 41, the receiving unit 412 and the embedded data acquisition unit 414 may be replaced with the receiving unit 210, and the Cam CPU 411 and the Raw processing unit 413 may be replaced with the information processing unit 220. In this case, the function of the information extraction unit 221 exerted by the embedded data acquisition unit 414 is exerted by the receiving unit 220.
[0155] 19, the video receiving device 4 includes a receiving unit 412 that receives a transmission signal including image data of a region of interest ROI in payload data and including ROI information in embedded data. The receiving unit 412 is configured to receive transmission data input from the video transmitting device 3. The receiving unit 412 receives the transmission data in accordance with the MIPI D-PHY standard, the MIPI C-PHY standard, or the MIPI CSI-2 standard. The receiving unit 412 is configured to generate various data from the input transmission data and output the data to the CamCPU 411, the Raw processing unit 413, and the embedded data acquisition unit 414.
[0156] The Raw processing unit 413 is configured to generate image data of the region of interest ROI based on information about the region of interest ROI (such as ROI information and image data included in payload data) input from the Cam CPU 411. The image data generated by the Raw processing unit 413 is unprocessed image data acquired by the photoelectric conversion unit 311, which is called Raw data, Raw image, or undeveloped data. The Raw processing unit 413 is configured to output the generated image data to the image processing unit 42 (described in detail later).
[0157] As shown in FIG. 19, the video receiving device 4 includes an embedded data acquisition unit (an example of a control unit) 414 that controls extraction of demosaic information, which is information used for demosaic processing of image data of the region of interest ROI, from a transmission signal (transmission data) received by the receiving unit 412. The control unit 41 having the embedded data acquisition unit 414 corresponds to an example of a control unit that controls extraction of demosaic information, which is information used for demosaic processing of image data of the region of interest ROI, from a transmission signal (transmission data) received by the receiving unit 412. The embedded data acquisition unit 414 is configured to extract the demosaic information from the ROI information included in the transmission signal (transmission data) input from the receiving unit 412. Since the ROI information is included in the embedded data, the embedded data acquisition unit 414 extracts the demosaic information from the embedded data included in the transmission signal (transmission data) input from the receiving unit 412. The embedded data acquisition unit 414 is configured to extract the color arrangement of the image data of the region of interest ROI or color information of the end of the image data of the region of interest ROI as the demosaic information. More specifically, the embedded data acquisition unit 414 is configured to extract, as the color array information, color information of the end of the region of interest ROI and information indicating which combination of odd and even rows and odd and even columns the end is, as demosaic information. In this embodiment, the embedded data acquisition unit 414 is configured to acquire color information of the upper left end as the end of the region of interest ROI, but as long as the information can specify the color array of the region of interest ROI, it may be any of the four corners or information of other locations. The embedded data acquisition unit 414 acquires color information of the pixel at, for example, the upper left end of the region of interest ROI, and even or odd row information and column information of the upper left end, and outputs them to the Cam CPU 411.
[0158] The embedded data acquisition unit 414 acquires various information included in the embedded data (e.g., the number of regions of interest ROI, region numbers and priorities of the regions of interest ROI, data lengths of the regions of interest ROI, and image formats of the regions of interest ROI, etc.) in addition to the demosaic information. The embedded data acquisition unit 414 also outputs the acquired various information to the Cam CPU 411.
[0159] 19, the CamCPU 411 has a coordinate determination unit 411a. The coordinate determination unit 411a is configured to determine the coordinates (position and size) and color arrangement of the region of interest ROI based on various information input from the embedded data acquisition unit 414. When information on multiple regions of interest ROI is input from the embedded data acquisition unit 414, the coordinate determination unit 411a determines the coordinates and color arrangement for each region of interest ROI. The coordinate determination unit 411a sets the coordinates of the upper left corner of the region of interest ROI for demosaic processing based on the coordinates of the upper left corner of the determined region of interest ROI.
[0160] The Cam CPU 411 outputs to the image processing unit 42 information on the coordinates and color arrangement of the region of interest ROI determined by the coordinate determination unit 411 a, and information such as the region number and priority of the region of interest ROI input from the embedded data acquisition unit 414 .
[0161] 19, the video receiving device 4 includes an image processing unit 42. The image processing unit 42 includes a demosaic processing unit 421 that executes demosaic processing of image data of the region of interest ROI using the demosaic information extracted by the embedded data acquisition unit 414. The image processing unit 42 also includes an image quality adjustment unit 422 that adjusts the image quality of the image data that has been subjected to the demosaic processing.
[0162] The demosaic processing unit 421 is configured to perform demosaic processing on image data input from the Raw processing unit 413 based on demosaic information (information on the coordinates and color array of the region of interest ROI) extracted by the embedded data acquisition unit 414 and input via the Cam CPU 411. As described with reference to Fig. 14 to Fig. 18, the demosaic processing unit 421 performs demosaic processing on image data before demosaic (corresponding to image data input from the Raw processing unit 413) shown to the left of the thick arrow in Fig. 18 based on the array pattern PA1 shown in Fig. 14B, for example. As a result, the demosaic processing unit 421 generates image data after demosaic shown to the right of the thick arrow in Fig. 18.
[0163] The demosaic processor 421 is also configured to perform boundary processing on image data arranged at the peripheral edge in the region of interest ROI. That is, the demosaic processor 421 performs boundary processing on pixels arranged at the contour (edge) of the region of interest ROI. The demosaic processor 421 performs boundary processing on image data of the first row in the region of interest ROI using image data of the second row in the region of interest ROI. The demosaic processor 421 also performs boundary processing on image data of the last row in the region of interest ROI using image data of the row before the last row in the region of interest ROI. Therefore, the demosaic processor 421 performs processing equivalent to performing boundary processing on image data included in image data of payload data of the first row of pixels included in the packet area of the transmission data using image data included in payload data of the second row of pixels included in the packet area. The demosaic processor 421 also performs processing equivalent to performing boundary processing on image data included in payload data of the last row of pixels included in the packet area of the transmission data using image data included in payload data of the last row of pixels included in the packet area.
[0164] The demosaic processing unit 421 is configured to output the image data that has been subjected to the demosaic processing to the image quality adjustment unit 422.
[0165] The image quality adjustment unit 422 is configured to adjust the image quality of the image data input from the demosaic processing unit 421 by performing RGB processing for adjusting gamma correction and white balance, and YC processing for adjusting the shading and brightness of the image quality. The image quality adjustment unit 422 is configured to output the image with the adjusted image quality to, for example, a display device (not shown). As a result, a desired image is displayed on the display device.
[0166] (De-mosaic processing method) Next, a demosaic processing method in the transmitting device, receiving device, and transmission system according to this embodiment will be described using Figures 20 and 21 while also referring to Figure 19. First, a flow of demosaic processing in the transmitting device, receiving device, and transmission system according to this embodiment will be described using Figure 20. Figure 20 is a flowchart showing an example of the flow of demosaic processing in the transmitting device, receiving device, and transmission system according to this embodiment.
[0167] (Step S31) As shown in FIG. 20, when the sensor CPU 321 provided in the video transmission device 3 detects a frame start trigger, it first determines a cut-out position for cutting out an image from the imaging area of the imaging unit 31, and proceeds to processing in step S33. In step S31, the sensor CPU 321 determines the cut-out position, i.e., the coordinates of the upper left end of the region of interest ROI and the image size (lengths in the X-axis direction and the Y-axis direction), and sets the determined coordinates and image size information in the embedded data. In addition, the sensor CPU 321 acquires demosaic information in the region of interest ROI (color information of the upper left end, and information on odd and even numbers in the matrix of the upper left end), includes it in the ROI information, and sets the ROI information in the embedded data. The sensor CPU 321 may include information on the color arrangement and arrangement pattern of the entire imaging area shown in FIG. 14A to FIG. 17B as demosaic information in the ROI information, and set the ROI information in the embedded data.
[0168] (Step S33) The sensor CPU 321 sets, in the transmitting unit 322, transmission data including embedded data in which the coordinates of the upper left corner of the region of interest ROI, the image size, and demosaic information are set, and ends the demosaic process.
[0169] The transmission data set in step S33 is transmitted from the video transmitting device 3 to the video receiving device 4 by hardware (HW) communication using MIPI.
[0170] The receiving unit 412 provided in the video receiving device 4 extracts embedded data from the received transmission data and outputs it to the embedded data acquiring unit 414. The embedded data acquiring unit 414 decodes the embedded data input from the receiving unit 412, acquires various information (e.g., the number of regions of interest ROI, region numbers and priorities of the regions of interest ROI, data lengths of the regions of interest ROI, and image formats of the regions of interest ROI), and outputs the acquired various information to the Cam CPU 411.
[0171] (Step S41) The Cam CPU 411 acquires the coordinates (position and size) of the region of interest ROI having the highest priority based on various information acquired and input by the embedded data acquisition unit 414 from the transmission data received by the receiving unit 412, triggered by the timing when the embedded data acquisition unit 414 decodes the embedded data, and proceeds to processing in step S43. In addition, in step S41, the Cam CPU 411 determines the color of the pixel at the upper left corner of the region of interest ROI having the highest priority and what combination of odd and even numbers the upper left corner has.
[0172] (Step S43) The Cam CPU 411 calculates the coordinates of the upper left corner of the region of interest ROI based on the acquired demosaic information of the region of interest ROI, and proceeds to the process of step S45.
[0173] (Step S45) The Cam CPU 411 sets the coordinates of the upper left end of the region of interest ROI for demosaic processing based on the coordinates of the upper left end of the region of interest ROI calculated in step S43. Furthermore, the Cam CPU 411 outputs the coordinates of the upper left end of the region of interest ROI for demosaic processing that have been set and demosaic information (information on the coordinates and color array of the region of interest ROI) to the demosaic processing unit 421, and proceeds to processing in step S47.
[0174] The demosaic processing unit 421 performs demosaic processing on the image data input from the Raw processing unit 413, using the demosaic information input from the Cam CPU 411. As a result, demosaic processing is performed on the image data of the region of interest ROI, which has the highest priority.
[0175] (Step S47) The Cam CPU 411 determines whether or not the processes from step S41 to step S45 have been performed on all regions of interest ROI input from the embedded data acquisition unit 414. If the Cam CPU 411 determines that the processes have been performed on all regions of interest ROI (Yes), it ends the demosaic process. On the other hand, if the Cam CPU 411 determines that the processes have not been performed on all regions of interest ROI (No), it returns to the process of step S41. The Cam CPU 411 repeats the processes from step S41 to step S47 until the demosaic process is completed on all regions of interest ROI.
[0176] Next, the processing timing of the demosaic process in the transmitting device, the receiving device, and the transmission system according to the present embodiment will be described with reference to FIG. 21. FIG. 21 is a diagram showing an example of a timing chart of the demosaic process in the transmitting device, the receiving device, and the transmission system according to the present embodiment. "Sensor V Sync" shown in FIG. 21 indicates a vertical synchronization signal input to the sensor CPU 321. "Sensor processing" shown in FIG. 21 indicates processing executed by the sensor CPU 321. "ISP Sync" shown in FIG. 21 indicates a vertical synchronization signal input to the Cam CPU 411. "IPS processing" shown in FIG. 21 indicates processing executed by the Cam CPU 411. The attention areas ROI-ε1 to ROI-ε3 shown in FIG. 21 are schematic representations of attention areas processed in one frame period. In FIG. 21, for ease of understanding, the attention area processed in the first frame period is the attention area ROI-ε1, but attention areas ROI-ε2 and ROI-ε3 processed in the second and third frame periods are also shown in order to compare the sizes of the attention areas. In FIG. 21, the passage of time is represented from left to right.
[0177] 21, when the sensor CPU 321 detects a frame start trigger at time t1, it executes the above-mentioned process of step S31 as the embedded setting process. That is, as the embedded setting process, the sensor CPU 321 sets a cutout position for cutting out an image from the imaging region of the imaging unit 31 and acquires demosaic information (color information of the upper left end, and information on odd and even numbers in the matrix of the upper left end) in the region of interest ROI-ε1.
[0178] At time t2 when the embedded setting process is completed, the sensor CPU 321 transmits transmission data including embedded data having the information set in the embedded setting process to the video receiving device 4 by communication through hardware (HW) using MIPI.
[0179] When the sensor CPU 321 starts transmitting the transmission data, it starts exposure and readout for that frame, that is, starts capturing an image with the imaging unit 31 .
[0180] At time t3 when the embedded data acquisition unit 414 finishes decoding the embedded data included in the transmission data that started receiving at time t2, the Cam CPU 411 starts calculating the coordinates and size of the region of interest ROI-ε1 and determining the color arrangement based on various information acquired and input by the embedded data acquisition unit 414, and at time t4, the calculation of the coordinates and size of the region of interest ROI-ε1 and the setting of the color arrangement are completed. That is, during the period from time t3 to time t4, the processes of steps S41 to S45 shown in FIG. 20 are executed once.
[0181] The video receiving device 4 performs demosaic processing and image quality adjustment as ISP processing from time t4.
[0182] Although a detailed description is omitted, the demosaic process is also performed on the regions of interest ROI-ε2 and ROI-ε3, which are different in position and size, at the same timing as the region of interest ROI-ε1.
[0183] 20 and 21, the video transmission system 10 can transmit transmission data having embedded data including demosaic information related to the region of interest ROI from the video transmission device 3 to the video reception device 4 by communication using MIPI. This allows the video transmission system 10 to perform demosaic processing for each region of interest ROI.
[0184] 5. Modification of the first embodiment of the present disclosure: Next, a transmitting device, a receiving device, and a transmission system according to a modified example of this embodiment will be described with reference to Fig. 22. Fig. 22 is a block diagram showing a schematic configuration of a video transmitting device 3, a video receiving device 4z, and a video transmission system 10z according to this modified example. Note that components having the same actions and functions as those of the video transmitting device 3, the video receiving device 4, and the video transmission system 10 according to this embodiment will be given the same reference numerals and will not be described.
[0185] 22, a video transmission system 10z according to this modification includes a video transmitting device 3 having the same configuration as the video transmitting device 3 according to this embodiment, and a video receiving device 4z having a part of the configuration different from that of the video receiving device 4 according to this embodiment. The video receiving device 4z according to this modification is characterized in that it includes a determination unit 423 that determines the coordinates and color arrangement of the region of interest ROI.
[0186] As shown in Fig. 22, the video receiving device 4z has a determination unit 423 to which various information output from the embedded data acquisition unit 414 is input. On the other hand, the CamCPU 411z does not have a coordinate determination unit. The determination unit 423 is configured with hardware. The determination unit 423 has a coordinate determination unit 423a and a control value generation unit 423b.
[0187] The coordinate determination unit 423a is configured to determine the coordinates (position and size) and color arrangement of the region of interest ROI based on various information input from the embedded data acquisition unit 414. When information on multiple regions of interest ROI is input from the embedded data acquisition unit 414, the coordinate determination unit 411a determines the coordinates and color arrangement for each region of interest ROI.
[0188] The control value generating unit 423b is configured to set the coordinates of the upper left corner of the region of interest ROI for demosaic processing, based on the coordinates of the upper left corner of the region of interest ROI determined by the coordinate determining unit 411a.
[0189] In this way, in the video receiving device 4z according to this modification, the determination unit that determines the coordinates of the region of interest ROI is configured in hardware, but the operation and function are the same as those of the video receiving device 4 according to this embodiment. Also, the video transmission system 10z according to this modification is the same as the video transmission system 10 according to this embodiment. Therefore, a description of the demosaic processing method in this modification will be omitted.
[0190] As described above, the transmitting device, receiving device, and transmission system according to this embodiment and the modified example can realize demosaic processing of a part of a region of interest (ROI) cut out from a captured image.
[0191] In the present embodiment and the modified example, the position, size, and number of parts to be cut out from the captured image are arbitrary. Therefore, the transmitting device, the receiving device, and the transmission system according to the present embodiment and the modified example can perform demosaicing of the ROI by cutting out the ROI in units of one pixel.
[0192] The transmitting device, receiving device, and transmission system according to this embodiment and the modified example are configured so that the coordinates and size of a portion cut out from an imaged area captured in the transmitting device can be transmitted to a downstream sensor CPU.
[0193] The receiving device according to this embodiment and the modified example is configured to receive the coordinates and size of a portion cut out from the imaging area of an imaging unit provided in the transmitting device, and use them to control the demosaic processing.
[0194] The receiving device according to this embodiment and its modified examples is configured to receive demosaic information (color arrangement information such as color information of the upper left corner) of the region of interest ROI, which is the cut-out portion transmitted from the transmitting device, and use it to control the demosaic processing.
[0195] The receiving device according to this embodiment and its modified example can calculate the color of the pixel at the start position (e.g., the upper left corner) of the demosaic from the coordinates and size of the region of interest ROI, and control the color designation of the first pixel (e.g., the pixel at the upper left corner) of the demosaic.
[0196] In the case where there are multiple regions of interest ROI in one frame (i.e., in one captured image), the transmitting device, receiving device, and transmission system according to this embodiment and its modified example can specify the color of the first pixel (e.g., the pixel at the upper left corner) of each of the multiple regions of interest ROI. This makes it possible to perform appropriate demosaic processing on each region of interest ROI.
[0197] 6. Second embodiment of the present disclosure: Next, a transmitting device, a receiving device, and a transmission system according to a second embodiment of the present disclosure will be described with reference to Fig. 23 and Fig. 24. First, a schematic configuration of a transmitting device, a receiving device, and a transmission system according to this embodiment will be described with reference to Fig. 23. Fig. 23 is a block diagram showing a schematic configuration of a video transmitting device (an example of a transmitting device) 5, a video receiving device (an example of a receiving device) 6, and a video transmission system (an example of a transmission system) 20 according to this embodiment. Note that components having the same actions and functions as those of the video transmitting device 3, the video receiving device 4, and the video transmission system 10 according to the first embodiment are given the same reference numerals, and description thereof will be omitted.
[0198] The video transmitting device 5, video receiving device 6 and video transmission system 20 according to this embodiment are configured to transmit a control signal indicating the color arrangement constraint to the video transmitting device 5 when the video receiving device 6 has a constraint on the color arrangement of the demosaic processing.
[0199] 23, the CamCPU 611 included in the video receiving device 6 is configured to be able to send a control signal for notifying the video transmitting device 5 of a demosaic color arrangement constraint. Except for being able to send the control signal, the video receiving device 6 has the same configuration as the video receiving device 4 according to the first embodiment and is configured to perform the same functions, so a description thereof will be omitted.
[0200] 23, a control signal notifying the demosaic color arrangement constraint sent from the video receiving device 6 is input to a sensor CPU 521 provided in the video transmitting device 5. When the control signal is input, the sensor CPU 521 does not transmit demosaic information related to the constrained color arrangement to the video receiving device 6 if the notified color arrangement is included in the color arrangement of the region of interest ROI.
[0201] For example, suppose that there is a restriction in the video receiving device 6 that demosaic processing cannot be performed when the number of horizontal pixels (pixels in the X-axis direction) in the region of interest ROI is odd. In this case, for example, the sensor CPU 521 may impose an even number restriction on the coordinates and size of the pixel (e.g., the pixel at the upper left corner) that is the starting point of the horizontal and vertical pixels of the region of interest ROI. For example, if the region of interest ROI has 151 horizontal pixels, the sensor CPU 521 converts the value (75.5) obtained by dividing 151 pixels by 2 into an integer (75), and doubles the value obtained by converting it into an integer. As a result, the number of horizontal pixels in the region of interest ROI becomes an even number (150 pixels in this example), so the video transmitting device 5 sends this information to the video receiving device 6 as demosaic information, and the video receiving device 6 can perform demosaic processing of the region of interest ROI.
[0202] The video transmitting device 5 has the same configuration and is configured to perform the same functions as the video transmitting device 3 of the first embodiment described above, except that it is capable of receiving a control signal notifying of the demosaic color arrangement constraints sent from the video receiving device 6 and executing the above-mentioned processing based on the control signal, and therefore a description thereof will be omitted.
[0203] (De-mosaic processing method) Next, a demosaic processing method in the transmitting device, receiving device, and transmission system according to this embodiment will be described using Fig. 24 while referring to Fig. 23. Fig. 24 is a flowchart showing an example of the flow of the demosaic processing in the transmitting device, receiving device, and transmission system according to this embodiment.
[0204] When the video transmission system 20 according to this embodiment is powered on and the video transmitting device 5 and the video receiving device 6 start operating, the video transmitting device 5 starts a sensor initialization process, and the video receiving device 6 starts an IPS initialization process.
[0205] (Step S60) As shown in Figure 24, when the CamCPU 611 provided in the video receiving device 6 starts the IPS initialization process, it sends an IPS constraint notification signal indicating that there are constraints on the color arrangement of the demosaic processing to the video transmitting device 5 via hardware (HW) communication using, for example, MIPI, and terminates the IPS initialization process.
[0206] (Step S50) 24, the sensor CPU 521 provided in the video transmission device 5 starts a sensor initialization process. If the sensor CPU 321 determines in step S50 that it has received an IPS restriction notification signal, it receives information about the restriction included in the IPS restriction notification signal and stores it in a storage area, and ends the sensor initialization process. On the other hand, if the sensor CPU 321 determines in step S50 that it has not received an IPS restriction notification signal, it ends the sensor initialization process without performing any special process.
[0207] (Step S51) As shown in Fig. 24, when the sensor CPU 521 detects a frame start trigger, it first executes a transmission coordinate control process. If the sensor CPU 521 stores information about the constraints on color arrangement of the demosaic process, it performs a predetermined process on the coordinates of the upper left corner of the region of interest ROI, the image size, or the color arrangement so that the demosaic process sent to the video receiving device 6 does not violate the constraints, and proceeds to the process of step S53. On the other hand, if the sensor CPU 521 does not store information about the constraints on color arrangement of the demosaic process, it executes the same process as step S31 in the first embodiment, and proceeds to the process of step S53.
[0208] (Step S53) The sensor Cam CPU 521 executes the embedded data transmission process and ends the demosaic process. The process in step S53 is the same as the process in step S33 in the first embodiment, and therefore the description thereof will be omitted.
[0209] (Steps S61 to S67) The CamCPU 411 starts the process of step S61 when the embedded data is decoded by the embedded data acquisition unit 414. The process of step S61 is similar to the process of step S41 in the first embodiment, the process of step S63 is similar to the process of step S43 in the first embodiment, the process of step S65 is similar to the process of step S45 in the first embodiment, and the process of step S67 is similar to the process of step S47 in the first embodiment. Therefore, a description of steps S61 to S67 will be omitted.
[0210] The video receiving device 6 has a certain restriction on the color arrangement of the demosaic processing. However, the sensor CPU 521 sets the demosaic information so as not to violate the restriction. Therefore, the Cam CPU 611 can perform the demosaic processing without judging whether the demosaic information violates the restriction or not, and the processing load of the demosaic processing is suppressed.
[0211] As described above, the transmission processing, reception processing, and transmission system according to this embodiment have certain constraints compared to the first embodiment described above, but can realize demosaic processing of a portion of a region of interest (ROI) extracted from a captured image.
[0212] The present disclosure is not limited to the above-described embodiment, and various modifications are possible. The video transmission devices 3 and 5 according to the first and second embodiments set the smallest rectangle including the object to be cut out as the region of interest ROI even if the object is not rectangular. Furthermore, the video transmission devices 3 and 5 are configured to include position information of the region of interest ROI (upper left end, length in the X-axis direction, length in the Y-axis direction) and demosaic information of the region of interest ROI (color information of the upper left end, etc.) in the embedded data, but the present disclosure is not limited to this.
[0213] For example, the video transmitting devices 3 and 5 may be configured to include position information and demosaic information of a target object in payload data for each pixel row, as in the base technology 2, and transmit the payload data to the video receiving devices 4 and 6. In this case, since the target object is not rectangular, there are cases where image data does not exist around the target pixel for demosaic processing, but demosaic processing can be performed even in such cases by interpolating image data using a process similar to boundary processing, for example.
[0214] The conversion area control unit 321b, the sensor CPU 321, or the control unit 32 provided in the video transmission device 3, 5 according to the first and second embodiments may be configured to control the area of the image data of the region of interest ROI so that the demosaic information satisfies a certain condition. That is, the conversion area control unit 321b, the sensor CPU 321, or the control unit 32 may control the image data of the region of interest ROI so that the demosaic information satisfies any one of the arrangement patterns PA1-PA4, PB1-PB4, PC1-PC4, and PD1-PD4 (an example of a certain condition) according to, for example, arrangement examples A to D of the color arrangement of the imaging element. In this way, the transmission device may control the region of interest so that it becomes a certain pattern for demosaic.
[0215] The present disclosure has been described above by citing the base technology, the embodiment, and the modified examples thereof, but the present disclosure is not limited to the above embodiment, etc., and various modifications are possible. Note that the effects described in this specification are merely examples. The effects of the present disclosure are not limited to the effects described in this specification. The present disclosure may have effects other than the effects described in this specification.
[0216] Furthermore, for example, the present disclosure can have the following configuration. (1) A control unit that controls acquisition of demosaic information that is information used for demosaic processing of image data of a region of interest (ROI); a transmitter that transmits the image data of the ROI as payload data and transmits ROI information as embedded data; A transmitting device comprising: (2) The demosaic information is included in the ROI information and is transmitted from the transmission unit. A transmitting device as described in (1) above. (3) The control unit acquires a color arrangement of the image data of the ROI or color information of an end portion of the image data of the ROI as the demosaic information. A transmitting device as described in (1) above. (4) The control unit controls the region of the image data of the ROI so that the demosaic information satisfies a certain condition. A transmitting device as described in (1) above. (5) The control unit acquires, as the demosaic information, color information of the end portion and information indicating which combination of odd-numbered rows and even-numbered rows and odd-numbered columns the end portion is made of. The transmitting device according to (4) above. (6) The transmission unit transmits a signal in accordance with the MIPI (Mobile Industry Processor Interface) D-PHY standard, the MIPI C-PHY standard, or the MIPI CSI (Camera Serial Interface)-2 standard. A transmitting device as described in (1) above. (7) A receiver that receives a transmission signal including image data of a region of interest (ROI) in payload data and including ROI information in embedded data; a control unit that controls extraction of demosaic information, which is information used for demosaic processing of image data of the ROI, from the transmission signal received by the receiving unit; a processing unit that executes demosaic processing of image data of the ROI using the demosaic information extracted by the control unit; A receiving device comprising: (8) The control unit extracts the demosaic information from the ROI information included in the transmission signal. A receiving device as described in (7) above. (9) The control unit extracts a color arrangement of the image data of the ROI or color information of an end portion of the image data of the ROI as the demosaic information. A receiving device as described in (7) above. (10) The control unit extracts, as the demosaic information, color information of the end portion of the ROI and information indicating which combination of odd-numbered rows and even-numbered rows and odd-numbered columns and even-numbered columns the end portion is. A receiving device as described in (9) above. (11) The processing unit performs boundary processing on image data arranged at the peripheral edge of the ROI. A receiving device as described in (7) above. (12) The receiving unit receives a signal according to the MIPI (Mobile Industry Processor Interface) D-PHY standard, the MIPI C-PHY standard, or the MIPI CSI (Camera Serial Interface)-2 standard. A receiving device as described in (7) above. (13) A transmitting device having a control unit that acquires demosaic information used for demosaic processing of image data of a region of interest (ROI), and a transmitting unit that transmits the image data as payload data and transmits ROI information as embedded data; a receiving device including a receiving unit that receives a transmission signal including image data of the ROI in the payload data and including the ROI information in the embedded data, a control unit that controls extraction of demosaic information used for demosaic processing of the image data of the ROI from the transmission signal received by the receiving unit, and a processing unit that executes demosaic processing of the image data of the ROI using the demosaic information extracted by the control unit; A transmission system comprising: (14) The transmission device transmits the demosaic information included in the ROI information from the transmission unit, The receiving device receives the transmission signal having the demosaic information at the receiving unit, and extracts the demosaic information from information of the ROI included in the transmission signal received at the receiving unit at the control unit. A transmission system as described in (13) above. (15) The transmission device transmits from the transmission unit a color arrangement of the image data of the ROI or color information of an end of the image data of the ROI as the demosaic information, The receiving device receives color information of the color array or the end of the image data of the ROI as the demosaic information at the receiving unit. A transmission system as described in (13) above. (16) The transmission device transmits, from the transmission unit as the demosaic information, color information of the end portion of the ROI and information indicating which combination of odd-numbered rows and even-numbered rows and odd-numbered columns and even-numbered columns the end portion is made of; The receiving device receives, as the demosaic information, the color information and information indicating which combination of odd-numbered rows and even-numbered rows and odd-numbered columns and even-numbered columns the end portion is. A transmission system as described in (15) above. (17) The transmission device transmits a signal conforming to the MIPI (Mobile Industry Processor Interface) D-PHY standard, the MIPI C-PHY standard, or the MIPI CSI (Camera Serial Interface)-2 standard from the transmission unit; The receiving device receives a signal according to the MIPI D-PHY standard, the MIPI C-PHY standard, or the MIPI CSI-2 standard at the receiving unit. A transmission system as described in (13) above.
[0217] Those skilled in the art will recognize that various modifications, combinations, subcombinations, and variations may occur to those skilled in the art depending on design requirements and other factors, and that such modifications, combinations, subcombinations, and variations are within the scope of the appended claims and their equivalents. [Explanation of symbols]
[0218] 1,10,10z,20 Video Transmission System 3,5,100 Video transmission device 4,4z,6,200 Video receiving device 31,110 Imaging section 32,41 Control section 42 Image Processing Department 100A CSI Transmitter 100B CCI Slave 111 Captured images 112,112a1,112a2,112a3,112a4,112b1,112b4,123a4,223A ROI image 112b Compressed image data 113,114 Location information 115 Priority 116,116a1,116a2 Transmission image 118 images 120,130 Image processing unit 120A, 120A1, 120A2, 130A, 147B Compressed image data 120B ROI information 120C Frame Information 121 ROI cutout 122 ROI Analysis Department 123 Detection unit 124 Priority setting section 125,131 Encoding section 126 Image processing control section 140 Transmitter 141 LINK control section 142 ECC generator 143 PH generation section 144,145 ROI data buffer 144 EBD Buffer 146 Normal image data buffer 147 Synthesis Section 147A Transmission Data 200A CSI Receiver 200B CCI Master 210 Receiving unit 211 Header Separator 212 Header Interpretation 213 Payload Separation Section 214 EBD Interpretation Division 214A EBD Data 215 ROI Data Separation Unit 215A, 215B payload data 220 Information Processing Department 221 Information extraction section 221A Extract information 222 ROI Decoder 222A Image data 223 ROI Image Generation Unit 224 Normal image decoding section 224A Normal Image 311 Photoelectric conversion unit 312 Signal conversion unit 313 Amplification Unit 321,521 Sensor CPU 321a Exposure control unit 321b Transformation domain control section 322 Transmitter 411,411z,611 CamCPU 411a Coordinate determination section 412 Receiving unit 413 Raw Processing Unit 414 Embedded Data Acquisition Unit 421 Demosaic Processing Unit 422 Image quality adjustment section 423 Judgment section 423a Coordinate determination unit 423b Control value generator 521 Sensor CPU A,B,C arrangement example ADC Analog to Digital Conversion AGC signal amplification Cb color difference component CCI Camera Control Interface CL Clock Lane
Claims
1. A control unit that controls acquisition of demosaic information that is information used for demosaic processing of image data of a region of interest (ROI); a transmitter that transmits the image data of the ROI as payload data and transmits ROI information as embedded data; Equipped with The control unit acquires, as the demosaic information, color information of an end portion of the image data of the ROI and information indicating which combination of odd-numbered rows and even-numbered rows and odd-numbered columns and even-numbered columns the end portion is. Transmitting device.
2. The demosaic information is included in the ROI information and is transmitted from the transmission unit. The transmitting device according to claim 1 .
3. The control unit acquires a color arrangement of the image data of the ROI as the demosaic information. The transmitting device according to claim 1 .
4. The control unit controls a region of the image data of the ROI so that the demosaic information satisfies a certain condition. The transmitting device according to claim 1 .
5. The transmitting unit transmits a signal according to the MIPI (Mobile Industry Processor Interface) D-PHY standard, the MIPI C-PHY standard, or the MIPI CSI (Camera Serial Interface)-2 standard. The transmitting device according to claim 1 .
6. A receiver that receives a transmission signal including payload data of an image of a region of interest (ROI) and embedded data of the ROI information; a control unit that controls extraction of demosaic information, which is information used for demosaic processing of image data of the ROI, from the transmission signal received by the receiving unit; a processing unit that executes demosaic processing of image data of the ROI using the demosaic information extracted by the control unit; Equipped with The control unit extracts, as the demosaic information, color information of an end portion of the image data of the ROI and information indicating which combination of odd-numbered rows and even-numbered rows and odd-numbered columns and even-numbered columns the end portion is. Receiving device.
7. The control unit extracts the demosaic information from the ROI information included in the transmission signal.
7. The receiving device according to claim 6.
8. The control unit extracts a color arrangement of the image data of the ROI as the demosaic information.
7. The receiving device according to claim 6.
9. The processing unit performs boundary processing on image data arranged at the peripheral edge of the ROI.
7. The receiving device according to claim 6.
10. The receiving unit receives a signal according to the MIPI (Mobile Industry Processor Interface) D-PHY standard, the MIPI C-PHY standard, or the MIPI CSI (Camera Serial Interface)-2 standard.
7. The receiving device according to claim 6.
11. A transmitting device having a control unit that acquires demosaic information used for demosaic processing of image data of a region of interest (ROI), and a transmitting unit that transmits the image data as payload data and transmits ROI information as embedded data; a receiving device including a receiving unit that receives a transmission signal including image data of the ROI in the payload data and including the ROI information in the embedded data, a control unit that controls extraction of demosaic information used for demosaic processing of image data of the ROI from the transmission signal received by the receiving unit, and a processing unit that performs demosaic processing of the image data of the ROI using the demosaic information extracted by the control unit; Equipped with The transmission device transmits, from the transmission unit, color information of an end portion of the image data of the ROI and information indicating which combination of odd-numbered rows and even-numbered rows and odd-numbered columns the end portion is, as the demosaic information; The receiving device receives, as the demosaic information, the color information and information indicating which combination of odd-numbered rows and even-numbered rows and odd-numbered columns and even-numbered columns the end portion is. Transmission system.
12. The transmitting device transmits the demosaic information included in the ROI information from the transmitting unit, The receiving device receives the transmission signal having the demosaic information at the receiving unit, and extracts the demosaic information from information of the ROI included in the transmission signal received at the receiving unit at the control unit. The transmission system according to claim 11.
13. The transmitting device transmits a color arrangement of the image data of the ROI from the transmitting unit as the demosaic information, The receiving device receives the color array as the demosaic information at the receiving unit. The transmission system according to claim 11.
14. the transmitting device transmits a signal conforming to the MIPI (Mobile Industry Processor Interface) D-PHY standard, the MIPI C-PHY standard, or the MIPI CSI (Camera Serial Interface)-2 standard from the transmitting unit; The receiving device receives a signal conforming to the MIPI D-PHY standard, the MIPI C-PHY standard, or the MIPI CSI-2 standard at the receiving unit. The transmission system according to claim 11.
Citation Information
Patent Citations
Image data control device
JP2012209831A
Image processing device, method thereof, and program
JP2013164834A
Video decoder, video transmission reception system, video decoding method and video transmission reception method
JP2014039219A
Image processing device, image processing system, imaging device and image processing method
JP2015231103A
Image processing apparatus, image processing method, and image processing system
JP2016201756A
