Transmitting device, receiving device, and communication system

Abstract

A sending device according to an embodiment of the present disclosure is provided with a cut-out unit, a derivation unit, a first processing unit, and a second processing unit. The cut-out unit cuts out, from image data obtained by imaging, one or more pieces of ROI image data included in the image data. The derivation unit derives ROI position information in the image data. The first processing unit generates first transmission data corresponding to a first transmission scheme on the basis of the one or more pieces of ROI image data, and one or more pieces of ROI position information in the image data. The second processing unit generates second transmission data corresponding to a second transmission scheme on the basis of the one or more pieces of ROI image data, and the one or more pieces of ROI position information in the image data.

Description

[Technical field]

[0001] The present disclosure relates to a transmitting device, a receiving device, and a communication system. [Background technology]

[0002] In recent years, there has been an increase in applications that require the transmission of large amounts of data. This puts a heavy load on the transmission system, and in the worst case scenario, the transmission system may crash, making it impossible to transmit data.

[0003] Conventionally, in order to avoid a breakdown in the transmission system, for example, instead of transmitting all of the captured images, the object of the image is identified and only a portion of the image cut out from the identified object is transmitted. Note that cutting out a portion of an image from a captured image is described, for example, in the following patent document: [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2016-201756 A [Patent Document 2] JP 2014-39219 A [Patent Document 3] JP 2013-164834 A [Patent Document 4] JP 2012-209831 A Summary of the Invention

[0005] Incidentally, as a method for transmission from an image sensor to an application processor, MIPI (Mobile Industry Processor Interface) CSI (Camera Serial Interface)-2, MIPI CSI-3, etc. may be used. Also, as a method for transmission from an application processor to a display, SLVS-EC (Scalable Low Voltage Signaling Embedded Clock) etc. may be used. It is desirable to provide a transmitting device, a receiving device, and a communication system that are compatible with a plurality of transmission methods.

[0006] A transmission device according to an embodiment of the present disclosure includes a cutout unit, a derivation unit, a first processing unit, and a second processing unit. The cutout unit cuts out one or more ROI image data included in image data from image data obtained by imaging. The derivation unit derives ROI position information in the image data. The first processing unit generates first transmission data corresponding to a first transmission method based on the one or more ROI image data and the one or more ROI position information in the image data. The second processing unit generates second transmission data corresponding to a second transmission method based on the one or more ROI image data and the one or more ROI position information in the image data.

[0007] In a transmitting device according to one embodiment of the present disclosure, a processing block that extracts one or more ROI image data contained in image data obtained by imaging and derives ROI position information in the image data is shared between the first transmission method and the second transmission method.

[0008] A receiving device according to an embodiment of the present disclosure includes a first processing unit, a second processing unit, and a generating unit. The first processing unit extracts one or more image data and one or more pieces of location information from first transmission data corresponding to a first transmission method. The second processing unit extracts one or more image data and one or more pieces of location information from second transmission data corresponding to a second transmission method. The generating unit generates one or more ROI image data included in captured image data obtained by imaging based on the one or more image data and the one or more pieces of location information extracted by the first processing unit or the second processing unit.

[0009] In the receiving device according to an embodiment of the present disclosure, a processing block that performs processing to generate one or more ROI image data is shared between the first transmission method and the second transmission method.

[0010] A communication system according to an embodiment of the present disclosure includes a transmitting device and a receiving device. The transmitting device has a common configuration with the above-described transmitting device. The receiving device has a common configuration with the above-described receiving device.

[0011] In the communication system according to an embodiment of the present disclosure, a processing block for extracting one or more ROI image data included in image data obtained by imaging and deriving ROI position information in the image data is shared between the first and second transmission methods. Furthermore, a processing block for generating one or more ROI image data is shared between the first and second transmission methods. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 is a diagram illustrating an example of a schematic configuration of a communication system. [Diagram 2] FIG. 2 is a diagram illustrating an example of a schematic configuration of an image sensor. [Diagram 3] FIG. 2 is a diagram illustrating an example of a schematic configuration of a processor. [Figure 4] FIG. 2 is a diagram illustrating an example of a packet configuration used in MIPI. [Diagram 5] FIG. 2 is a diagram illustrating an example of the configuration of transmission data used in MIPI. [Figure 6] FIG. 2 is a diagram illustrating an example of the configuration of transmission data used in MIPI. [Figure 7] 1 is a diagram illustrating an example of the configuration of a packet used in SLVS-EC. [Figure 8] 1 is a diagram illustrating an example of the configuration of transmission data used in SLVS-EC. [Figure 9] 1 is a diagram illustrating an example of a format of transmission data used in SLVS-EC. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. The following description is a specific example of the present disclosure, and the present disclosure is not limited to the following embodiment.

[0014] <1. Preferred embodiment> [composition] In recent years, mobile devices such as smartphones and camera devices have been handling increasingly large volumes of image data, and there is a demand for faster data transmission within devices or between different devices with lower power consumption. To meet these demands, the standardization of high-speed interface standards such as the C-PHY standard and the D-PHY standard, established by the MIPI Alliance, is underway as connection interfaces for mobile devices and camera devices. The C-PHY standard and the D-PHY standard are interface standards for the physical layer (PHY) of communication protocols. In addition, CSI for camera devices exists as an upper protocol layer of the C-PHY standard and the D-PHY standard. In addition, a proprietary standard, SLVS-EC, exists as CSI for camera devices.

[0015] A communication system 1000 according to an embodiment of the present disclosure is a system that transmits and receives signals according to either the MIPI CSI-2 standard or the MIPI CSI-3 standard, and the SLVS-EC standard. The communication system 1000 can be applied to various electronic devices, such as communication devices such as smartphones, drones (devices capable of remote control or autonomous operation), moving objects such as automobiles, computers such as PCs (Personal Computers), tablet devices, and game consoles.

[0016] 1 shows a schematic configuration example of a communication system 1000 according to the present embodiment. The communication system 1000 is applied to the transmission of a data signal, a clock signal, and a control signal, and includes an image sensor 100 (transmitting device) and a processor 200 (receiving device).

[0017] The image sensor 100 and the processor 200 are electrically connected by a data bus B1. The data bus B1 is a signal transmission path that connects the image sensor 100 and the processor 200. Data indicating an image transmitted from the image sensor 100 (hereinafter referred to as "image data") is transmitted from the image sensor 100 to the processor 200 via the data bus B1. The image sensor 100 and the processor 200 may be electrically connected by a bus (control bus B2). The control bus B2 is another signal transmission path that connects the image sensor 100 and the processor 200, and is a transmission path different from the data bus B1. The control data transmitted from the processor 200 is transmitted from the processor 200 to the image sensor 100 via the control bus B2.

[0018] The image sensor 100 has an imaging function and a transmission function, and transmits image data generated by imaging. The image sensor 100 serves as a transmission device in the communication system 1000. The image sensor 100 includes any type of image sensor device capable of generating an image, such as an "imaging device such as a digital still camera, a digital video camera, or a stereo camera," an "infrared sensor," or a "distance image sensor," and has a function of transmitting the generated image. The image generated by the image sensor 100 corresponds to data indicating the sensing result of the image sensor 100. An example of the configuration of the image sensor 100 will be described in detail later with reference to FIG. 2.

[0019] The image sensor 100 transmits data corresponding to a region set in the image data (hereinafter also referred to as "image data of the region") by a transmission method described below. The control related to the transmission of the image data of the region is performed, for example, by a component (described below) that functions as an image processing unit in the image sensor 100. The region set in the image is called ROI (Region Of Interest). Hereinafter, the region set in the image will be referred to as "ROI". Also, the image data of the region will be referred to as "image data of the ROI".

[0020] Examples of processes related to setting an area in an image include any process capable of identifying a portion of an area in an image (or any process capable of cutting out a portion of an area from an image), such as "a process of detecting an object from an image and setting an area that includes the detected object" or "a process of setting an area specified by operating an arbitrary operating device."

[0021] The image sensor 100 transmits image data of the ROI, i.e., transmits a portion of the image data, thereby reducing the amount of data involved in transmission compared to transmitting the entire image data. Therefore, by transmitting image data of the ROI, the image sensor 100 achieves various effects due to the reduction in the amount of data, such as shortening the transmission time and reducing the load related to transmission in the communication system 1000. Note that the image sensor 100 can also transmit the entire image data.

[0022] The processor 200 receives data transmitted from the image sensor 100 and processes the received data. The processor 200 serves as a receiving device in the communication system 1000. An example of a configuration related to processing of data transmitted from the image sensor 100 (a configuration for serving as a receiving device) will be described in detail later with reference to FIG. 3.

[0023] The processor 200 is composed of one or more processors, each of which is composed of an arithmetic circuit such as an MPU (Micro Processing Unit), and various processing circuits. The processor 200 performs various processes, such as a process related to recording control of image data on a recording medium, a process related to display control of an image on a display screen of a display device, and a process for executing any application software. An example of the process related to recording control is a process of transmitting control data including a recording command and data to be recorded on a recording medium to a recording medium. An example of the process related to display control is a process of transmitting control data including a display command and data to be displayed on a display screen to a display device. The processor 200 may control functions in the image sensor 100, for example, by transmitting control information to the image sensor 100. The processor 200 can also control data transmitted from the image sensor 100, for example, by transmitting area designation information to the image sensor 100.

[0024] (Packet structure) Next, an example of a packet structure used for transmitting an image from the image sensor 100 to the processor 200 in the communication system 1000 will be described. In the communication system 1000, image data captured by the image sensor 100 is divided into partial image data in units of rows, and the partial image data for each row is transmitted using one or more packets. The same applies to image data of the ROI.

[0025] Fig. 4 shows an example of a packet structure used for transmitting image data in the communication system 1000. Fig. 4 shows an example of a packet structure used when transmitting image data in accordance with the MIPI CSI-2 standard or the MIPI CSI-3 standard. Figs. 5 and 6 show an example of transmission data 137A transmitted from the image sensor 100 to the processor 200 in the communication system 1000. Figs. 5 and 6 show an example of transmission data 137A used when transmitting image data in accordance with the MIPI CSI-2 standard or the MIPI CSI-3 standard.

[0026] As shown in Fig. 4, a packet used to transmit image data is defined as a series of data in a data stream that starts in low power mode LP and ends in low power mode LP. The packet includes a packet header PH, payload data, and a packet footer PF, arranged in this order. The payload data (hereinafter also simply referred to as "payload") includes pixel data of a partial image in units of rows.

[0027] The packet header PH is, for example, a packet header of the PayloadData of a LongPacket. The LongPacket refers to a packet arranged between the packet head PH and the packet footer PF. The PayloadData of a LongPacket refers to the main data transmitted between devices. The packet header PH includes, for example, DI, WC, and ECC (Error-Correcting Code). The DI is an area for storing a data identifier. The DI includes a VC (Virtual Channel) number and DataType (data type of each ROI). The VC is a concept introduced for packet flow control, and is a mechanism for supporting multiple independent data streams sharing the same link. The WC is an area for indicating the end of the packet to the processor 200 in the number of words. The WC includes, for example, the Payload length. The Payload length is, for example, the number of bytes included in the Payload of a LongPacket, for example, the number of bytes for each ROI. The ECC is PayloadDataECC information including a value for performing error detection or correction on the PayloadData. The ECC includes an error correction code.

[0028] The transmission data 137A is composed of an image data frame as shown in, for example, FIG. 5 and FIG. 6. The image data frame usually has a header area, a packet area, and a footer area. In FIG. 5 and FIG. 6, the footer area is omitted for convenience. In the image data frame, the header area R1 includes header information including EmbeddedData and header ECC information for performing error detection or correction on the header information. EmbeddedData refers to additional information that can be embedded in the header or footer of the image data frame. At this time, EmbeddedData includes a frame number, the number of ROIs, and ROI information. The header ECC information includes a value for performing error detection or correction on the header information. The header ECC information includes an error correction code.

[0029] The frame number is an identifier of the transmission data 137 A. The number of ROIs is the total number of ROIs included in the transmission data 137 A. The ROI information is information about the ROI provided for each ROI included in the transmission data 137 A.

[0030] The ROI information includes, for example, the region number (or priority) of one or more ROIs included in the image data, and the position information of one or more ROIs in the image data. The region number of the ROI is an identifier given to each ROI. The priority of the ROI is an identifier given to each ROI, and is discrimination information that can discriminate which of the multiple ROIs in the image data has had its overlapping region omitted.

[0031] The position information of the ROI includes, for example, the upper left end coordinate (Xa, Ya) of the ROI, the length of the ROI in the X-axis direction, and the length of the ROI in the Y-axis direction. The length of the ROI in the X-axis direction is, for example, the physical area length XLa of the ROI in the X-axis direction. The length of the ROI in the Y-axis direction is, for example, the physical area length YLa of the ROI in the Y-axis direction. The physical area length refers to the physical length (data length) of the ROI. The position information of the ROI may include the coordinates of a position different from the upper left end of the ROI. The position information of the ROI further includes, for example, the output area length XLc of the ROI in the X-axis direction and the output area length YLc of the ROI in the Y-axis direction. The output area length is, for example, the physical length (data length) of the ROI after the resolution of the ROI is changed by thinning processing, pixel addition, or the like.

[0032] The ROI information may further include, for each ROI, sensing information, exposure information, gain information, AD (Analog-Digital) word length, image format, and the like, in addition to the position information. The sensing information refers to the calculation contents for the object included in the ROI, supplementary information for subsequent signal processing of the image data of the ROI, and the like. The exposure information refers to the exposure time of the ROI. The gain information refers to the gain information of the ROI. The AD word length refers to the word length of data per pixel that has been AD converted in the ROI. The image format refers to the format of the image of the ROI.

[0033] 5 and 6, in the data frame, the packet area R2 contains the PayloadData of a LongPacket for each line, and further contains a packet header PH and a packet footer PF between which the PayloadData of the LongPacket is sandwiched. Furthermore, a low power mode LP is contained between which the packet header PH and the packet footer PF are sandwiched.

[0034] Packet area R2 also includes compressed image data 137B. Compressed image data 137B is composed of one compressed image data or multiple compressed image data. In FIG. 5, a packet group closer to packet header PH includes compressed image data 135C (135C1) of a certain ROI, and a packet group farther from packet header PH includes compressed image data 135C (135C2) of another ROI. These two compressed image data 135C1 and 135C2 constitute compressed image data 137B. PayloadData of LongPacket for each line includes pixel data for one line in compressed image data 137B.

[0035] Fig. 7 shows an example of a packet structure used for transmitting image data in the communication system 1000. Fig. 7 shows an example of a packet structure used when transmitting image data in accordance with the SLVS-EC standard. Fig. 8 shows an example of transmission data 147A transmitted from the image sensor 100 to the processor 200 in the communication system 1000.

[0036] As shown in FIG. 7, a packet used to transmit image data is defined as a series of data in a data stream that starts with a start code and ends with an end code. The packet also contains a header and payload data arranged in that order. A footer may also be added after the payload data. The payload data contains pixel data of a partial image in units of lines. The header contains various information related to the lines corresponding to the partial image contained in the payload. The footer contains additional information (optional).

[0037] The footer of the packet may include a CRC option or a payload data ECC option. The packet may include, for example, any of the following elements (1), (2), or (3). (1) Packet header + payload data (2) Packet header + payload data + packet footer (3) Packet header + payload data with ECC

[0038] The packet footer optionally has payload data ECC information for performing error detection on the payload data. Packets are assembled at the TX LINK layer and disassembled at the RX LINK layer to extract the payload data and other auxiliary information. The following describes the options in the packet footer (footer options). In high-speed serial data transmission using the built-in clock, random data errors may occur in some of the pixel data transferred as payload data due to the bit error characteristics of the PHY layer. Therefore, it is necessary to consider the possibility of pixel data being destroyed. The payload data error detection and correction function detects such damage to pixel data, corrects the destroyed parts, and improves the effective bit error performance of the entire interface.

[0039] The packet footer further includes payload data ECC information for optional error correction on the payload data. The performance and cost of the error correction circuitry can be optimized by this function to compensate for the difference between the system level requirements of error tolerance and the bit error characteristics at the PHY layer. This function is also optional and can be set by a configuration register (ECC option).

[0040] Here, the information contained in the header will be described. As shown in Fig. 7, the header contains "Frame Start", "Frame End", "Line Valid", "Line Number", "EBD Line", "Data ID", "Reserved", and "Header ECC" in this order.

[0041] Frame Start is 1-bit information indicating the beginning of a frame. For example, the Frame Start of the header of a packet used to transmit pixel data of the first row of image data to be transmitted is set to a value of 1, and the Frame Start of the header of a packet used to transmit pixel data of other rows is set to a value of 0. Frame Start is an example of "information indicating the start of a frame."

[0042] Frame End is 1-bit information indicating the end of a frame. For example, the Frame End of the header of a packet whose payload contains pixel data of the last line of an effective pixel area among the data of an image to be transmitted is set to a value of 1, and the Frame End of the header of a packet used to transmit pixel data of other lines is set to a value of 0. Frame End is an example of "information indicating the end of a frame."

[0043] The Frame Start and Frame End correspond to an example of frame information, which is information relating to a frame.

[0044] Line Valid is 1-bit information indicating whether the line of pixel data stored in the payload is a line of valid pixels. A value of 1 is set to Line Valid in the header of a packet used to transmit pixel data of a row within the valid pixel area, and a value of 0 is set to Line Valid in the header of a packet used to transmit pixel data of other lines. Line Valid is an example of "information indicating whether the corresponding row is valid or not."

[0045] The Line Number is 13-bit information that indicates the line number of a line composed of pixel data stored in the payload.

[0046] The EBD Line is 1-bit information indicating whether or not a row has embedded data. That is, the EBD Line corresponds to an example of "information indicating whether or not a row has embedded data".

[0047] The Data ID is 4-bit information for identifying each piece of data (i.e., data included in the payload) when transferring data divided into multiple streams. The Data ID corresponds to an example of "identification information of data included in the payload."

[0048] Line Valid, Line Number, EBD Line, and Data ID constitute Line Information, which is information about a line.

[0049] Reserved is a 27-bit area for extension. In the following, the area indicated as Reserved will also be referred to as the "extension area." The total data size of the header information is 6 bytes.

[0050] As shown in Fig. 7, the Header ECC placed following the header information includes a CRC (Cyclic Redundancy Check) code, which is a 2-byte error detection code calculated based on the 6-byte header information. That is, the Header ECC corresponds to an example of "information for detecting or correcting errors in the header information". In addition, the Header ECC includes two pieces of information following the CRC code, which are the same as the 8-byte information that is a combination of the header information and the CRC code.

[0051] The header is placed at the beginning of a packet and is used to store additional information other than pixel data that is transferred before the payload data. The header consists of header information and header ECC, and the additional information is stored in the header information. The header ECC stores a CRC to detect errors in the header information, and two repetitions of the combination of the header information and the CRC. If an error occurs during the transfer of the header information, the error is detected by the CRC, and the correct header information can be reproduced on the RX side using other header information (where no error was detected) that was transferred repeatedly.

[0052] Frame information is mainly transferred to establish frame synchronization within the system. Similarly, line information is used to establish line synchronization, and this information is assumed to be used to reproduce the synchronization relationship between each image stream on the RX side when multiple image streams are transferred simultaneously using multiple SLVS-EC interfaces. As mentioned above, header ECC is used as a measure against header information transfer errors. Also, frame information and line information are input from the application layer (CIS) and transferred directly to the receiving application layer (DSP) without being processed within the interface. The same transfer process is performed for reserved bits. In contrast, header ECC is generated within the interface, added to other information, and transferred, and this information is assumed to be used by the RX side link layer.

[0053] In other words, one packet header contains three sets of the same header information and CRC code. The total data size of the header is 24 bytes, consisting of the first 8-byte set of header information and CRC code, the second 8-byte set of header information and CRC code, and the third 8-byte set of header information and CRC code.

[0054] Here, the extension area (Reserved) provided in the header of a packet will be described with reference to Fig. 8. As shown in Fig. 8, in the extension area, information indicating a type according to information transmitted in a packet is set as a header information type (Header Info Type) in the first three bits. In accordance with the type of header information, the format of information set in the remaining 24-bit area of ​​the extension area excluding the three bits specifying the type of header information (i.e., the type of information and the position where the information is set) is determined. This allows the receiving side to recognize what information is set at what position in the other areas of the extension area other than the area where the type of header information is specified by checking the type of header information, and to read out the information.

[0055] For example, FIG. 8 shows an example of setting the type of header information and a method of using the extension area according to the setting, in which the payload length of a packet (in other words, the length of a line) is variable. Specifically, in the example shown in FIG. 8, a value according to the type of header information when the payload length is variable is set. More specifically, in the example shown in FIG. 8, "001" is set for the type of header information. That is, in this case, the type of header information corresponding to "001" means the type when the payload length is variable. Also, in the example shown in FIG. 8, 14 bits in the extension area are assigned to "Line Length". "Line Length" is information for notifying the payload length. With this configuration, the receiving side can recognize that the payload length is variable based on the value set as the type of header information, and can recognize the payload length by reading the value set as "Line Length" in the extension area.

[0056] FIG. 9 shows an example of the format of transmission data 147A transmitted in accordance with the SLVS-EC method standard. In FIG. 9, a series of packets indicated by reference symbol A1 is a schematic representation of packets in which image data of an ROI is transmitted. A series of packets indicated by reference symbols A2 and A3 correspond to packets different from packets for transmitting image data of an ROI. In the following description, when distinguishing between the packets indicated by reference symbols A1, A2, and A3, they are also referred to as "packet A1", "packet A2", and "packet A3", respectively, for convenience. That is, in a period in which data for one frame is transmitted, a series of packets A2 is transmitted before a series of packets A1 is transmitted. A series of packets A3 may also be transmitted after a series of packets is transmitted.

[0057] In the example shown in Fig. 9, at least a part of the series of packets A2 is used to transmit Embedded Data. For example, the Embedded Data may be stored in the payload of the packet A2 and transmitted. As another example, the Embedded Data may be stored in an area of ​​the packet A2 separate from the payload and transmitted.

[0058] The embedded data corresponds to additional information additionally transmitted by the image sensor 100 (in other words, information embedded by the image sensor 100), and includes, for example, information on the imaging conditions of the image, information on the ROI, etc. The embedded data includes, for example, information such as "ROI ID", "top left coordinate", "height", "width", "AD word length (AD bit)", "exposure", "gain", and "sensing information".

[0059] The information of "ROI ID", "top left coordinate", "height", and "width" corresponds to information about a region (ROI) set in an image, and is used, for example, in restoring the image of the region at the receiving side. Specifically, "ROI ID" is identification information for identifying each region. "Top left coordinate" corresponds to coordinates that serve as an index of the position of the region set in the image, and indicates the top left vertex coordinate of the rectangular range in which the region is set. Furthermore, "height" and "width" indicate the height (vertical width) and width (horizontal width) of the rectangular range in which the region is set. Note that, among the Embedded Data, information about the region (ROI) such as the above-mentioned "ROI ID", "top left coordinate", "height", and "width" corresponds to an example of "region information" included in the first packet (e.g., packet A2).

[0060] The "Exposure" information indicates the exposure time for imaging the region (ROI). The "Gain" information indicates the gain for imaging the region. The AD word length (AD bit) indicates the word length of data per pixel that has been AD converted in the region. Examples of sensing information include the contents of calculations for objects (subjects) contained in the region, and supplementary information for subsequent signal processing of the image of the region.

[0061] 9, at least a part of packet A2 is used to transmit Embedded Data, but at least a part of packet A3 may be used to transmit Embedded Data instead of packet A2. In the following description, Embedded Data is also referred to as "EBD."

[0062] In Fig. 9, "SC" indicates "Start Code" and is a group of symbols indicating the start of a packet. The Start Code is added to the beginning of a packet. The Start Code is represented by, for example, four symbols, K28.5, K27.7, K28.2, and K27.7, which are combinations of three types of K Characters.

[0063] "EC" stands for "End Code" and is a group of symbols that indicates the end of a packet. The End Code is added to the end of a packet. The End Code is expressed by four symbols, for example, K28.5, K27.7, K30.7, and K27.7, which are combinations of three types of K characters.

[0064] "PH" indicates a "Packet Header", and corresponds to, for example, the header described with reference to FIG. 7. "FS" indicates an FS (Frame Start) packet. "FE" indicates an FE (Frame End) packet.

[0065] "DC" stands for "Deskew Code" and is a symbol group used to correct data skew between lanes, that is, the timing difference between data received by each lane on the receiving side. Deskew Code is expressed by four symbols, for example, K28.5 and Any**.

[0066] "IC" stands for "Idle Code" and is a group of symbols that are repeatedly transmitted during periods other than the transmission of packet data. The idle code is represented by, for example, D character D00.0 (00000000), which is an 8B10B code.

[0067] "DATA" indicates the area data stored in the payload (i.e., pixel data of the portion corresponding to the area set in the image).

[0068] "XY" corresponds to information indicating the position of the left end of the partial area corresponding to the area data stored in the payload (position in the image) as X and Y coordinates. Note that hereinafter, the X and Y coordinates indicating the position of the left end of the partial area indicated by "XY" are also simply referred to as the "XY coordinates of the partial area."

[0069] The X and Y coordinates of the partial area are stored at the beginning of the payload of the packet A1. In addition, the X and Y coordinates of the partial area may be omitted in the packet A1 transmitted later if there is no change in the X coordinate of the corresponding partial area between consecutively transmitted packets A1 and the Y coordinate is incremented by +1. This control will be described later separately with a specific example.

[0070] In addition, in SLVS-EC, when transmitting area data of partial areas corresponding to each of a row in which a plurality of areas spaced apart from each other in the horizontal direction are set, packets A1 for each of the plurality of areas are generated and transmitted separately. That is, for a row in which two areas spaced apart from each other in the horizontal direction are set, two packets A1 are generated and transmitted.

[0071] (Image sensor 100) Fig. 2 shows an example of the configuration of the image sensor 100. The configuration shown in Fig. 2 corresponds to a specific example of a CSI transmitter. The image sensor 100 includes, for example, an imaging unit 110, a common processing unit 120, a MIPI processing unit 130, an SLVS-EC processing unit 140, and a transmission unit 150. The image sensor 100 transmits to the processor 200 via a data bus B1 either one of transmission data 137A and transmission data 147A generated by performing a predetermined process on image data 111 obtained by the imaging unit 110.

[0072] The imaging unit 110 converts an optical image signal obtained through, for example, an optical lens into image data. The imaging unit 110 includes, for example, a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor. The imaging unit 110 has an analog-to-digital conversion circuit and converts analog image data into digital image data. The data format after conversion may be a YCbCr format in which the color of each pixel is expressed by a luminance component Y and color difference components Cb and Cr, or may be an RGB format. The imaging unit 110 outputs image data 111 (digital image data) obtained by imaging to the common processing unit 120.

[0073] The common processing unit 120 is a circuit that performs a predetermined process on the image data 111 input from the imaging unit 110. When a control signal instructing to cut out an ROI is input from the processor 200 via the control bus B2, the common processing unit 120 performs a predetermined process on the image data 111 input from the imaging unit 110. Thereby, the common processing unit 120 generates various data (one or more ROI image data 112, one or more ROI information 116, frame information 117) and outputs them to the MIPI processing unit 130 and the SLVS-EC processing unit 140. When a control signal instructing to output a normal image is input from the processor 200 via the control bus B2, the common processing unit 120 performs a predetermined process on the image data 111 input from the imaging unit 110. Thereby, the common processing unit 120 generates image data 119 and outputs it to the MIPI processing unit 130 and the SLVS-EC processing unit 140.

[0074] The MIPI processing unit 130 is a circuit that generates transmission data 137A corresponding to the transmission method of the MIPI CSI-2 standard or the MIPI CSI-3 standard based on various data (one or more ROI image data 112, one or more ROI information 116, and frame information 117) input from the common processing unit 120. The SLVS-EC processing unit 140 is a circuit that generates transmission data 147A corresponding to the transmission method of the SLVS-EC standard based on various data (one or more ROI image data 112, one or more ROI information 116, and frame information 117) input from the common processing unit 120. The transmission unit 150 transmits a packet (either the transmission data 137A or the transmission data 147A) transmitted from either the MIPI processing unit 130 or the SLVS-EC processing unit 140 to the processor 200 via the data bus B1 for each row.

[0075] The common processing unit 120 includes, for example, an ROI cutout unit 121, an ROI analysis unit 122, an overlap detection unit 123, a priority setting unit 124, and an image processing control unit 125.

[0076] When a control signal instructing to extract an ROI is input from the processor 200 via the control bus B2, the ROI extraction unit 121 specifies one or more objects of the image capture target included in the image data 111 input from the imaging unit 110, and sets an ROI for each specified object. The ROI is, for example, a rectangular area including the specified object. The ROI extraction unit 121 extracts image data of one or more ROIs (one or more ROI image data 112) from the image data 111. The ROI extraction unit 121 extracts one or more ROI image data 112 included in the image data 111 from the image data 111. The ROI extraction unit 121 further assigns a region number as an identifier to each set ROI. For example, when two ROIs are set in the image data 111, the ROI extraction unit 121 assigns region number 1 to one ROI (ROI1) and region number 2 to the other ROI (ROI2). The ROI cutout unit 121 stores the assigned identifier (region number) in the storage unit, for example. The ROI cutout unit 121 stores each ROI image data 112 cut out from the image data 111 in the storage unit, for example. The ROI cutout unit 121 further stores the identifier (region number) assigned to each ROI in association with the ROI image data 112 in the storage unit, for example. When a control signal instructing output of a normal image is input from the processor 200 via the control bus B2, the ROI cutout unit 121 performs a predetermined process on the image data 111 input from the imaging unit 110, thereby generating image data 119.

[0077] The ROI analysis unit 122 derives, for each ROI, position information 113 of the ROI in the image data 111. For example, the ROI analysis unit 122 stores the derived position information 113 in a storage unit. For example, the ROI analysis unit 122 stores the derived position information 113 in the storage unit in association with an identifier (region number) assigned to the ROI.

[0078] When a plurality of objects to be photographed are identified in the image data 111, the overlap detection unit 123 detects an overlap region (ROO (Region Of Overlap)) where two or more ROIs overlap each other based on the position information 113 of the plurality of ROIs in the image data 111. That is, the overlap detection unit 123 derives the position information 114 of the overlap region ROO in the image data 111 for each overlap region ROO. For example, the overlap detection unit 123 stores the derived position information 114 in a storage unit. For example, the overlap detection unit 123 stores the derived position information 114 in the storage unit in association with the overlap region ROO. The overlap region ROO is, for example, a rectangular region that is the same size as or smaller than the smallest ROI among the two or more ROIs that overlap each other.

[0079] The priority setting unit 124 assigns a priority 115 to each ROI in the image data 111. For example, the priority setting unit 124 stores the assigned priority 115 in a storage unit. For example, the priority setting unit 124 stores the assigned priority 115 in the storage unit in association with the ROI. The priority setting unit 124 may assign the priority 115 to each ROI separately from the region number assigned to each ROI, or may use the region number assigned to each ROI instead of the priority 115. For example, the priority setting unit 124 may associate the priority 115 with the ROI and store it in the storage unit, or may associate the region number assigned to each ROI with the ROI and store it in the storage unit.

[0080] The priority 115 is an identifier of each ROI, and is discrimination information that can discriminate which of the multiple ROIs in the image data 111 has had the overlapping area ROO omitted. For example, the priority setting unit 124 assigns 1 as the priority 115 to one ROI and 2 as the priority 115 to the other ROI in two ROIs each including an overlapping area ROO. In this case, when generating the transmission image data 135A described later, the overlapping area ROO is omitted for the ROI with the larger priority 115 value. The priority setting unit 124 may assign the same number as the region number assigned to each ROI as the priority 115 to the ROI. For example, the priority setting unit 124 associates the priority 115 assigned to each ROI with the ROI image data 112 and stores it in the storage unit. The priority setting unit 124 outputs the region number or priority 115 assigned to each ROI to the MIPI processing unit 130 and the SLVS-EC processing unit 140.

[0081] The image processing control unit 125 generates the ROI information 116 and the frame information 117, and outputs them to the MIPI processing unit 130 and the SLVS-EC processing unit 140. The ROI information 116 includes, for example, each position information 113. The ROI information 116 further includes, for example, at least one of the data type of each ROI, the number of ROIs included in the image data 111, the position information 114 of the overlapping area, the area number (or priority 115) of each ROI, the data length of each ROI, and the image format of each ROI. The frame information 117 includes, for example, the number of the virtual channel assigned to each frame, the data type of each ROI, the payload length for each line, and the like. The data type includes, for example, YUV data, RGB data, or RAW data. The data type further includes, for example, ROI format data or normal format data.

[0082] The MIPI processing unit 130 is a circuit that generates and transmits transmission data 137A corresponding to the transmission method of the MIPI CSI-2 standard or the MIPI CSI-3 standard based on various data (one or more ROI image data 112, one or more ROI information 116, frame information 117) input from the common processing unit 120. The MIPI processing unit 130 transmits the ROI information 116 for each ROI in the image data 111 by EmbeddedData. When a control signal instructing to cut out the ROI is input from the processor 200 via the control bus B2, the MIPI processing unit 130 further transmits the image data of each ROI (compressed image data 135C) by PayloadData of a LongPacket. At this time, the MIPI processing unit 130 transmits the image data of each ROI (compressed image data 135C) by a virtual channel common to each other. The MIPI processing unit 130 also transmits image data of each ROI (compressed image data 135C) by an image data frame, and transmits ROI information 116 for each ROI in the header of the image data frame. The MIPI processing unit 130 also transmits normal image data (compressed image data 135D) by PayloadData of a LongPacket when a control signal instructing output of a normal image is input from the processor 200 via the control bus B2.

[0083] The MIPI processing unit 130 includes, for example, a LINK control unit 131, an ECC generation unit 132, a PH generation unit 133, an EBD buffer 134, an encoding unit 135, an ROI data buffer 136, and a synthesis unit 137.

[0084] The LINK control unit 131 outputs, for example, the frame information 117 to the ECC generation unit 132 and the PH generation unit 133 for each line. The ECC generation unit 132 generates an error correction code for one line based on, for example, data of the line in the frame information 117 (for example, the virtual channel number, the data type of each ROI, the payload length for each line, etc.). The ECC generation unit 132 outputs, for example, the generated error correction code to the PH generation unit 133. The PH generation unit 133 generates a packet header PH for each line using, for example, the frame information 117 and the error correction code generated by the ECC generation unit 132. The PH generation unit 143 outputs the generated packet header PH to the synthesis unit 147.

[0085] The EBD buffer 134 temporarily stores one or more pieces of ROI information 116, and outputs the one or more pieces of ROI information 116 to the synthesis unit 137 as EmbeddedData at a predetermined timing.

[0086] When a control signal instructing to extract an ROI is input from the processor 200 via the control bus B2, the encoding unit 135 generates one or more transmission image data 135A based on one or more ROI image data 112 obtained from the image data 111 and the priority 115 corresponding to the one or more ROI image data 112. The encoding unit 135 generates multiple transmission image data 135A by omitting the image data 135B from the multiple ROI image data 112 obtained from the image data 111 so that the image data 135B of the overlapping area ROO is not included in the multiple ROI image data 112 obtained from the image data 111 in a redundant manner.

[0087] The encoding unit 135 further encodes one or more pieces of transmission image data 135A to generate compressed image data 135C when a control signal instructing cutting out an ROI is input from the processor 200 via the control bus B2. The encoding unit 135 compresses one or more pieces of transmission image data 135A using a compression format conforming to the JPEG standard, for example, to generate compressed image data 135C. The encoding unit 135 encodes the image data 119 to generate compressed image data 135D when a control signal instructing output of a normal image is input from the processor 200 via the control bus B2. The encoding unit 135 compresses the image data 119 using a compression format conforming to the JPEG standard, for example, to generate compressed image data 135E.

[0088] The ROI data buffer 136 temporarily stores the compressed image data 135C or the compressed image data 135D, and outputs the compressed image data 135C or the compressed image data 135D to the synthesis unit 137 as PayloadData of a LongPacket at a predetermined timing.

[0089] The synthesis unit 137 generates transmission data 137A based on various input data (packet header PH, ROI information 116, and compressed image data 135C or compressed image data 135D). The synthesis unit 137 outputs the generated transmission data 137A to the transmission unit 150. That is, the synthesis unit 137 includes DataType (data type of each ROI) in the packet header PH of PayloadData of LongPacket and sends it. The synthesis unit 137 also sends image data of each ROI (compressed image data 135C) on a common virtual channel. The synthesis unit 137 synthesizes the ROI information 116 into the transmission data 137A as EmbeddedData. The synthesis unit 137 synthesizes header information including EmbeddedData and header ECC information for performing error detection or correction on the header information into the transmission data 137A. The synthesis unit 137 synthesizes PayloadDataECC information for performing error detection or correction on PayloadData into the transmission data 137A. The synthesis unit 137 synthesizes the compressed image data 135C or the compressed image data 135D into the transmission data 137A as PayloadData.

[0090] For example, in the packet area R2 of the transmission data 137A, the synthesis unit 137 arranges the compressed image data 135C by dividing it into pixel rows of the compressed image data 135C. Therefore, the packet area R2 of the transmission data 137A does not include overlapping compressed image data corresponding to the image data 135B in the overlap area ROO. Also, for example, in the packet area R2 of the transmission data 137A, the synthesis unit 137 omits pixel rows of the image data 111 that do not correspond to each transmission image data 135A. Therefore, the packet area R2 of the transmission data 137A does not include pixel rows of the image data 111 that do not correspond to each transmission image data 135A. Note that in the packet area R2 of FIG. 6, the area surrounded by a dashed line corresponds to the compressed image data of the image data 135B in the overlap area ROO.

[0091] The boundary between the packet group closer to the packet header PH (e.g., 1(n) in FIG. 6) and the packet group farther from the packet header PH (e.g., 2(1) in FIG. 6) is specified by the physical region length XLa1 of the ROI image data 112 corresponding to the compressed image data of the packet group closer to the packet header PH (e.g., 1(n) in FIG. 6). In the compressed image data corresponding to the image data 135B of the overlapping region ROO included in the packet group closer to the packet header PH (e.g., 1(n) in FIG. 6), the start position of the packet is specified by the physical region length XLa2 of the ROI image data 112 corresponding to the packet group farther from the packet header PH (e.g., 2(1) in FIG. 6).

[0092] For example, when generating PayloadData of a LongPacket for each line in the packet area R2 of the transmission data 137A, the synthesis unit 137 may include, for example, ROI information 116 in PayloadData of the LongPacket in addition to pixel data for one line in the compressed image data 135C. That is, the synthesis unit 137 may include the ROI information 116 in PayloadData of the LongPacket and transmit it. At this time, the ROI information 116 includes at least one of the number of ROIs (number of ROIs) included in the image data 111, the region number (or priority 115) of each ROI, the data length of each ROI, and the image format of each ROI. It is preferable that the ROI information 116 is arranged at the end of the PayloadData of the LongPacket on the packet header PH side (that is, at the beginning of the PayloadData of the LongPacket).

[0093] The SLVS-EC processing unit 140 is a circuit that generates and transmits transmission data 147A corresponding to the transmission method of the SLVS-EC standard based on various data (one or more ROI image data 112, one or more ROI information 116, frame information 117) input from the common processing unit 120. The SLVS-EC processing unit 140 further transmits image data of each ROI (compressed image data 145C) as PayloadData when a control signal instructing to cut out an ROI is input from the processor 200 via the control bus B2. The SLVS-EC processing unit 140 also transmits normal image data (compressed image data 145D) when a control signal instructing to output a normal image is input from the processor 200 via the control bus B2.

[0094] The SLVS-EC processing unit 140 includes, for example, a LINK control unit 141, an ECC generation unit 142, a PH generation unit 143, an EBD buffer 144, an encoding unit 145, an ROI data buffer 146, and a synthesis unit 147.

[0095] The LINK control unit 141 outputs, for example, the frame information 117 to the ECC generation unit 142 and the PH generation unit 143 for each line. For example, the ECC generation unit 142 generates an error correction code for one line based on data (for example, Frame Start, Frame End, etc.) of the frame information 117. For example, the ECC generation unit 142 outputs the generated error correction code to the PH generation unit 143. For example, the PH generation unit 143 generates a packet header for each line using the frame information 117 and the error correction code generated by the ECC generation unit 142. When transmitting area data, as described above, the PH generation unit 143 sets information indicating that area information (for example, area data) is transmitted as the type of header information in the extension area of ​​the packet header. Then, the PH generation unit 143 sets information indicating that area data is transmitted using the payload in at least a part of the extension area. In addition, for a packet in which the coordinates of the area are inserted in the payload, the PH generation unit 143 sets information indicating that the coordinates of the area are transmitted using the payload in at least a part of the extension area. The PH generation unit 143 outputs the generated packet header to the synthesis unit 147. Note that the PH generation unit 143 may place the data of the area in EmbeddedData instead of in the Payload.

[0096] The EBD buffer 144 temporarily stores the additional information (one or more pieces of ROI information 116) transmitted from the common processing unit 120, and outputs the one or more pieces of ROI information 116 to the synthesis unit 147 as EmbeddedData at a predetermined timing.

[0097] When a control signal instructing to cut out an ROI is input from the processor 200 via the control bus B2, the encoding unit 145 generates one or more transmission image data 145A based on one or more ROI image data 112 obtained from the image data 111 and the priority 115 corresponding to the one or more ROI image data 112. The encoding unit 145 generates multiple transmission image data 145A obtained by omitting the image data 145B from the one or more ROI image data 112 obtained from the image data 111 so that the image data 145B of the overlapping area ROO is not included in the multiple ROI image data 112 obtained from the image data 111 in a redundant manner.

[0098] The encoding unit 145 further encodes one or more pieces of transmission image data 145A to generate compressed image data 145C when a control signal instructing cutting out an ROI is input from the processor 200 via the control bus B2. The encoding unit 145 compresses one or more pieces of transmission image data 145A using a compression format conforming to the JPEG standard, for example, to generate compressed image data 145C. The encoding unit 145 encodes the image data 119 to generate compressed image data 145D when a control signal instructing output of a normal image is input from the processor 200 via the control bus B2. The encoding unit 145 compresses the image data 119 using a compression format conforming to the JPEG standard, for example, to generate compressed image data 145D.

[0099] The ROI data buffer 146 temporarily stores the compressed image data 145C or the compressed image data 145D, and outputs the compressed image data 145C or the compressed image data 145D to the synthesis unit 147 at a predetermined timing.

[0100] The synthesis unit 147 generates transmission data 147A based on the various input data (packet header, additional information, and compressed image data 145C or compressed image data 145D). The synthesis unit 147 synthesizes the ROI information 116 into the transmission data 147A as EmbeddedData. The synthesis unit 147 synthesizes header information including EmbeddedData and header ECC information for performing error detection or correction on the header information into the transmission data 147A. The synthesis unit 147 outputs the generated transmission data 147A to the transmission unit 150. The synthesis unit 147 synthesizes PayloadDataECC information for performing error detection or correction on PayloadData into the transmission data 147A. The synthesis unit 147 synthesizes the compressed image data 145C or compressed image data 145D into the transmission data 147A as PayloadData.

[0101] For example, when three ROIs (ROI1, ROI2, ROI3) are set, the synthesis unit 147 transmits region information of each ROI as part of additional information (embedded data) in packet A2, and transmits region data corresponding to each ROI row by row in packet A1.

[0102] The transmitting unit 150 transmits either the transmission data 137A or the transmission data 147A to the processor 200 via the data bus B1. The transmitting unit 150 transmits a packet transferred from either the combining unit 137 or the combining unit 147 to the processor 200 via the data bus B1 for each row. The output of the transmitting unit 150 is connected to, for example, an output pin P1 connected to the data bus B1.

[0103] (Processor 200) Next, the processor 200 will be described. Fig. 3 shows an example of the configuration of the processor 200. The configuration shown in Fig. 3 corresponds to a specific example of a CSI receiver. The processor 200 is, for example, a device that receives a signal in accordance with a standard common to the image sensor 100 (for example, either the MIPI CSI-2 standard or the MIPI CSI-3 standard, and the SLVS-EC standard).

[0104] The processor 200 includes, for example, an MIPI processing unit 210, an SLVS-EC processing unit 220, and a common processing unit 230. The MIPI processing unit 210 and the SLVS-EC processing unit 220 are circuits that receive either the transmission data 137A or the transmission data 147A output from the image sensor 100 via a data bus B1, perform a predetermined process on the received transmission data, generate various data (112, 116, 117), and output the various data (112, 116, 117) to the common processing unit 230. The input of the MIPI processing unit 210 and the input of the SLVS-EC processing unit 220 are connected to, for example, a common input pin P2 connected to the data bus B1. This makes it possible to reduce the number of input pins compared to a case where separate input pins are provided. The data formats of the various data (112, 116, 117) output from the MIPI processing unit 210 and the various data (112, 116, 117) output from the SLVS-EC processing unit 220 are the same.

[0105] The common processing unit 230 is a circuit that generates an ROI image 233A based on various data (112, 116, 117) received from either the MIPI processing unit 210 or the SLVS-EC processing unit 220. The common processing unit 230 is a circuit that generates a normal image 234A based on data (119) received from either the MIPI processing unit 210 or the SLVS-EC processing unit 220. Since the data formats of the output of the MIPI processing unit 210 and the output of the SLVS-EC processing unit 220 are the same, the common processing unit 230 is not provided with a dedicated processing circuit for the output of the MIPI processing unit 210 or a dedicated processing circuit for the output of the SLVS-EC processing unit 220. In other words, the common processing unit 230 is capable of processing both the outputs of the MIPI processing unit 210 and the SLVS-EC processing unit 220 using a common processing circuit.

[0106] The MIPI processing unit 210 extracts one or more ROI image data 112 and one or more ROI information 116 from the transmission data 137A compatible with the MIPI CSI-2 standard or the MIPI CSI-3 standard. The MIPI processing unit 210 has, for example, a header separation unit 211, a header interpretation unit 212, a payload separation unit 213, an EBD interpretation unit 214, an ROI data separation unit 215, an information extraction unit 216, an ROI decoding unit 217, and a normal image decoding unit 218.

[0107] The header separation unit 211 receives the transmission data 137A from the image sensor 100 via the data bus B1. That is, the header separation unit 211 receives the transmission data 137A including the ROI information 116 for each ROI in the image data 111 in the EmbeddedData and including the image data of each ROI (compressed image data 135C) in the PayloadData of the LongPacket. The header separation unit 211 separates the received transmission data 137A according to the rules defined in the MIPI CSI-2 standard or the MIPI CSI-3 standard. The header separation unit 211 separates the received transmission data 137A into a header region R1 and a packet region R2.

[0108] The header interpretation unit 212 identifies the position of the PayloadData of the LongPacket included in the packet area R2 based on the data (specifically, EmbeddedData) included in the header area R1. The payload separation unit 213 separates the PayloadData of the LongPacket included in the packet area R2 from the packet area R2 based on the position of the PayloadData of the LongPacket identified by the header interpretation unit 212.

[0109] The EBD interpretation unit 214 outputs the EmbeddedData as EBD data 214A to the information extraction unit 216. The EBD interpretation unit 214 further determines, from the data type included in the EmbeddedData, whether the image data included in the PayloadData of the LongPacket is compressed image data 135C of the ROI image data (ROI image data 112) or compressed image data 135D of the normal image data (image data 119). The EBD interpretation unit 214 outputs the determination result to the ROI data separation unit 215.

[0110] When the image data included in the PayloadData of the LongPacket is compressed image data 135C of the image data of the ROI (ROI image data 112), the ROI data separation unit 215 outputs the PayloadData of the LongPacket as PayloadData 215A to the ROI decoding unit 217. When the image data included in the PayloadData is compressed image data 135D of the normal image data (image data 119), the ROI data separation unit 215 outputs the PayloadData of the LongPacket as PayloadData 215B to the normal image decoding unit 218. When the PayloadData of the LongPacket includes ROI information 116, PayloadData 215A includes the ROI information 116 and one line of pixel data of the compressed image data 135C.

[0111] The information extraction unit 216 extracts one or more pieces of ROI information 116 from the EmbeddedData included in the EBD data 214A. For example, the information extraction unit 216 extracts, for example, the number of ROIs included in the image data 111, the region number (or priority 115) of each ROI, the data length of each ROI, and the image format of each ROI from the EmbeddedData included in the EBD data 214A. That is, the transmission data 137A includes the region number (or priority 115) of the ROI corresponding to each ROI image data 112 as discrimination information that can discriminate which of the multiple ROI image data 112 obtained from the transmission data 137A has omitted the image 118 of the overlapping region ROO.

[0112] The information extraction unit 216 extracts, for example, from EmbeddedData included in the EBD data 214A, the coordinates (for example, upper left coordinate (Xa1, Ya1)), length (for example, physical region length XLa1, YLa1), and region number 1 (or priority 115 (=1)) of an ROI corresponding to a certain ROI image data 112. The information extraction unit 216 further extracts, for example, from EmbeddedData included in the EBD data 214A, the coordinates (for example, upper left coordinate (Xa2, Ya2)), length (for example, physical region length XLa2, YLa2), and region number 2 (or priority 115 (=2)) of an ROI corresponding to another ROI image data 112.

[0113] The normal image decoding unit 218 decodes the PayloadData 215B to generate normal image data 218A. The ROI decoding unit 217 decodes the compressed image data 137B included in the PayloadData 215A to generate image data 217A. This image data 217A is composed of one or more ROI image data 112.

[0114] The SLVS-EC processing unit 220 extracts one or more ROI image data 112 and one or more ROI information 116 from the transmission data 147A conforming to the SLVS-EC standard. The SLVS-EC processing unit 220 has, for example, a header separation unit 221, a header interpretation unit 222, a payload separation unit 223, an EBD interpretation unit 224, an ROI data separation unit 225, an information extraction unit 226, an ROI decoding unit 227, and a normal image decoding unit 228.

[0115] The header separation unit 221 receives the transmission data 147A from the image sensor 100 via the data bus B1. The header separation unit 211 separates the received transmission data 147A into the transmission data 147A in accordance with the rules defined by the SLVS-EC standard.

[0116] The header interpretation unit 222 interprets the contents indicated by the header data. As a specific example, the header interpretation unit 222 recognizes the format of information set in other areas of the extension area other than the first three bits according to the type of header information set in the first three bits of the extension area of ​​the packet header. Then, the header interpretation unit 222 reads out various information set in the extension area according to the recognition result of the format. This makes it possible for the header interpretation unit 222 to recognize, for example, that information (e.g., area data) of an area (ROI) is transmitted or that coordinates of an area are transmitted using a payload based on the information set in the extension area. Then, the header interpretation unit 222 notifies the payload separation unit 223 of the setting recognized according to the read result of the various information set in the extension area. Specifically, when the header interpretation unit 222 recognizes that information (e.g., area data) of an area (ROI) is transmitted or that coordinates of an area are transmitted using a payload, it notifies the payload separation unit 223 of the recognition result.

[0117] The payload separation unit 223 separates the additional information and image data (normal data or area data) from the payload data based on the interpretation result in the header interpretation unit 222. For example, when the packet to be processed is packet A2 or A3, the payload separation unit 223 may separate the additional information (embedded data) from the packet.

[0118] As another example, when the packet to be processed is packet A1, the payload separation unit 223 separates image data from the payload data. For example, when area data is stored in the payload, the payload separation unit 223 may separate the area data from the payload data according to the interpretation result of the packet header. At this time, the payload separation unit 223 may separate the coordinates of the area inserted at the beginning of the payload (i.e., the XY coordinates of the partial area) according to the interpretation result of the packet header. When normal data is stored in the payload, the payload separation unit 223 may separate the normal data from the payload data according to the interpretation result of the packet header.

[0119] The payload separation unit 223 transmits the additional information, among the various data separated from the payload data, to the EBD interpretation unit 224. Furthermore, the payload separation unit 223 transmits the image data (region data or normal data), among the various data separated from the payload data, to the ROI data separation unit 225. Furthermore, at this time, the payload separation unit 223 may transmit the region data to the ROI data separation unit 225 in association with the coordinates of the region corresponding to the region data (i.e., the XY coordinates of the partial region).

[0120] The EBD interpretation unit 224 interprets the contents of the additional information (Embedded Data) and outputs the interpretation result 224A of the additional information to the information extraction unit 226. The EBD interpretation unit 224 may also transmit the interpretation result 224A of the additional information to the ROI data separation unit 225. The format of the additional information (Embedded Data) is as described above with reference to FIG. 9.

[0121] When the image data transmitted from the payload separation unit 223 is compressed image data 120A of the ROI image data 112, the ROI data separation unit 225 outputs the image data separated from the payload data as PayloadData 215A to the ROI decoding unit 227. When the image data transmitted from the payload separation unit 223 is compressed image data 130A of the normal image data, the ROI data separation unit 225 outputs the image data separated from the payload data to the normal image decoding unit 228 as PayloadData 215B.

[0122] The information extraction unit 226 extracts the ROI information 116 from the interpretation result 224A of the additional information. For example, the information extraction unit 226 extracts the number of ROIs included in the image data 111, the region number (or the priority 115) of each ROI, the data length of each ROI, and the image format of each ROI from the interpretation result 224A of the additional information. That is, the transmission data 147A includes the region number (or the priority 115) of the ROI corresponding to each ROI image data 112 as discrimination information capable of discriminating for which of the multiple ROI image data 112 obtained from the transmission data 147A the image 118 of the overlapping region ROO has been omitted.

[0123] The normal image decoding unit 228 decodes the PayloadData 215B to generate normal image data 228A. The ROI decoding unit 227 decodes the compressed image data 147B included in the PayloadData 225A to generate image data 227A. This image data 227A is composed of one or more ROI image data 112.

[0124] The common processing unit 230 generates one or more ROI image data 112 included in the image data 111 based on the output of either the MIPI processing unit 210 or the SLVS-EC processing unit 220. The common processing unit 230 also generates image data 111 as a normal image based on the output of either the MIPI processing unit 210 or the SLVS-EC processing unit 220. When the processor 200 inputs a control signal instructing the image sensor 100 to cut out an ROI via the control bus B2, the common processing unit 230 executes image processing based on the generated multiple ROI image data 112. When the processor 200 inputs a control signal instructing the image sensor 100 to output a normal image via the control bus B2, the common processing unit 230 executes image processing based on the generated image data 111.

[0125] The common processing unit 230 includes, for example, three selection units 231, 232, and 234, an ROI image generation unit 233, and an image processing unit 235.

[0126] The two selection units 231, 232 select one of the outputs of the MIPI processing unit 210 (information extraction unit 216, ROI decoding unit 217) and the SLVS-EC processing unit 220 (information extraction unit 226, ROI decoding unit 227) and outputs it to the ROI image generation unit 233. The selection unit 234 selects one of the outputs of the MIPI processing unit 210 (normal image decoding unit 218) and the SLVS-EC processing unit 220 (normal image decoding unit 228) and outputs it to the image processing unit 235.

[0127] The ROI image generating unit 233 generates an image (ROI image data 112) of each ROI in the image data 111 based on the outputs of the two selecting units 231 and 232. When the processor 200 inputs a control signal instructing the image sensor 100 to cut out an ROI via the control bus B2, the image processing unit 235 performs image processing using the image (ROI image data 112) of each ROI in the image data 111. On the other hand, when the processor 200 inputs a control signal instructing the image sensor 100 to output a normal image via the control bus B2, the image processing unit 235 performs image processing using the image data 111.

[0128] [effect] Next, effects of communication system 1000 according to the present embodiment will be described.

[0129] In recent years, there has been an increase in applications that require the transmission of large amounts of data. This puts a heavy load on the transmission system, and in the worst case scenario, the transmission system may crash, making it impossible to transmit data.

[0130] Conventionally, in order to avoid a breakdown in the transmission system, for example, instead of transmitting all captured images, the object being photographed is identified and only a portion of the image extracted from the identified object is transmitted.

[0131] Incidentally, MIPI CSI-2, MIPI CSI-3, etc. are sometimes used as a method for transmitting from an image sensor to an application processor, and SLVS-EC, etc. are sometimes used as a method for transmitting from an application processor to a display.

[0132] When attempting to transmit a region (ROI (Region Of Interest)) extracted from a captured image using these methods, in order to support multiple transmission methods, it was necessary to provide a transmitter and receiver for each transmission method to be supported, and improvements were needed in terms of cost and device size.

[0133] On the other hand, in this embodiment, both the image sensor 100 and the processor 200 support two transmission methods (MIPI CSI-2 or MIPI CSI-3, and SLVS-EC). Here, in the image sensor 100, a processing block (common processing unit 120) that can be shared between MIPI CSI-2 or MIPI CSI-3 and SLVS-EC is shared. Furthermore, in the processor 200, a processing block (common processing unit 230) that can be shared between MIPI CSI-2 or MIPI CSI-3 and SLVS-EC is shared.

[0134] Specifically, in the image sensor 100, a processing block that performs a process of extracting ROI image data 112 from image data 111 obtained by the imaging unit 110 and a process of generating information (ROI information 116 and frame information 117) required for generating transmission data 137A, 147A corresponding to each transmission method is shared. Also, in the processor 200, a processing block that performs a process of generating (reconstructing) the ROI image data 112 is shared. This makes it possible to reduce the circuit scale and reduce costs compared to a case where separate processing blocks are provided to support MIPI CSI-2 or MIPI CSI-3 and SLVS-EC.

[0135] <2. Modifications> In the above embodiment, MIPI CSI-2 or MIPI CSI-3, and SLVS-EC are given as multiple transmission methods that can be supported. However, even if the above embodiment is made to support multiple transmission methods other than these, it is possible to standardize the processing block on the transmitting device side that performs the process of extracting ROI image data from image data obtained by imaging and the process of generating information (ROI information and frame information) necessary for generating transmission data. Also, it is possible to standardize the processing block on the receiving device side that performs the process of generating (restoring) ROI image data.

[0136] Although the present disclosure has been described above by way of the embodiment and its modified examples, the present disclosure is not limited to the above embodiment and can be modified in various ways. Note that the effects described in this specification are merely examples. The effects of the present disclosure are not limited to the effects described in this specification. The present disclosure may have effects other than those described in this specification.

[0137] Furthermore, for example, the present disclosure can have the following configuration. (1) A cutout unit that cuts out one or more ROI (Region Of Interest) image data included in image data obtained by imaging; A derivation unit that derives ROI position information in the image data; a first processing unit that generates first transmission data corresponding to a first transmission method based on the one or more ROI image data and one or more ROI position information in the image data; a second processing unit that generates second transmission data corresponding to a second transmission method based on the one or more ROI image data and one or more ROI position information in the image data; Equipped with Transmitting device. (2) The first processing unit combines the one or more pieces of ROI position information into the first transmission data as EmbeddedData, The second processing unit combines the one or more pieces of ROI position information into the second transmission data as EmbeddedData. A transmitting device as described in (1). (3) The first processing unit combines header information including the EmbeddedData and header ECC information for performing error detection or correction on the header information into the first transmission data, The second processing unit combines header information including the EmbeddedData and header ECC information for performing error detection or correction on the header information into the second transmission data. A transmitting device as described in (2). (4) The first processing unit combines the one or more ROI image data as PayloadData into the first transmission data; The second processing unit combines the one or more ROI image data with the second transmission data as PayloadData. A transmitting device as described in (1). (5) The first processing unit combines PayloadDataECC information for performing error detection or correction on the PayloadData with the first transmission data, The second processing unit combines PayloadDataECC information for performing error detection or correction on the PayloadData with the second transmission data. A transmitting device as described in (4). (6) The first processing unit combines the one or more ROI position information as EmbeddedData and the one or more ROI image data as PayloadData into the first transmission data; The second processing unit combines the one or more ROI position information as EmbeddedData and the one or more ROI image data as PayloadData into the second transmission data; The first processing unit combines header information including the EmbeddedData, header ECC information for performing error detection or correction on the header information, and at least one of PayloadDataECC information for performing error detection or correction on a packet footer or the PayloadData, into the first transmission data; The second processing unit combines, into the second transmission data, at least one of header information including the EmbeddedData, header ECC information for performing error detection or correction on the header information, and PayloadDataECC information for performing error detection or correction on a packet footer or the PayloadData. A transmitting device as described in (1). (7) a first processing unit that extracts one or more pieces of image data and one or more pieces of position information from first transmission data corresponding to a first transmission method; a second processing unit that extracts one or more pieces of image data and one or more pieces of position information from second transmission data corresponding to a second transmission method; a generation unit that generates one or more ROI image data included in captured image data obtained by imaging, based on the one or more image data and the one or more position information extracted by the first processing unit or the second processing unit; Equipped with Receiving device. (8) The first processing unit extracts the one or more pieces of position information from EmbeddedData included in the first transmission data, The second processing unit extracts the one or more pieces of position information from EmbeddedData included in the second transmission data. A receiving device as described in (7). (9) The first processing unit extracts the one or more image data from PayloadData included in the first transmission data, The second processing unit extracts the one or more image data from PayloadData included in the second transmission data. A receiving device as described in (7). (10) A transmitting device and a receiving device are provided, The transmitting device A cutout unit that cuts out one or more ROI image data included in image data obtained by imaging; A derivation unit that derives ROI position information in the image data; a first processing unit that generates first transmission data corresponding to a first transmission method based on the one or more ROI image data and one or more ROI position information in the image data; a second processing unit that generates second transmission data corresponding to a second transmission method based on the one or more ROI image data and one or more ROI position information in the image data; having The receiving device includes: a first processing unit that extracts one or more pieces of image data and the one or more pieces of position information from the first transmission data; a second processing unit that extracts the one or more pieces of image data and the one or more pieces of position information from the second transmission data; a generating unit that generates one or more ROI image data included in the image data based on the one or more image data and the one or more position information extracted by the first processing unit or the second processing unit; have Communication systems.

[0138] According to a transmitting device of one embodiment of the present disclosure, a processing block that extracts one or more ROI image data contained in image data obtained by imaging and derives ROI position information in the image data is made common to the first transmission method and the second transmission method, thereby making it possible to support multiple transmission methods.

[0139] In addition, according to a receiving device of an embodiment of the present disclosure, a processing block that performs processing to generate one or more ROI image data is made common to the first transmission method and the second transmission method, thereby making it possible to support multiple transmission methods.

[0140] Furthermore, according to a communication system of an embodiment of the present disclosure, a processing block that extracts one or more ROI image data contained in image data obtained by imaging and derives ROI position information in the image data is made common to the first transmission method and the second transmission method, and a processing block that performs processing to generate one or more ROI image data is made common to the first transmission method and the second transmission method, thereby making it possible to support a plurality of transmission methods.

[0141] Note that the effects of the present disclosure are not necessarily limited to the effects described herein, and may be any of the effects described in this specification.

[0142] This application claims priority based on Japanese Patent Application No. 2019-139884, filed on July 30, 2019 in the Japan Patent Office, the entire contents of which are incorporated herein by reference.

[0143] Those skilled in the art will recognize that various modifications, combinations, subcombinations, and variations may occur to those skilled in the art depending on design requirements and other factors, and that such modifications are within the scope of the appended claims and their equivalents.

Claims

1. A cutout unit that cuts out one or more ROI (Region Of Interest) image data included in image data obtained by imaging; A derivation unit that derives ROI position information in the image data; a first processing unit that generates first transmission data corresponding to a first transmission method based on the one or more ROI image data and one or more ROI position information in the image data; a second processing unit that generates second transmission data corresponding to a second transmission method based on the one or more ROI image data and one or more ROI position information in the image data; a transmitting unit that transmits either the first transmission data or the second transmission data to a processor; Equipped with Transmitting device.

2. The first processing unit combines the one or more pieces of ROI position information into the first transmission data as EmbeddedData, The second processing unit combines the one or more pieces of ROI position information into the second transmission data as EmbeddedData. The transmitting device according to claim 1 .

3. The first processing unit combines header information including the EmbeddedData and header ECC information for performing error detection or correction on the header information into the first transmission data, The second processing unit combines header information including the EmbeddedData and header ECC information for performing error detection or correction on the header information into the second transmission data. The transmitting device according to claim 2.

4. The first processing unit combines the one or more ROI image data with the first transmission data as PayloadData, The second processing unit combines the one or more ROI image data with the second transmission data as PayloadData. The transmitting device according to claim 1 .

5. The first processing unit combines PayloadDataECC information for performing error detection or correction on the PayloadData with the first transmission data, The second processing unit combines PayloadDataECC information for performing error detection or correction on the PayloadData with the second transmission data. The transmitting device according to claim 4.

6. The first processing unit synthesizes the one or more ROI position information as EmbeddedData and the one or more ROI image data as PayloadData into the first transmission data, The second processing unit synthesizes the one or more ROI position information as EmbeddedData and the one or more ROI image data as PayloadData into the second transmission data, The first processing unit combines header information including the EmbeddedData, header ECC information for performing error detection or correction on the header information, and at least one of PayloadData ECC information for performing error detection or correction on a packet footer or the PayloadData, into the first transmission data, The second processing unit combines, into the second transmission data, at least one of header information including the EmbeddedData, header ECC information for performing error detection or correction on the header information, and PayloadData ECC information for performing error detection or correction on a packet footer or the PayloadData. The transmitting device according to claim 1 .

7. A receiving unit that receives either first transmission data corresponding to a first transmission method or second transmission data corresponding to a second transmission method; a first processing unit that extracts one or more pieces of image data and one or more pieces of position information from the first transmission data; a second processing unit that extracts one or more pieces of image data and one or more pieces of position information from the second transmission data; a generation unit that generates one or more ROI image data included in captured image data obtained by imaging, based on the one or more image data and the one or more position information extracted by the first processing unit or the second processing unit; Equipped with Receiving device.

8. The first processing unit extracts the one or more pieces of position information from EmbeddedData included in the first transmission data, The second processing unit extracts the one or more pieces of position information from EmbeddedData included in the second transmission data.

8. The receiving device according to claim 7.

9. The first processing unit extracts the one or more image data from PayloadData included in the first transmission data, The second processing unit extracts the one or more image data from PayloadData included in the second transmission data.

8. The receiving device according to claim 7.

10. A transmitting device and a receiving device are provided. The transmitting device A cutout unit that cuts out one or more ROI image data included in image data obtained by imaging; A derivation unit that derives ROI position information in the image data; a first processing unit that generates first transmission data corresponding to a first transmission method based on the one or more ROI image data and one or more ROI position information in the image data; a second processing unit that generates second transmission data corresponding to a second transmission method based on the one or more ROI image data and one or more ROI position information in the image data; a transmitting unit that transmits either the first transmission data or the second transmission data to the receiving device; having The receiving device includes: a receiving unit that receives either the first transmission data or the second transmission data; a first processing unit that extracts one or more pieces of image data and the one or more pieces of position information from the first transmission data; a second processing unit that extracts the one or more pieces of image data and the one or more pieces of position information from the second transmission data; a generating unit that generates one or more ROI image data included in the image data based on the one or more image data and the one or more position information extracted by the first processing unit or the second processing unit; have Communication systems.

Images