Data processing device, data processing method, and event data output sensor

Abstract

This disclosure relates to a data processing device, a data processing method, and an event data output sensor capable of facilitating processing on a reception side. This data processing device comprises: a data generation unit that generates data in which the timing of output is irregular and the data length for each prescribed data unit is not constant; and a packet generation processing unit that generates a packet in which data is stored in one or more data units. The packet generation processing unit generates a packet with a packet structure in which one of a plurality of types of headers provided when data is stored in a packet is always arranged at the head of the payload of each packet. The present technology can be applied, for example, to an EVS that generates event data.

Description

Data processing device, data processing method, and event data output sensor

[0001] The present disclosure relates to a data processing device, a data processing method, and an event data output sensor, and in particular to a data processing device, a data processing method, and an event data output sensor that make it easier to process data on the receiving side.

[0002] While solid-state imaging devices such as CMOS image sensors have traditionally been required to achieve higher image quality, development is underway to add functions that use metadata, such as EVS (Event-based Vision Sensor).For example, EVS detects brightness changes in each pixel as events in real time and outputs event data, which can result in irregular output timing and inconsistent data lengths for the event data included in each line.

[0003] Patent document 1 proposes an EVS that transmits event data in a frame structure in which event data indicating the contents of the event is made part of the payload data, and line information related to the event data added to the line is stored at the beginning of the payload data.

[0004] International Publication No. 2023 / 058670

[0005] However, when transmitting data with irregular output timing and variable data length, such as the event data described above, it becomes difficult for the receiving side to process such data accordingly. Therefore, there is a need for a format definition that allows the receiving side to easily process the data.

[0006] The present disclosure has been made in view of such circumstances, and aims to make it possible to further facilitate processing on the receiving side.

[0007] A data processing device according to a first aspect of the present disclosure includes a data generation unit that generates data whose output timing is irregular and whose data length for each specified data unit is not constant, and a packet generation processing unit that generates packets that store the data in one or more of the data units, and the packet generation processing unit generates the packets in a packet structure in which one of multiple types of headers that are provided when storing the data in the packet is always placed at the beginning of the payload of each of the packets.

[0008] A data processing method of a first aspect of the present disclosure includes a data processing device generating data having irregular output timing and inconsistent data length for each specified data unit, and generating packets storing the data in one or more of the data units, wherein the packets are generated with a packet structure in which one of multiple types of headers provided when storing the data in the packet is always placed at the beginning of the payload of each of the packets.

[0009] In a first aspect of the present disclosure, data is generated that has irregular output timing and a variable data length for each predetermined data unit, and packets are generated that store the data in one or more data units. The packets are then generated with a packet structure in which one of multiple types of headers provided when storing the data in the packet is always placed at the beginning of the payload of each packet.

[0010] An event data output sensor according to a second aspect of the present disclosure includes a data generation unit that detects the occurrence of an event, which is a change in a luminance signal indicating the luminance of light received by a plurality of pixels arranged in an array, and generates event data indicating the content of the event; and a packet generation processing unit that generates packets storing the event data for one or more of the lines, with a line of the pixels as the data unit of the event data, and the packet generation processing unit generates the packets with a packet structure in which a header that is provided when storing the event data in the packet, either a line header that is provided for each line, or a frame header that is provided for each sub-frame corresponding to one screen of the pixel area in which the pixels are provided, is always placed at the beginning of the payload of each of the packets.

[0011] In a second aspect of the present disclosure, the occurrence of an event, which is a change in a luminance signal indicating the luminance of light received by a plurality of pixels arranged in an array, is detected, event data indicating the content of the event is generated, and a packet storing the event data for one or more lines is generated with a line of pixels as the data unit of the event data.The packets are then generated with a packet structure in which a header, which is provided when storing the event data in a packet, is always located at the beginning of the payload of each packet, either a line header provided for each line or a frame header provided for each sub-frame corresponding to one screen of a pixel area in which the pixels are provided.

[0012] 1 is a block diagram showing an example configuration of an embodiment of an EVS to which the present technology is applied. FIG. 2 is a diagram showing a first example packet structure of event data. FIG. 3 is a diagram showing a third example packet structure of event data. FIG. 4 is a diagram illustrating a frame header, a line header, a line footer, and a frame footer. FIG. 5 is a diagram illustrating a Line Num stored in a line header. FIG. 6 is a diagram illustrating identification of a line header and a frame header. FIG. 7 is a diagram illustrating padding insertion provided in a line header. FIG. 8 is a flowchart illustrating a first processing example of a packet generation process. FIG. 9 is a flowchart illustrating a second processing example of a packet generation process. FIG. 10 is a diagram illustrating a determination to store one line of event data across packets. FIG. 11 is a flowchart illustrating a third processing example of a packet generation process. FIG. 12 is a diagram illustrating a case in which one line of event data is stored without spanning packets. FIG. 13 is a diagram illustrating a case in which one line of event data is stored across packets. FIG. 14 is a diagram illustrating an application example in which output of padding and dummy data is omitted. FIG. 15 is a diagram illustrating an example in which a packet is output after event data has accumulated. FIG. 16 is a diagram illustrating an application example in which a timestamp is stored in embedded data of image data. 1 is a diagram showing an example of a format used in SLVS-EC data transmission; FIG. 2 is a diagram showing an example of a packet structure when the present technology is applied to SLVS-EC; and FIG. 3 is a diagram showing an example of use using an image sensor.

[0013] Hereinafter, specific embodiments to which the present technology is applied will be described in detail with reference to the drawings.

[0014] <Configuration Example of EVS> FIG. 1 is a block diagram showing a configuration example of an embodiment of an EVS to which the present technology is applied.

[0015] As shown in FIG. 1, the EVS 11 includes a sensor unit 21 , an event detection unit 22 , an event information generation processing unit 23 , a packet generation processing unit 24 , and a data output unit 25 .

[0016] The sensor unit 21 is configured to include a pixel region 31 in which a plurality of pixels are arranged in an array, each of which detects the brightness of received light and outputs a brightness signal indicating the brightness, and a line control unit 32 and a column control unit 33 which control the output of the brightness signal from each pixel provided in the pixel region 31. Hereinafter, a row of a plurality of pixels arranged horizontally in the pixel region 31 will be referred to as a line, and one frame of event data corresponding to one screen of the pixel region 31 will be referred to as one subframe.

[0017] The event detection unit 22 detects the occurrence of an event, which is a change in the luminance signal output from the sensor unit 21, generates event data indicating the content of the event (for example, data indicating whether the luminance value has changed to the positive or negative side from the reference value), and supplies the event data to the event information generation processing unit 23 and the packet generation processing unit 24. For example, the event detection unit 22 supplies the event data for each subframe to the event / line count unit 41, and supplies the event data for each line to the event count unit 42, the data amount calculation unit 44, the compression unit 45, and the packet generation processing unit 24. Note that the event data generated by the event detection unit 22 is generated when the occurrence of an event is detected, and therefore has the characteristics that the timing of output from the event detection unit 22 is irregular and the data length for each line is not constant.

[0018] The event information generation processing unit 23 generates event information (valid line number information, event number analysis information, line data amount information, and compressed data) that is information related to the event detected by the event detection unit 22, and supplies it to the packet generation processing unit 24. The event information generation processing unit 23 is configured to include an event / line counting unit 41, an event counting unit 42, an event number analysis unit 43, a data amount calculation unit 44, and a compression unit 45.

[0019] The event line counting unit 41 counts the number of lines containing pixels in which an event has occurred (hereinafter referred to as the number of valid lines) for each subframe of event data supplied from the event detection unit 22. The event line counting unit 41 then supplies valid line number information indicating the number of valid lines for each subframe to the packet generation processing unit 24.

[0020] The event counting unit 42 obtains the data length of the event data for each line by counting the number of pixels in which an event has occurred for each line of event data supplied from the event detection unit 22. The event counting unit 42 then supplies data length information indicating the data length of the event data for each line to the event number analysis unit 43.

[0021] The event number analysis unit 43 analyzes whether the data length of the event data for each line is in bytes based on the data length information supplied from the event count unit 42. Then, the event number analysis unit 43 supplies the packet generation processing unit 24 with a signal (Byte_Sig signal / no_Byte_Sig signal) indicating whether the data length of the event data for the line to be processed is in bytes, and event number analysis information including the data length information.

[0022] The data amount calculation unit 44 performs a calculation to calculate the data amount of all the event data, regardless of whether an event has occurred, for each line of event data supplied from the event detection unit 22. Then, the data amount calculation unit 44 supplies line data amount information indicating the data amount for each line to the packet generation processing unit 24 and the compression unit 45.

[0023] The compression unit 45 compresses one line of event data supplied from the event detection unit 22 using a predetermined compression method, and supplies the compressed data obtained by compressing the event data line by line to the packet generation processing unit 24. For example, the compression unit 45 can compress the event data by switching between multiple compression methods based on the line data amount information supplied from the data amount calculation unit 44.

[0024] The packet generation processing unit 24 generates packets that store the event data supplied from the event detection unit 22 for one or more lines based on the event information (valid line number information, event number analysis information, line data amount information, and compressed data) supplied from the event information generation processing unit 23, and supplies the packets to the data output unit 25. For example, as will be described later with reference to FIG. 2, the packet generation processing unit 24 generates packets that store the event data in a packet structure in which either a frame header or a line header is always placed at the beginning of the payload of each packet. The packet generation processing unit 24 is configured with a header generation unit 51 and a packet generation unit 52.

[0025] The header generation unit 51 generates various headers to be provided in packets storing event data, based on the event information supplied from the event information generation processing unit 23. For example, the header generation unit 51 generates a packet header, which is a header of a packet generated by the packet generation unit 52, a frame header, which is a header provided for each subframe, and a line header, which is a header provided for each line.

[0026] For example, the header generation unit 51 sets the number of valid lines to Valid Line Num in the frame header based on the valid line number information. As will be described later with reference to FIG. 6, the header generation unit 51 sets the number of lines included in a packet to Line Num in the first line header of the packet, and sets the line number in the packet to Line Num in the second or subsequent line header of the packet. As will be described later with reference to FIG. 7, the header generation unit 51 sets 0 to Header Indicator in the frame header and 1 to Header Indicator in the line header. As will be described later with reference to FIG. 8, when padding has been inserted, the header generation unit 51 sets 1 to Padding Insertion in the line header. As will be described later with reference to FIG. 11, when one line of event data is stored across packets, the header generation unit 51 sets 1 to Data Straddle in the line header.

[0027] The packet generator 52 generates a packet storing the event data supplied from the event detector 22 based on the event information supplied from the event information generator 23 .

[0028] For example, the packet generation unit 52 can generate packets storing multiple lines of event data based on data length information included in the event count analysis information. If the event data is not in bytes, the packet generation unit 52 inserts padding so that the read line length is in bytes based on a signal (Byte_Sig signal / no_Byte_Sig signal) included in the event count analysis information. As will be described later with reference to FIG. 11 , if the previous packet cannot store one line of event data, the packet generation unit 52 stores the remaining portion of the event data that cannot be stored in the previous packet in the next packet. When no event occurs, the packet generation unit 52 generates packets storing dummy data so that the packet output interval is constant. In addition to generating packets storing event data, the packet generation unit 52 may also generate packets storing embedded data.

[0029] The data output unit 25 outputs the packet storing the event data supplied from the packet generation processing unit 24 to an application processor (not shown) via, for example, D-PHY of MIPI (Mobile Industry Processor Interface), which is a high-speed interface standard for mobile devices. Note that the present technology is not limited to D-PHY and can also be applied to the MIPI CSI (Camera Serial Interface)-2 standard. For example, the data output unit 25 may output the packet storing the event data via A-PHY, which is a standard for automotive devices.

[0030] The EVS11 configured as described above can generate packets that store event data using a packet structure in which either a frame header or a line header is always placed at the beginning of the payload of each packet, thereby facilitating processing by the application processor (not shown) that receives the packets.

[0031] <Example of Event Data Packet Structure> The packet structure of a packet storing event data output from the EVS 11 will be described with reference to FIGS.

[0032] 2 to 4 show the packet structure of a packet storing one frame's worth of event data, with one frame including multiple subframes. One frame is composed of, for example, multiple subframes that can be output at a time corresponding to the frame rate of an image sensor that captures images in parallel with the event data detection by the EVS 11. As shown in the figures, the subframe is composed of multiple packets from a frame start (FS) to a frame end (FE). For example, one subframe is composed of multiple packets that store event data in a number corresponding to the number of lines in the pixel area 31. As shown in the figures, the subframe is composed of multiple packets from a packet having a frame header (FH) to the packet immediately preceding the packet having the next frame header. The frame header may be output in synchronization with an external signal, or may be output internally synchronized according to a period defined by internal settings.

[0033] A packet consists of one or more lines of event data stored in the payload from the packet header (PH) to the packet footer (PF). In addition to event data, the packet payload can also store dummy data and padding inserted so that the read line length is in bytes.

[0034] The frame header is placed at the beginning of the payload of the first packet that makes up each subframe. That is, the frame header is placed after the packet header of the first packet that makes up the subframe. The line header (LH) is placed at the beginning of the event data in line units. The line header is also placed at the beginning of dummy data.

[0035] FIG. 2 is a diagram showing a first example of the packet structure of a packet in which event data is stored.

[0036] 2, in the first packet structure example, the first packet constituting a subframe has a packet header followed by a frame header, a line header followed by the frame header, and event data followed by the line header. Also, in the second and subsequent packets constituting a subframe, the packet header followed by a line header, and event data followed by the line header.

[0037] That is, in the first packet structure example, either a frame header or a line header is always placed at the beginning of the payload of each packet, and event data is also stored in the first packet that makes up the subframe.

[0038] In this way, the first packet structure example has a configuration in which a header (either a frame header or a line header) is always placed at the beginning of the packet payload, and this simpler packet structure can facilitate processing by the application processor that receives the packet output from data output unit 25. For example, the application processor can read the header and perform processing to receive event data, assuming that a header is always placed at the beginning of the packet payload.

[0039] In the first packet structure example, event data is stored in the payload of the first packet of the subframe, and therefore two types of headers, a frame header and a line header, are placed in the first packet of the subframe.

[0040] FIG. 3 is a diagram illustrating a second example of the packet structure of a packet in which event data is stored.

[0041] 3, in the second packet structure example, the first packet constituting a subframe has a frame header following the packet header, and dummy data is stored in the payload after the frame header. Also, the second and subsequent packets constituting a subframe have a line header following the packet header, and event data follows the line header.

[0042] That is, in the second packet structure example, either a frame header or a line header is always placed at the beginning of the payload of each packet, and the first packet constituting a subframe is configured without storing event data. As a result, in the second packet structure example, only one type of header (either a frame header or a line header) is placed in one packet. That is, only a frame header is placed in the first packet constituting a subframe, and only line headers are placed in the second and subsequent packets constituting a subframe.

[0043] In this way, the second packet structure example has a configuration in which a header (either a frame header or a line header) is always placed at the beginning of the packet payload, and only one type of header is placed in one packet, making the packet structure simpler than the first packet structure example described above, and thereby making it easier to process in the application processor that receives the packets output from data output unit 25. For example, the application processor can read the headers and perform processing to receive event data, assuming that only a frame header is placed in the first packet that makes up a subframe, and that only line headers are placed in the second and subsequent packets that make up the subframe.

[0044] In addition, the second packet structure example allows, for example, the area where dummy data stored in the first packet constituting a subframe is provided to be used as a spare for extending the frame header, thereby enabling further enhancement of scalability.

[0045] In addition, in the first packet structure example described above, event data is also stored in the payload of the first packet that constitutes a subframe, which reduces waste of data bandwidth compared to the second packet structure example, and enables event data to be transmitted more efficiently.

[0046] FIG. 4 is a diagram showing a third example of the packet structure of a packet in which event data is stored.

[0047] As shown in FIG. 4, in the third packet structure example, a frame footer (FF) is placed at the end of the payload of the last packet that makes up a subframe, and a line footer (LF) is placed at the end of the event data for each line.

[0048] For example, the frame footer and line footer store error detection information such as CRC (Cyclic Redundancy Check), parity, checksum, ECC (Error-Correcting Code), etc. The frame footer also stores valid line number information indicating the number of valid lines in one subframe.

[0049] In this way, the third packet structure example can be realized more easily in terms of design by arranging information on the number of valid lines, error detection information, and the like in the footer.

[0050] The frame header, line header, line footer, and frame footer will be described with reference to FIG.

[0051] The frame header includes a Header Indicator, a Valid Line Num, a Timestamp, a Frame Count, a SubFrame Count, and a Reserve.

[0052] The Header Indicator is an identifier for distinguishing between a line header and a frame header. For example, 0 is set in the Header Indicator for a frame header.

[0053] Valid Line Num indicates the number of valid lines in one subframe. For example, if Valid Line Num is set to 100, this indicates that the line containing the pixel in which an event occurred in that subframe is 100. As described above, the event line count unit 41 supplies valid line number information indicating the number of valid lines for each subframe to the packet generation processing unit 24, and in the packet generation processing unit 24, the header generation unit 51 can set the valid line number in Valid Line Num in accordance with the valid line number information. By storing Valid Line Num in this way, an application processor that receives the event data can detect errors by, for example, comparing the number of valid lines set in Valid Line Num with the number of valid lines in one subframe that was actually received.

[0054] The Timestamp is a count value that counts time, the Frame Count is a count value that counts frames, and the SubFrame Count is a count value that counts subframes. Note that these count values ​​return to 0 after reaching their maximum value. The Reserve is provided as a reserve for expansion.

[0055] The line header includes a Header Indicator, a Line Num, a Y_address, a Data Volume, a Compression, a Padding Insertion, a Data Straddle, and a Reserve.

[0056] The Header Indicator is an identifier for distinguishing between a line header and a frame header. For example, the Header Indicator is set to 1 in a line header.

[0057] In the first line header of a packet, Line Num indicates the number of lines contained in the packet, and in the second or subsequent line headers of a packet, it indicates the line number within the packet (a value that is incremented within the packet).

[0058] Y_address indicates the address of the read line. Data Volume indicates the number of pixels or the amount of data read in one line, and the number of pixels or the amount of data read in one line varies depending on the compression method used by the compression unit 45.

[0059] Compression indicates the compression method used by the compression unit 45. In addition to the compression method, Compression may also indicate non-compression, a signal processing method, etc. For example, signal processing methods indicated by Compression include optical flow calculation that represents the movement of an object in an image using a vector, feature point extraction that extracts feature points in an image, and ROI (Region of Interest) extraction that extracts an ROI in an image. Note that Compression may be provided in the frame header instead of the line header.

[0060] Padding Insertion is an identifier for identifying whether padding has been inserted so that the read line length is in bytes. For example, when Padding Insertion is set to 0, it indicates that padding has not been inserted, and when Padding Insertion is set to 1, it indicates that padding has been inserted.

[0061] Data Straddle is an identifier that indicates whether one line of event data is stored across to the next packet. For example, when Data Straddle is set to 0, this indicates that one line of event data is not stored across to the next packet, and when Data Straddle is set to 1, this indicates that one line of event data is stored across to the next packet. Reserve is provided as a spare for expansion.

[0062] The line footer includes a Footer Indicator and a CRC.

[0063] The Footer Indicator is an identifier for distinguishing between a line footer and a frame footer, and for example, in a line footer, the Footer Indicator is set to 1. The CRC is error detection information, and as described above, error detection information other than the CRC may also be used.

[0064] The frame footer stores a Footer Indicator, a Valid Line Num, and an ECC.

[0065] The Footer Indicator is an identifier for distinguishing between a line footer and a frame footer; for example, in a frame footer, the Footer Indicator is set to 0. The Valid Line Num indicates the number of valid lines in one subframe, similar to the Valid Line Num in the frame header. ECC is an error correction code.

[0066] The Line Number stored in the line header will be described with reference to FIG.

[0067] 6, in a configuration in which three lines of event data are stored in one packet, the Line Num of the first line header is set to 3, indicating the number of lines included in the packet, the Line Num of the second line header is set to 1, indicating the line number within the packet, and the Line Num of the third line header is set to 2, indicating the line number within the packet.

[0068] For example, in the packet generation processing unit 24, the packet generation unit 52 generates a packet storing multiple lines of event data based on the data length information included in the event count analysis information. At this time, the header generation unit 51 sets the number of lines included in the packet generated by the packet generation unit 52 to the Line Num of the line header arranged at the first position of the packet, and sets the line number in the packet to the Line Num of the line header arranged at the second or subsequent position of the packet.

[0069] In this way, by providing a Line Num in the line header that indicates the number of lines contained in the packet, an application processor that receives the event data can, for example, detect errors early by comparing the number of lines set in Line Num with the number of lines in a single packet that was actually received.

[0070] With reference to FIG. 7, the distinction between line headers and frame headers will be described.

[0071] 7A, a new identifier (Header Indicator) for identifying a line header and a frame header can be provided in each line header and frame header. For example, in the packet generation processing unit 24, the header generation unit 51 sets an identifier (Header Indicator: 0) indicating a frame header for the header placed at the beginning of the first packet constituting a subframe, and sets an identifier (Header Indicator: 1) indicating a line header for headers other than the header placed at the beginning of the first packet constituting a subframe.

[0072] Alternatively, as shown in B of Fig. 7, the Data ID provided in the packet header can be used to distinguish between line headers and frame headers. The Data ID is a number for identifying the type of data in the packet (e.g., JPEG data, Raw data, etc.). For example, if the Data ID is set to 0, it indicates that the type of data in the packet is JPEG data, and if the Data ID is set to 1, it indicates that the type of data in the packet is Raw data.

[0073] When Data ID is set to 2, this indicates that the type of data in the packet is event data for the first packet that makes up a subframe, and the header located at the beginning of that packet can be identified as a frame header. When Data ID is set to 3, this indicates that the type of data in the packet is event data for the second or subsequent packet that makes up a subframe, and the header located at the beginning of that packet can be identified as a line header. In this way, by using the Data ID in the packet header, it is possible to identify line headers and frame headers more quickly than, for example, using the Header Indicator in the line header or frame header.

[0074] 7C, both the Data ID in the packet header and the identifiers (Header Indicators) in the line and frame headers may be used to distinguish between line and frame headers, thereby enabling error detection based on whether the Data ID and Header Indicator match.

[0075] The padding insertion provided in the line header will be described with reference to FIG.

[0076] For example, when the event data is in bytes, as shown in A of Fig. 8, it is not necessary to insert padding. Therefore, in this case, the header generation unit 51 sets the Padding Insertion field in the line header to 0, which indicates that no padding has been inserted.

[0077] On the other hand, as shown in B of Fig. 8, when the event data is not in bytes, padding is inserted so that the read line length is in bytes. Therefore, in this case, the header generation unit 51 sets Padding Insertion in the line header to 1, which indicates that padding has been inserted, and the packet generation unit 52 inserts padding so that the read line length is in bytes.

[0078] 8C, even if the event data is not in byte units, if padding insertion is not required, padding is not inserted. For example, in a communication standard that does not packetize in byte units, information indicating that padding insertion is not required (Padding_on: 0) is set in an internal register of the packet generation processing unit 24. Therefore, in this case, the header generation unit 51 sets the Padding Insertion stored in the line header to 0, indicating that padding insertion is not performed, and the packet generation unit 52 does not insert padding.

[0079] <Example of Packet Generation Processing> An example of packet generation processing for generating packets in the EVS 11 will be described with reference to the flowcharts shown in FIGS.

[0080] FIG. 9 is a flowchart illustrating a first processing example of the packet generation processing.

[0081] In step S11, the sensor unit 21 outputs luminance signals from a plurality of pixels arranged in the pixel region 31 under the control of the line control unit 32 and the column control unit 33. The event detection unit 22 acquires the luminance signals output from the sensor unit 21 and detects the occurrence of an event, which is a change in the luminance signal. When the event detection unit 22 detects the occurrence of an event, it generates event data indicating the content of the event and supplies one subframe's worth of event data to the event information generation processing unit 23 and the packet generation processing unit 24.

[0082] In step S12, the event counting unit 42 of the event information generating unit 23 calculates the data length of the event data supplied in step S11 from the event detecting unit 22, for each line to be processed, starting from the first line of the subframe. The event counting unit 42 then supplies data length information indicating the data length of the event data for each line to the event number analyzing unit 43.

[0083] In step S13, the event number analysis unit 43 in the event information generation processing unit 23 determines whether the data length of the event data is in bytes based on the data length information supplied from the event count unit 42 in step S12.

[0084] In step S13, if the event number analysis unit 43 determines that the data length of the event data of the line to be processed is in bytes, the process proceeds to step S14.

[0085] In step S14, the event number analysis unit 43 supplies the packet generation processing unit 24 with event number analysis information including a Byte_Sig signal indicating that the data length of the event data of the line to be processed is in bytes.

[0086] In step S15, in the packet generation processing unit 24, the header generation unit 51 sets the Padding Insertion provided in the line header of the line to be processed to 0 in accordance with the Byte_Sig signal, indicating that no padding has been inserted.

[0087] In step S16, in the packet generation processing unit 24, the packet generation unit 52 stores the event data for the line to be processed in a packet without inserting padding in accordance with the Byte_Sig signal. Then, when data of a data size that can be stored in the payload has been stored in the packet, the packet generation unit 52 places a packet footer at the end of the packet to generate a packet and outputs the packet to the data output unit 25.

[0088] On the other hand, if the event number analysis unit 43 determines in step S13 that the data length of the event data of the line to be processed is not in bytes, the process proceeds to step S17.

[0089] In step S17, the event number analysis unit 43 supplies the packet generation processing unit 24 with event number analysis information including a no_Byte_Sig signal indicating that the data length of the event data of the line to be processed is not in bytes.

[0090] In step S18, in the packet generation processing unit 24, the header generation unit 51 sets the Padding Insertion provided in the line header of the line to be processed to 1 in accordance with the no_Byte_Sig signal, indicating that padding has been inserted.

[0091] In step S19, in the packet generation processing unit 24, the packet generation unit 52 inserts padding for the line to be processed in accordance with the no_Byte_Sig signal and stores the event data in a packet. Then, when data of a data size that can be stored in the payload has been stored in the packet, the packet generation unit 52 places a packet footer at the end of the packet to generate a packet and outputs the packet to the data output unit 25.

[0092] After processing step S16 or S19, the process proceeds to step S20, where the event information generation processing unit 23 determines whether or not all of the lines for one subframe supplied from the event detection unit 22 in step S11 have been processed.

[0093] In step S20, if the event information generation processing unit 23 determines that not all lines for one subframe have been processed, the process returns to step S12, and the next line is processed, and the same process is repeated thereafter.

[0094] On the other hand, if the event number analysis unit 43 determines in step S20 that all lines have been processed, the process returns to step S11, and the same process is repeated for the next subframe.

[0095] As described above, even in a communication standard where packetization is performed in byte units, for example, when the event data for each line is not in byte units due to the scene or compression method, the EVS 11 can insert padding to generate packets containing event data so that the read line length is in byte units. This allows the application processor receiving the event data to easily perform encoding or decoding processing because the read line length is in byte units. Furthermore, the application processor can determine whether padding has been inserted according to the padding insertion information provided in the line header.

[0096] FIG. 10 is a flowchart illustrating a second processing example of the packet generation processing.

[0097] In steps S31 to S37, the same processes as in steps S11 to S27 in Fig. 9 are performed. After the process of step S37, the process proceeds to step S38, where the packet generation processing unit 24 determines whether or not it is necessary to insert padding based on the information (Padding_on) set in the internal register.

[0098] If the packet generation processing unit 24 determines in step S38 that padding needs to be inserted (Padding_on: 1), the process proceeds to step S39. In steps S39 and S40, the same processes as in steps S18 and S19 in FIG. 9 are performed.

[0099] On the other hand, if the packet generation processing unit 24 determines in step S38 that it is not necessary to insert padding (Padding_on: 0), the process proceeds to step S41.

[0100] In step S41, the header generating unit 51 of the packet generating unit 24 sets the Padding Insertion stored in the line header of the line to be processed to 0, which indicates that no padding has been inserted.

[0101] In step S42, the packet generation unit 52 in the packet generation processing unit 24 stores the event data for the line being processed in a packet without inserting padding. Then, when data of a data size that can be stored in the payload has been stored in the packet, the packet generation unit 52 places a packet footer at the end of the packet to generate a packet and outputs the packet to the data output unit 25.

[0102] After processing step S36, step S40, or step S42, the process proceeds to step S43, where the event information generation processing unit 23 determines whether all of the lines for one subframe supplied from the event detection unit 22 in step S31 have been processed.

[0103] In step S43, if the event information generation processing unit 23 determines that not all lines for one subframe have been processed, the process returns to step S32, and the next line is processed, and the same process is repeated thereafter.

[0104] On the other hand, if the event number analysis unit 43 determines in step S43 that all lines have been processed, the process returns to step S31, and the same process is repeated for the next subframe.

[0105] As described above, the EVS 11 can generate packets that store event data without inserting padding, for example, in a communication standard that does not packetize in byte units.

[0106] <Example of Processing in Which One Line of Event Data is Storing Across Packets> A processing in which one line of event data is stored across packets will be described with reference to FIGS.

[0107] In the EVS 11, event data is generated in response to the occurrence of an event, so the data size of one line of event data is not necessarily constant. On the other hand, the maximum data size that can be stored in the payload of one packet (hereinafter referred to as one packet size) is set to a constant size. Therefore, when multiple lines of event data are stored in a packet and the amount of data exceeds one packet size, it is necessary to generate the packet so that one line of event data is stored across packets.

[0108] The packet generator 52 stores the remaining part of the event data that exceeds the size of one packet of the previous packet and cannot be stored in the previous packet, across packets, in the next packet. At this time, the header generator 51 sets the Data Straddle provided in the line header placed at the beginning of the event data stored in the previous packet to 1, indicating that the event data for one line is stored across the next packet.

[0109] FIG. 11A is a diagram illustrating a case where one line of event data is stored without spanning packets.

[0110] For example, when the packet generation unit 52 stores the first line of event data in a packet, if the data size of the data stored in the packet at that time (hereinafter referred to as the valid data size of the packet) is equal to or smaller than a threshold set in a register, for example, the packet generation unit 52 determines that the second line of event data should be stored in the packet. Then, if the valid data size of the packet at the time the second line of event data is stored in the packet exceeds the threshold, the packet generation unit 52 inserts padding into the remaining payload of the packet and places a packet footer at the end of the packet to generate the packet.

[0111] In other words, even if the effective data size of the packet at the time the second line of event data is stored in the packet is less than the size of one packet and there is remaining data in the packet payload, the event data of the next line is not stored and the payload is filled with padding to make it one packet size. In this way, when one line of event data is stored without spanning across packets, the header generation unit 51 sets 0 to the Data Straddle of the line header, indicating that one line of event data is not stored spanning across the next packet.

[0112] FIG. 11B is a diagram illustrating a case where one line of event data is stored across packets.

[0113] For example, if the effective data size of a packet at the time when the first line of event data is stored in the packet is equal to or smaller than a threshold, the packet generation unit 52 determines that the second line of event data should be stored in that packet. If the effective data size of a packet at the time when the second line of event data is stored in the packet exceeds the size of one packet, the packet generation unit 52 stores the remaining portion of the event data that exceeds the size of one packet and cannot be stored in the packet in the next packet across packets. In this way, when one line of event data is stored across packets, the header generation unit 51 sets the Data Straddle of the line header located at the beginning of the portion of event data stored in the previous packet to 1, indicating that one line of event data is stored across packets.

[0114] FIG. 12 is a flowchart illustrating a third example of the packet generation process.

[0115] In step S51, the sensor unit 21 outputs luminance signals from a plurality of pixels arranged in the pixel region 31 under the control of the line control unit 32 and the column control unit 33. The event detection unit 22 acquires the luminance signals output from the sensor unit 21 and detects the occurrence of an event, which is a change in the luminance signal. When the event detection unit 22 detects the occurrence of an event, it generates event data indicating the content of the event and supplies one subframe's worth of event data to the event information generation processing unit 23 and the packet generation processing unit 24.

[0116] In step S52, the packet generation processing unit 24 stores, in the packet to be processed, one line of event data among the one subframe of event data supplied in step S51 from the event detection unit 22. For example, the packet generation processing unit 24 stores, in the packet, the event data of multiple lines of event data included in one subframe, in order from the first line.

[0117] In step S53, the packet generation processing unit 24 determines whether the effective data size of the packet to be processed is larger than a threshold value.

[0118] In step S53, if the packet generation processing unit 24 determines that the effective data size of the packet to be processed is not greater than the threshold (i.e., is less than the threshold), the processing returns to step S52, the next line of event data is stored in the packet to be processed, and the same processing is repeated thereafter.

[0119] On the other hand, if the packet generation processing unit 24 determines in step S53 that the effective data size of the packet to be processed is larger than the threshold value, the process proceeds to step S54.

[0120] In step S54, the packet generation processing unit 24 determines whether the effective data size of the packet to be processed is less than the size of one packet.

[0121] If the packet generation processing unit 24 determines in step S54 that the effective data size of the packet to be processed is less than the size of one packet, the process proceeds to step S55.

[0122] In step S55, the packet generation processing unit 24 inserts padding into the remaining payload of the packet to be processed, filling the payload with padding to make it one packet size.

[0123] In step S56, the packet generation processing unit 24 generates a packet by placing a packet footer at the end of the packet to be processed, the payload of which has been filled with padding, and outputs the packet to the data output unit 25. At this time, the header generation unit 51 sets 0 to Data Straddle in the line header.

[0124] On the other hand, if the packet generation processing unit 24 determines in step S54 that the effective data size of the packet to be processed is not less than one packet size (i.e., is equal to or greater than one packet size), the process proceeds to step S57.

[0125] In step S57, the packet generation processing unit 24 stores a portion of the event data that can be stored up to the size of one packet of the packet to be processed, places a packet footer at the end of the packet, generates the packet, and outputs it to the data output unit 25. At this time, the header generation unit 51 sets 1 to Data Straddle in the line header.

[0126] In step S58, the packet generation processing unit 24 carries over the remaining part of the event data that could not be stored in the packet to be processed from the beginning of the next packet to be processed, and the process returns to step S52. In other words, in this case, in step S52, this carried-over part of the event data is stored in the next packet, and the same process is repeated thereafter.

[0127] After the process of step S56, the process proceeds to step S59, where the packet generation processing unit 24 determines whether or not the event data of all lines for one subframe supplied from the event detection unit 22 in step S51 has been stored in the packet.

[0128] In step S59, if the packet generation processing unit 24 determines that the event data for all lines of one subframe has not been stored in the packet, the processing returns to step S52, the event data for the next line is stored in the packet, and the same processing is repeated thereafter.

[0129] On the other hand, if in step S59 the packet generation processing unit 24 determines that the event data for all lines of one subframe has been stored in the packet, the processing returns to step S51, and the same processing is repeated for the next subframe.

[0130] A case where one line of event data is stored without spanning packets will be described with reference to FIG.

[0131] As shown in A of FIG. 13, the packet generation unit 52 stores the event data of the first line in a packet in step S52 of FIG. 11, and as shown in B of FIG. 13, if the effective data size of the packet is not larger than the threshold, the result is determined to be No in step S53 of FIG. 11, and the processing returns to step S52.

[0132] As shown in C of FIG. 13, the packet generation unit 52 stores the event data of the second line in a packet in step S52 of FIG. 11, and as shown in D of FIG. 13, if the effective data size of the packet is larger than the threshold, the result of determination in step S53 of FIG. 11 is Yes, and the processing proceeds to step S54.

[0133] If the effective data size of the packet is less than one packet size, as shown in E of Fig. 13, the packet generation unit 52 determines "Yes" in step S54 of Fig. 11, and the process proceeds to step S55. In step S55, the packet generation unit 52 inserts padding into the remaining payload of the packet until it reaches the packet size.

[0134] Then, in step S56, the packet generation unit 52 generates a packet by placing a packet footer at the end of the packet whose payload has been filled with padding, and outputs the packet as shown in F of Fig. 13. At this time, the header generation unit 51 sets 0 to the Data Straddle of the line header of the event data of the second line.

[0135] A case where one line of event data is stored across packets will be described with reference to FIG.

[0136] As shown in A of FIG. 14, the packet generation unit 52 stores the event data of the first line in a packet in step S52 of FIG. 11, and as shown in B of FIG. 14, if the effective data size of the packet is not larger than the threshold, the result is determined to be No in step S53 of FIG. 11, and the processing returns to step S52.

[0137] As shown in C of FIG. 14, in step S52 of FIG. 11, the packet generation unit 52 stores the event data of the second line in a packet, and if the effective data size of the packet is greater than the threshold, the result of step S53 of FIG. 11 is Yes, and the processing proceeds to step S54.

[0138] If the valid data size of the packet is not less than one packet size, as shown in D of Fig. 14, the packet generation unit 52 determines No in step S54 of Fig. 11 and proceeds to step S57. In step S57, as shown in E of Fig. 14, the packet generation unit 52 stores a portion of the event data that can be stored up to the packet size of one packet, places a packet footer at the end of the packet, generates a packet, and outputs it to the data output unit 25. At this time, the header generation unit 51 sets 1 to Data Straddle in the line header.

[0139] Then, in the next step S52, the packet generating unit 52 stores the carried-over part of the event data in the next packet, as shown in F of FIG.

[0140] <Application Examples> Application examples will be described with reference to FIGS.

[0141] FIG. 15 is a diagram illustrating an application example in which padding and dummy data output are omitted.

[0142] For example, the EVS 11 can generate event data when the event detection unit 22 detects the occurrence of an event, and output a packet containing the event data. Therefore, when the event detection unit 22 does not detect the occurrence of an event, the EVS 11 has no event data to output.

[0143] On the other hand, EVS 11 must control the output of event data to satisfy conditions such as uniform packet size (equal length) and periodic packet output with a constant packet output interval, in accordance with constraints for facilitating processing in the application processor (not shown) that receives the event data. Therefore, as shown in the upper part of FIG. 15 , EVS 11 controls the output of packets by inserting padding to make the packet data size constant, or by generating and transmitting packets containing dummy data, thereby periodically outputting packets. The padding and dummy data may be, for example, signals consisting of all "0"s or all "1"s, or signals such as "010101...".

[0144] On the other hand, the system may be configured so that it is not necessary to satisfy conditions such as uniform data size (equal length) for each packet and periodic packet output with a constant output interval between packets. In this case, the EVS 11 can omit padding and dummy data output, as shown in the lower part of Figure 15. Furthermore, the EVS 11 may output a packet after a predetermined amount of event data (e.g., one frame's worth of event data) has accumulated. This eliminates the need for padding, as shown in Figure 16.

[0145] An application example in which a time stamp is stored in embedded data of image data will be described with reference to FIG.

[0146] For example, in a configuration in which image data is output from an image sensor (not shown) and event data is output from the EVS 11 in parallel, a timestamp that is common to the image sensor and the EVS 11 can be used.

[0147] For example, a timestamp can be stored in the embedded data (EMB) included in the first packet of a frame of image data output from the image sensor, and a timestamp can be stored in the frame header of the event data output from the EVS 11. By using a timestamp common to the image sensor and the EVS 11 in this way, an application processor (not shown) that receives the event data can easily understand the relative relationship between the image data and the event data for each subframe. For example, the application processor can easily understand which subframe corresponds to the beginning of the image data frame.

[0148] 17, the timestamp stored in the embedded data included in the first packet of the second frame of image data indicates 1050. The timestamp stored in the frame header of the last subframe of the first frame of event data indicates 1000, and the timestamp stored in the frame header of the first subframe of the second frame of event data indicates 1100.

[0149] Therefore, the application processor can easily understand that the subframe in which the timestamp indicating 1100 is stored corresponds to the beginning of the frame of image data in which the timestamp indicating 1050 is stored.

[0150] As shown in Fig. 5, the timestamp is stored in the frame header and is output in subframe units, meaning that all event data within a subframe indicates the same time (count).

[0151] Therefore, as shown in A of Fig. 18, by allocating a timestamp area (bits) in the line header, the timestamp can be switched on a line-by-line basis, making it possible to have time information on a line-by-line basis.

[0152] Also, by having timestamp bits in both the frame header and the line header, for example, the upper bits may be assigned to the frame header timestamp and the lower bits to the line header timestamp, which makes it possible to reduce the bit width in the line header.

[0153] For example, if 16-bit information is required for the timestamp, and a 16-bit area is reserved for the line header timestamp, as shown in B of Figure 18, a 16-bit timestamp must be sent for each line header.

[0154] In contrast, as shown in Figure 18C, if the frame header timestamp and line header timestamp form a 16-bit area, the line header timestamp area can be reduced to 8 bits by utilizing the frame header area.

[0155] The data unit of the event data will be described with reference to FIG.

[0156] As described above, in this embodiment, the process of storing event data in packets in units of lines of the pixel region 31 (see A in Fig. 19 ) has been described, but the process is not limited to such units of lines. For example, instead of units of lines, event data may be stored in packets in units of columns of the pixel region 31, as shown in B in Fig. 19 , or in units of groups of the pixel region 31, as shown in C in Fig. 19 .

[0157] The present technology is not limited to the EVS 11 that generates event data, but may also be applied to a system that transmits data whose output timing is irregular and whose data length for each predetermined data unit is not constant. For example, the present technology can also be applied to sensors that do not output data aperiodically, such as edge AI (artificial intelligence) sensors, or sensors whose data length is variable.

[0158] Furthermore, the EVS11 can employ, for example, either a scan-type event detection unit 22 that outputs event data regardless of whether an event occurs or not, or an arbiter-type event detection unit 22 that outputs event data only when an event occurs.

[0159] <SLVS-EC> The SLVS-EC, which is one of the high-speed communication IFs, will be described with reference to FIGS. 20 and 21. FIG.

[0160] In the above-described embodiment, the EVS 11 transmits data in accordance with MIPI D-PHY. However, the present technology can be applied to other standards, such as SLVS-EC (Scalable Low Voltage Signaling with Embedded Clock), which is one of the high-speed communication IFs.

[0161] For example, in the SLVS-EC, the data output unit 25 of the EVS 11 transmits data for each pixel generated based on the output of the event detection unit 22. Hereinafter, the transmission path for transmitting data from the EVS 11 to the application processor will be referred to as a lane, where appropriate.

[0162] The application processor receives the pixel data transmitted via the transmission path from the EVS 11. In this manner, data is transmitted and received between the EVS 11 and the application processor.

[0163] In the SLVS-EC, an application layer, a link layer, and a physical layer are defined according to the content of signal processing. The link layer processing and the physical layer processing are performed by the EVS 11 and the application processor, respectively.

[0164] Link layer processing, for example, involves processing to achieve the following functions: 1. Pixel data - byte data conversion 2. Payload data error correction 3. Transmission of packet data and auxiliary data 4. Payload data error correction using packet footers 5. Lane management 6. Protocol management for packet generation Meanwhile, physical layer processing, for example, involves processing to achieve the following functions: 1. Control code generation and extraction 2. Bandwidth control 3. Skew control between lanes 4. Symbol placement 5. Symbol coding for bit synchronization 6. SERDES (SERializer / DESerializer) 7. Clock generation and recovery 8. SLVS (Scalable Low Voltage Signaling) signal transmission

[0165] FIG. 20 is a diagram showing an example of a format used for data transmission in SLVS-EC.

[0166] The effective pixel area is the area of ​​effective pixels in one frame of an image captured by the image sensor. A margin area is arranged on the left side of the effective pixel area.

[0167] A front dummy area is arranged above the effective pixel area. In the example of Fig. 20, Embedded Data is arranged in the front dummy area. The Embedded Data includes information on setting values ​​related to image capture by the image sensor, such as shutter speed, aperture value, and gain. In addition to information on setting values ​​related to image capture, various additional information such as contents, format, and data size is also arranged as Embedded Data as appropriate. The Embedded Data is additional information added to the image data of each frame.

[0168] A rear dummy area is arranged below the effective pixel area, and embedded data may be arranged in the rear dummy area.

[0169] The image data area is composed of the effective pixel area, margin area, front dummy area, and rear dummy area.

[0170] A header is added before each line that makes up the image data area, and a start code is added before the header. In addition, a footer is optionally added after each line that makes up the image data area, and a control code such as an end code is added after the footer. If a footer is not added, a control code such as an end code is added after each line that makes up the image data area.

[0171] For each frame of image generated based on the output of the event detection unit 22, data transmission is performed using frame data in the format shown in FIG.

[0172] The upper band in Figure 20 shows the structure of packets used to transmit the frame data shown below. If the horizontal data arrangement is considered to be a line, the payload of a packet stores data that constitutes one line of the image data area. The entire frame data of one frame is transmitted using a number of packets equal to or greater than the number of pixels in the vertical direction of the image data area. Furthermore, the entire frame data of one frame is transmitted by sending packets that store data on a line-by-line basis, for example, starting with the data arranged on the top line.

[0173] A packet is constructed by adding a header and a footer to a payload that stores one line of data. At least a start code and an end code, which are control codes, are added to each packet.

[0174] As shown in the lower left of FIG. 20, the header contains additional information about the data stored in the payload, such as Frame Start, Frame End, Line Valid, and Line Number.

[0175] Frame Start is 1-bit information that indicates the beginning of a frame. The Frame Start of the header of a packet used to transmit the first line of frame data is set to a value of 1, and the Frame Start of the header of a packet used to transmit the other lines of data is set to a value of 0.

[0176] Frame End is 1-bit information indicating the end of a frame. The Frame End of the header of a packet containing the data of the last line of frame data is set to a value of 1, and the Frame End of the header of a packet used to transmit data of other lines is set to a value of 0.

[0177] "Line Valid" is one bit of information that indicates whether the line of data stored in the packet is a line in the valid pixel area. A value of 1 is set to "Line Valid" in the header of a packet used to transmit pixel data of a line within the valid pixel area, and a value of 0 is set to "Line Valid" in the header of a packet used to transmit data of other lines.

[0178] The Line Number is 13-bit information that indicates the line number of the line on which the data stored in the packet is arranged.

[0179] Even when EVS11 is a high-speed communication IF that complies with a standard different from SLVS-EC, frame data containing image data for each frame is generated, and data transmission is carried out using packets that store the data for each line of the frame data.

[0180] FIG. 21 is a diagram showing an example of a packet structure when this technology is applied to SLVS-EC.

[0181] As shown in Fig. 21, in the packet structure when this technology is applied to SLVS-EC, a packet is configured with a start code and packet header placed before the frame header and line header, and an end code, deskew code, and idle code placed at the end of the event data (and padding). Also, embedded data is stored in the first packet, and blanking data is added to the top and bottom of these packets.

[0182] <Regarding Error Detection Information> As described above with reference to FIG. 4, in the present technology, error detection information can be stored in the frame footer and the line footer.

[0183] For example, the error detection information stored in the frame footer and line footer is used to detect and correct errors in the timestamps (see FIG. 18 described above) stored in the line headers or frame headers of the event data. If an error occurs in each subframe or each line, the error detection information can be used to detect and correct the error. In particular, in this technology, it is important to detect and correct errors that occur in each line.

[0184] For example, image data output from an image sensor conventionally contains error detection information such as ECC, but does not usually contain a timestamp, making it impossible to detect and correct errors in the timestamp. Also, in conventional EVS, error detection information is stored in the packet footer, so while it is possible to detect and correct errors in the timestamp within a packet, it is not possible to detect and correct errors in the timestamp of each subframe or each line.

[0185] In this way, by storing error detection information in the frame footer and line footer and making it possible to detect and correct errors that occur in each subframe or each line, this technology can be applied to applications that require greater error tolerance, for example.

[0186] For example, if a bandwidth can be secured in the frame header, line header, etc., the error detection information may be stored in the frame header, line header, etc. In the above-mentioned SLVS-EC, error countermeasures for packet data are performed by the ECC option or footer CRC, but the error detection information may also be stored in the footer or header.

[0187] <Example of Use of Image Sensor> FIG. 22 is a diagram showing an example of use of the image sensor (EVS) described above.

[0188] The image sensor described above can be used in various cases for sensing light such as visible light, infrared light, ultraviolet light, and X-rays, for example, as follows.

[0189] ・Devices for taking images for viewing purposes, such as digital cameras and mobile devices with camera functions. ・Devices for traffic purposes, such as in-vehicle sensors that take images of the front, rear, surroundings, and interior of a car for safe driving such as automatic stopping, and for recognizing the driver's state, surveillance cameras that monitor moving vehicles and roads, and distance measuring sensors that measure distances between vehicles. ・Devices for home appliances such as TVs, refrigerators, and air conditioners that take images of user gestures and operate the device according to those gestures. ・Devices for medical and healthcare purposes, such as endoscopes and devices that take images of blood vessels by receiving infrared light. ・Devices for security purposes, such as surveillance cameras for crime prevention and cameras for person authentication. ・Devices for beauty purposes, such as skin measuring devices that take images of the skin and microscopes that take images of the scalp. ・Devices for sports purposes, such as action cameras and wearable cameras for sports, etc. ・Devices for agricultural purposes, such as cameras to monitor the condition of fields and crops.

[0190] <Examples of Combinations of Configurations> The present technology can also be configured as follows. (1) A data processing device comprising: a data generation unit that generates data whose output timing is irregular and whose data length for each predetermined data unit is not constant; and a packet generation processing unit that generates packets storing the data in one or more of the data units, wherein the packet generation processing unit generates the packets with a packet structure in which one of multiple types of headers provided when storing the data in the packets is always placed at the beginning of the payload of each of the packets. (2) The data generation unit detects the occurrence of an event that is a change in a luminance signal that indicates the luminance of light received by multiple pixels arranged in an array, and generates, as the data, event data that indicates the content of the event, and the packet generation processing unit always places, at the beginning of the payload of each of the packets, either a line header that is provided for each line of the pixels, with the line of the pixels being the data unit, or a frame header that is provided for each sub-frame corresponding to one screen of a pixel area in which the pixels are provided. (3) The data processing device according to (2) above, wherein the packet generation processing unit places the frame header at the beginning of the payload of the first packet of the subframe, and also stores the event data for each line in the payload of that packet. (4) The data processing device according to (2) or (3) above, wherein the packet generation processing unit places the frame header at the beginning of the payload of the first packet of the subframe, and stores dummy data in the payload of that packet without storing the event data. (5) The data processing device according to any of (2) to (4) above, wherein the frame header is provided with information indicating the number of valid lines, which is the number of lines including the pixel in which the event occurred in that subframe. (6) The data processing device according to any of (2) to (5) above, wherein the line header placed first in the packet is provided with information indicating the number of lines included in that packet.(7) The data processing device according to (6) above, wherein the line headers arranged second or later in the packet are provided with information indicating the line number within that packet. (8) The data processing device according to any of (2) to (7) above, wherein the packet generation processing unit arranges a frame footer at the end of the payload of the last packet constituting the subframe, and arranges a line footer at the end of the event data for each line. (9) The data processing device according to (8) above, wherein the frame footer is provided with information indicating the number of valid lines, which is the number of lines including the pixel in which the event occurred in that subframe. (10) The data processing device according to any of (2) to (9) above, wherein the frame header and the line header are provided with identifiers for identifying the line header and the frame header, respectively. (11) The data processing device according to any of (2) to (10) above, wherein the packet generation processing unit, if the event data is not in byte units, inserts padding so that the read line length is in byte units. (12) The data processing device according to (11) above, wherein the line header is provided with an identifier for identifying whether the padding has been inserted so that the read line length is in bytes. (13) The data processing device according to any of (2) to (12) above, wherein, when the packet generation processing unit is unable to store one line of the event data in the previous packet, the packet generation processing unit stores the remaining part of the event data that cannot be stored in the previous packet in the next packet, and the line header is provided with an identifier indicating whether the one line of the event data is stored across the next packet. (14) The data processing device according to any of (2) to (13) above, wherein the packet generation processing unit generates the packet storing dummy data when the event does not occur so that the packet output intervals are constant. (15) The data processing device according to any of (2) to (14) above, wherein the frame header is provided with a timestamp for counting time.(16) The data processing device according to (15), wherein the timestamp is common to a timestamp stored in embedded data included in the first packet of a frame of image data output from an image sensor in parallel with the output of the event data. (17) The data processing device according to (15) or (16), wherein higher-order bits of the timestamp are assigned to the frame header and lower-order bits of the timestamp are assigned to the line header. (18) A data processing method, comprising: a data processing device generating data whose output timing is irregular and whose data length for each predetermined data unit is not constant; and generating packets that store the data in one or more of the data units, wherein the packets are generated in a packet structure in which one of a plurality of types of headers provided when storing the data in the packet is always placed at the beginning of the payload of each of the packets. (19) An event data output sensor comprising: a data generation unit that detects the occurrence of an event, which is a change in a luminance signal indicating the luminance of light received by a plurality of pixels arranged in an array, and generates event data indicating the content of the event; and a packet generation processing unit that uses a line of the pixels as a data unit of the event data and generates a packet storing the event data of one or a plurality of the lines, wherein the packet generation processing unit generates the packet with a packet structure in which a header that is provided when storing the event data in the packet, either a line header provided for each line, or a frame header provided for each sub-frame corresponding to one screen of a pixel area in which the pixels are provided, is always placed at the beginning of the payload of each of the packets.

[0191] It should be noted that the present embodiment is not limited to the above-described embodiment, and various modifications are possible within the scope of the gist of the present disclosure. Furthermore, the effects described in this specification are merely examples and are not intended to be limiting, and other effects may also be obtained.

[0192] REFERENCE SIGNS LIST 11 EVS, 21 Sensor unit, 22 Event detection unit, 23 Event information generation processing unit, 24 Packet generation processing unit, 25 Data output unit, 31 Pixel area, 32 Line control unit, 33 Column control unit, 41 Event / line count unit, 42 Event count unit, 43 Event number analysis unit, 44 Data amount calculation unit, 45 Compression unit, 51 Header generation unit, 52 Packet generation unit

Claims

1. A data processing device comprising: a data generation unit that generates data whose output timing is irregular and whose data length for each specified data unit is not constant; and a packet generation processing unit that generates packets that store the data in one or more of the data units, wherein the packet generation processing unit generates the packets in a packet structure in which one of multiple types of headers that are provided when storing the data in the packet is always placed at the beginning of the payload of each of the packets.

2. The data processing device according to claim 1, wherein the data generation unit detects the occurrence of an event, which is a change in a luminance signal indicating the luminance of light received by a plurality of pixels arranged in an array, and generates event data indicating the content of the event as the data, and the packet generation processing unit always places, at the beginning of the payload of each of the packets, either a line header provided for each line of pixels, with the line being the data unit, or a frame header provided for each sub-frame corresponding to one screen of the pixel area in which the pixels are arranged.

3. The data processing device according to claim 2, wherein the packet generation processing unit places the frame header at the beginning of the payload of the first packet of the subframe, and stores the event data for each line in the payload of that packet as well.

4. The data processing device according to claim 2, wherein the packet generation processing unit places the frame header at the beginning of the payload of the first packet of the subframe, and stores dummy data in the payload of that packet instead of the event data.

5. The data processing device according to claim 2, wherein the frame header includes information indicating the number of valid lines, which is the number of lines that include the pixel in which the event occurred in that subframe.

6. The data processing device according to claim 2, wherein the line header arranged first in the packet is provided with information indicating the number of lines included in the packet.

7. The data processing device according to claim 6, wherein the second or subsequent line header in the packet contains information indicating the number of the line within the packet.

8. The data processing device according to claim 2, wherein the packet generation processing unit places a frame footer at the end of the payload of the last packet that constitutes the subframe, and places a line footer at the end of the event data for each line.

9. The data processing device according to claim 8, wherein the frame footer is provided with information indicating the number of valid lines, which is the number of lines that include the pixel in which the event occurred in that subframe.

10. The data processing device according to claim 2, wherein the frame header and the line header are provided with identifiers for identifying the line header and the frame header, respectively.

11. The data processing device according to claim 2, wherein, when the event data is not in byte units, the packet generation processing section inserts padding so that the read line length is in byte units.

12. The data processing device according to claim 11, wherein the line header is provided with an identifier for identifying whether or not the padding has been inserted so that the read line length is in bytes.

13. The data processing device according to claim 2, wherein, when the packet generation processing unit is unable to store one line of the event data in the previous packet, the remaining portion of the event data that cannot be stored in the previous packet is stored in the next packet, and the line header is provided with an identifier indicating whether one line of the event data is stored across the next packet.

14. The data processing device according to claim 2, wherein said packet generation processing unit generates said packets containing dummy data when said event does not occur so that said packet output intervals are constant.

15. The data processing device according to claim 2, wherein the frame header is provided with a timestamp for counting time.

16. The data processing device according to claim 15, wherein the timestamp is the same as a timestamp stored in embedded data included in the first packet of a frame of image data output from an image sensor in parallel with the output of the event data.

17. The data processing device according to claim 15, wherein the most significant bits of the timestamp are assigned to the frame header, and the least significant bits of the timestamp are assigned to the line header.

18. A data processing method comprising: a data processing device generating data whose output timing is irregular and whose data length for each predetermined data unit is not constant; and generating packets that store the data in one or more of the data units, wherein the packets are generated in a packet structure in which one of multiple types of headers provided when storing the data in the packet is always placed at the beginning of the payload of each of the packets.

19. An event data output sensor comprising: a data generation unit that detects the occurrence of an event, which is a change in a luminance signal indicating the luminance of light received by a plurality of pixels arranged in an array, and generates event data indicating the content of the event; and a packet generation processing unit that generates packets storing the event data for one or a plurality of said lines, with said pixel lines being the data unit of said event data, wherein said packet generation processing unit generates said packets with a packet structure in which a header that is provided when storing the event data in said packets, either a line header provided for each said line, or a frame header provided for each sub-frame equivalent to one screen of the pixel area in which said pixels are provided, is always placed at the beginning of the payload of each said packet.

Images