Sensor device, data structure, and information processing device

Abstract

A sensor device according to the present invention is provided with: a pixel array unit in which pixels are two-dimensionally arranged; a determination processing unit that determines whether or not each line data output from the pixel array unit is effective line data to be processed in subsequent image processing; an additional information generation unit that generates additional information on the basis of the result of the determination by the determination processing unit; an output data generation unit that generates output data for each line by using the line data and the additional information; and an output unit that outputs the output data.

Description

Sensor device, data structure, information processing device

[0001] The present technology relates to a sensor device, a data structure, and an information processing device for data for each line output from a pixel array unit.

[0002] Systems are known that perform image processing on image data to detect objects, defects, etc. In such systems, it is sometimes necessary to perform image processing efficiently. For example, Patent Document 1 listed below discloses a technology for adding a flag to image data that indicates the shift direction of the image.

[0003] Japanese Patent Application Laid-Open No. 2000-004404

[0004] Incidentally, high-speed image sensors exceeding 30 fps or 60 fps are becoming widespread. When performing image processing on frame images as image data output at such high speeds, it is desirable to reduce the amount of calculation required for the image processing.

[0005] The present technology has been made in view of such problems, and aims to reduce the processing load of image processing on frame images.

[0006] A sensor device according to the present technology includes a pixel array unit in which pixels are two-dimensionally arranged, a determination processing unit that determines whether each line data output from the pixel array unit is valid line data to be processed in a subsequent image processing stage, an additional information generation unit that generates additional information based on the result of the determination by the determination processing unit, an output data generation unit that generates output data for each line using the line data and the additional information, and an output unit that outputs the output data, thereby enabling a subsequent information processing device to determine whether or not to perform image processing for each line data.

[0007] The data structure related to the present technology is a data structure of output data output from a sensor device having a pixel array section in which pixels are arranged two-dimensionally, and is generated based on line data output from the pixel array section and additional information indicating the determination result of whether or not each of the line data is valid line data to be processed in subsequent image processing.

[0008] The information processing device according to the present technology includes a receiving processing unit that receives output data from a sensor device, the output data being generated based on line data output from a pixel array unit in which pixels are arranged two-dimensionally and additional information indicating a determination result as to whether or not each of the line data is valid line data to be processed in subsequent image processing; a judgment processing unit that determines whether or not a predetermined image processing can be performed on the line data based on the additional information included in the output data received by the receiving processing unit; and an image processing unit 15 (15A, 15B) that performs the predetermined image processing on the line data based on the determination result of the judgment processing unit.

[0009] FIG. 1 is a block diagram showing an example of the configuration of a detection system for the present technology, and is a block diagram showing the configuration of the detection system for first and second embodiments. FIG. 2 is a diagram showing an overview of the detection system. FIG. 3 is a diagram showing an example of determining whether a target subject is imaged for each pixel block region. FIG. 4 is a diagram for explaining changes in average luminance value between frames in a pixel block region in which a target subject is imaged. FIG. 5 is a diagram showing an example of additional information for each line. FIG. 6 is a diagram showing an example of the data structure of output data. FIG. 7 is a diagram showing the data structure of packet data compliant with the SLVS-EC standard. FIG. 8 is a diagram for explaining that multiple pixel data are stored in the payload region of packet data. FIG. 9 is a flowchart showing an example of processing executed by the detection system regarding detection of a target subject and generation of additional information, etc. FIG. 10 is a flowchart showing an example of processing executed by an outside-sensor processing unit. FIG. 11 is a flowchart showing an example of processing executed by the detection system regarding identification of a processing target region and image processing. FIG. 12 is a diagram for explaining an example of further setting an effective range and an invalid range in effective line data. FIG. 13 is a diagram showing an example of a signal output for distinguishing between an effective range and an invalid range. FIG. 14 is a diagram for explaining a processing target region that is set as the range to be processed in subsequent image processing. FIG. 15 is a block diagram showing the configuration of a detection system for a third embodiment.

[0010] Hereinafter, with reference to the accompanying drawings, embodiments of the present technology will be described in the following order: <1. First embodiment> <1-1. System configuration> <1-2. Process of generating additional information> <1-3. Regarding output data> <1-4. Processing flow> <2. Second embodiment> <3. Third embodiment> <4. Modification> <5. Summary> <6. Present technology>

[0011] 1. First embodiment A first embodiment of a sensor device 1 of the present technology will be described. The sensor device 1 is included in a camera device 2, and configures a detection system S together with an information processing device 3 at a subsequent stage.

[0012] <1-1. System Configuration> The detection system S detects a subject to be detected in a captured image obtained by imaging, determines an area to be processed in subsequent image processing, and performs image processing on a partial area in the frame image FG based on the subject detection result and the determined area. The detection system S is applied to, for example, a system for detecting defects in products, or a system for reading one-dimensional codes or two-dimensional codes attached to a subject.

[0013] FIG. 1 shows an example of the configuration of a detection system S according to the first embodiment.

[0014] The detection system S includes a camera device 2 and an information processing device 3 (see FIGS. 1 and 2).

[0015] The camera device 2 includes a sensor device 1 and an external processing unit 4 at a subsequent stage.

[0016] The sensor device 1 is configured as, for example, a CCD (Charge Coupled Device) type or CMOS (Complementary Metal Oxide Semiconductor) type image sensor, and includes an imaging unit 5, a detection processing unit 6, an additional information generation unit 7, an output data generation unit 8, and an output unit 9.

[0017] The imaging unit 5 includes a pixel array section 5a in which pixels having photoelectric conversion elements as light receiving sections such as photodiodes are arranged two-dimensionally, and a readout circuit that reads out electrical signals obtained by photoelectric conversion from each pixel included in the pixel array section 5a, and is capable of outputting the electrical signals as imaging data in the form of RAW data.

[0018] The imaging data output from the imaging unit 5 is, for example, data for each line that combines data for multiple pixels included in the same row (line). This data for each line is referred to as "line data LD." Note that data for one pixel included in the line data LD may also be referred to as "pixel data."

[0019] The line data LD output from the pixel array unit 5a may be not only data such as color images or monochrome images, but also two-dimensional data having values ​​for each pixel, such as distance images, EVS data, heat map images, spectral images, and polarized images.

[0020] For example, a downstream image processing unit can perform image processing to inspect the quality of a product based on imaging data such as color images or monochrome images. Furthermore, a downstream image processing unit can perform image processing to detect defects (holes, chips, etc.) in the shape of a product based on imaging data as a distance image. Furthermore, a downstream image processing unit can perform image processing to detect the degree of doneness of a food product based on imaging data as a heat map image.

[0021] In addition, it is possible to perform image processing in a downstream image processing unit to detect defects in the color of a product based on imaging data such as spectral images and polarized images. These are merely examples, and image processing for various other purposes is also possible.

[0022] In this embodiment, the image processing unit at the subsequent stage is provided in an information processing device 3 provided outside the camera device 2.

[0023] The readout circuit included in the pixel array unit 5a performs, for example, CDS (Correlated Double Sampling) processing, AGC (Automatic Gain Control) processing, etc. on the electrical signals obtained by photoelectric conversion, and further performs A / D (Analog / Digital) conversion processing.

[0024] The image data after A / D conversion is output to the detection processing unit 6 .

[0025] The imaging data provided to the detection processing unit 6 is digital data (RAW data) that has undergone A / D conversion processing and is output from the imaging unit 5. Alternatively, data that has undergone preprocessing or the like on the RAW data may be provided to the detection processing unit 6.

[0026] In the pre-processing, clamping processing for clamping the R, G, and B black levels to a predetermined level and correction processing between the R, G, and B color channels are performed on the imaging data.

[0027] The detection processing section 6 performs processing for detecting a target subject OB that is to be subjected to subsequent image processing for each line data LD.

[0028] Various detection methods can be considered for the detection processing unit 6, and for example, detection may be performed for each line data LD. Alternatively, as will be described later, the detection processing unit 6 may divide data (frame image FG as RAW data) obtained by collecting each line data LD into one frame of image data into pixel block regions Ar, and detect the target subject OB for each pixel block region Ar.

[0029] The additional information generating unit 7 generates additional information AD for each line data LD, which indicates whether or not the target subject OB is included, based on the detection result of the detection processing unit 6. The determination result as the additional information AD can be expressed by 1-bit flag information.

[0030] The output data generation unit 8 generates output data OD to be output to a subsequent processing unit using the line data LD and additional information AD. The generated output data OD may be, for example, data conforming to the SLVS (Scalable Low Voltage Signaling) standard (including SLVS-EC (SLVS with Embedded Clock)), data conforming to the MIPI (Mobile Industry Processor Interface) standard, or data having any other configuration.

[0031] The output unit 9 outputs the output data OD generated by the output data generation unit 8 to a subsequent processing unit, specifically to the extra-sensor processing unit 4. The output data OD output from the output unit 9 to the outside of the sensor device 1 is data including one line of line data LD and its additional information AD. That is, the output unit 9 outputs the same number of output data OD as the number of line data LD for one frame of imaging.

[0032] The outside-sensor processing unit 4 is configured to include a sensor image receiving unit 10 , a camera signal processing unit 11 , and a camera image transmitting unit 12 .

[0033] The sensor image receiving unit 10 receives output data OD as a sensor image output from the sensor device 1, and supplies it to a camera signal processing unit 11 at the subsequent stage.

[0034] The camera signal processing unit 11 performs development processing including the above-mentioned pre-processing, synchronization processing, YC generation processing, etc. on the sensor image output from the sensor device 1. Furthermore, the development processing may include resolution conversion processing, codec processing, etc. as necessary.

[0035] In the synchronization process, a color separation process is performed so that the image data for each pixel contains all R, G, and B color components. For example, in the case of an image sensor using a Bayer array color filter, a demosaic process is performed as the color separation process. In the YC generation process, a luminance (Y) signal and a color (C) signal are generated (separated) from the R, G, and B image data. In the resolution conversion process, a resolution conversion process is performed on the image data that has been subjected to various signal processes.

[0036] In codec processing, the imaging data that has undergone the various processes described above is subjected to encoding processing and file generation for recording or communication, for example. In codec processing, it is possible to generate moving image file formats such as MPEG-2 (Moving Picture Experts Group) and H.264. It is also possible to generate still image file formats such as JPEG (Joint Photographic Experts Group), TIFF (Tagged Image File Format), and GIF (Graphics Interchange Format).

[0037] The camera image transmission unit 12 performs processing to transmit the image data (camera image data) obtained by processing by the camera signal processing unit 11 to a device external to the camera device 2. In this example, the camera image transmission unit 12 transmits the camera image data to the information processing device 3 included in the detection system S.

[0038] Next, an example of the configuration of the information processing device 3 will be described with reference to Fig. 1. The information processing device 3 performs predetermined image processing on the camera image data received from the camera device 2. At this time, the information processing device 3 selects a portion of the camera image data and performs the image processing.

[0039] The information processing device 3 includes a reception processing unit 13 , a determination processing unit 14 , an image processing unit 15 , a storage processing unit 16 , and a display processing unit 17 .

[0040] The reception processing unit 13 receives camera image data from the camera device 2. The received camera image data is supplied to the storage processing unit 16 and stored in a storage unit (not shown).

[0041] The received camera image data is also supplied to the determination processing unit 14 .

[0042] The determination processing unit 14 determines the lines to be subjected to image processing using the additional information AD for each line data LD related to the supplied camera image data. That is, the determination processing unit 14 determines whether or not image processing can be performed for each line data LD in the camera image data.

[0043] The determination result is supplied to the image processing unit 15 together with the camera image data.

[0044] The image processing unit 15 performs image processing on the area (line) to be processed in the camera image data based on the determination result of the determination processing unit 14. As described above, the image processing performed by the image processing unit 15 includes processing for detecting the target subject OB and determining whether or not there is a defect or other problem, and processing for detecting and reading a two-dimensional code as the target subject OB.

[0045] More specifically, the image processing unit 15 performs image processing on the line data LD that is determined to be the target of image processing based on the additional information AD. In other words, the image processing unit 15 does not perform image processing on the line data LD that is determined not to be the target of image processing based on the additional information AD.

[0046] The processing results from the image processing unit 15 are supplied to the storage processing unit 16 and stored in a storage unit (not shown).

[0047] The image processing unit 15 also supplies the results of the image processing to the display processing unit 17. This makes it possible to present the results of the image processing to the user of the detection system S.

[0048] <1-2. Process for Generating Additional Information> The process for generating additional information AD will be described with reference to FIG. 3 and other figures.

[0049] Figure 3 shows an image of a target subject OB traveling on a conveyor belt captured by the camera device 2 shown in Figure 2, and shows a frame image FG generated from one frame of line data LD output from the imaging unit 5.

[0050] The detection processing unit 6 divides the frame image FG into a plurality of pixel block areas Ar and detects the target subject OB for each pixel block area Ar. The detection processing unit 6 designates the pixel block areas Ar in which the target subject OB is captured as "valid blocks Bv" and the pixel block areas Ar in which the target subject OB is not captured as "invalid blocks Bi."

[0051] The detection processing unit 6 determines valid blocks Bv and invalid blocks Bi using the average luminance value calculated for each pixel block region Ar. One pixel block region Ar is, for example, an area of ​​64 pixels in total, consisting of 8 pixels vertically and 8 pixels horizontally.

[0052] The detection processing unit 6 detects the target subject OB when one frame's worth of line data LD is received from the imaging unit 5, or when an amount of line data LD that can be processed for the pixel block area Ar is received from the imaging unit 5. That is, in the example shown in Fig. 3, the detection processing may be executed every time all of the line data LD included in the frame image FG is received, or every time eight pieces of line data LD are received.

[0053] The detection processing unit 6 compares the average brightness value calculated for the pixel block area Ar with a threshold value, and if the deviation between the average brightness value and the reference value is greater than a predetermined value, it determines that the target subject OB has been detected and determines the pixel block area Ar as an effective block Bv.

[0054] In this example, the reference value used for comparison with the average brightness value may be the average brightness value of the pixel block area Ar in which only the conveyor belt is captured. The reference value may also be different for each pixel block area Ar. For example, the reference value may be set for each pixel block area Ar using a frame image FG captured when no goods or products are being transported on the conveyor belt.

[0055] In this example, the line data LD including part of one effective block Bv is eight line data LD. These eight line data LD are set as "effective line data LDv" that are the line data LD that are the subject of subsequent image processing. The additional information generating unit 7 sets a valid flag Fv as additional information AD for each of these effective line data LDv.

[0056] There are other possible methods for determining the valid blocks Bv and invalid blocks Bi by the detection processing unit 6. For example, the valid blocks Bv may be determined based on the change in the average luminance value between the previous frame image FG and the current frame image FG.

[0057] An example is shown in Fig. 4. Fig. 4 shows the current frame image FG, and the position of the target subject OB in the previous frame image FG is indicated by a broken line.

[0058] Focus is placed on two pixel block areas Ar1 and Ar2. Pixel block area Ar1 is one of the pixel block areas Ar in which a portion of the target subject OB was captured in the previous frame image. Pixel block area Ar2 is one of the pixel block areas Ar in which a portion of the target subject OB was captured in the current frame image FG.

[0059] In this case, both pixel block regions Ar1 and Ar2 are pixel block regions Ar whose average luminance values ​​have changed. Therefore, the detection processing unit 6 may determine both pixel block regions Ar1 and Ar2 as effective blocks Bv.

[0060] Alternatively, the detection processing unit 6 may determine the effective blocks Bv based on the direction of change in the average brightness value. For example, consider a case where the target subject OB is a subject with a brightness value greater than black. In this case, by changing the color of the conveyor belt to black, the pixel block area Ar2 where the target subject OB is not included in the image will have a positive change in the average brightness value.

[0061] On the other hand, in the pixel block area Ar1 where the target subject OB has changed from being captured to not being captured, the change in average luminance value is determined to be in the negative direction. Therefore, by determining the pixel block area Ar where the average luminance value has changed in the positive direction as the effective block Bv, it is possible to determine only the line data LD in which the target subject OB has been captured as the effective line data LDv to be processed.

[0062] On the other hand, if the target subject OB is a color close to black, the color of the conveyor belt is made white, and the pixel block area Ar whose average brightness value has changed in the negative direction is determined as the effective block Bv, thereby making it possible to set appropriate additional information AD.

[0063] Furthermore, the frame images FG used in the detection processing unit 6 are raw data before synchronization processing. Therefore, if the color of the target subject OB is red, it is possible to appropriately determine the effective blocks Bv by calculating the change in the average luminance value of red in the pixel block area Ar. In this case, it is desirable to set the color of the belt conveyor so that the luminance value of red when the belt conveyor is imaged differs from the luminance value of red when the target subject OB is imaged.

[0064] <1-3. Regarding output data> The detection result of the target subject OB for each line data LD is added to the line data LD as additional information AD. The additional information AD is information indicating whether or not a detection has occurred, and can be expressed as 1-bit flag information.

[0065] An example of additional information AD is shown in Fig. 5. The hatched rows are rows in which "1" indicating valid line data LDv is set as additional information AD, while the non-hatched rows are rows in which "0" indicating invalid line data LDi is set as additional information AD.

[0066] By including such additional information AD, which is made into one-bit flag information, in the packet data for transmitting the line data LD, it is possible to communicate to a downstream processing unit whether or not the line data LD is to be subjected to image processing.

[0067] An example of output data generated using the line data LD and additional information AD will be described with reference to Fig. 6. In various packet data formats, each packet is provided with a packet header PH, a payload area, and a packet footer PF.

[0068] The payload area in one packet includes, for example, one line data LD.

[0069] The configuration of the packet header PH varies depending on the specifications. For example, in the SLVS-EC standard, the packet header PH includes "Frame Start," "Frame End," "Line Valid," "Line Number," "EBD Line," "Data ID," "Header Info Type," "Reserved," "Line Length," "Reserved," and "ECC" (see FIG. 7).

[0070] By storing additional information AD in unused areas or extended areas, specifically areas marked "Reserved," among these items, the line data LD and additional information AD can be included in the packet data without changing the size of the packet header PH.

[0071] Furthermore, in accordance with the MIPI specifications, among the "Data ID," "Word Count," and "ECC" provided in the packet header PH, the channel information and data type information stored in "Data ID" can be included. Furthermore, the additional information AD can be stored in an area such as "Word Count." Since the packet size for transmitting one piece of line data LD is known if the size of the pixel array unit 5a is known, the additional information AD can be stored instead.

[0072] As described above, the additional information AD, which is one-bit flag information, can be stored in a region of the packet data that is separate from the line data LD. Alternatively, the additional information AD can be included in the line data LD.

[0073] For example, if the horizontal size of the pixel array unit 5a is n pixels, the line data LD contains n pieces of pixel data (see FIG. 8). In the figure, pixel data (0) indicates the first (first) pixel data, and pixel data (n-1) indicates the nth (last) pixel data.

[0074] Here, it is conceivable to replace the pixel data at both ends of the line data LD with the additional information AD, or more preferably, to replace the pixel data at the beginning of the line data LD with the additional information AD.

[0075] Of the pixels arranged in the same row, the first pixel (the pixel at the left end in the horizontal direction) and the last pixel (the pixel at the right end in the horizontal direction) are pixels that are unlikely to contain the target subject if the angle of view of the camera device 2 is appropriately set. Therefore, by replacing the pixel data of such edge pixels among the pixel data included in the line data LD with additional information AD, it becomes possible to transmit both the line data LD and the additional information AD without changing the packet size while maintaining the effectiveness of the image processing at the subsequent stage.

[0076] It is also possible to treat one bit in the pixel data of the first pixel or the last pixel as additional information AD. In this case, the LSB of the pixel data may be assigned to the additional information AD. This minimizes the effect on the pixel data of the first pixel or the last pixel, and makes it possible to treat the pixel data as original pixel data.

[0077] 9 shows an example of processing executed by the sensor device 1 in the camera device 2, and Fig. 10 shows an example of processing executed by the outside-sensor processing unit 4 in the camera device 2. Also, Fig. 11 shows an example of processing executed by the information processing device 3, which is a device downstream of the camera device 2.

[0078] 9 and 10 are processes executed for each capture of a frame image FG, which is repeated from the start of operation to the stop of operation of the detection system S. Also, Fig. 11 shows processes that are repeated from the start of operation of the detection system S to the stop of operation, and are executed multiple times for each capture of a frame image FG, equal to the number of line data LD.

[0079] The imaging unit 5 of the sensor device 1 of the camera device 2 performs imaging processing in step S101 of FIG.

[0080] After the image capturing process is completed, the detection processing unit 6 of the sensor device 1 performs a target subject detection process in step S102. The process of detecting the target subject OB is performed for each pixel block area Ar based on the average luminance value, as described above, for example.

[0081] The additional information generating unit 7 of the sensor device 1 performs processing to generate additional information AD in step S103.

[0082] In step S104, the output data generating unit 8 of the sensor device 1 generates output data (packet data, which is data of the sensor image) using the line data LD and the additional information AD.

[0083] In step S105, the output unit 9 of the sensor device 1 outputs the output data to the out-of-sensor processing unit 4 at the subsequent stage.

[0084] Next, the sensor image receiving unit 10 of the outside-sensor processing unit 4 of the camera device 2 receives the sensor image in step S201 of FIG.

[0085] In step S202, the camera signal processing unit 11 of the outside-sensor processing unit 4 performs signal processing, such as the synchronization processing and YC generation processing described above.

[0086] In step S203, the camera image transmitting unit 12 of the outside-sensor processing unit 4 transmits the camera image to the information processing device 3, which is determined to be the subsequent device.

[0087] The reception processing unit 13 of the information processing device 3 performs a process of receiving this camera image in step S301 of Fig. 11. Specifically, data including one line data LD of the camera image is received.

[0088] The determination processing unit 14 of the information processing device 3 extracts the additional information AD corresponding to the line data LD received in step S302, and determines whether the received line data LD is valid line data LDv.

[0089] The information processing device 3 performs branching processing based on the determination result of the additional information AD in step S303. Specifically, if the additional information AD indicates valid line data LDv, the image processing unit 15 of the information processing device 3 performs image processing in step S304. As described above, this image processing includes processing to detect the target subject OB and determine whether or not there is a defect or other problem, and processing to detect and read a two-dimensional code representing the target subject OB.

[0090] Note that the processing of step S304 may be executed after waiting for a series of pixel data to be accumulated. For example, taking the frame image FG shown in Fig. 5 as an example, if two target subjects OB are captured in the frame image FG, after receiving a plurality of line data LD capturing one of the target subjects OB, the image processing of step S304 is performed on the area represented by the plurality of line data LD. Then, after receiving a plurality of line data LD capturing the other subject OB, the image processing of step S304 is performed again on the area represented by the plurality of line data LD.

[0091] Furthermore, the image processing of step S304 may be performed after all valid line data LDv in one frame image FG has been received.

[0092] Next, in step S305, the storage processing unit 16 of the information processing device 3 performs processing to store the results of the image processing. Then, in step S306, the display processing unit 17 of the information processing device 3 performs processing to display the results of the image processing and the frame image FG.

[0093] On the other hand, if the additional information AD indicates invalid line data LDi in step S303, the information processing device 3 discards the received line data LD in step S307, or performs processing to use the received line data LD as complementary information in the image processing of step S305 executed on the valid line data LDv. The processing to use the received invalid line data LDi as complementary information is, for example, processing to store the received invalid line data LDi in a storage unit.

[0094] 2. Second Embodiment The configuration of a camera device 2A of a detection system SA in a second embodiment is similar to that of the camera device 2 shown in Fig. 1. Therefore, only the different parts will be described to avoid redundant explanation.

[0095] The camera device 2A includes a sensor device 1A and an outside-sensor processing unit 4A.

[0096] The detection processing unit 6A of the sensor device 1A not only determines whether each line data LD contains pixel data of a target subject OB that is the subject of subsequent image processing, but also determines the range in the line data LD that is the subject of image processing as the valid range Rv and the range that is not the subject of image processing as the invalid range Ri.

[0097] 12, for example, additional information AD for distinguishing between valid line data LDv and invalid line data LDi for each line data LD is generated by the additional information generating unit 7. Furthermore, the detection processing unit 6A sets the pixel range in which the target subject OB in the valid line data LDv is imaged as the valid range Rv, and the pixel range in which the target subject OB is not imaged as the invalid range Ri.

[0098] The valid range Rv and invalid range Ri differ for each line data LD. Furthermore, the valid range Rv is not set for invalid line data LDi.

[0099] Furthermore, the valid range Rv and the invalid range Ri may be set for each of the pixel block regions Ar described above.

[0100] Information for specifying the valid range Rv and the invalid range Ri, together with the line data LD and the additional information AD, is output from the sensor device 1A to the outside-sensor processing unit 4A. The information for specifying the valid range Rv and the invalid range Ri may be, for example, information specifying a pixel at the boundary between the valid range Rv and the invalid range Ri, information indicating a pixel block area Ar belonging to the valid range Rv (or the invalid range Ri), or information indicating the start pixel of the valid range Rv and the number of pixels included in the valid range Rv.

[0101] Here is an example: A synchronization signal synchronized with the transmission of the line data LD is supplied from the sensor device 1A to the outside-sensor processing unit 4A.

[0102] The outside-sensor processing unit 4A is capable of determining the valid range Rv and the invalid range Ri in the line data LD based on the received synchronization signal.

[0103] A specific description will be given with reference to Fig. 13. The output unit 9 of the sensor device 1A outputs a pixel clock signal S_clk, packet data, a pixel data enable signal S_data_en, and a valid data signal S_data_vld to the extra-sensor processing unit 4A. Fig. 13 is a timing chart showing the period during which pixel data out of the packet data is being transmitted.

[0104] The line data LD includes n pieces of pixel data, each of which is shown as Pix(0) to Pix(n-1) in the drawing.

[0105] The valid range Rv in the line data LD is the range from Pix(f+1) to Pix(g) and the range from Pix(h+1) to Pix(m). That is, the invalid range Ri is the range from Pix(0) to Pix(f), the range from Pix(g+1) to Pix(h), and the range from Pix(m+1) to Pix(n-1).

[0106] During transmission of the line data LD stored in the payload area of ​​the packet data, the pixel data enable signal S_data_en is set to 1 (High), and the rest is set to 0 (Low).

[0107] Furthermore, during transmission of pixel data in the valid range Rv, the valid data signal S_data_vld is set to 1 (High), and during transmission of pixel data in the invalid range Ri, the valid data signal S_data_vld is set to 0 (Low).

[0108] The sensor image receiving unit 10 of the sensor outside processing unit 4A receives the pixel clock signal S_clk, the pixel data enable signal S_data_en, and the valid data signal S_data_vld together with the packet data, and obtains the line data LD and additional information AD. The obtained line data LD and additional information AD are provided to the information processing device 3A via the camera image transmitting unit 12A.

[0109] The reception processing unit 13A of the information processing device 3A provides the received line data LD and additional information AD to the storage processing unit 16, and further provides the storage processing unit 16 with information about the effective range Rv in the line data LD.

[0110] The reception processing unit 13A also supplies the line data LD, additional information AD, and information specifying the effective range Rv to the determination processing unit 14A.

[0111] The determination processing unit 14A determines the lines to be processed using the additional information AD for each line data LD related to the supplied camera image data. Furthermore, the determination processing unit 14A determines the effective range Rv of the effective line data LDv. That is, the determination processing unit 14A performs processing to identify the rectangular processing target area ArT to be subjected to subsequent image processing (see FIG. 14).

[0112] The image processing unit 15A obtains the desired results by performing image processing on the processing target area ArT identified by the determination processing unit 14A. In other words, the image processing unit 15A does not perform image processing on the invalid line data LDi or the invalid range Ri in the valid line data LDv.

[0113] The results obtained by the image processing are provided to the storage processing unit 16 and the display processing unit 17, where they are stored and presented to the user.

[0114] 3. Third Embodiment In the above-described example, the image processing unit 15 that performs predetermined image processing is provided in the information processing device 3 (3A) downstream of the camera device 2 (2A). In the detection system SB of this embodiment, the camera device 2B is provided with the image processing unit 15B. An example configuration of the detection system SB is shown in FIG. 15. Note that the same components as those in the first and second embodiments are designated by the same reference numerals, and descriptions thereof will be omitted where appropriate.

[0115] The detection system SB includes a camera device 2B and an information processing device 3B. The information processing device 3B is a device for presenting the image processing results to the user.

[0116] The camera device 2B includes the sensor device 1 and an outside-sensor processing unit 4B.

[0117] The sensor device 1 includes an imaging unit 5, a detection processing unit 6, an additional information generating unit 7, an output data generating unit 8, and an output unit 9. The configuration of each unit is the same as in the first embodiment. That is, the sensor device 1 outputs output data OD (sensor image data) that conforms to a predetermined standard and includes line data LD and additional information AD.

[0118] The outside-sensor processing unit 4B includes a sensor image receiving unit 10B, a determination processing unit 14B, an image processing unit 15B, and an information transmitting unit 18.

[0119] The sensor image receiving unit 10B receives the output data OD output from the sensor device 1 and supplies it to the subsequent determination processing unit 14B.

[0120] The determination processing unit 14B determines the valid line data LDv that is the target of image processing using the additional information AD for each line data LD related to the supplied sensor image. The determination processing unit 14B supplies information on the valid line data LDv to the downstream image processing unit 15B, thereby urging the image processing unit 15B to perform image processing on the valid line data LDv.

[0121] The image processing unit 15B obtains the desired results by performing image processing on the valid line data LDv identified by the determination processing unit 14B. The image processing unit 15B also avoids image processing on the invalid line data LDi. Note that the invalid line data LDi that are not subject to image processing may be discarded by the determination processing unit 14B or may be used as complementary data for the image processing of the valid line data LDv.

[0122] The processing results by the image processing unit 15B and partial image data for the valid line data LDv that are the processing target are transmitted to the information transmitting unit 18.

[0123] The information transmitting unit 18 transmits the information received from the image processing unit 15B to the information processing device 3B. Note that the information transmitting unit 18 may transmit the entire sensor image related to the frame to the information processing device 3B.

[0124] The information processing device 3B includes a reception processing unit 13, a storage processing unit 16, and a display processing unit 17. The reception processing unit 13 supplies partial image data and information on the results of image processing for the valid line data LDv received from the camera device 2B to the storage processing unit 16, thereby storing the information in the storage unit. The reception processing unit 13 also supplies the information received from the camera device 2B to the display processing unit 17, thereby realizing a presentation process for the user.

[0125] The flow of processing in the third embodiment will be described. The sensor device 1 of the camera device 2B executes the processes of steps S101 to S105 shown in Fig. 9 to detect the target subject OB, generate additional information AD, generate output data OD, and so on.

[0126] Furthermore, the outside-sensor processing unit 4B of the camera device 2B performs the processes of steps S301, S302, S303, S304, and S307 shown in FIG. 11, thereby performing image processing on a partial area of ​​the frame image FG.

[0127] Furthermore, the information processing device 3B executes the processes of steps S305 and S306 shown in FIG. 11, thereby performing the process of storing information and the process of presenting information to the user.

[0128] 4. Modifications The detection system S (SA, SB) in the above-described example performs processing to determine the area to be processed within the sensor device 1 (1A). Therefore, the data output from the sensor device 1 (1A) may be limited to pixel data within the range determined as the effective line data LDv and the effective range Rv of the frame image FG. This reduces the amount of communication required for sending and receiving pixel data, thereby reducing the processing load and enabling effective use of communication bandwidth.

[0129] In the first embodiment, the camera device 2 and the information processing device 3 are described as separate devices, but other configurations are also possible. For example, the camera device 2 may be configured as a camera module including the sensor device 1 and the extra-sensor processing unit 4, the information processing device 3 may be configured as an extra-module processing unit provided outside the camera module, and the camera module and the extra-module processing unit may be provided within a single device.

[0130] 5. Summary As described in the above examples, the sensor device 1 (1A) includes a pixel array section 5a in which pixels are arranged two-dimensionally, a detection processing section 6 (6A) that determines whether each line data LD output from the pixel array section 5a is valid line data LDv to be processed in subsequent image processing, an additional information generation section 7 that generates additional information AD based on the results of the determination by the determination processing section (detection processing section 6, 6A), an output data generation section 8 that generates output data OD for each line using the line data LD and the additional information AD, and an output section 9 that outputs the output data OD. This allows the subsequent image processing section 15 (15A, 15B) to determine whether to perform image processing for each line data LD. This avoids image processing of invalid line data LDi and reduces the processing load. Furthermore, performing image processing only on valid line data LDv facilitates high frame rate output, enabling suitable image processing for fast-moving subjects (target subjects OB). It should be noted that the camera device 2 (2A, 2B) including the sensor device 1 (1A) described above can also be regarded as the sensor device of the present technology.

[0131] As described with reference to Figures 5 to 8, in the sensor device 1 (1A), the additional information AD may be flag information for each line data LD. This minimizes the amount of data for the additional information AD. This prevents an increase in the amount of output data OD, reducing the processing load on the downstream image processing unit 15 (15A, 15B).

[0132] As described with reference to Figures 5 to 7, the output data generation unit 8 in the sensor device 1 (1A) may generate output data OD by storing additional information AD in an area (e.g., a packet header area) separate from the area (e.g., a payload area) where line data LD is stored. This allows the line data LD and additional information AD to be transmitted separately. Therefore, the line data LD can be used in subsequent image processing without losing any part of it.

[0133] As described with reference to Figures 5 to 7, the output data generation unit 8 in the sensor device 1 (1A) may use a header area (header area of ​​packet data) as the separate area in which the additional information AD is stored. This allows the additional information AD to be transmitted without increasing the size of the payload area in which the line data LD is stored. Furthermore, by storing the additional information AD in an unused area or an extended area in the header area, it is possible to suppress an increase in the data size of the packet data without having to change the header size.

[0134] As described with reference to Fig. 8 etc., the output data generating unit 8 in the sensor device 1 (1A) may generate the output data OD by including the additional information AD in part of the line data LD, thereby making it possible to transmit both the line data LD and the additional information AD without changing the configuration of the header or the configuration of the payload area.

[0135] As described with reference to FIG. 8 and other figures, the output data generation unit 8 of the sensor device 1 (1A) may include additional information AD in the pixel data of the first pixel in the line data LD. Among the pixel data included in the line data LD, the pixel data of the first pixel is likely to be pixel data that does not capture the target subject OB. By including additional information AD in such pixel data, it is possible to include the additional information AD in the packet data while minimizing the impact on subsequent image processing. Furthermore, it is possible to prevent the size of the packet data from increasing due to the inclusion of additional information AD. Note that additional information AD may also be included in the pixel data of the last pixel in the line data LD. However, by including additional information AD in the pixel data of the first pixel, it is possible to determine whether the remaining data in the line data LD is unnecessary when the beginning of the line data LD is read. Therefore, it is possible to reduce the processing load associated with the analysis of the line data LD. Furthermore, by using one-bit flag information as the additional information AD, only one bit of information from the pixel data is used as flag information, thereby reducing the impact on the image. By embedding additional information AD in the LSB (Least Significant Bit) of the first pixel, it is possible to minimize loss of image information.

[0136] As described with reference to Figures 12 to 14, the determination processing unit (detection processing unit 6, 6A) in the sensor device 1 (1A) may determine the valid range Rv and invalid range Ri in the line data LD determined to be valid line data LDv, and the output unit 9 may output a first synchronization signal (pixel data enable signal S_data_en) synchronized with the line data LD in the output data OD and a second synchronization signal (valid data signal S_data_vld) synchronized with the output timing of the data of the valid range Rv and invalid range Ri. This allows the downstream image processing unit 15 (15A, 15B) to identify the valid range Rv and invalid range Ri in the line data LD based on the input second synchronization signal (valid data signal S_data_vld). This allows for more efficient image processing.

[0137] As described with reference to Figures 3 and 4, the determination processing unit (detection processing unit 6, 6A) in the sensor device 1 (1A) may determine whether each pixel block region Ar, consisting of a plurality of pixels in each row and column, is an effective block Bv to be subjected to subsequent image processing, and may determine line data LD containing part of the effective block Bv as effective line data LDv. This enables the determination of validity and invalidity to be performed with higher accuracy than if it were performed line by line. Therefore, it is possible to reduce the possibility of unnecessary image processing being performed in the subsequent image processing unit 15 (15A, 15B), thereby reducing the processing load.

[0138] As described with reference to Figures 3 and 4, the determination processing unit (detection processing unit 6, 6A) in the sensor device 1 (1A) may calculate the average brightness value of the pixel block region Ar and determine the valid blocks Bv. For example, in a system that detects defective products in a situation where products are traveling on a conveyor belt, the average brightness value of the pixel block region Ar in which the product is captured differs from the average brightness value of the pixel block region Ar in which only the conveyor belt is captured. Therefore, it is possible to distinguish between the pixel block regions Ar to be processed through simple processing. This enables efficient image processing in the downstream image processing unit 15 (15A, 15B) while suppressing an increase in the processing load on the sensor device.

[0139] As described with reference to Figures 3 and 4, the determination processing unit (detection processing unit 6, 6A) in the sensor device 1 (1A) may calculate the change in average luminance value between frames and determine the effective block Bv. As a result, the pixel block area Ar whose average luminance value has changed is determined to be the effective block Bv. Therefore, the effective block Bv can be detected efficiently.

[0140] As described with reference to Figures 3 and 4, the determination processing unit (detection processing unit 6, 6A) in the sensor device 1 (1A) may determine a pixel block area Ar as an effective block Bv if the calculated change in average luminance value between frames is a change in a predetermined direction. The change direction can be a positive direction in which luminance values ​​increase or a negative direction in which luminance values ​​decrease. For example, consider a situation in which a non-black product is being transported on a black conveyor belt. The change direction for the pixel block area Ar in which the product is newly captured is a positive direction in which luminance values ​​increase, while the change direction for the pixel block area Ar in which the product has passed and only the conveyor belt is captured is a negative direction in which luminance values ​​decrease. In such a case, the effective block Bv that is the target of image processing in the subsequent image processing unit 15 (15A, 15B) is the pixel block area Ar in which a change in luminance values ​​is observed in the positive direction. Therefore, by determining a pixel block area Bv as an effective block Bv if the change between frames is a change in a predetermined direction, the number of pixel block areas Ar that are the target of subsequent image processing can be reduced, thereby reducing the processing load. In addition, when a dark product is flowing on a white conveyor belt, the same effect can be achieved by using the pixel block area Ar in which the direction of change in the average brightness value is negative as the effective area.

[0141] As described with reference to Figures 12 to 14, the determination processing unit (detection processing unit 6, 6A) in the sensor device 1 (1A) may determine the valid range Rv and the invalid range Ri in the line data LD determined to be valid line data LDv, and determine the range of the valid line data LDv that is included in the valid block Bv as the valid range Rv. This makes it possible to determine the valid pixel range and the invalid pixel range at a finer granularity than if it were done for each line, and to distinguish the valid range Rv to be processed in the line data LD from other ranges in the image processing unit 15 (15A, 15B) at a subsequent stage. Therefore, the range to be processed in the image processing at the subsequent stage can be narrowed, and the processing load can be reduced.

[0142] The data structure of the present technology is a data structure of output data OD output from a sensor device 1 (1A) having a pixel array section 5a in which pixels are arranged two-dimensionally, and is generated based on line data LD output from the pixel array section 5a and additional information AD indicating a determination result of whether each line data LD is valid line data LDv to be processed in subsequent image processing. By using such a data structure, it becomes possible to distinguish between line data LD to be processed in the received information processing device and line data LD that is not. Therefore, it is possible to reduce the burden of subsequent image processing and shorten the processing time.

[0143] As described with reference to Figures 1, 15, etc., the information processing device 3 (3A) of the present technology or the extra-sensor processing unit 4B as an information processing device includes: a reception processing unit 13, 13A (or a sensor image receiving unit 10B) that receives output data OD from the sensor device 1 (1A) that is generated based on line data LD output from a pixel array unit 5a in which pixels are arranged two-dimensionally and additional information AD that indicates a determination result as to whether or not each piece of line data LD is valid line data LDv to be processed in subsequent image processing; a determination processing unit 14 (14A, 14B) that determines whether or not predetermined image processing can be performed on the line data LD based on the additional information AD included in the output data OD received by the reception processing unit 13, 13A (or the sensor image receiving unit 10B); and an execution processing unit (image processing unit 15, 15A, 15B) that executes predetermined image processing on the line data LD based on the determination result of the determination processing unit 14 (14A, 14B). This allows the information processing device 3 (3A) or the outside-sensor processing unit 4B as an information processing device to distinguish the line data LD to be subjected to image processing, thereby enabling the necessary image processing to be reliably performed.

[0144] As described with reference to Figures 1, 15, etc., the execution processing unit (image processing unit 15, 15A, 15B) in the information processing device 3 (3A) or the extra-sensor processing unit 4B as an information processing device according to the present technology does not need to perform predetermined image processing on the line data LD when it is determined that the line data LD is not valid line data LDv based on the additional information AD corresponding to the line data LD. This makes it possible to limit the data that is the subject of image processing to a portion, thereby reducing the processing load on the information processing device. Furthermore, by performing image processing only on the valid line data LDv, it becomes easy to support output at a high frame rate, and it is possible to realize suitable image processing for a subject that moves at high speed.

[0145] 11, 15, etc., the information processing device 3 (3A) of the present technology or the determination processing unit 14 (14A, 14B) of the outside-sensor processing unit 4B as an information processing device may discard the line data LD when it is determined that the line data LD is not valid line data LDv based on the additional information AD corresponding to the line data LD. This allows only a portion of the line data LD to be stored, thereby enabling effective use of the storage area.

[0146] The effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0147] Furthermore, the above-described examples may be combined in any manner, and even when various combinations are used, the various effects described above can be obtained.

[0148] <6. The Present Technology> The present technology may also have the following configurations. (1) A sensor device comprising: a pixel array unit in which pixels are two-dimensionally arranged; a determination processing unit that determines whether each line data output from the pixel array unit is valid line data to be processed in subsequent image processing; an additional information generation unit that generates additional information based on a result of the determination by the determination processing unit; an output data generation unit that generates output data for each line using the line data and the additional information; and an output unit that outputs the output data. (2) The sensor device described in (1) above, in which the additional information is flag information for each line data. (3) The sensor device described in any of (1) to (2) above, in which the output data generation unit stores the additional information in an area separate from an area in which the line data is stored and generates the output data. (4) The sensor device described in (3) above, in which the output data generation unit uses the separate area as a header area. (5) The sensor device according to any of (1) to (2), wherein the output data generation unit generates the output data by including the additional information in part of the line data. (6) The sensor device according to (5), wherein the output data generation unit includes the additional information in pixel data of a first pixel in the line data. (7) The sensor device according to any of (1) to (6), wherein the determination processing unit determines a valid range and an invalid range in the line data determined to be the valid line data, and the output unit outputs a first synchronization signal synchronized with the line data in the output data and a second synchronization signal synchronized with output timing of data of the valid range and the invalid range. (8) The sensor device according to any of (1) to (7), wherein the determination processing unit determines whether each pixel block area consisting of a plurality of pixels in each of the row and column directions is a valid block to be processed in the subsequent image processing, and determines the line data including a portion of the valid block to be the valid line data. (9) The sensor device according to (8), wherein the determination processing unit calculates an average luminance value of the pixel block region and determines the effective block.(10) The sensor device according to (9), wherein the determination processing unit calculates a change between frames in the average luminance value and determines the effective block. (11) The sensor device according to (10), wherein the determination processing unit determines the pixel block area as the effective block when the calculated change between frames is a change in a predetermined direction. (12) The sensor device according to (11), wherein the determination processing unit determines an effective range and an invalid range in the line data determined to be the effective line data, and determines a range of the effective line data that is included in the effective block as the effective range. (13) A data structure of output data output from a sensor device having a pixel array unit in which pixels are two-dimensionally arranged, wherein the data structure is generated based on the line data output from the pixel array unit and additional information indicating a determination result as to whether or not each of the line data is effective line data to be processed in subsequent image processing. (14) An information processing device comprising: a reception processing unit that receives from a sensor device output data generated based on line data output from a pixel array unit in which pixels are two-dimensionally arranged and additional information indicating a determination result of whether or not each of the line data is valid line data to be processed in subsequent image processing, a determination processing unit that decides whether or not to perform predetermined image processing on the line data based on the additional information included in the output data received by the reception processing unit, and an image processing unit that performs the predetermined image processing on the line data based on the determination result of the determination processing unit. (15) The information processing device described in (14) above, wherein the image processing unit does not perform the predetermined image processing on the line data when it is determined that the line data is not valid line data based on the additional information corresponding to the line data. (16) The information processing device described in any of (14) to (15) above, wherein the determination processing unit discards the line data when it is determined that the line data is not valid line data based on the additional information corresponding to the line data.

[0149] 1, 1A Sensor device 3, 3A Information processing device 4B Outside sensor processing unit (information processing device) 5a Pixel array unit 6, 6A Detection processing unit 7 Additional information generation unit 8 Output data generation unit 9 Output unit 10B Sensor image receiving unit (reception processing unit) 13, 13A Reception processing unit 14, 14A, 14B Determination processing unit 15, 15A, 15B Image processing unit (execution processing unit) LD Line data LDv Effective line data Ar, Ar1, Ar2 Pixel block area AD Additional information OD Output data Bv Effective block Rv Effective range Ri Invalid range

Claims

1. A sensor device comprising: a pixel array section in which pixels are arranged two-dimensionally; a judgment processing section which judges whether each line data output from the pixel array section is valid line data to be subjected to subsequent image processing; an additional information generation section which generates additional information based on the result of the judgment by the judgment processing section; an output data generation section which generates output data for each line using the line data and the additional information; and an output section which outputs the output data.

2. The sensor device according to claim 1, wherein the additional information is flag information for each of the line data.

3. The sensor device according to claim 1, wherein the output data generating section generates the output data by storing the additional information in an area separate from an area in which the line data is stored.

4. The sensor device according to claim 3, wherein the output data generating section sets the separate area as a header area.

5. The sensor device according to claim 1, wherein the output data generating section generates the output data by including the additional information in a portion of the line data.

6. The sensor device according to claim 5, wherein the output data generating section includes the additional information in pixel data of a leading pixel in the line data.

7. The sensor device of claim 1, wherein the judgment processing unit judges the valid range and invalid range in the line data judged to be the valid line data, and the output unit outputs a first synchronization signal synchronized with the line data in the output data and a second synchronization signal synchronized with the output timing of the data of the valid range and the invalid range.

8. The sensor device according to claim 1, wherein the judgment processing unit judges whether each pixel block area consisting of a plurality of pixels in both the row and column directions is an effective block to be processed in the subsequent image processing, and judges the line data which includes a portion of the effective block to be the effective line data.

9. The sensor device according to claim 8, wherein the determination processing section calculates an average luminance value of the pixel block area to determine whether the block is effective.

10. The sensor device according to claim 9, wherein the determination processing section calculates a change in the average luminance value between frames to determine whether the block is valid.

11. The sensor device according to claim 10, wherein the determination processing unit determines the pixel block area to be the effective block when the calculated change between frames is a change in a predetermined direction.

12. The sensor device according to claim 11, wherein the judgment processing unit judges the valid range and invalid range of the line data judged to be the valid line data, and judges the range of the valid line data that is included in the valid block as the valid range.

13. A data structure for output data output from a sensor device having a pixel array section in which pixels are arranged two-dimensionally, the data structure being generated based on line data output from the pixel array section and additional information indicating the result of determining whether each line data is valid line data to be processed in subsequent image processing.

14. An information processing device comprising: a receiving processing unit that receives output data from a sensor device, which is generated based on line data output from a pixel array unit in which pixels are arranged two-dimensionally and additional information indicating a judgment result as to whether or not each of the line data is valid line data to be processed in subsequent image processing; a judgment processing unit that determines whether or not a predetermined image processing can be performed on the line data based on the additional information included in the output data received by the receiving processing unit; and an image processing unit that performs the predetermined image processing on the line data based on the judgment result of the judgment processing unit.

15. The information processing device according to claim 14, wherein the image processing unit does not execute the predetermined image processing on the line data when it is determined that the line data is not valid line data based on the additional information corresponding to the line data.

16. The information processing device according to claim 14, wherein said determination processing section discards said line data when it determines that said line data is not valid line data based on said additional information corresponding to said line data.

Images