Sensor device and signal processing method

Abstract

A sensor device has: an array sensor having a plurality of imaging elements of visible light or invisible light arranged one-dimensionally or two-dimensionally; an image processing unit that performs image processing on an image signal obtained as a result of imaging by the array sensor; and a threshold setting unit. The threshold setting unit is configured so that: a threshold for parameter setting is set for all or some parameters used for image processing for imaging by the array sensor or image processing by the image processing unit, and processing using parameters changed based on the threshold is performed.

Description

Technical Field

[0001] The present technology relates to a sensor device, a parameter setting method and a signal processing method, and in particular to the technical field of a sensor device having a function of processing an image signal obtained by an array sensor. Background Art

[0002] In order to achieve object detection using an array sensor including an array of what are generally called imaging elements, it is known to capture an image using the array sensor, transmit an image signal of the captured image to a processor outside the array sensor, and perform object detection computation processing on the external processor side.

[0003] Non-Patent Literature 1 identified below discloses a technology related to face recognition processing for performing face recognition from an image.

[0004] Furthermore, Patent Document 1 indicated below discloses a technique that includes setting a region of interest within an area of ​​an image captured by an image sensor.

[0005] Furthermore, Patent Document 2 identified below discloses a technology of changing an operating frequency according to movement of a monitoring area.

[0006] Prior art literature

[0007] Patent Literature

[0008] Patent Document 1: JP 2014-225868A

[0009] Patent Document 2: JP 2007-318262A

[0010] Non-patent literature

[0011] Non-patent document 1: Yoshihiko Kawai, Face Recognition Technique for TV Program Video, NHK Science & Technology Research Laboratories / No.164 / 2017.8P23-P30 Summary of the invention

[0012] Technical Problems to be Solved by the Invention

[0013] The image quality of an array sensor including an image sensor has been adjusted so as to produce a beautiful picture viewed by a viewer. However, when the subject that makes the determination is constituted by a computer as in the case of image recognition, there arises a problem that even in the case of an image captured with beautiful image quality from the viewer's perspective, the accuracy of recognition of the category (classification or genre) of the imaging target is reduced.

[0014] In this case, considering processing such as object detection using image recognition, it is preferable to implement image processing suitable for the purpose of object detection rather than creating beautiful pictures.

[0015] Furthermore, it is preferable to reduce the load of processing, storage, and transmission while achieving the purpose of object detection and the like.

[0016] Therefore, the present disclosure proposes a technology that can obtain an image signal suitable for the purpose of object detection by image recognition or other purposes, and can also reduce the processing load and the like.

[0017] Solution to the problem

[0018] A sensor device according to a first disclosed technology relates to a sensor device, which includes: an array sensor, including a plurality of visible light or invisible light imaging elements arranged in one dimension or two dimensions; an image processing unit, which performs image processing on an image signal obtained by image capture using the array sensor; and a threshold setting unit, which sets a threshold for parameter setting for all or some parameters used in image capture processing associated with image capture performed by the array sensor or image processing performed by the image processing unit, so as to realize processing using parameters based on the threshold setting.

[0019] A threshold is set based on which the parameter is determined.

[0020] Examples of parameters associated with the image capture process include exposure time of the array sensor, readout timing from the array sensor, and frame rate.

[0021] Examples of parameters related to image processing include the resolution and the number of color levels of the image.

[0022] The sensor device according to the above-mentioned first disclosed technology may be configured so that the threshold value setting unit sets the threshold value corresponding to the category of the object detected from the image signal.

[0023] Specifically, the image processing unit performs image processing on the image signal obtained by the array sensor. In this case, the threshold value for setting the image processing parameter is set based on the category of the object detected in the image signal.

[0024] Note that the object detected from the image signal refers to an object that may become a detection target for the purpose of recognition from the image. Any object can be designated as an object of the detection target referred to herein. For example, the target object can be any object, such as a person, an animal, a moving object (e.g., a car, a bicycle, and an airplane), a natural object (e.g., vegetables and plants), an industrial product / part, a building, an organization, a mountain, a sea, a river, a star, the sun, and a cloud.

[0025] Classification refers to the classification of objects to be identified using image recognition. For example, the class divides the objects to be detected into several categories, such as "person", "car", "plane", "ship", "truck", "sign", "bird", "cat", "dog", "deer", "frog", and "horse".

[0026] Then, a threshold value considered appropriate is set according to the category such as "person", "car", and "sign".

[0027] The sensor device according to the above-mentioned first disclosed technology may be configured so that the threshold value setting unit sets the threshold value based on a learning process performed on the image signal.

[0028] For example, the threshold value is obtained by performing a learning process as local learning on the sensor device side.

[0029] The sensor device according to the above-mentioned first disclosed technology may be configured to cause the threshold setting unit to set the threshold so that a predetermined ratio is obtained as the confidence level of the object detection.

[0030] For example, in the case where an object is detected (eg, a person is detected) from an image signal output from the sensor device, the threshold value is set to a level required for the detection accuracy to satisfy the detection.

[0031] The sensor device according to the above-mentioned first disclosed technology may be configured such that the threshold value setting unit is provided in the device having the same housing as that of the image processing unit.

[0032] For example, the threshold setting unit is provided in a unit serving as a sensor device, or is provided in a terminal device or an imaging device including the sensor device.

[0033] A sensor device according to a second disclosed technology relates to a sensor device, including: an array sensor including a plurality of detection elements arranged one-dimensionally or two-dimensionally; a signal processing unit that acquires a detection signal acquired by the array sensor and performs signal processing on the detection signal; and a calculation unit that detects an object from the detection signal acquired by the array sensor and gives an instruction of region information generated based on the detection of the object to the signal processing unit as region information associated with acquisition of the detection signal acquired from the array sensor or associated with signal processing of the detection signal. The calculation unit sets an active region of the detection signal acquired from the array sensor based on information associated with previous region information, and detects the object using the detection signal of the active region.

[0034] Thus, object detection used to generate region information is accomplished based on information associated with the region designated as the active region rather than the entire region of the array sensor.

[0035] Note that information associated with region information refers to information associated with an object detection region based on which the region information is generated, or the region information itself.

[0036] The sensor device according to the above-mentioned second disclosed technology may be configured to cause the calculation unit to set the active area so that a plurality of pieces of area information generated in a previous predetermined cycle each contain the detection area detected by object detection based on the generated corresponding area information.

[0037] In other words, in the case where object detection is performed for generating region information, the active region is defined as the region where the detection target object appears.

[0038] The sensor device according to the second disclosed technology described above may be configured so that the signal processing unit includes an acquisition unit that selectively acquires detection signals from the detection elements of the array sensor, and the acquisition unit acquires the detection signals of the detection elements selected based on the area information received from the calculation unit as a frame of detection signals. In a state where the acquisition unit has selected the detection elements based on the area information, in the case where the target object is not detected in the detection signals acquired from the array sensor by object detection, the calculation unit instructs the acquisition unit to acquire the detection signals of the active area from the array sensor in a subsequent frame.

[0039] Specifically, when the target object is not detected from a frame in which only information associated with some detection elements is acquired, the calculation unit uses the acquisition unit to restore the detection signal acquisition to a state in which the detection signal is acquired from the active area.

[0040] The sensor device according to the above-mentioned second disclosed technology may be configured so that an object is detected from the detection signal of the active area in a target frame designated as a key frame in the detection signal obtained from the array sensor, and area information is generated based on the detection of the object.

[0041] Specifically, the calculation unit selects a frame corresponding to the key frame according to a predetermined selection algorithm to perform the process of active area setting.

[0042] The sensor device according to the above-mentioned second disclosed technology can be configured to enable a computing unit to identify the category of an object detected in a detection signal obtained from an array sensor, and generate area information corresponding to the object using a template, the area information indicating an acquisition area of ​​the detection signal corresponding to the identified category, and the computing unit includes a threshold setting unit, which sets a threshold for all or some parameters of signal processing performed by a signal processing unit or acquiring a detection signal using an array sensor, and sets parameters for processing performed on the acquisition area indicated by the template based on the threshold.

[0043] The threshold value is set to change the parameter of the process performed on the acquisition area indicated by the template based on the threshold value.

[0044] According to the third disclosed technology, the sensor device includes: an array sensor, including a plurality of detection elements arranged one-dimensionally or two-dimensionally; a signal processing unit, which acquires a detection signal obtained by the array sensor and performs signal processing on the detection signal; and a threshold setting unit, which sets a threshold of a frame rate according to a category identified for an object detected from the detection signal obtained by the array sensor, so that processing using a frame rate set based on the threshold is performed according to the category identified for an object detected from the detection signal obtained by the array sensor.

[0045] A threshold corresponding to the category is set to change the frame rate based on the threshold.

[0046] The sensor device according to the above-mentioned third disclosed technology may be configured so that the threshold value setting unit sets the threshold value to maintain a frame rate at which the object is tracked from the image.

[0047] According to the image signal output from the sensor device, in object detection, the frame rate at which object tracking can be maintained is different for each category. Therefore, object tracking is maintained according to the category.

[0048] The sensor device according to the above-mentioned third disclosed technology can be configured to perform control based on the setting value of the frame rate of the first mode or the second mode according to the result of object detection, and the setting value of the frame rate set by the threshold setting unit is used as the setting value of the frame rate of one of the first mode and the second mode.

[0049] For example, the value of the readout interval of the array sensor, etc. is stored as a setting value to give an instruction of the frame rate. In one mode, the frame rate is a rate set according to a threshold value.

[0050] The sensor device according to the above-mentioned first, second and third disclosed technologies may include an output unit that outputs the image signal subjected to the image processing by the image processing unit to an external device.

[0051] Specifically, the image signal subjected to image processing using the parameters selected based on the category recognition is transmitted and output to the external device. For example, it is assumed that the external device is an external processor that detects the object, or a processor in the cloud.

[0052] Furthermore, the sensor device according to the above-mentioned first, second and third disclosed technologies may be configured such that the output unit also transmits information associated with a category identification of the image signal to be output.

[0053] Information associated with the category identification, such as the identified category and the number of objects (the number of objects), is output together with the output image signal.

[0054] In addition, the sensor device according to the above-mentioned first, second and third disclosed technologies can be configured so that the image processing unit performs at least one of color correction, gamma correction, color grayscale processing, gain processing, contour emphasis processing, data compression processing, frame rate conversion, resolution conversion, aspect ratio conversion, contrast adjustment processing, clarity adjustment processing, grayscale adjustment processing or sampling rate change processing as image processing of the image signal.

[0055] Specifically, the image processing unit changes color, brightness, hue, clarity, data size (resolution or frame rate), and the like.

[0056] The image processing method according to the first disclosed technology described above relates to a signal processing method performed by a sensor device, including: an array sensor including a plurality of visible light or invisible light imaging elements arranged one-dimensionally or two-dimensionally; and an image processing unit that performs image processing on an image signal obtained by image capture using the array sensor. The signal processing method includes: setting a threshold value for parameter setting for all or some parameters used in image capture processing associated with image capture performed by the array sensor or image processing performed by the image processing unit, so as to implement processing using parameters that change based on the threshold value.

[0057] For example, an image signal indicating the minimum necessary quality or the like for processing of object detection or the like can be output by setting (changing) a parameter using a threshold value.

[0058] The image processing method according to the above-mentioned second disclosed technology relates to a signal processing method performed by a sensor device, including: an array sensor including a plurality of detection elements arranged in one dimension or two dimensions; and a signal processing unit, which acquires a detection signal obtained by the array sensor and performs signal processing on the detection signal. The signal processing method includes: detecting an object from the detection signal obtained by the array sensor, and giving an instruction to generate region information based on the detection of the object to the signal processing unit as region information associated with the acquisition of the detection signal obtained from the array sensor or associated with the signal processing of the detection signal; performing processing to set an active region of the detection signal obtained from the array sensor based on the previous region information; and using the detection signal of the active region as the detection signal obtained by the array sensor to detect the object.

[0059] In this case, it is not necessary to perform the object detection process for setting the region information on the entire frame of the image.

[0060] The image processing method according to the third disclosed technology relates to a signal processing method performed by a sensor device, including: an array sensor including a plurality of detection elements arranged one-dimensionally or two-dimensionally; and a signal processing unit that acquires a detection signal obtained by the array sensor and performs signal processing on the detection signal. The signal processing method includes: setting a threshold value of a frame rate according to a category identified for an object detected from the detection signal obtained by the array sensor, so that processing using a frame rate set based on the threshold value is performed according to the category identified for an object detected from the detection signal obtained by the array sensor.

[0061] A frame rate suitable for detecting the target class is applied by setting (changing) the frame rate using a threshold value. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 is a block diagram of a sensor device applicable to first to fifth and seventh embodiments of the present technology;

[0063] Figure 2 is an illustrative diagram depicting the relationship between image recognition accuracy and image quality;

[0064] Figure 3 is an explanatory diagram of an outline of the classification image adaptation process according to the first embodiment;

[0065] Figure 4 is an explanatory diagram of an image quality parameter set according to the first embodiment;

[0066] Figure 5 is a flowchart of the classification image adaptation process according to the first embodiment;

[0067] Figure 6is an explanatory diagram of the object detection process according to the first embodiment;

[0068] Figure 7 is an explanatory diagram of a region cropping framework according to the second embodiment;

[0069] Figure 8 is an explanatory diagram of an overview of the region clipping analysis according to the second embodiment;

[0070] Fig. 9 is a flow chart of the region clipping analysis according to the second embodiment;

[0071] Fig.10 is an explanatory diagram of calculating a region of interest (ROI) from a bounding box according to the second embodiment;

[0072] Fig.11 is an explanatory diagram of another example of ROI according to the second embodiment;

[0073] Fig.12 is an explanatory diagram of a high-level ROI according to the third embodiment;

[0074] Fig.13 is an explanatory diagram of a high-level ROI according to the third embodiment;

[0075] Fig.14 is a flow chart of a region clipping analysis using advanced ROI according to a third embodiment;

[0076] Fig.15 is an explanatory diagram of intelligent compression according to a fourth embodiment;

[0077] Fig.16 is a flow chart of intelligent compression according to a fourth embodiment;

[0078] Fig.17 is an explanatory diagram of active sampling according to the fifth embodiment;

[0079] Fig.18 is a flow chart of active sampling according to the fifth embodiment;

[0080] Fig.19 is an explanatory diagram of threshold value setting for image adaptation processing according to the sixth embodiment;

[0081] Fig. 20 is an explanatory diagram of threshold setting according to categories according to the sixth embodiment;

[0082] Fig.21 is a block diagram of a sensor device applicable to the sixth to ninth embodiments;

[0083] Fig. 22 is a flowchart of a process of performing parameter setting according to a threshold value according to a sixth embodiment;

[0084] Fig.23 is an explanatory diagram of threshold value setting according to the sixth embodiment;

[0085] Fig.24 is a flowchart of threshold value calculation according to the sixth embodiment;

[0086] Fig.25 is a block diagram of another configuration applicable to the sixth to ninth embodiments;

[0087] Fig.26 is a block diagram of yet another configuration applicable to the sixth to ninth embodiments;

[0088] Fig. 27 is an explanatory diagram of active region cutting according to the seventh embodiment;

[0089] Fig.28 is another explanatory diagram of active area cutting according to the seventh embodiment;

[0090] Fig.29 is a flow chart of active area clipping according to the seventh embodiment;

[0091] Fig.30 is a flowchart of key frame effective area calculation according to the seventh embodiment;

[0092] Fig.31 is an explanatory diagram of threshold setting of an advanced ROI (Advanced ROI: AROI) according to the eighth embodiment;

[0093] Fig.32 is a flow chart of a region clipping analysis using an advanced ROI according to an eighth embodiment;

[0094] Fig.33 is an explanatory diagram of active sampling according to a ninth embodiment;

[0095] Fig.34 is another explanatory diagram of active sampling according to the ninth embodiment;

[0096] Fig.35 is a flow chart of active sampling according to the ninth embodiment;

[0097] Fig.36 is a block diagram depicting an example of a schematic configuration of a vehicle control system;

[0098] Fig.37 1 is a diagram to help explain an example of the installation positions of the vehicle exterior information detection section and the imaging section. DETAILED DESCRIPTION

[0099] The embodiments will be described below in the following order.

[0100] <1. Configuration of sensor device>

[0101] <2. First embodiment: classification image adaptation>

[0102] <3. Second Embodiment: Region Clipping>

[0103] <4. Third Example: Region Cropping Using AROI>

[0104] <5. Fourth embodiment: intelligent compression>

[0105] <6. Fifth embodiment: active sampling>

[0106] <7. Sixth Embodiment: Image Adaptation by Threshold Setting>

[0107] <8. Seventh embodiment: active area trimming>

[0108] <9. Eighth Embodiment: Region Cropping Using Threshold Setting and AROI>

[0109] <10. Ninth embodiment: Active sampling by threshold setting>

[0110] <11. Application examples for mobile objects>

[0111] <12. Summary and revision>

[0112] Note that in the following embodiments, the sensor device 1 is described by way of example as having an imaging element array and serving as an image sensor for outputting an image signal as a detection signal. Specifically, it is assumed that the sensor device 1 of the embodiment has an object detection function realized by image analysis and constitutes a device called a smart array sensor.

[0113] <1. Configuration of sensor device>

[0114] Figure 1 Depicted is a configuration example of the sensor device 1. Note that Figure 1 Also depicted are a processor 11 and an external sensor 12, each of which serves as an external device that establishes data communication with the sensor device 1. The processor 11 is assumed to be any type of processor that is communicably connected to the sensor device 1.

[0115] The sensor device 1 includes an image sensor device as hardware, a storage area such as a dynamic random access memory (DRAM), and a component constituting an artificial intelligence (AI) function processor. In this case, these three components form a structure such as a three-layer laminated structure, one layer having a configuration generally referred to as horizontal mounting, and a two-layer (e.g., the same layer containing the DRAM and the AI ​​function processor) laminated structure constituting an integrated device.

[0116] like Figure 1 As shown, the sensor device 1 includes an array sensor 2, an analog-to-digital converter (ADC) / pixel selector 3, a buffer 4, a logic unit 5, a memory 6, an interface unit 7 and a calculation unit 8.

[0117] The ADC / pixel selector 3 , the buffer 4 , and the logic unit 5 constitute an example of a signal processing unit 30 that performs signal processing to output a detection signal obtained by the array sensor 2 to the outside.

[0118] The array sensor 2 includes visible light or non-visible light imaging elements constituting detection elements. A plurality of imaging elements are arranged one-dimensionally or two-dimensionally. For example, the array sensor 2 includes a large number of imaging elements arranged two-dimensionally in the row direction and the column direction, and is configured to output a two-dimensional image signal by photoelectric conversion performed by each imaging element.

[0119] Although the following description assumes that the array sensor 2 outputs a two-dimensional image signal as an image sensor, the array sensor 2 included in the sensor device 1 may have other configurations, for example, a sensor array module including an array of acoustic wave detection elements and a sensor array module including an array of tactile information detection elements.

[0120] The ADC / pixel selector 3 converts the electric signal photoelectrically converted by the array sensor 2 into digital data, and outputs an image signal in the form of digital data.

[0121] In addition, the ADC / pixel selector 3 having the function of selecting pixels (imaging elements) of the array sensor 2 is also capable of reading only the photoelectric conversion signals of the pixels selected by the array sensor 2, converting the read signals into digital data, and outputting the converted signals.

[0122] Therefore, the ADC / pixel selector 3 normally outputs digital data of photoelectric conversion signals of all effective pixels constituting one frame of image, but can also output digital data of photoelectric conversion signals of only selected pixels.

[0123] The ADC / pixel selector 3 reads the image signal in units of frames. The image signal of each frame is temporarily stored in the buffer 4, and is read and provided to the processing performed by the logic unit 5 at an appropriate time.

[0124] The logic unit 5 performs various types of necessary signal processing (image processing) on ​​the input image signal of each frame.

[0125] For example, it is assumed that the logic unit 5 realizes image quality adjustment by executing processes such as color correction, gamma correction, color grayscale processing, gain processing, contour emphasis processing, contrast adjustment processing, sharpness adjustment processing, and grayscale adjustment processing.

[0126] It is also assumed that the logic unit 5 performs processing for changing the size of data, for example, data compression processing, resolution conversion, frame rate conversion, aspect ratio conversion, and sampling rate change.

[0127] Parameters are set for each process performed by the logic unit 5 so as to be used for the corresponding process. For example, the set values ​​include correction coefficients for color and brightness, gain values, compression rates, frame rates, resolutions, processing target areas, and sampling rates. The logic unit 5 performs necessary processing using the parameters set for the corresponding processes. According to the present embodiment, in some cases, the calculation unit 8 sets these parameters as described below.

[0128] The image signal processed by the logic unit 5 is stored in the memory 6 .

[0129] The image signal stored in the memory 6 is transmitted and output to the processor 11 and other devices through the interface unit 7 at necessary timing.

[0130] Note that the memory 6 is assumed to be composed of a DRAM, a static random access memory (SRAM), a magnetoresistive random access memory (MRAM), or the like.

[0131] Note that MRAM is a memory that uses magnetism to store data. MRAM that includes a tunnel magnetoresistance (TMR) element instead of a magnetic core is known. The TMR element includes a thin insulating layer of several atoms with an extremely small thickness, and is sandwiched between magnets, and changes resistance according to the magnetization direction of the magnetic layer. Even after power is turned off, the magnetization direction of the TMR element does not change, thereby constituting a non-volatile memory. With the increase in miniaturization, it is necessary to increase the write current. Therefore, for the miniaturization of the memory cell, it is known to adopt an STT-MRAM using a spin injection magnetization reversal system (STT: spin torque transfer), which does not use a magnetic field but uses an electron flow each having a balanced spin to achieve writing.

[0132] It goes without saying that specific examples of the memory 6 include not only these examples but also other types of storage elements.

[0133] The processor 11 outside the sensor device 1 performs image analysis and image recognition processing on the image signal transmitted from the sensor device 1 to perform necessary object detection and the like.

[0134] The processor 11 can also refer to detection information obtained by the external sensor 12 .

[0135] Note that the processor 11 may be connected to the sensor device 1 by wire or wirelessly.

[0136] The processor 11 described herein may be provided in the same housing as the sensor device 1. For example, it is assumed that the processor 11 is a processor included in an imaging device or a terminal device including the sensor device 1.

[0137] Alternatively, the processor 11 may be provided on a device separate from the sensor device 1. For example, the processor 11 may be a built-in device included in an information processing device, a terminal device, an image editing device, a monitoring device, a communication device, etc., which is connected to the imaging device or the terminal device including the sensor device 1 via a cable, wireless communication, etc.

[0138] Furthermore, the processor 11 may be, for example, a processor included in a cloud computing system, and may be configured to establish network communication with the sensor device 1 or a device including the built-in sensor device 1 .

[0139] For example, the computing unit 8 is composed of, for example, an AI processor. In addition, as shown, the computing unit 8 includes a key frame selection unit 81, an object region identification unit 82, a category identification unit 83, and a parameter selection unit 84, each as an executable computing function. Note that these computing functions can be composed of multiple processors.

[0140] The key frame selection unit 81 performs processing for selecting key frames among image signal frames forming a video according to a predetermined algorithm or instruction.

[0141] Furthermore, in some cases, the key frame selection unit 81 also performs processing for switching the mode associated with the frame rate (the idle mode and the normal mode in the fifth embodiment).

[0142] The object area identification unit 82 performs processing of detecting an object area corresponding to a detection candidate from a frame of an image signal photoelectrically converted by the array sensor 2 and read by the ADC / pixel selector 3, and processing of identifying an area around an object (bounding box) in an image (frame) of an object of the detection target.

[0143] The object detected from the image signal refers to an object that becomes a detection target in order to be identified from the image. Which object will be designated as the detection target is different according to the detection purpose, processing power, application type, etc. of the sensor device 1 or the processor 11. Any object can be designated as the object of the detection target referred to in this article. Examples of detection target objects include animals, moving bodies (e.g., cars, bicycles, and airplanes), natural objects (e.g., vegetables and plants), industrial products or parts, buildings, institutions, mountains, seas, rivers, stars, the sun, and clouds. These examples are only partial examples, and any object can correspond to the detection target object.

[0144] In addition, as will be described in the second embodiment, the object area identification unit 82 performs processing for calculating a region of interest as region information indicating a region to be designated as a processing target (region of interest (ROI)) based on a bounding box, control of the ADC / pixel selector 3 based on the ROI, and other processing in some cases.

[0145] The category identification unit 83 classifies the object detected by the object region identification unit 82 .

[0146] The category in this article refers to the category of the identified object using image recognition. For example, the objects to be detected are divided into classes, such as "person", "car", "plane", "ship", "truck", "bird", "cat", "dog", "deer", "frog" and "horse".

[0147] As will be described in the first embodiment, the parameter selection unit 84 stores parameters for signal processing for each category and selects one or more corresponding parameters using the category of the detection object, the bounding box, etc. identified by the category identification unit 83. Thereafter, the parameter selection unit 84 sets one or more parameters for the logic unit 5.

[0148] Furthermore, as in the third embodiment, the parameter selection unit 84 stores a template of an advanced ROI (Advanced ROI: AROI) calculated in advance for each category based on the category of the ROI calculated from the bounding box, and in some cases, performs processing for selecting the template.

[0149] Furthermore, in the fifth embodiment, the parameter selection unit 84 stores the setting values ​​of the idle mode and the normal mode, and performs a process of selecting the setting values ​​based on object detection, and in some cases, controls the signal processing unit 30 .

[0150] These functions executed by the calculation unit 8 are processes not typically performed in an array sensor. According to this embodiment, object detection, category recognition, and control based on these are performed within the array sensor. In this way, an image signal suitable for detection purposes is provided to the processor 11, and a reduction in data volume is achieved without degrading detection performance.

[0151] Note that the interface unit 7 outputs the image signal to the processor 11 and is also capable of outputting information associated with the object detected by the calculation unit 8, category information, the number of detected objects, information associated with the selected parameters, etc. as metadata together with the image signal, or independently of the image signal. Additionally, for example, the interface unit 7 is also capable of outputting only the category information.

[0152] Furthermore, for example, it is also possible to adopt an instruction from the processor 11 to the interface unit 7 to give necessary information, allowing the output of information corresponding to the instruction from the interface unit 7.

[0153] <2. First Embodiment: Classification Image Adaptation>

[0154] The process of classification image adaptation will be described as the process of the first embodiment that can be performed by the sensor device 1 having the Figure 1 configuration.

[0155] According to image quality adjustment, the accuracy of image recognition is variable. For example, the accuracy of image recognition achieved through deep learning is improved through image quality control.

[0156] Furthermore, it is well known that the image quality required for image recognition (i.e., the image quality that improves object detection accuracy) does not have to be consistent with the quality of a beautiful image for a viewer.

[0157] For example, Figure 2 A in Figure 2 depicts an example of a high-quality image for a viewer, while

[0158] B in Figure 2 is assumed to be an image with slightly lower image quality for the viewer's perception due to, for example, a reduction in the number of gray levels. Figure 2 However, according to the object detection result of analyzing the image in

[0159] A using a neural network, the flower is misidentified as a fish, while the flower in the image in

[0160] Furthermore, such image quality suitable for object detection is not image quality adjusted by a uniform parameter, but image quality that is different for each object corresponding to the detection target. For example, the desired image quality adjustment state is different for each of the detection of a person and the detection of a car. In other words, the value of the desired parameter for image quality adjustment is different for each detection target.

[0161] Therefore, in the classification image adaptation process, appropriate parameters (image quality adjustment values) are stored for each class of objects that can be designated as targets. Subsequently, object detection and class recognition of the detected object are performed on the image captured by the array sensor 2, and parameters are selected and set for the logic unit 5 according to the recognized class. Then, the logic unit 5 performs processing based on the selected parameters for the corresponding image.

[0162] Figure 3 Describes the outline of the classification image adaptation process described above. Note that Figure 3 Indicates from Figure 1 The extracted part of the configuration used to explain the summary.

[0163] According to the sensor device 1, the imaged object light is focused on the array sensor 2 using the imaging optical system 40 to capture an image. The image signal G thus obtained is processed by the logic unit 5 and is also supplied to the calculation unit 8.

[0164] The calculation unit 8 performs processing for detecting an object corresponding to the candidate and identifying the region of the object using the object region identification unit 82. The object region identification unit 82 also calculates a bounding box of a necessary object region.

[0165] In addition, the calculation unit 8 performs category recognition on the detected object using the category recognition unit 83. In the case where a plurality of objects or a plurality of types of objects are detected, the category of each object and type is recognized to classify the objects and types into categories. For example, in the case of this figure, category recognition and classification are performed in a manner that one object in the "vehicle" category, five objects in the "person" category, and one object in the "traffic light" category are defined.

[0166] The class information and bounding box information thus obtained are supplied to the parameter selection unit 84. The parameter selection unit 84 selects one parameter set from the stored parameter sets PR1, PR2 and other parameter sets using the class information, etc. For example, the figure depicts a state where the parameter set PR4 has been selected.

[0167] Note that a parameter set refers to a collection of a plurality of parameter values ​​(eg, gain setting value, color correction coefficient, number of grayscales, compression ratio, and frame rate) that are stored and used for processing of the logic unit 5 .

[0168] The selected parameter set PR4 is set for the logic unit 5. The logic unit 5 performs various types of signal processing on the image signal G using the respective parameters indicated in the parameter set PR4.

[0169] The array sensor outputs all or any one of the output data (eg, image signal, category, number of objects, and presence or absence of a target category) in response to a request from the processor 11 .

[0170] Furthermore, the processor 11 can send various types of instructions to the sensor device 1 .

[0171] As described above, according to the sensor device 1, the computing unit 8 has a function of category recognition based on object detection (object classification function), and performs classification image quality adaptation (parameter selection suitable for the target category based on object detection), which adaptively sets parameters for the logic unit 5 according to the output from the category recognition unit.

[0172] For parameter sets, appropriate parameters (image quality setting values) are pre-generated through pre-learning and stored for each category using deep learning.

[0173] For example, to generate a parameter set for the “human” category, a large number of images of humans are used as learning data SD to perform deep learning to generate a parameter set PR1 having the highest image recognition rate considering human recognition, such as Figure 4 As shown in A.

[0174] For each of the other categories, parameter sets PR2, PR3, and other parameter sets each having the highest image recognition rate are similarly generated using deep learning.

[0175] Afterwards, if Figure 4 As shown in B, the generated parameter sets PR1, PR2, PR3 and other parameter sets corresponding to the respective categories are stored in a manner that can be selected by the parameter selection unit 84.

[0176] Figure 5 A specific example of the classification image adaptation process performed by the sensor device 1 is described. Figure 6 Describe them one by one.

[0177] Figure 5 The process performed by the sensor device 1 (mainly by the calculation unit 8 ) after the output of the image signal in units of one frame from the array sensor 2 starts in step S100 .

[0178] exist Figure 5 The processing performed by the computing unit 8 is performed by Figure 1The key frame selection unit 81, the object region identification unit 82, the category identification unit 83 and the parameter selection unit 84 are shown as processing performed by their respective functions. Note that this definition applies to Fig. 9 , Fig.14 , Fig.16 and Fig.18 .

[0179] In step S101 , the calculation unit 8 (key frame selection unit 81 ) performs processing for selecting a key frame at a timing corresponding to a key frame selection algorithm.

[0180] The sensor device 1 identifies the category of the image capture target by selecting a key frame from the image signal in frames as the pixel array output signal of the array sensor 2 and then performing image recognition. The selection of the key frame is achieved based on a key frame selection algorithm. In this way, a still image (a certain frame) is selected.

[0181] This article will give an example of a keyframe selection algorithm.

[0182] Initially, an approach that can be adopted is to select one frame for each interval of the specified time. For example, one frame is designated as a key frame every 30 seconds. Needless to say, the 30-second interval is just an example.

[0183] In addition, a key frame may be selected at a timing specified by a command from outside the sensor device 1 (e.g., the processor 11). For example, it is assumed that a key frame is selected according to an instruction from a device or apparatus on which the sensor device 1 is installed. For example, in the case where the sensor device 1 is installed in a car, a key frame is selected at the time when the car starts to drive after parking in a parking lot.

[0184] In addition, the key frame selection method may be changed according to the situation. For example, in the case where the sensor device 1 is mounted on a car, the interval of the key frame is changed for each of the stop state, the normal driving state, and the high-speed driving state.

[0185] After selecting a key frame, the calculation unit 8 (object region identification unit 82 ) detects the position of the candidate object in the key frame in step S102 .

[0186] Specifically, the calculation unit 8 searches for candidates of the object to be detected in the image of the key frame, and obtains one or more candidate positions (position coordinates in the image).

[0187] For example, suppose Figure 6 The input image depicted in A is designated as a key frame. The computing unit 8 detects a portion in the image that looks like an object to be detected. For example, Figure 6Each region in B and C of 6 is designated as a part that looks like a detected object. These regions are candidate regions of the object.

[0188] exist Figure 5 In step S103, the computing unit 8 (class identification unit 83) classifies the detected object. Specifically, the computing unit 8 identifies each candidate class of the object as a class for classification.

[0189] As mentioned above, each category herein refers to a category of an object to be recognized using image recognition.

[0190] For example, to achieve category recognition such as "person" and "flower", Figure 6 6D and 6E are shown.

[0191] exist Figure 5 In step S104, the calculation unit 8 checks whether the target category exists among the categories obtained as a result of category identification.

[0192] The target category is a category specifically selected from the categories and set by the processor 11. For example, in the case where the target category is set to "person", it is assumed that when a person is recognized, the sensor device 1 starts a specified process.

[0193] Furthermore, it is preferred that multiple target categories be allowed to be specified.

[0194] For example, in a state where "person" and "flower" have been specified as target categories, if "person" or "flower" exists in the category identified in step S103, the calculation unit 8 advances the processing flow from step S104 to step S105.

[0195] On the other hand, in the case where the target class does not exist, the calculation unit 8 returns to step S101 to select the next key frame.

[0196] In the case where the flow proceeds to step S105 by the presence of the target class, the calculation unit 8 (object region identification unit 82) calculates the correct position coordinates (bounding box) around the object region classified into the class.

[0197] Figure 6 6F and 6G both depict a bounding box 20. The bounding box 20 is specified by a minimum coordinate value Xmin and a maximum coordinate value Xmax defining the range of the region on the X axis, and a minimum coordinate value Ymin and a maximum coordinate value Ymax defining the range of the region on the Y axis.

[0198] exist Figure 5 In step S106 , the calculation unit 8 (parameter selection unit 84 ) selects a parameter set based on the category and number of objects and the area of ​​the bounding box 20 .

[0199] For example, in the presence of one target class, a parameter set corresponding to that class is selected.

[0200] In the case where objects in multiple types of target classes appear in the screen, the following example can be adopted.

[0201] For example, you can select a parameter set that corresponds to the category that contains the most objects among the categories.

[0202] Alternatively, in a case where objects in a plurality of types of target categories appear in the screen, a parameter set corresponding to a category containing an object having the largest area of ​​the bounding box 20 may be selected.

[0203] Alternatively, in the case where objects in a plurality of types of target categories appear in the screen, a parameter set corresponding to a category containing objects having the largest sum of areas of bounding boxes 20 calculated for each category may be selected.

[0204] Alternatively, in the case where objects of multiple types of target categories appear in the screen, a parameter set corresponding to a category with the highest priority may be selected, the highest priority being assigned based on the sum (or maximum value) of the number of objects in each category and the areas of the respective bounding boxes 20 .

[0205] Needless to say, other parameter set selection methods are also possible. In either method, it is sufficient if the main object in the screen is selected, or a parameter set corresponding to the category of the object to be detected preferentially is selected.

[0206] In step S107 , the calculation unit 8 (parameter selection unit 84 ) performs processing for setting the selected parameter set for the logic unit 5 .

[0207] Through this processing, the logic unit 5 is then allowed to perform various types of image processing using parameter sets set for image signals of respective frames sequentially input.

[0208] The processed image signal, set parameters, information associated with the recognized category, and the like are temporarily stored in the DRAM 6 .

[0209] In step S108, in response to a request from the processor 11, the sensor device 1 outputs all or at least any one of an image signal (still image or moving image), category identification information (e.g., category, number of objects, and presence or absence of a target category), a parameter set used, and other information.

[0210] Specifically, any one of the information temporarily stored in the DRAM 6 is read and transmitted via the interface unit 7 according to a request from the processor 11 .

[0211] Note that this processing in step S108 may be performed under the control of the computing unit 8, or may be performed in response to the access of the processor 11 to the DRAM 6 via the interface unit 7. In the case where the computing unit 8 does not control the interface unit 7, after completing step S107, the processing performed by the computing unit 8 returns to step S101.

[0212] Through the aforementioned processing, image signals for which parameters are set according to the presence of target classes as objects contained in the image are supplied to the processor 11. These image signals are image signals on which image processing suitable for detecting objects of the corresponding target classes has been performed.

[0213] Furthermore, if information associated with the detected category (target category) or the number of objects is also provided to the processor 11 , the information can be used as effective information for the object detection process performed by the processor 11 .

[0214] In these ways, the processor 11 can achieve high-accuracy object detection.

[0215] Note that a method of use may also be employed in which simplified category setting is performed within the sensor device 1 and detailed recognition is performed outside the sensor device 1. For example, a method of using the processor 11 to perform processing up to face recognition and license plate recognition without using the sensor device 1 may also be employed.

[0216] Furthermore, although it is assumed Figure 5 The processing example in step S102 detects each part that looks like an object ( Figure 6 B and C of 6), and in step S103, category recognition is performed ( Figure 6 D and E of 6), and then in step S105, a bounding box 20 ( Figure 6 F and G of 6). However, it is not required to adopt these processes. For example, the following process can be adopted. When each part that looks like an object is detected at the stage of step S102, a bounding box 20 is set. Thereafter, category recognition is performed in step S103. If there is a target category, the process proceeds from step S104 to step S106.

[0217] <3. Second Embodiment: Region Clipping>

[0218] Region clipping will be described as allowing Figure 1 The sensor device 1 of the configuration performs the processing of the second embodiment.

[0219] For the image signal detected by the array sensor 2, it is generally considered that information associated with all pixels of each frame is transmitted to the processor 11 and image recognition is performed.

[0220] However, when information associated with all pixels of all frames is transmitted to the processor 11 to perform object detection using the processor 11, the amount of transmitted information increases significantly, especially as the clarity of the image captured by the array sensor 2 increases. In this case, transmission time is also required. In addition, in the case of cloud transmission, the increase in communication volume significantly affects the cost and time of communication. In addition, the load on the storage capacity imposed on the processor 11 and the cloud also increases. In addition, the analysis processing load and processing time are further increased. In addition, the object detection performance may deteriorate.

[0221] Therefore, the second embodiment attempts to improve processing efficiency in the following manner. When a necessary object is recognized in an image of a certain frame, in each of the next frame and frames after the next frame, an image signal is acquired and transmitted at the pixel level of an approximate area of ​​the corresponding object, and the presence of information associated with pixels in other areas is eliminated.

[0222] Figure 7 Describes the outline.

[0223] Figure 7 A depicts an image of a specific frame F1. In the case where "person" is set as an object to be detected, a region of the person is detected in the image of the frame F1. Subsequently, each region where a person is detected is designated as a region of interest 21 corresponding to a region of interest (ROI).

[0224] In frames F2, F3, and up to Fn following frame F1, only pixels within the region designated as ROI 21 are read from the array sensor 2. This image corresponds to an image containing only information associated with a portion of ROI 21, such as Figure 7 As shown in B.

[0225] Thereafter, the calculation unit 8 performs analysis based on the image signal including the above-mentioned partial pixel information, or the image is transmitted to the processor 11 and analyzed by the processor 11 .

[0226] Specifically, Figure 8 A schematically depicted in FIG. 1 , a specific frame F1 that appears at a rate of one every N frames is designated as an image containing information associated with all valid pixels in the image signal obtained by the array sensor 2. Subsequently, the entire screen is scanned using the computing unit 8 to detect the presence or absence and position of the target object. Thereafter, the ROI 21 is set.

[0227] When acquiring the next frame F2, an image signal is acquired, wherein only the pixels in the ROI 21 designated as the target area are AD converted, as shown in FIG. Figure 8 As shown in B. Note that each square separated by the grid represents a pixel.

[0228] As described above, for example, the target object is detected by scanning the entire screen in only one frame every N frames, and Figure 8 As shown in C, in each of the subsequent frames F2, F3, F4, etc., image analysis is performed only on the detection area of ​​the target object detected in the previous frame.

[0229] This processing reduces the amount of analysis data and the amount of communication data without reducing the accuracy of object detection as an application target, thereby reducing power consumption of the sensor device 1 and increasing the speed of image analysis associated with object detection of the entire system in which the sensor device 1 is installed.

[0230] Fig. 9 FIG. 1 depicts an example of a process for region clipping analysis performed by the computing unit 8 of the sensor device 1 . Fig.10 Describe them one by one.

[0231] The calculation unit 8 (key frame selection unit 81 ) determines in step S201 whether the current time is the object detection key frame recording timing.

[0232] The object detection key frame recording timing herein refers to a time for acquiring information associated with the entire effective pixel area of ​​the array sensor 2 for object detection.

[0233] The object detection key frame recording timing may be determined according to a command issued from the outside of the sensor device 1 (eg, the processor 11). For example, it is assumed that the object detection key frame recording timing is determined at intervals of 60 seconds according to an instruction of 60 seconds.

[0234] At the object detection key frame recording timing, the calculation unit 8 advances the process to step S202 to obtain AD-converted image data of the entire effective pixel area of ​​the array sensor 2. For example, the calculation unit 8 causes the ADC / pixel selector 3 to output the image signal of the entire effective pixel area in one frame received from the array sensor 2.

[0235] The calculation unit 8 (the object region identification unit 82 ) detects the position of the candidate object in the acquired image in step S203 .

[0236] For example, when frame F1 is designated as an object detection keyframe, such as Fig.10 As shown in FIG. 1A , a candidate region 23 of an object is detected within the image of frame F1 . In this case, each region of the image containing a “person” or a “tree” is designated as a candidate region 23 .

[0237] exist Fig. 9 In step S204, the calculation unit 8 (class identification unit 83) classifies the detected objects into candidates.

[0238] For example, the computing unit 8 identifies each type of object in the candidate area 23 as “person”, “tree”, etc. Fig.10 As shown in B.

[0239] exist Fig. 9 In step S205, the calculation unit 8 checks whether the target category exists among the categories obtained as a result of category recognition.

[0240] For example, in the case where "person" is specified as the target class, it is considered that the target class exists as Fig.10 In this case, the calculation unit 8 changes the process from Fig. 9 Step S205 in advances to step S206.

[0241] On the other hand, in the case where the target class does not exist, the calculation unit 8 returns the flow to step S201 to wait for the next object detection key frame recording timing.

[0242] exist Fig. 9 In step S206 in FIG. 8 , the calculation unit 8 (the object region identification unit 82) calculates the bounding box 20 at the correct position coordinates of the region surrounding the object classified into the category specified as the target category.

[0243] For example, Fig.10 0044] FIGURE C depicts an example of a bounding box 20 of an image of a person corresponding to a target class. Specifically, the bounding box 20 is calculated as a more accurate region of an object corresponding to the target class.

[0244] exist Fig. 9 In step S207 , the calculation unit 8 (the object region identification unit 82 ) calculates the ROI based on the bounding box 20 .

[0245] Fig.10 D depicts the ROI 21 and the bounding box 20. The ROI 21 is calculated after expanding the horizontal and vertical dimensions (x×y) of the bounding box 20 (ax×by). For the horizontal and vertical dimensions, the ratios of expansion a and b can be set independently. The expansion ratio can be fixed or specified by the outside of the sensor device 1 (e.g., the processor 11).

[0246] The calculation unit 8 (object region identification unit 82 ) notifies the ADC / pixel selector 3 of the ROI thus calculated.

[0247] In response to the notification, the ADC / pixel selector 3 performs analog-to-digital conversion corresponding to only the pixels of the ROI 21 in the array sensor 2 and outputs the converted pixels.

[0248] exist Fig. 9In step S208, the computing unit 8 acquires image data of the next frame containing information associated only with pixels in the ROI 21. Thereafter, the computing unit 8 performs the processing in steps S203 and S204 on the acquired frame.

[0249] Fig.10 FIG. 4 schematically depicts AD conversion of only pixels in the ROI 21 among the entire effective pixels (each square separated by a grid in each figure represents one pixel).

[0250] like Fig.10 As shown in FIG. 2A , the AD conversion allows the computing unit 8 to acquire an image of a frame F2 having information associated only with a portion corresponding to the ROI 21 .

[0251] Thereafter, the computing unit 8 detects the position of the candidate object in the image of frame F2 and calculates Fig. 9 In steps S203 and S204, the objects are classified. Fig.10 In the case of F, a person is detected. Therefore, the process proceeds to steps S206 and S207 to calculate a new bounding box 20 and calculate a new ROI 21 based on the bounding box 20. Fig.10 In F, the newly obtained ROI is represented as "ROI 21 (new)".

[0252] Note that the ROI 21 is generated by expansion of the bounding box 20 to handle the movement of the object corresponding to the imaging object (or the change of the imaging object direction of the imaging device).

[0253] For example, Fig.10 The position of the person in frame F2 in E changes from Fig.10 The position of the person in the frame F1 in A is changed to the right. However, in a state where the ROI 21 is set as a wide area, the possibility of acquiring an image of the target person in the frame F2 increases even if only the pixels within the ROI 21 are moved.

[0254] As described above, the ROI 21 is obtained by expanding the bounding box 20 to detect the target object also in the next frame, but the expansion ratios a and b when the horizontal and vertical sizes (x×y) are expanded may be set to ratios corresponding to the frame rate.

[0255] For example, the time between frames increases at a low frame rate. In this case, the amount of movement of an object such as a person increases. Therefore, the size of the ROI 21 can be expanded to be larger than that in the case of a high frame rate.

[0256] Furthermore, the ROI 21 is recalculated (a new ROI 21 (new) is generated) for each frame to cope with the movement of the object corresponding to the imaging subject (or the change in the imaging subject direction of the imaging device).

[0257] Due to human movement, Fig.10 A person is detected at a position shifted to the right by the image of F. Therefore, a ROI 21 is obtained by calculating a new bounding box 20 around the person area to update the ROI in a manner consistent with the movement of the person indicated by the ROI 21 (new).

[0258] In step S207, the calculation unit 8 notifies the ADC / pixel selector 3 of the new ROI 21 (NEW). As a result, only the pixels within the new ROI 21 (NEW) of the next frame are AD converted (see Fig.10 G).

[0259] Furthermore, the calculation unit 8 similarly acquires image signals of only information associated with pixels within the ROI 21 (new) in step S208 , and performs the processing in step S203 and subsequent steps.

[0260] This process is repeatedly performed until it is determined in step S205 that there is no target class. Fig.10 When the image signal of the frame Fn containing information associated with the area of ​​the person is moved as indicated in the frame Fn of the H portion, the image signal of the frame Fn containing information associated with the area of ​​the person can also be acquired by updating the position of the ROI 21 according to the person corresponding to the imaging object based on the ROI 21 calculated in the previous frame F(n-1) not shown.

[0261] If the detected person moves out of the frame and becomes difficult to detect, it is difficult to obtain the target category. Therefore, the calculation unit 8 returns the process from step S205 to step S201 to wait for the next object detection key frame recording timing.

[0262] According to the region cropping analysis process performed as described above, at the object detection key frame recording timing, the data of all valid pixels are included in the image signal of the key frame, but only the pixels required for object detection are included in the subsequent frames. In this way, a significant reduction in the data amount of the image signal and the formation of an image suitable for detecting the target object can be achieved. In addition, by reducing the number of pixels to be read by the array sensor 2, a reduction in power consumption can also be achieved.

[0263] In addition, Fig. 9 In the case of the processing example of , ROI 21 is set for each object, each object corresponds to one target category, and the area of ​​ROI 21 corresponding to each object is the target read from the array sensor 2. In this case, the target is limited to the object detected in the object detection key frame. For example, even when a new object (e.g., a person) appears as an imaging object at the time of frames F2 and F3, it is possible that the image of this person is not acquired.

[0264] For example, when the purpose of use is to track and analyze objects found in object detection key frames at specific time intervals, this situation does not cause a problem. However, in the case of application to a surveillance system or the like to monitor all persons who appear as imaging subjects, it is preferred that objects appearing in frames other than the object detection key frames are also designated as detection targets.

[0265] Therefore, for example, even when detecting an object of the target category continues (ie, even if “yes” continues to be determined in step S205 ), the process can always be returned to step S202 at predetermined time intervals to acquire image signals of all valid pixels.

[0266] Also preferably, the processor 11 or the like specifies a time interval for acquiring image signals of all valid pixels.

[0267] Alternatively, the peripheral portion of the image may also be continuously designated as a region of the AD conversion target separately from the ROI 21. In this case, when a new object enters the frame as an imaging object, the new object is detected and the ROI 21 is given.

[0268] The above is a case where a rectangular region of the ROI 21 is generated by expanding the bounding box 20. However, the ROI 21 is not limited to a rectangular region.

[0269] For example, semantic segmentation, i.e., pixel-wise object region detection, can be used to compute ROIs from regions of objects of the target class 21.

[0270] Fig.11 An ROI 21 based on semantic segmentation is depicted. This is an example of setting the ROI 21 having a non-rectangular shape by extending a pixel region corresponding to an object (eg, a person).

[0271] For example, in the case of a truck with a protrusion and a cyclist, there is a case where the ROI 21 having a rectangular shape has a missing portion or is too large. By forming a non-rectangular ROI 21 based on the object position at the pixel level, the possibility of generating an ROI 21 that achieves data volume reduction and necessary information acquisition can be increased.

[0272] Note that by performing a combination of the region cropping of the second embodiment and the classified image adaptation processing in the above-described first embodiment, the effects of data amount reduction and improvement in detection accuracy can be more effectively obtained.

[0273] <4. Third embodiment: region cropping using AROI>

[0274] Region cropping using Advanced ROI (also denoted as “AROI”) will be described as being able to be performed by Figure 1The sensor device 1 of the configuration performs the processing of the third embodiment.

[0275] AROI is an ROI set using a template defined for each category.

[0276] The array sensor 2 (image sensor) consumes the highest power through photoelectric conversion. Therefore, it is preferable to reduce the number of pixels for photoelectric conversion as much as possible to achieve power consumption reduction.

[0277] Furthermore, the image signals obtained using the array sensor 2 are generated for the purpose of image analysis and are therefore not visible to the viewer. Therefore, these image signals do not need to be recognizable or form a beautiful image for the viewer. In other words, it is important to form an image that enables accurate target detection.

[0278] For example, in the second embodiment described above, the category is identified for the detected object. However, for this category identification, it is only necessary to set the minimum necessary area corresponding to the category as ROI. Therefore, Fig.12 and 13 Each of the AROIs depicted in 22.

[0279] Fig.12 An AROI 22 generated using a template corresponding to the class "person" for a person image region is depicted. Each grid in the figure represents a pixel. It is assumed that each dark pixel represents a pixel specified by the AROI.

[0280] For example, a template corresponding to the category "person" is a template that defines a face part as a part requiring high-density pixels and a body part as a part requiring low-density pixels to cover the whole.

[0281] also, Fig.13 Depicted is an AROI 22 generated using a template corresponding to the class "car". This example is for an image of the rear of a car. For example, the portion where the number plate is located is defined as the portion that requires high density pixels, while the other portions are defined as the portion that requires low density pixels to cover the entirety.

[0282] In actual situations, the category "person" can be divided into "person facing sideways", "person facing front", "person sitting", etc. to further divide the templates, and the category "car" can be divided into "side image", "front image", "rear image", etc. to further divide the templates.

[0283] In this way, a template is selected according to the category, and the selected template is enlarged or reduced according to the area size within the actual frame to generate the AROI 22 .

[0284] Fig.14An example of processing using AROI 22 is depicted.

[0285] Note that the processing from step S201 to step S206 is similar to Fig. 9 The corresponding processing in , so it will not be explained again.

[0286] In step S206 , the calculation unit 8 (the object region identification unit 82 ) calculates the bounding box 20 if the object of the target class exists in the image signals of all valid pixels obtained at the object detection key frame recording timing.

[0287] Then, the calculation unit 8 (parameter selection unit 84 ) selects an AROI template that is calculated and stored in advance based on the category in step S210 .

[0288] For example, in a state where "person" is the target category, when a person exists in an image, a template of "person" is selected.

[0289] In step S211 , the calculation unit 8 (the object region identification unit 82 ) calculates the AROI 22 based on the bounding box 20 .

[0290] For example, the template resized according to the size of the bounding box 20 is designated as the AROI 22 .

[0291] Thereafter, the calculation unit 8 (the object region identification unit 82 ) notifies the ADC / pixel selector 3 of the AROI 22 (the pattern and region of the AROI) thus calculated.

[0292] In response to the notification, the ADC / pixel selector 3 performs AD conversion corresponding to only the pixels of the AROI 22 in the array sensor 2 and outputs the converted pixels.

[0293] In step S212, the calculation unit 8 acquires image data of the next frame containing information associated only with pixels within the AROI 22. Thereafter, the calculation unit 8 performs the processing in steps S203 and S204 on the acquired frame.

[0294] The following processing flow is similar to the reference Fig. 9 Explain the corresponding processing flow.

[0295] By generating the AROI 22 using the template set according to the category as described above, even if the photoelectrically converted pixels are significantly reduced, information that enables accurate object detection according to the category can be obtained.

[0296] Note that the point where the object detection key frame recording timing is constantly set at certain time intervals and the point where the peripheral portion of the image is constantly designated as the area of ​​the AD conversion target as described in the second embodiment also applies to the third embodiment.

[0297] Furthermore, by performing a combination of the region cropping of the third embodiment using the AROI 22 as described above and the classified image adaptation processing in the first embodiment, the effects of data amount reduction and detection accuracy improvement can be more effectively obtained.

[0298] <5. Fourth embodiment: intelligent compression>

[0299] The process of intelligent compression will be described as being able to be performed by Figure 1 The sensor device 1 of the configuration performs the processing of the fourth embodiment.

[0300] Smart compression is a process of specifying a detection target object, compressing the target object at a low compression rate, and compressing objects other than the target object at a high compression rate.

[0301] Fig.15 A specific example is described.

[0302] Fig.15 A depicts a state where, in a case where the category “car” is detected as an object category from an image of a specific frame, an ROI 21 is generated for each car.

[0303] Fig.15 B is a state in which the image signal is compressed at a low compression rate in each region of the ROI 21 and is compressed at a high compression rate in other regions.

[0304] In this way, a reduction in the amount of analysis data and a reduction in the amount of communication data are achieved without reducing the detection accuracy of an object corresponding to a target of an object detection application.

[0305] Furthermore, attempts are made to reduce power consumption of the sensor device 1 and to increase the speed of image analysis associated with object detection of the entire system in which the sensor device 1 is installed.

[0306] Fig.16 Depicted is a processing example of smart compression.

[0307] Note that the processing from step S201 to step S206 is similar to Fig. 9 However, compared with the above-mentioned region cropping case, there are some differences. Therefore, the processing corresponding to these points will also be involved.

[0308] The calculation unit 8 (key frame selection unit 81 ) determines in step S201 whether the current time is the object detection key frame recording timing.

[0309] At the object detection key frame recording timing, the calculation unit 8 advances the flow to step S202 to acquire AD-converted image data of the entire effective pixel area of ​​the array sensor 2 .

[0310] However, in the case of smart compression, the ADC / pixel selector 3 reads (AD converts) the signals of all pixels from the array sensor 2 for each frame.

[0311] The calculation unit 8 (the object region identification unit 82) detects a position corresponding to a candidate object in the image acquired in step S201. In the subsequent step S204, the calculation unit 8 (the class identification unit 83) classifies the detected object as a candidate.

[0312] In step S205 , the calculation unit 8 checks whether or not a target class exists among the classes obtained as a result of class identification.

[0313] In step S206 , the calculation unit 8 (the object region identification unit 82 ) calculates the bounding box 20 if the object of the target class exists in the image signals of all valid pixels obtained at the object detection key frame recording timing.

[0314] In step S220, the calculation unit 8 (the object region identification unit 82) calculates the ROI 21 based on the bounding box 20. In this case, for example, the ROI 21 may also be defined by extending the bounding box 20.

[0315] The calculation unit 8 (object region identification unit 82 ) notifies the logic unit 5 of the ROI 21 thus calculated.

[0316] In response to the notification, the logic unit 5 performs compression processing at a low compression rate in the pixel region corresponding to the ROI 21 and performs compression processing at a high compression rate in other pixel regions on the image signal read from the array sensor 2 in step S221 .

[0317] Then, the compressed image signal is written into the DRAM 6 and transmitted to the processor 11 via the interface unit 7 .

[0318] The necessary area specified by the ROI 21 has been compressed at a low compression rate on the side of the processor 11. Therefore, by the presence of sufficient information, accurate object detection can be achieved.

[0319] As for the processing performed by the calculation unit 8, the flow returns to step S203 after completing step S220. Thereafter, the calculation unit 8 detects the position of the candidate object in the next frame, and identifies the category of the detected object in step S204.

[0320] According to the fourth embodiment of the present invention, all valid pixels are read from the array sensor 2 in each frame. Therefore, even in the case where the flow returns to step S203 after steps S220 and S211, the calculation unit 8 is allowed to scan the range of all valid pixels and detect candidate objects in step S203. By scanning the range of all valid pixels and detecting candidate objects, the appearance of new objects of the target class during the key frame recording timing can be continuously processed.

[0321] However, in this case, if the computing unit 8 detects the candidate object only in the area within the region of interest 22 , the processing load imposed on the computing unit 8 can be reduced.

[0322] The calculation unit 8 updates the ROI 21 in steps S206 and S220 according to the confirmation of the existence of the target class.

[0323] Therefore, the area compressed by the logic unit 5 with a low compression rate is also updated according to the position of the object in each frame.

[0324] If it is determined in step S205 that there is no target class, the processing flow executed by the calculation unit 8 returns to step S202 to wait for the object detection key frame recording timing.

[0325] Fig.16 The above-described processing presented in realizes intelligent compression processing that compresses at a low compression rate in a portion necessary for analysis, ie, within the ROI 21 of each portion containing an object of the target class, and compresses at a high compression rate in another portion.

[0326] Note that the point where the object detection key frame recording timing is constantly set at certain time intervals and the point where the ROI based on semantic segmentation is generated as described in the second embodiment also applies to the fourth embodiment described herein.

[0327] Furthermore, by performing a combination of the intelligent compression processing of the fourth embodiment described above and the classified image adaptation processing in the first embodiment, the effects of reducing the amount of data and improving the detection accuracy can be more effectively obtained.

[0328] <6. Fifth embodiment: active sampling>

[0329] Active sampling will be described as being possible by Figure 1 The sensor device 1 of the configuration performs the processing of the fifth embodiment.

[0330] Active sampling refers to a process of dynamically changing the frame rate according to the presence or absence of a target object. This process is considered to achieve compression of the data body in the time axis direction according to the presence or absence of the target object. In addition, this process achieves a reduction in power consumption of the sensor device 1.

[0331] Will refer to Fig.17 A summary describing active sampling.

[0332] This paper assumes that people are detected from captured images under the assumption that “people” are assigned as the target class. For example, consider the case where an external image is captured from inside a building through an entrance using a surveillance camera.

[0333] Fig.17 A depicts a state where the captured image does not contain a person. In this case, the frame rate is set to a low rate, for example, 1 frame per second (fps).

[0334] Fig.17 B depicts a state where a person is detected in a captured image. In this case, the frame rate is set to a high rate, for example, 100 fps.

[0335] In other words, the frame rate is dynamically changed subject to the constraints of a specific detection target so that the frame rate is lowered when not needed (when no person is detected) and increased when needed (when a person is detected) to obtain a large amount of information.

[0336] Fig.18 Depicted is a processing example of active sampling.

[0337] For example, in step S301 , the computing unit 8 (key frame selection unit 81 ) sets video capture for the ADC / pixel selector 3 according to the setting of the idle mode pre-stored in the computing unit 8 .

[0338] The parameter selection unit 84 in the computing unit 8 is configured to store, for example, settings for the idle mode and settings for the normal mode.

[0339] Active sampling includes the settings of idle mode and normal mode. Idle mode is the mode selected before determining that the captured screen contains an object of the target category.

[0340] In this idle mode, video capture is performed at a lower frame rate than in normal mode.

[0341] It may be adopted that the idle mode starts in response to a command from outside the sensor device 1. Alternatively, the idle mode may be started in response to a command of an interval of the idle mode data acquisition time issued from outside the sensor device 1. For example, in the case of a 60-second instruction, the object detection key frame recording timing is set at intervals of 60 seconds.

[0342] The normal mode is a normal video capture mode. For example, the normal mode responds to a command of an interval of a normal mode data acquisition time issued from outside the sensor device 1.

[0343] Video capture is usually performed at a higher frame rate than the idle mode. For example, in the case of a 0.01 second instruction, the normal mode becomes a mode in which image capture is performed at intervals of 0.01 seconds (100 fps).

[0344] Therefore, if the idle mode is set to 1 fps, video capture is performed at intervals of 1 second in response to the idle mode instruction issued from the calculation unit 8 to the ADC / pixel selector 3 in step S301.

[0345] Note that the settings of the idle mode and the normal mode do not necessarily need to be stored in the computing unit 8 , but may be stored in an external memory of the computing unit 8 .

[0346] Needless to say, the frame rates of the idle mode and normal mode are just examples.

[0347] Furthermore, preferably, the setting values ​​of the idle mode and the normal mode are rewritable from an external device such as the processor 11 .

[0348] The calculation unit 8 (the object region identification unit 82 ) detects the position of the candidate object in the acquired image in step S302 .

[0349] In step S303 , the calculation unit 8 (class identification unit 83 ) classifies the detected objects into candidates.

[0350] In step S304 , the calculation unit 8 checks whether or not there is a target class among the classes obtained as a result of class identification.

[0351] If there is no target category, the calculation unit 8 performs the processing in steps S301, S302 and S303. Specifically, the image of the next frame in the idle mode is acquired, and the position detection and category recognition of the candidate object are similarly performed. In this case, for example, assuming that image capture is achieved at 1fps, these processes are performed on the image after one second.

[0352] For example, in a state where “person” has been specified as the target class, in the event that “person” exists as the recognized class, the calculation unit 8 advances the processing flow from step S304 to step S305 .

[0353] The calculation unit 8 (key frame selection unit 81) sets the video capture for the ADC / pixel selector 3 according to the storage setting of the normal mode, and gives an instruction for image capture in the normal mode.

[0354] Therefore, if the normal mode is set to 100 fps, video capturing is performed at intervals of 0.01 seconds, for example.

[0355] In this way, the calculation unit 8 performs the processing in steps S302 and S303 in a state where the mode has been switched to the normal mode.

[0356] As long as the target class appears in the captured image, the normal mode continues. However, when the target class no longer exists, the process returns to step S301 to switch to the idle mode.

[0357] The above processing is performed as active sampling. In this way, data volume compression is achieved by reducing the frame rate, especially in the period without target categories, and thus power consumption is reduced.

[0358] According to this example, the calculation unit 8 instructs the ADC / pixel selector 3 to change the frame rate to achieve the frame rate change. However, the calculation unit 8 may instruct the logic unit 5 to change the frame rate.

[0359] For example, the readout from the array sensor 2 is always performed at 100 fps. In case of idle mode, the calculation unit 8 instructs the logic unit 5 to perform frame thinning. In this way, a reduction in the amount of data associated with the transfer to the processor 11 can be achieved.

[0360] Note that by combining the aforementioned active sampling process with the second, third, and fourth embodiments, a more effective data amount reduction effect can be produced without reducing detection accuracy.

[0361] Furthermore, by combining the active sampling process and the classification image adaptation process of the first embodiment, it is possible to achieve an improvement in detection accuracy and an effective reduction in the amount of data.

[0362] <7. Sixth Embodiment: Image Adaptation by Threshold Setting>

[0363] The process of image adaptation will be described as that of the sixth embodiment. The example described herein is an example of further adding the idea of ​​parameter variation corresponding to threshold setting to the process of classified image adaptation of the first embodiment.

[0364] The examples of parameters mentioned herein are assumed to be parameters used for image processing by the logic unit 5. For example, the parameters used for image processing by the logic unit 5 are set (adjusted or changed) so that the threshold value set for the sensor device 1 can be satisfied.

[0365] Furthermore, these parameters are also assumed to be parameters used for image capture processing, such as the signal read out by the ADC / pixel selector 3 and the exposure operation performed by the array sensor 2. For example, control parameters of the operation of the image capture processing performed by the ADC / pixel selector 3 or the array sensor 2 are set (adjusted or changed) so that the threshold value input to the sensor device 1 can be satisfied.

[0366] According to the first embodiment described above, the parameters used by the logic unit 5 are selected according to the class identification. However, the selected parameters may be set (adjusted or changed) based on a threshold value.

[0367] Alternatively, the parameters are not limited to the parameters selected on the class identification. As long as the parameters are used by the logic unit 5, the ADC / pixel selector 3 or the array sensor 2, the parameters can be set based on the threshold value.

[0368] Specific examples of parameters associated with the image capture process and image processing are presented below and are automatically set based on thresholds as described above.

[0369] For example, parameters associated with image processing are those presented below by way of example.

[0370] -Aspect ratio of the image

[0371] - Resolution

[0372] - Number of color levels (number of colors or digits)

[0373] -Contrast adjustment value

[0374] -Sharpness adjustment value

[0375] - Grayscale adjustment value

[0376] - Gamma correction value

[0377] - Sample rate conversion ratio

[0378] The parameters of the aspect ratio and resolution of the image are also reflected in ROI 21.

[0379] The number of gradations, contrast adjustment value, sharpness adjustment value, grayscale adjustment value, gamma correction value, and resolution are parameters associated with image quality.

[0380] The sampling rate conversion ratio is a parameter of time resolution.

[0381] Furthermore, parameters associated with the image capturing process include the following.

[0382] - Sampling rate

[0383] - Resolution (e.g., the resolution set during ADC / Pixel Selector 3 reading)

[0384] -Shutter speed (exposure time) of array sensor 2

[0385] It goes without saying that the parameters automatically set based on the threshold value include parameters other than the above-mentioned examples.

[0386] These parameters according to the thresholds are set for purposes such as reducing the amount of data, increasing processing speed, reducing power consumption, etc., while ensuring the actual accuracy of the output of the object detection when, for example, the processor 11 performs object detection based on learning using a deep neural network (DNN: Deep Neural Network).

[0387] Therefore, parameters are set so that the necessary accuracy level of target detection is maintained even after a reduction in the amount of image capture data achieved by changes in parameters such as resolution and the number of colors.

[0388] Reference Fig.19 Describe the idea of ​​threshold-based parameter setting.

[0389] For example, assume that an image output by capturing an image of a person using the sensor device 1 is full-color image data containing information associated with all pixels (all effective pixels) of the array sensor 2, and is output at a frame rate of 60fps (frames per second), for example.

[0390] Furthermore, it is assumed that correct detection of a person is achieved at a rate of 98% as a confidence level CR of 0.98 by performing object detection from the image data using the processor 11 etc. The confidence level is a confidence level that an object is correctly determined and detected.

[0391] On the other hand, by outputting image data with a slightly reduced resolution, a slightly reduced number of color levels, and a frame rate of 30 fps, it is assumed that detection is achieved with a confidence CR of 0.92.

[0392] Furthermore, by outputting image data with a further reduced resolution, a further reduced number of color levels, and a frame rate of 15 fps, it is assumed that detection is achieved with a confidence level CR of 0.81.

[0393] Furthermore, by outputting image data with a significantly reduced resolution, a significantly reduced number of color levels, and a frame rate of 10fps, it is assumed that detection is achieved with a confidence level CR of 0.58.

[0394] These are examples given for explanation purposes only. In this way, the confidence level changes depending on changes in parameters associated with image capture or image quality, such as resolution, number of colors, and temporal resolution image data corresponding to the analysis target. In other words, the accuracy of image analysis and object detection changes.

[0395] At the same time, preferably, the confidence of the object detection is the highest possible ratio. However, the highest ratio may not be obtained in actual situations.

[0396] For example, suppose from Figure 7A shows an approximate number of people detected in a bird's-eye view image captured in a park. In this case, such a high accuracy is not necessary. For example, in cases where a rough detection result of a few people, about 10 people, about 20 people, etc. is desired, a confidence CR of only about 0.6 may be sufficient.

[0397] On the other hand, in the case where it is desired to accurately monitor entry of a person or the like using a crime prevention camera or the like, a confidence level CR of about 0.95 may be required.

[0398] Furthermore, a confidence CR of 0.70 may be sufficient during the day, but a confidence of approximately 0.9 may be required at night.

[0399] Therefore, the confidence level CR required as the accuracy of object detection is different for each of various factors, such as the purpose of detection, target, type of device / application, season, and region.

[0400] Furthermore, the confidence level varies according to the analysis capability and learning level of the processor 11, and also varies according to the detection target and category.

[0401] Taking these into consideration, for example, by determining a threshold based on a desired appropriate confidence level and changing parameters according to the threshold, an image signal matching a request such as object detection can be output.

[0402] This paper assumes that Fig.19 A confidence level of 0.80 or higher is required in the example.

[0403] In this case, a parameter that realizes a threshold value of 0.80 or higher is calculated as the confidence CR and is set as a parameter used by the logic unit 5 etc. Specifically, a parameter that is higher than the threshold value but reduces the amount of data to a relatively small value is set.

[0404] For example, as shown, parameters such as resolution, number of tone levels, and frame rate are set to achieve a confidence CR of 0.81.

[0405] In this case, a significant reduction in the amount of data can be achieved, and the necessary object detection accuracy can be maintained compared to the case where parameters that achieve the confidence level CR of 0.98 are set and an image signal is output.

[0406] Note that the "threshold" may be considered as the value required for a desired confidence level. However, from the perspective of calculating a threshold for parameter adjustment, the "threshold" may be a parameter value used to obtain a desired "threshold" confidence level.

[0407] Specifically, from a technical point of view, the following processing methods [1] and [2] are assumed as processing for “setting a threshold value of a parameter, and performing processing using the parameter set based on the threshold value”.

[0408] [1] A threshold value of an index value is calculated, for example, a confidence level suitable for a usage mode or usage environment, and a parameter is set that is actually used as a parameter value for obtaining an index value that exceeds the threshold value of the index value. In other words, the threshold value of the parameter is set from the perspective of the index value of object detection.

[0409] [2] A threshold value of a parameter required to obtain a value such as an index value of confidence is calculated, and the parameter actually used is calculated based on the threshold value. In other words, the threshold value of the parameter is set from the perspective of the value of the parameter itself.

[0410] According to the present embodiment, for example, by setting a threshold based on confidence in the manner described in [1] or [2] above, parameters suitable for reducing the amount of image data to the minimum possible amount are actually used. These parameters are calculated in real time (for example, regularly during image capture) to dynamically change the parameters.

[0411] For example, according to the application, target category, and image capture environment of the sensor device 1, appropriate thresholds or parameters corresponding to these thresholds are calculated using DNN processing, and parameter changes are made. In this way, speed increase, power consumption reduction, and accuracy improvement suitable for applications, etc. are achieved.

[0412] Specifically, according to the parameter adjustment in the example explained in the sixth embodiment, a threshold value corresponding to the confidence level of object detection is defined, and a setting value of a parameter that is closest to but not lower than the threshold value is calculated.

[0413] Furthermore, it is appropriate to set a threshold value and a parameter corresponding to the threshold value for each category.

[0414] Fig. 20 A depicts an image classified as a “face”, while Fig. 20 FIG. 8B depicts an image classified as a “road sign” class.

[0415] Fig. 20 A depicts the following situation of a human face. In the case where the image signal indicates 8-bit RGB data 16777216 colors and a data volume of 1.5MB, a confidence CR of 0.99 is obtained. In the case where the image signal indicates 8-bit grayscale 256 colors and a data volume of 540KB, a confidence CR of 0.75 is obtained. In the case where the image signal indicates two colors of white-black and a data volume of 180KB, a confidence CR of 0.58 is obtained.

[0416] In this case, for example, assuming that the threshold of the confidence CR of a face is set to 0.74, the right image is inappropriate. Therefore, the parameters in the center image are suitable for parameter setting.

[0417] Fig. 20 B describes the following of the road sign. In the case where the image signal indicates 8-bit RGB data 16777216 colors and a data volume of 1.4MB, a confidence CR of 0.99 is obtained. In the case where the image signal indicates 8-bit grayscale 256 colors and a data volume of 520KB, a confidence CR of 0.95 is obtained. In the case where the image signal indicates two colors of white and black and a data volume of 110KB, a confidence CR of 0.88 is obtained.

[0418] In this case, for example, assuming that the threshold of the confidence CR of the road sign is set to 0.85, all images are suitable. The parameters of the right image are suitable for the parameter setting.

[0419] As described above, for example, for each object category, the detection accuracy of the image signal quality is different from the required accuracy. Therefore, it is appropriate to perform threshold setting and parameter change according to the category.

[0420] A specific example of the sixth embodiment will be described below.

[0421] first, Fig.21 An example of the configuration of the sensor device 1 is depicted. Note that Figure 1 The same constituent elements that correspond to the same constituent elements in the drawings are given the same reference numerals, and the same explanation is not repeated.

[0422] Fig.21 Configuration and Figure 1 The configuration in FIG. 8 is different in that the threshold value setting unit 85 is provided as a calculation function included in the calculation unit 8 constituted by, for example, an AI processor.

[0423] The threshold setting unit 85 serves as a DNN engine and performs processing for setting thresholds for all or some parameters of one of the following items: image processing performed by the logic unit 5, and image capture processing associated with image capture performed by the array sensor 2 (processing performed by the array sensor 2 and the ADC / pixel selector 3).

[0424] Furthermore, the threshold setting unit 85 enables all or some of the logic unit 5 , the array sensor 2 , and the ADC / pixel selector 3 to perform processing using parameters based on the threshold change.

[0425] Specifically, for example, the threshold value setting unit 85 changes the parameters used for the image processing of the logic unit 5 based on the threshold value, and sets the changed parameters for the logic unit 5 .

[0426] Alternatively, for example, the threshold setting unit 85 changes parameters used for image capture processing (e.g., exposure operation performed by the array sensor 2, readout processing and AD conversion processing performed by the ADC / pixel selector 3) based on the threshold, and sets the changed parameters for the array sensor 2 and the ADC / pixel selector 3.

[0427] Fig. 22 An example of processing performed by the calculation unit 8 of the sensor device 1 thus configured is presented. Fig. 22 The above processing performed by the threshold setting unit 85 is added to Figure 5 An example of the process of image adaptation for classification. Figure 5 The corresponding processing in Fig. 22 The processing in is given the same step number, and repeated explanation is omitted.

[0428] In this article Fig. 22 In the process performed by the threshold setting unit 85, steps S150 and S151 are added to Figure 5 is being processed.

[0429] In step S150, the calculation unit 8 determines whether the current time is the threshold calculation timing. If the current time is the threshold calculation timing, the flow proceeds to step S151. If the current time is not the threshold calculation timing, the flow proceeds to step S101.

[0430] Examples of the threshold calculation timing include the following.

[0431] a. Every predetermined time interval: for example, every hour from the time the image is captured

[0432] b. Each scheduled set time: for example, 0:00 am every day

[0433] c. Every predetermined number of times the target category appears: For example, every 1,000 times the target category appears

[0434] d. Every predetermined period of image capture of the target category: For example, every five hours of image capture of the target category

[0435] e. Time from external command: for example, an instruction from a device / equipment including the sensor device 1 (for example, the processor 11 ).

[0436] For example, these types of timings are determined in step S150. When the current time is determined as the threshold calculation timing, the calculation unit 8 calculates the threshold according to the threshold calculation strategy in step S151. Specifically, the threshold is determined, and a parameter corresponding to the threshold is set.

[0437] The threshold calculation strategy (threshold) is divided into several strategies according to the type or manner of understanding the parameters associated with the image capture process and image processing, and is taken into consideration when calculating the threshold, and is different for each application. The following is an example of the threshold calculation strategy.

[0438] -Use the inflection point of the confidence drop curve

[0439] The data size decreases as the resolution decreases. In this case, the advantage of reduced computational cost can also be achieved. However, on the contrary, the confidence level decreases.

[0440] exist Fig.23 In A, the horizontal axis represents resolution, and the vertical axis represents confidence.

[0441] As shown, at low resolutions equivalent to or below a certain resolution (inflection point), the confidence level is significantly reduced. Thus, for example, when the resolution is changed, an inflection point of a curve representing the relationship between confidence level and resolution is obtained. The inflection point or a point near the inflection point is considered as a threshold to set the parameters for reducing the resolution.

[0442] -Use the maximum confidence value

[0443] Depending on the class of object detection classification, confidence does not necessarily increase with the number of colors. For each object class, there is an optimal number of colors that achieves the highest confidence.

[0444] exist Fig.23 In B, the horizontal axis represents the number of color levels, and the vertical axis represents the confidence.

[0445] In the case where a confidence peak is observed as shown, a threshold is calculated based on the maximum value. For example, the maximum value (the peak value of the relationship curve between the confidence and the number of color levels) is considered as the threshold, or a predetermined range around the maximum value (for example, a value obtained by reducing the confidence by a predetermined percentage) is considered as the threshold. Subsequently, the parameter of the number of color levels is set according to the threshold.

[0446] - Use the remaining battery

[0447] Parameter settings enabling image capturing for N time periods are obtained based on the battery remaining amount, and parameters resulting in a confidence level becoming highest (or becoming equal to or higher than a predetermined rate) are selected and set from among the obtained settings.

[0448] For example, the threshold value, eg, the confidence level, may be lowered according to the remaining battery level in order to obtain the longest possible image capture time, and the parameters may be set according to the confidence level.

[0449] - Keep track of objects using temporal resolution

[0450] Object tracking in this article refers to tracking and identifying a specific detection object (object) along the forward direction of the frame in consecutive frames of the image signal.

[0451] Generally, as the temporal resolution of the image signal decreases, the computational cost required for object tracking increases.

[0452] Based on a threshold indicating a parameter for maintaining object tracking, the temporal resolution and other parameters are determined, prioritizing reduction in computational cost of object tracking.

[0453] For example, any of the strategies exemplified above is adopted, and a predetermined algorithm is used to calculate the threshold. Fig.24 An example of the threshold value calculation process performed by the calculation unit 8 is depicted.

[0454] Depend on Fig.24 The processing performed by the computing unit 8 in Fig.21 The processing is performed by the respective functions of the object region identifying unit 82, the category identifying unit 83, and the threshold value setting unit 85 depicted in FIG.

[0455] In step S160 , the calculation unit 8 acquires image signals in units of one frame from the array sensor 2 .

[0456] In step S161 , the calculation unit 8 (the object region identification unit 82 ) detects the position of the candidate object in the acquired frame.

[0457] Specifically, the calculation unit 8 searches for candidates of the object to be detected in the image of the frame to obtain one or more candidate positions (position coordinates in the image).

[0458] In step S162, the computing unit 8 (class identification unit 83) classifies the detected object. Specifically, the computing unit 8 identifies a class for each candidate for the classified object.

[0459] In step S163 , the calculation unit 8 checks whether or not the target class exists among the classes obtained as a result of class identification.

[0460] For example, as described above, the target class is a class set by the processor 11. Specifically, a class specified as a target for object detection by the processor 11 is assumed to be the target class.

[0461] If there is no target category, the calculation unit 8 returns the process to step S160 to obtain image data of the next frame.

[0462] In a case where the target class exists, the calculation unit 8 advances the process from step S163 to step S164.

[0463] In the case where the flow proceeds to step S164, the calculation unit 8 (object region identification unit 82) calculates the correct position coordinates (bounding box 20) around the object region classified into the categories. Figure 6 The descriptions of F and G of 6 and other figures regarding the bounding box 20 apply to the bounding box 20 herein.

[0464] In step S165 , the calculation unit 8 (threshold setting unit 85 ) calculates the threshold and the parameter while changing the parameters of the image capturing process or the image processing for the object class surrounded by the bounding box 20 according to the threshold calculation strategy set in advance.

[0465] In step S166 , the calculation unit 8 (threshold value setting unit 85 ) records information indicating the threshold value, parameter, target class, and threshold value calculation strategy calculated in step S165 in association with each other.

[0466] For example, the computing unit 8 records the information in a recording area within the computing unit 8 , records the information in a predetermined area of ​​the memory 6 , or transmits the information to the processor 11 and records the information in the processor 11 .

[0467] In this way, thresholds and parameters corresponding to the target class are set.

[0468] For example, in Fig. 22 In step S151 of , the threshold and the parameters are set in the above-described manner. Therefore, the parameter set associated with a certain target class changes for each threshold calculation timing.

[0469] When the target class is a person, for example, all or some of the parameter sets corresponding to the person are changed according to a threshold value.

[0470] Fig. 22 Steps S101 to S107 in are similar to Figure 5 Therefore, in this case, the parameter set corresponding to the target category is selected. In the subsequent step S107, the parameter set is set for the logic unit 5.

[0471] The parameter set that has been set for the logic unit 5 at this time is a parameter set that is suitable for the target class, and is a parameter set that is changed based on the threshold value calculated by the aforementioned processing in step S151.

[0472] As described above, the calculation unit 8 (threshold value setting unit 85 ) gives instructions for necessary processing, such as instructions for transferring parameters to the logic unit 5 and instructions for changing parameters, so as to change the parameters used by the logic unit 5 .

[0473] In this manner, parameters associated with the image processing or image capture process are changed to values ​​that reduce the amount of image data to the minimum possible amount based on the threshold value.

[0474] Therefore, the image signal output from the interface unit 7 has image quality and the like maintaining the necessary accuracy of object detection required by the processor 11 and also has a small data amount.

[0475] Although Fig. 22 An example of processing is to add the idea of ​​parameter change based on threshold setting to Figure 5 An example of the classification image adaptation process, but Fig. 22 The processing example does not need to be combined with the processing of classification image adaptation.

[0476] For example, a processing example including only steps S150, S151, S107, and S108 may also be adopted ( Fig. 22 , in which the processing example of steps S101 to S107 is removed).

[0477] Specifically, in step S151, parameters for image processing of the logic unit 5, or parameters for image capture processing of the array sensor 2 or ADC / pixel selector 3 are set based on the threshold value calculation.

[0478] Furthermore, at the time of step S107 , parameters according to the threshold setting are set for the logic unit 5 , the array sensor 2 , or the ADC / pixel selector 3 .

[0479] In other words, the calculation unit 8 (threshold setting unit 85) transmits the parameters set according to the threshold to some or all of the logic unit 5, array sensor 2 and ADC / pixel selector 3, or gives parameter change instructions to some or all of these units.

[0480] In this case, for example, such a process of sequentially changing the default parameter sets of, for example, the logic unit 5, the array sensor 2, and the ADC / pixel selector 3 is implemented based on threshold calculation rather than on the idea of ​​using a parameter set corresponding to each category.

[0481] The configuration including the threshold setting unit 85 is not limited to Fig.21 Instead, it can be, for example Fig.25 and Fig.26 The configuration example depicted in .

[0482] Fig.25 A configuration example of the terminal device 100 is depicted in which the calculation unit 8 is provided separately from the sensor apparatus 1. Note that various types of apparatuses such as an information processing terminal and an imaging apparatus terminal are applicable to the terminal device 100.

[0483] The computing unit 8 is provided in the terminal device 100 as a chip separate from the sensor device 1 , and communicates with the sensor device 1 via the interface unit 7 .

[0484] Furthermore, the calculation unit 8 includes a threshold setting unit 85 constituting a DNN engine for threshold setting.

[0485] in this case, Fig.25 The computing unit 8 is also capable of executing Fig. 22 The above situation is handled similarly.

[0486] Notice, Fig.25 The configuration example described in the above also applies to the configuration examples corresponding to the first to fifth embodiments. Figure 1 The calculation unit 8 may be a unit separate from the sensor device 1 .

[0487] Fig.26 The configuration example is an example in which the threshold setting unit 85 constituting the DNN engine for threshold setting is constituted by a processor or the like provided independently from the sensor device 1 and the calculation unit 8 .

[0488] For example, the terminal device 100 has a configuration including a sensor apparatus 1 (including a calculation unit 8 ), a processor 11 , an external sensor 12 , and a threshold setting unit 85 .

[0489] In this case, the threshold setting unit 85 can similarly communicate with the sensor device 1 via the interface unit 7 and can cooperate with the calculation unit 8 to perform the same Fig. 22 The processing is similar to that of .

[0490] In addition, although not shown, in the Fig.25 In the illustrated configuration in which the sensor device 1 and the computing unit 8 are separated from each other, the threshold setting unit 85 may also be constituted by a separate processor or the like.

[0491] Note that similarly to the threshold setting unit 85, a configuration may be adopted in which the key frame selection unit 81, the object region identification unit 82, the category identification unit 83, the parameter selection unit 84, etc. are provided outside the sensor device 1 or the calculation unit 8. This applies to Figure 1 Modifications of the configuration shown.

[0492] <8. Seventh embodiment: active area trimming>

[0493] In the seventh embodiment, an example in which further efficient processing is achieved using the ROI of the above-described second embodiment is described.

[0494] Note that the processing of the seventh embodiment described below is applicable to Figure 1 , Fig.21 , Fig.25 and Fig.26 Any of the configurations depicted in .

[0495] According to the example described in the second embodiment above, Figure 7 and Figure 8 As shown, a ROI 21 is set for an object corresponding to a detection target, and only pixels within a region designated as the ROI 21 are read from the array sensor 2.

[0496] It is noted herein that in some cases the region designated as ROI 21 is focused on a specific area within the image.

[0497] Fig. 27 A depicts an example of an image captured by, for example, a surveillance camera set inside a building. Assume that an ROI 21 is set for a person as a detection target. The figure indicates the position of a bounding box 20 in the image. The ROI 21 is specified based on the bounding box 20 set in advance within a predetermined period.

[0498] In this case, for example, the setting position of the bounding box 20 (and the ROI 20 ) in the previous predetermined period is located in the area near the floor in the image.

[0499] In other words, detection processing for detecting a person in the image region around the ceiling is unnecessary because a person does not appear in the region near the ceiling in the image.

[0500] So, for example, Fig. 27 As shown in B, the area where the "person" corresponding to the detection target appears (i.e., the area where the bounding box 20 has been previously set in a predetermined period) is designated as the active area RA, and the area where the "person" corresponding to the detection target does not appear (i.e., the area where the bounding box 20 has not been previously set in a predetermined period) is designated as the inactive area DA.

[0501] Fig.28 A depicts an example of an image captured by a monitoring camera that monitors, for example, a car as a detection target on a highway, and indicates the position of a bounding box 20 set in advance in a predetermined cycle.

[0502] In this case, the car appears close to the road. Therefore, it can be similar to Fig. 27 Those in B are used to set the active area RA and the inactive area DA.

[0503] As in Fig. 27 B and Fig.28 In the example depicted in B of FIG. 1 , an active area RA is set, and an object is detected from the detection signal of the active area RA in the pixel captured by the array sensor 2. Thereafter, similarly to the second embodiment, an instruction to designate the ROI 21 generated based on the detection of the object as a region associated with the acquisition of the detection signal or the signal processing of the detection signal is provided to the signal processing unit 30.

[0504] Specifically, an object is detected from an object detection key frame by partial photoelectric conversion performed not by full-screen scanning but based on history information associated with object detection.

[0505] Note that the object detection key frame herein refers to a frame that acquires information from all effective pixel areas of the array sensor 2 for object detection in the second embodiment. The processing in the seventh embodiment is a processing that acquires information only from the pixel area of ​​the active area RA in the key frame.

[0506] Fig.29 An example of processing performed by the computing unit 8 is depicted. Note that Fig. 9 The same processing is assigned the same step number to the corresponding processing in .

[0507] In step S250, the calculation unit 8 determines whether the current time is the active area calculation timing of the key frame. If the current time is the calculation timing, the flow proceeds to step S161. If the current time is not the calculation timing, the flow proceeds to step S201.

[0508] For example, the following can be used as the calculation timing of the active area RA of the key frame.

[0509] a. Every predetermined time interval: for example, every hour from the start of image capture

[0510] b. Each scheduled set time: for example, 0:00 am every day

[0511] c. Every predetermined number of times the target category appears: For example, every 1,000 times the target category appears

[0512] d. Every predetermined period of image capture of the target category: For example, every five hours of image capture of the target category

[0513] e. Time from external command: for example, an instruction from a device / equipment including the sensor device 1 (for example, the processor 11 ).

[0514] In the case where the flow proceeds to step S161 based on the determination that the current time is the calculation timing, the calculation unit 8 calculates the active area RA of the key frame.

[0515] Fig.30 An example of a calculation process of calculating the active area RA of a key frame is depicted.

[0516] In step S271 , the calculation unit 8 (the object region identification unit 82 ) calculates pixels in the appearance region of the array sensor 2 in which the bounding box 20 of the target class appears in the previous predetermined period.

[0517] In this case, all pixels in each bounding box 20 that has appeared correspond to pixels in the corresponding appearance area. Preferably, a range including all bounding boxes 20 that have appeared is set, and all pixels in the range are designated as pixels of the appearance area.

[0518] Alternatively, a range including all bounding boxes 20 that have appeared may be expanded in the peripheral direction, and all pixels in the expanded range may be designated as pixels of the appearance area.

[0519] The pixel range containing all appearance areas of the bounding box 20 thus calculated is designated as the active area RA.

[0520] In step S272, the calculation unit 8 (object area identification unit 82) records the calculated pixel area together with the category name as the active area RA of the key frame. For example, the calculation unit 8 records the pixel area in a recording area within the calculation unit 8, records the pixel area in a predetermined area in the memory 6, or transmits the pixel area to the processor 11 and records the pixel area in the processor 11.

[0521] In this way, the active area RA corresponding to the target category is set.

[0522] exist Fig.29 In step S201, the calculation unit 8 (key frame selection unit 81) determines in step S201 whether the current time is the object detection key frame recording timing.

[0523] Similar to the second embodiment ( Fig. 9 ), the object detection key frame recording timing in this article refers to the timing for acquiring information from the array sensor 2 for object detection.

[0524] The object detection key frame recording timing may be determined according to a command issued from outside the sensor device 1 (eg, of the processor 11). For example, it is assumed that the object detection key frame recording timing is determined at intervals of 60 seconds according to an instruction of 60 seconds.

[0525] At the object detection key frame recording timing, the calculation unit 8 advances the process to step S252 to obtain the AD-converted image data of the pixels in the active area RA of the array sensor 2. For example, the calculation unit 8 causes the ADC / pixel selector 3 to output the image signal of the active area RA in one frame received from the array sensor 2.

[0526] The calculation unit 8 (object region identification unit 82) detects the position of the candidate object in the acquired image in step S203. The processing from step S203 to step S208 is similar to Fig. 9 The corresponding processing in .

[0527] According to the above Fig.29 In the process in step S203, the object detection in step S203 may not be performed in the entire effective pixel area of ​​one frame, but may be performed only in the active area RA. In addition, the active area RA is an area where an object of the target category can be detected. In other words, the area other than the active area RA is an area where basically no object of the target category is detectable.

[0528] Therefore, by reducing the number of pixels read from the object detection key frame and reducing the detection range, it is possible to achieve improved processing efficiency, reduced power consumption, etc.

[0529] Although the active area RA is set based on the history of the bounding box 20 in the above example, the active area RA may be set based on the history of the ROI 21. In this case, the history of the pixel position of the ROI movable per frame may also be added (refer to Fig.10 Explanation of ROI 21 (new)).

[0530] <9. Eighth Embodiment: Region Cropping Using Threshold Setting and AROI>

[0531] In the eighth embodiment, a processing example of further improving the efficiency of region cropping using the AROI 22 in the third embodiment is described. Note that the processing of the eighth embodiment described herein is applicable to Fig.21 , Fig.25 and Fig.26 Any of the configurations depicted in .

[0532] The eighth embodiment sets the parameters of the object (category) or part corresponding to the target to be detected based on the set threshold value when using the AROI 22 using the template. Specifically, the idea of ​​the threshold value in the sixth embodiment is adopted. The threshold value is determined based on the object detection accuracy calculated by the DNN to set the parameters.

[0533] For example, the resolution distribution of the region of interest within the AROI 22 is determined according to a threshold value set based on the confidence level.

[0534] Fig.31 An example is schematically depicted. This paper will consider the case where a person is assigned as the target class and the case where a face is assigned as the target class.

[0535] It is assumed that the relationship (first resolution)>(second resolution)>(third resolution) holds.

[0536] Assume that the confidence CR of face detection is 0.95 at the first resolution, 0.86 at the second resolution, and 0.66 at the third resolution.

[0537] Assume that the confidence CR for human detection is 0.98 at the first resolution, 0.81 at the second resolution, and 0.65 at the third resolution.

[0538] In the case where the threshold value thF for face detection is 0.85, the second resolution is selected as a parameter suitable for reducing the amount of image data to the minimum possible amount, and image processing is performed on the pixels within the template.

[0539] Furthermore, in the case where the threshold value thP for human body detection is 0.80, the second resolution is selected as a parameter suitable for reducing the amount of image data to the minimum possible amount, and image processing is performed on the pixels within the template.

[0540] In these cases, the second solution is applicable to both cases. However, depending on the case, assume that in the case of face detection, the first resolution is set at a threshold value thF set to 0.94, or for human body detection, the third resolution is set at a threshold value thP set to 0.60.

[0541] In other words, in the case of using the AROI 22 , a threshold is set for each object class so as to set parameters of image processing, reading processing, and the like for pixels within the AROI 22 .

[0542] Fig.32 An example of processing performed by the calculation unit 8 is depicted.

[0543] Fig.32 Steps S250 and S251 in are similar to Fig.29 The calculation unit 8 calculates the active area RA ( Fig.30 in the process).

[0544] Steps S260 and S261 are similar to Fig. 22 Specifically, in step S260, the calculation unit 8 (threshold setting unit 85) determines whether the current time is the threshold calculation timing. If the current time is the threshold calculation timing, then in step S261, the calculation unit 8 (threshold setting unit 85) calculates the threshold ( Fig.24 in the process).

[0545] In this case, the calculation unit 8 (the object region identification unit 82) performs the processing in steps S160 to S164 as similar to the above. Fig.24Thereafter, in step S165, the calculation unit 8 (threshold setting unit 85) calculates a threshold based on the confidence obtained from the data corresponding to the pixel area of ​​the AROI pattern as a template while changing the resolution of the target class surrounded by the bounding box 20, and sets parameters based on the calculated threshold.

[0546] In this case, the parameters of the AROI pattern as a template corresponding to the category are set according to the threshold value. For example, the resolution of the area of ​​the AROI pattern is set.

[0547] In step S166, the calculation unit 8 (threshold setting unit 85) records information indicating the threshold, target category, AROI mode and necessary parameters, and threshold strategy in association with each other. For example, the calculation unit 8 records the information in a recording area within the calculation unit 8, records the information in a predetermined area of ​​the memory 6, or transmits the information to the processor 11 and records the information in the processor 11.

[0548] exist Fig.32 In step S201 of the present invention, the calculation unit 8 determines whether the current time is the object detection key frame recording timing. When it is determined that the current time is the object detection key frame recording timing, the calculation unit 8 advances the process to step S252 to obtain the AD-converted image data of the pixels in the active area RA of the array sensor 2. For example, the calculation unit 8 causes the ADC / pixel selector 3 to output the image signal of the active area RA in one frame received from the array sensor 2.

[0549] The calculation unit 8 (object region identification unit 82) detects the position of the candidate object in the acquired image in step S203. Steps S203 to S206 herein are similar to Fig. 9 .

[0550] After calculating the bounding box 20 in step S206 , the calculation unit 8 (parameter selection unit 84 ) selects the AROI pattern calculated and recorded based on the threshold value in step S262 .

[0551] In step S211 , the calculation unit 8 (object region identification unit 82 ) calculates the actual AROI 22 based on the bounding box 20 . Specifically, the actual pixel region corresponding to the selected AROI pattern is obtained. For example, a template whose size is adjusted according to the size of the bounding box 20 is designated as the AROI 22 .

[0552] Thereafter, the calculation unit 8 (the object region identification unit 82 ) notifies the ADC / pixel selector 3 of the AROI 22 (the pattern and region of the AROI) thus calculated.

[0553] In response to the notification, the ADC / pixel selector 3 performs AD conversion corresponding to only the pixels of the AROI 22 in the array sensor 2 and outputs the converted pixels.

[0554] In step S212, the calculation unit 8 acquires image data of the next frame containing information associated only with pixels within the AROI 22. Thereafter, the calculation unit 8 performs the processing in steps S203 and S204 on the acquired frame.

[0555] As described above, for example, parameters corresponding to the confidence level are also set for the AROI 22 to set parameters such as resolution, etc. In this way, the efficiency of the image capturing process and the image processing can be improved while maintaining the object detection accuracy.

[0556] <10. Ninth embodiment: Active sampling by threshold setting>

[0557] In the ninth embodiment, an example is described which adds a method for determining the temporal resolution based on the object detection accuracy calculated by the DNN to the active sampling method explained in the fifth embodiment.

[0558] Specifically, a process of dynamically changing the frame rate based on the average movement amount of the object category per unit time is performed.

[0559] Note that the processing of the ninth embodiment described herein is applicable to Fig.21 , Fig.25 and Fig.26 Any of the configurations depicted in .

[0560] In the fifth embodiment described above, the normal mode and the idle mode are prepared, and the idle mode is selected to set a low frame rate during a period in which the target class is not detected in the captured image. In addition, when the target class is confirmed to exist, the normal mode is selected to increase the frame rate and increase the amount of information.

[0561] In addition to the aforementioned processing, the ninth embodiment sets the frame rate in the normal mode according to the target category.

[0562] Fig.33 A depicts an example of an image in the case where the sensor device 1 is included in a monitoring camera capturing an image of a highway. The figure includes a car designated as an object class and a bounding box 20. The dashed arrow indicates the direction of movement of a certain car.

[0563] Fig.33 B presents the movement of the car captured by the image as the change in the position (pixel position) of the bounding box 20 in the image of consecutive frames. This movement is taken into account for many cars. Assume that the average movement is 1152 pixels / second.

[0564] In this case, it is assumed that the computational sampling rate used to maintain object tracking (tracking the target object in consecutive frame images) is 46fps.

[0565] Next reference Fig.34 A depicts an example of an image in which the sensor device 1 is included in a surveillance camera set up inside a building. The image contains each person assigned to a target class and a bounding box 20 corresponding to the person. The dotted arrow indicates the direction of movement of a certain person.

[0566] Fig.34 B presents the movement of the person captured by the image as the change in the position (pixel position) of the bounding box 20 in the image of consecutive frames. This movement is checked for each of the many people. Assume that the average movement is 192 pixels / second.

[0567] In this case, assume that the computational frame rate used to maintain object tracking is 5fps.

[0568] For example, the frame rate used to maintain object tracking where a car is designated as the target class is different than the frame rate used to maintain object tracking where a person is designated as the target class.

[0569] Therefore, by obtaining the frame rate for maintaining object tracking using a DNN based on the target category and then obtaining a threshold for the frame rate (the lower limit of the allowed frame rate), the accuracy of object detection for detecting the target object and tracking the target object can be maintained while reducing the amount of data to the minimum possible amount.

[0570] Note that the frame rate is determined according to the setting of the readout timing of the array sensor 2 and the setting of the sampling rate of the ADC / pixel selector 3 .

[0571] Fig.35 An example of processing performed by the calculation unit 8 is depicted.

[0572] Steps S350 and S351 are similar to Fig. 22 Specifically, in step S350, the calculation unit 8 (threshold setting unit 85) determines whether the current time is the threshold calculation timing. If the current time is the threshold calculation timing, then in step S351, the calculation unit 8 (threshold setting unit 85) calculates the threshold ( Fig.24 in the process).

[0573] As in this case Fig.24In the threshold value calculation processing in the calculation unit 8 (object region identification unit 82), the calculation unit 8 performs the processing in steps S160 to S164 similarly to the above. Subsequently, the calculation unit 8 (threshold value setting unit 85) calculates the threshold value (frame rate as the threshold value) for maintaining object tracking while changing the frame rate of the target class surrounded by the bounding box 20 in step S165.

[0574] Thereafter, the calculation unit 8 (threshold value setting unit 85) records information indicating the threshold value calculated in step S165, the target class, and the threshold value calculation strategy used for the threshold value calculation in association with each other in step S166. For example, the calculation unit 8 records the information in a recording area within the calculation unit 8, records the information in a predetermined area of ​​the memory 6, or transmits the information to the processor 11 and records the information in the processor 11.

[0575] In this way, for example, a parameter based on a threshold corresponding to a target class is set, ie, the lowest possible value of the frame rate for maintaining object tracking.

[0576] Fig.35 Steps S301 to S106 in are similar to Fig.18 .

[0577] For example, in step S301 , the calculation unit 8 (key frame selection unit 81 ) inputs the video capture setting to the ADC / pixel selector 3 according to the setting of the idle mode pre-stored in the calculation unit 8 .

[0578] Therefore, if the idle mode is set to 1 fps, video capturing is performed at intervals of 1 second, for example.

[0579] The calculation unit 8 (the object region identification unit 82 ) detects the position of the candidate object in the acquired image in step S302 .

[0580] In step S303 , the calculation unit 8 (class identification unit 83 ) classifies the detected objects into candidates.

[0581] In step S304 , the calculation unit 8 checks whether or not there is a target class among the classes obtained as a result of class identification.

[0582] If the target category does not exist, the calculation unit 8 repeats the processing in steps S301 , S302 , and S303 together with steps S350 and S351 .

[0583] When the threshold calculation timing comes during this period, the process in step S351 is performed.

[0584] In a case where the target class exists in step S304 , the calculation unit 8 advances the process from step S304 to step S352 .

[0585] The calculation unit 8 (key frame selection unit 81) specifies the parameters stored in the processing in step S351 as parameters in the normal mode setting, sets video capture for the ADC / pixel selector 3, and gives an image capture instruction in the normal mode.

[0586] For example, in the case where people are specified as the target class, Fig.34 Object tracking shown in C is maintained at 5fps. When the frame rate is set to 5fps, the frame rate in normal mode is set to 5fps.

[0587] In this way, the calculation unit 8 performs the processing in steps S302 and S303 in a state where the mode has been switched to the normal mode.

[0588] As long as there is an object class in the captured image, the normal mode continues. However, when there is no longer an object class, the process returns to step S301 after steps S350 and S351 to switch to the idle mode.

[0589] The above process is performed as active sampling. In this way, data volume compression is achieved by reducing the frame rate, especially in the period without the target class, and thus the power consumption is reduced.

[0590] Furthermore, even in normal mode, the processing is performed at a frame rate adapted according to the target category. Therefore, depending on the category, the frame rate is reduced to a fairly low frame rate (e.g., the above-mentioned 5fps). Therefore, even in normal mode, data volume compression and power consumption reduction can be achieved.

[0591] According to this example, the calculation unit 8 instructs the ADC / pixel selector 3 to change the frame rate to achieve the frame rate change. However, the calculation unit 8 may instruct the logic unit 5 to change the frame rate.

[0592] For example, readout from the array sensor 2 is continuously performed at 100 fps. However, the logic unit 5 is instructed to perform frame refinement according to the parameters set in idle mode or normal mode. In this way, a reduction in the amount of data associated with transmission to the processor 11 can be achieved.

[0593] <11. Application examples for mobile objects>

[0594] The technology according to the present disclosure is applicable to various types of products. For example, the technology according to the present disclosure can be implemented in the form of a device installed on any type of mobile body, such as a car, an electric car, a hybrid electric car, a motorcycle, a bicycle, a personal mobility device, an airplane, a drone, a ship, and a robot.

[0595] Fig.36: is a block diagram depicting an example of a schematic configuration of a vehicle control system as an example of a moving body control system to which the technology according to the embodiment of the present disclosure can be applied.

[0596] The vehicle control system 12000 includes a plurality of electronic control units connected to each other via a communication network 12001. Fig.36 In the example shown, the vehicle control system 12000 includes a driving system control unit 12010, a body system control unit 12020, an outside information detection unit 12030, an inside information detection unit 12040, and an integrated control unit 12050. In addition, a microcomputer 12051, a sound / image output section 12052, and an in-vehicle network interface (I / F) 12053 are shown as a functional configuration of the integrated control unit 12050.

[0597] The driving system control unit 12010 controls the operation of devices related to the vehicle driving system according to various programs. For example, the driving system control unit 12010 serves as a control device for a driving force generating device (e.g., an internal combustion engine, a driving motor, etc.) for generating a driving force for the vehicle, a driving force transmitting mechanism for transmitting the driving force to the wheels, a steering mechanism for adjusting the steering angle of the vehicle, a braking device for generating a braking force for the vehicle, etc.

[0598] The body system control unit 12020 controls the operation of various devices provided to the body according to various programs. For example, the body system control unit 12020 is used as a control device for a keyless entry system, a smart key system, a power window device, or various lights (e.g., a headlight, a reverse light, a brake light, a turn signal, a fog light, etc.). In this case, instead of a key, a radio wave transmitted from a mobile device or a signal of various switches may be input to the body system control unit 12020. The body system control unit 12020 receives these input radio waves or signals and controls the door lock device, the power window device, the lights, etc. of the vehicle.

[0599] The vehicle exterior information detection unit 12030 detects information about the exterior of the vehicle including the vehicle control system 12000. For example, the vehicle exterior information detection unit 12030 is connected to the imaging unit 12031. The vehicle exterior information detection unit 12030 causes the imaging unit 12031 to image the exterior of the vehicle and receives the imaged image. Based on the received image, the vehicle exterior information detection unit 12030 can perform a process of detecting an object such as a person, a vehicle, an obstacle, a sign, a character on a road surface, or a process of detecting a distance therefrom.

[0600] The imaging unit 12031 is an optical sensor that receives light and outputs an electrical signal corresponding to the amount of light received. The imaging unit 12031 may output the electrical signal as an image, or may output the electrical signal as information about the measured distance. In addition, the light received by the imaging unit 12031 may be visible light, or may be invisible light such as infrared rays.

[0601] The in-vehicle information detection unit 12040 detects information about the interior of the vehicle. The in-vehicle information detection unit 12040 is connected to, for example, a driver state detection unit 12041 that detects the driver's state. The driver state detection unit 12041 includes, for example, a camera that images the driver. Based on the detection information input from the driver state detection unit 12041, the in-vehicle information detection unit 12040 can calculate the driver's fatigue level or the driver's concentration level, or can determine whether the driver is dozing off.

[0602] The microcomputer 12051 can calculate the control target value of the driving force generation device, the steering mechanism, or the braking device based on the information about the inside or outside of the vehicle obtained by the vehicle exterior information detection unit 12030 or the vehicle interior information detection unit 12040, and output a control command to the driving system control unit 12010. For example, the microcomputer 12051 can perform cooperative control aimed at realizing the functions of the advanced driver assistance system (ADAS), which include collision avoidance or vibration reduction of the vehicle, following driving based on following distance, vehicle speed keeping driving, vehicle collision warning, vehicle lane departure warning, etc.

[0603] In addition, the microcomputer 12051 can perform collaborative control aimed at automatic driving by controlling the driving force generating device, steering mechanism, braking device, etc. based on information about the outside or inside of the vehicle obtained by the outside information detection unit 12030 or the inside information detection unit 12040, which enables the vehicle to drive autonomously without relying on the driver's operation, etc.

[0604] Furthermore, the microcomputer 12051 may output a control command to the body system control unit 12020 based on the information about the exterior of the vehicle obtained by the exterior information detection unit 12030. For example, the microcomputer 12051 may perform cooperative control aimed at preventing glare by controlling the headlights to change from high beam to low beam, for example, according to the position of a preceding vehicle or an oncoming vehicle detected by the exterior information detection unit 12030.

[0605] The sound / image output unit 12052 sends an output signal of at least one of sound and image to an output device capable of visually or auditorily notifying the vehicle occupants or the outside of the vehicle of information. Fig.36In the example of FIG. 1 , an audio speaker 12061, a display portion 12062, and a dashboard 12063 are shown as output devices. The display portion 12062 may include, for example, at least one of an in-vehicle display and a head-up display.

[0606] Fig.37 The figure depicts an example of the installation position of the imaging unit 12031.

[0607] exist Fig.37 In the figure, the imaging unit 12031 includes imaging units 12101, 12102, 12103, 12104 and 12105.

[0608] The imaging units 12101, 12102, 12103, 12104 and 12105 are, for example, disposed at positions on the front nose, side mirrors, rear bumper and rear door of the vehicle 12100 and at positions on the upper part of the windshield inside the vehicle. The imaging unit 12101 disposed at the front nose and the imaging unit 12105 disposed at the upper part of the windshield inside the vehicle mainly obtain images in front of the vehicle 12100. The imaging units 12102 and 12103 disposed on the side mirrors mainly obtain images on the sides of the vehicle 12100. The imaging unit 12104 disposed on the rear bumper or rear door mainly obtains images of the rear of the vehicle 12100. The imaging unit 12105 disposed at the upper part of the windshield inside the vehicle is mainly used to detect vehicles ahead, pedestrians, obstacles, signals, traffic signs, lanes, etc.

[0609] By the way, Fig.37 An example of the shooting range of the imaging units 12101 to 12104 is depicted. Imaging range 12111 represents the imaging range of the imaging unit 12101 provided to the front nose. Imaging ranges 12112 and 12113 respectively represent the imaging ranges of the imaging units 12102 and 12103 provided to the side mirrors. Imaging range 12114 represents the imaging range of the imaging unit 12104 provided to the rear bumper or the rear door. For example, by superimposing the image data imaged by the imaging units 12101 to 12104, a bird's-eye view image of the vehicle 12100 viewed from above is obtained.

[0610] At least one of the imaging sections 12101 to 12104 may have a function of obtaining distance information. For example, at least one of the imaging sections 12101 to 12104 may be a stereo camera composed of a plurality of imaging elements, or may be an imaging element having pixels for phase difference detection.

[0611] For example, the microcomputer 12051 can determine the distance to each three-dimensional object within the imaging ranges 12111 to 12114 and the time change of the distance (relative speed relative to the vehicle 12100) based on the distance information obtained from the imaging units 12101 to 12104, thereby extracting the nearest three-dimensional object that is particularly present on the driving path of the vehicle 12100 and is traveling at a predetermined speed (e.g., equal to or greater than 0 km / hour) in the same direction as the vehicle 12100 as the leading vehicle. In addition, the microcomputer 12051 can pre-set the following distance to be maintained in front of the leading vehicle, and perform automatic braking control (including follow-up stop control), automatic acceleration control (including follow-up start control), etc. Therefore, cooperative control for automatic driving can be performed, which allows the vehicle to travel autonomously without relying on the operation of the driver or the like.

[0612] For example, the microcomputer 12051 can classify the three-dimensional object data on the three-dimensional object into three-dimensional object data of two-wheeled vehicles, standard-sized vehicles, large vehicles, pedestrians, utility poles, and other three-dimensional objects based on the distance information obtained from the imaging units 12101 to 12104, extract the classified three-dimensional object data, and use the extracted three-dimensional object data to automatically avoid obstacles. For example, the microcomputer 12051 identifies obstacles around the vehicle 12100 as obstacles that the driver of the vehicle 12100 can visually identify and obstacles that the driver of the vehicle 12100 is difficult to visually identify. Then, the microcomputer 12051 determines the collision risk indicating the risk of collision with each obstacle. When the collision risk is equal to or higher than the set value and therefore there is a possibility of collision, the microcomputer 12051 outputs a warning to the driver via the audio speaker 12061 or the display unit 12062, and performs forced deceleration or evasive steering via the driving system control unit 12010. The microcomputer 12051 can thereby assist driving to avoid collisions.

[0613] At least one of the imaging units 12101 to 12104 may be an infrared camera that detects infrared rays. The microcomputer 12051 may, for example, identify a pedestrian by determining whether a pedestrian exists in the imaging images of the imaging units 12101 to 12104. For example, such identification of pedestrians is performed by extracting a process of feature points in the imaging images of the imaging units 12101 to 12104 as infrared cameras and determining whether it is a pedestrian by performing a pattern matching process on a series of feature points representing the contour of the object. When the microcomputer 12051 determines that a pedestrian exists in the imaging images of the imaging units 12101 to 12104 and thus identifies the pedestrian, the sound / image output unit 12052 controls the display unit 12062 so that a square contour line for emphasis is displayed so as to be superimposed on the identified pedestrian. The sound / image output unit 12052 may also control the display unit 12062 so that an icon or the like representing a pedestrian is displayed at a desired position.

[0614] An example of a vehicle control system to which the technology according to the present disclosure is applicable has been described above. For example, the technology according to the present disclosure is applicable to the imaging unit 12031 in the above configuration. Specifically, the sensor device 1 of the present disclosure is applied to an image sensor mounted on the imaging unit 12031 to perform a part or all of segmented image adaptation, area cropping, intelligent compression, and active sampling. In this way, image processing suitable for information detection outside the vehicle, reducing the processing load by appropriately reducing the amount of information without reducing the detection accuracy, etc. can be achieved.

[0615] <12. Summary and revision>

[0616] According to the above-described embodiments, the following advantageous effects are produced.

[0617] As described in the first and sixth embodiments, the sensor device 1 includes: an array sensor 2 including a plurality of visible light or invisible light imaging elements arranged one-dimensionally or two-dimensionally; a logic unit 5 (image processing unit) that performs image processing using parameters corresponding to instructions of an image signal obtained by image capturing using the array sensor 2; and a calculation unit 8. The calculation unit 8 identifies the category of an object detected from the image signal obtained by image capturing using the array sensor 2, selects parameters for image processing based on the identified category, and performs processing settings of the logic unit 5 using the selected parameters.

[0618] Specifically, the logic unit 5 performs image processing on the image signal obtained by the array sensor 2. In this case, parameters for the image processing are set based on the category recognition of the object detected in the image signal.

[0619] In the case of detecting an object from an image, an image with high quality for a viewer is not necessarily an image with high recognition accuracy. Furthermore, the preferred image quality is different for each category of the object to be recognized. Specifically, considering visual recognition, an image that has undergone image processing based on normal parameter settings to achieve high image quality does not necessarily have an image quality suitable for object detection. Furthermore, the preferred image processing parameters are different for each category of the object to be recognized.

[0620] Therefore, a parameter set is reserved in advance for each category, and the parameter set to be used is selected according to the category identification of the object detected in the captured image. In this way, image processing suitable for detecting the target object is achieved. Images subjected to image processing in this way achieve an improvement in object detection accuracy.

[0621] In addition, image quality adjustment suitable for object detection is different from image quality adjustment for producing beautiful images for viewers. Therefore, for example, a diffusion filter or the like for producing beautiful images with priority is not used. Therefore, the parameters to be set are generally parameters that reduce the processing load.

[0622] In addition, the amount of data is generally reduced according to the parameters corresponding to the category (for example, parameters associated with grayscale change or compression). In this case, processing delays caused by heavy calculation loads on the processor 11 side and increased power consumption of the entire system can be avoided.

[0623] The sensor device 1 of each embodiment includes the interface unit 7 (output unit) that outputs the image signal subjected to the image processing using the logic unit 5 (image processing unit) to an external device.

[0624] The sensor device 1 performs image processing corresponding to the object category using the logic unit 5 provided inside, and sends and outputs the processed image to the processor 11 ( Figure 5 In this way, the necessity of image quality adjustment by the processor 11 to improve object detection is eliminated. Therefore, while reducing the processing load of the processor 11, highly accurate object detection can be achieved.

[0625] In addition, considering the visual recognition of the viewer, the parameter set selected based on the category recognition is not necessarily the parameter set for obtaining the maximum image quality. The data amount of the processed image is reduced according to the situation. Specifically, an image is output, which does not necessarily have the maximum image quality for the viewer, but has an image quality suitable for the object recognized by the processor. Therefore, in some cases, the data amount of the image signal to be transmitted can be reduced.

[0626] Therefore, the communication cost can be reduced without reducing the accuracy of object detection. The transmission delay caused in the case of performing calculations in the cloud can also be improved.

[0627] According to the above description, the interface unit 7 (output unit) of the embodiment also transmits information associated with the category identification of the image signal to be output ( Figure 5 . , (a) in step S108.

[0628] In this way, the processor 11 corresponding to the output destination, the artificial intelligence processor in the cloud, or the like is able to detect the object after recognizing the category, thus achieving more accurate object detection.

[0629] According to the above description in the first and sixth embodiments, the calculation unit 8 performs, in one frame of the image signal, an object region recognition process for detecting an object region corresponding to a candidate of an object as a detection target, a category recognition process for recognizing a category of an object in the object region detected by the object region recognition process, and a parameter selection process for selecting a parameter based on a recognition result of the category recognition process and inputting the processing setting to the logic unit 5 (see Figure 1 and Figure 5 ).

[0630] Specifically, the calculation unit 8 includes an object region identification unit 82 for performing object region identification processing, a category identification unit 83 for performing category identification processing, and a parameter selection unit 84 for performing parameter selection processing. This configuration realizes detection of candidate objects from one frame of an image signal, category identification, and parameter selection based on category identification.

[0631] According to the above description in the first and sixth embodiments, the calculation unit 8 performs, in one frame of the image signal, an object region recognition processing for detecting an object region corresponding to a candidate of an object as a detection target, a category recognition processing for recognizing a category of an object in the object region detected by the object region recognition processing, a processing for determining whether a target category exists in the category recognized by the category recognition processing, and a parameter selection processing for selecting a parameter based on the recognition result of the target category in a case where the target category exists and inputting the processing setting to the logic unit 5 (see Figure 1 , 5 and 22).

[0632] By determining the presence of the target category, parameter selection based on the object category that is the target of image recognition can be effectively performed. In other words, by eliminating parameter selection based on the detection of non-target objects, processing efficiency is increased, and inappropriate parameters are not set without performing useless parameter selection. Therefore, the image recognition accuracy of the target object is improved.

[0633] According to the first and sixth embodiments, a plurality of target categories can be set. In the case where objects in a plurality of target categories exist in one frame, parameter selection processing is performed based on a selected target category (see Figure 5 and 22 . (step S106 in FIG. 106 ).

[0634] By allowing multiple target categories to be set, image recognition of target objects corresponding to multiple categories can be processed.

[0635] Furthermore, for example, in the case where objects in multiple target classes exist in one frame, a dominant or main target class is selected. In this way, an appropriate parameter set is provided by selecting parameters based on one target class.

[0636] Furthermore, according to the above-mentioned first and sixth embodiments ( Figure 5 and 22 In steps S105 and S106 in the above example, a bounding box 20 around the object is calculated for the object area corresponding to the target category. In the case where there are multiple target categories, one target category is selected based on the area of ​​the bounding box 20.

[0637] In the case where there are objects in multiple target categories, the area of ​​each object is specified using the bounding box 20, and the main area of ​​the object in each target category in one frame is obtained to determine the target category to be processed preferentially. In this way, more appropriate parameter selection can be achieved.

[0638] According to the first and sixth embodiments described above, the calculation unit 8 selects a parameter set including a plurality of parameters and set for each category based on the identified category. Specifically, a set of parameters for various types of processing performed by the logic unit is stored. The parameter set is selected and set for the logic unit (see Figure 4 , 5 and step S107 in 22).

[0639] In this way, a plurality of parameters suitable for the target category (target category) can be set as parameters suitable for various types of processing performed by the logic unit 5 .

[0640] According to the first embodiment presented above, the parameter set is a collection of a plurality of parameters obtained by performing deep learning using an image of a corresponding object of each category.

[0641] For example, image processing parameters for achieving a high image recognition rate for human recognition are obtained by deep learning using human images as learning data, and the obtained parameter set is designated as a parameter set corresponding to a class called “human” (see Figure 4 ).

[0642] In this way, an appropriate parameter set can be prepared for each category, and parameter selection suitable for target image recognition can be achieved by selecting an appropriate parameter set.

[0643] According to the first and sixth embodiments described above, the calculation unit 8 identifies the category of a frame designated as a key frame in an image signal obtained by image capturing using the array sensor 2, and selects parameters for image processing based on the identified category (see Figure 5 and 22 1 and 2).

[0644] This category identification is performed not for all frames but only for key frames. Therefore, the processing load imposed on the calculation unit 8 does not increase excessively. In addition, an appropriate parameter selection state can be maintained by selecting key frames based on an appropriate selection algorithm.

[0645] According to the above-described first and sixth embodiments, key frames are frames obtained at predetermined time intervals.

[0646] In this way, parameter selection is performed regularly. For example, one frame is designated as a key frame every thirty seconds. In this case, the processing performed by the computing unit 8 does not become too cumbersome and appropriate parameter settings can be maintained.

[0647] Furthermore, it is preferable to set the interval of the key frames according to various types of situations, such as a target object of image recognition, a usage environment of the sensor device 1 , a usage purpose, and a type of device including the sensor device 1 .

[0648] Furthermore, an example of specifying a frame obtained at a time based on a command from an external device has been described.

[0649] For example, the key frame is set in response to an instruction from the processor 11 corresponding to the image output destination, etc. In this way, key frame selection and parameter selection can be implemented according to the purpose of the recognition processing performed by the device corresponding to the output destination of the information associated with the image signal or class recognition, etc. For example, the key frame is set according to the type, purpose, etc. of the device including the sensor device 1.

[0650] In this case, assuming that the device including the sensor device 1 is an in-vehicle device, for example, at the time when the car starts to travel, the interval of the key frames may be reduced.

[0651] The logic unit 5 of each embodiment performs processing such as color correction, gamma correction, color grayscale processing, gain processing, contour emphasis processing, data compression processing, frame rate conversion, resolution conversion, aspect ratio conversion, contrast adjustment processing, clarity adjustment processing, grayscale adjustment processing or sampling rate change processing as image processing of the image signal.

[0652] In this way, image quality adjustment and data volume conversion are achieved. In this case, image quality adjustment and data size adjustment (e.g., resolution and frame rate) suitable for the object category to be identified are performed by setting parameters for the aforementioned processing. As a result, an image and data size suitable for object detection of the target category are obtained, and unnecessary increases in image quality and data volume are reduced. These help reduce communication costs, increase processing speed, improve object detection accuracy, etc.

[0653] According to the above-described respective embodiments, the interface unit 7 (output unit) outputs any or all items of information indicating the image signal subjected to the image processing by the logic unit 5, the recognized category information, the number of detected objects, and the presence or absence of the target category in response to a request from an external device. This is a common operation performed in the first to fifth embodiments.

[0654] Specifically, the interface unit 7 outputs information required by the processor 11, information indicating the category recognized by the calculation unit 8, the number of objects, and the presence or absence of the target category among the information items indicating the image signal subjected to the image processing of the logic unit 5 in response to a request from the processor 11, a processor in the cloud, etc. In this way, unnecessary information transmission is avoided. Therefore, a reduction in communication volume and a reduction in power consumption are also achieved.

[0655] Furthermore, information corresponding to processing performed by the processor 11 or the like may be provided.

[0656] Although the image signal is the target in each embodiment, it is assumed that the array sensor 2 is composed of an array of acoustic wave detection elements or a tactile sensor element array. In this case, the interface unit 7 outputs these detection signals (detection signals processed by the logic unit 5) in response to a request from an external device.

[0657] As described in the second, third, fourth, seventh and eighth embodiments, the sensor device 1 includes an array sensor 2 including a plurality of detection elements arranged one-dimensionally or two-dimensionally, and a signal processing unit 30, which acquires a detection signal obtained by the array sensor 2 and performs signal processing on the detection signal. The calculation unit 8 detects an object from the detection signal obtained by the array sensor 2, and gives an instruction to generate region information (ROI 21 or AROI 22) based on the detection of the object to the signal processing unit 30 as region information associated with the acquisition of the detection signal obtained from the array sensor 2 or associated with the signal processing of the detection signal.

[0658] Specifically, the signal processing unit 30 performs signal processing on the detection signal obtained by the array sensor 2, and the processed detection signal is output from the interface unit 7. In this case, the region information associated with the acquisition of the detection signal from the array sensor 2 or the signal processing performed by the signal processing unit 30 is set based on the object detection.

[0659] As in the embodiment, for detecting an object from an image, information associated with all pixels of each frame is not always required. For example, in the case of detecting a person, only detection information indicating the region containing the person within the frame is required. Therefore, the calculation unit 8 generates the ROI 21 or AROI 22 based on the object detection to implement the processing performed by the signal processing unit 30, that is, the acquisition of the detection signal from the array sensor 2 using the ADC / pixel selector 3, and the compression processing performed by the logic unit 5 based on the ROI 21 or AROI 22.

[0660] In this way, a reduction in the amount of data of the processing target and an improvement in the processing speed can be achieved, and an image signal can also be obtained without reducing the detection accuracy.

[0661] Note that object detection can be performed not only on image signals but also on detection signals obtained from an array sensor in the form of acoustic wave detection signals or tactile detection signals to give an instruction to the signal processing unit on area information generated based on the detection of the object as area information associated with the acquisition of the detection signal obtained from the array sensor or associated with signal processing of the detection signal.

[0662] In this way, a reduction in the amount of data of the processing target and an improvement in the processing speed can be achieved, and even in the case of using an acoustic wave sensor array or a contact sensor array, an effect of acquiring a detection signal without reducing the detection accuracy can be obtained.

[0663] In the case of the second, third, fourth, seventh and eighth embodiments, the interface unit 7 (output unit) outputs the detection signal subjected to the signal processing of the signal processing unit 30 to the external device.

[0664] An image signal obtained by performing AD conversion on only part of the pixels using the ROI 21 or the AROI 22 or an image signal compressed at a compression rate changed for each region using the ROI 21 is output to the processor 11, etc. Therefore, the amount of data to be transmitted is greatly reduced. As a result, a reduction in communication cost and a reduction in transmission time are achieved. In addition, information required for object detection is included. Therefore, the object detection accuracy of the processor 11, etc. is not reduced. In addition, by reducing the amount of data, the processing load imposed on the processor 11 is reduced.

[0665] Furthermore, processing delays caused by heavy calculation loads on the processor 11 side and increases in power consumption of the entire system can be avoided.

[0666] According to various embodiments, the signal processing unit 30 includes an ADC / pixel selector 3 (acquisition unit) that selectively acquires detection signals from the detection elements of the array sensor 2 .

[0667] Furthermore, according to the second, third, seventh and eighth embodiments, the ADC / pixel selector 3 acquires the detection signal of the detection element selected based on the ROI 21 or AROI 22 received from the calculation unit 8 as a frame of detection signals (see Fig. 9 , Fig.14 , Fig.29 and Fig.32 ).

[0668] The ADC / pixel selector 3 performs AD conversion on only the photoelectric conversion signal within the range specified by the ROI 21 or the AROI 22 in the next frame in which the object has been detected and acquires the photoelectric conversion signal. In this way, the amount of data for one frame can be significantly reduced. In addition, by setting the ROI 21 or the AROI 22 based on the object detection, the information associated with the pixels required for the object detection can be appropriately obtained.

[0669] According to the second, third, seventh, and eighth embodiments described above, in a state where the ADC / pixel selector 3 (acquisition unit) does not select a detection element using the region information (ROI 21 or AROI 22), the calculation unit 8 detects an object from the detection signal acquired by the array sensor 2, and gives an instruction to generate the designated ROI 21 or AROI 22 based on the detection of the object using the ADC / pixel selector 3 to the signal processing unit 30 as region information for acquiring the detection signal in the next frame from the array sensor 2 (see Fig. 9 , Fig.14 , Fig.29 and Fig.32 ).

[0670] By detecting an object from an image signal containing all valid pixels in one frame, the object to be detected can be accurately detected in the captured image. In addition, by generating ROI 21 or AROI 22 based on the detected object and providing ROI 21 or AROI 22 to ADC / pixel selector 3, information associated with only pixels required for object detection can be acquired from the next frame. Therefore, while reducing the amount of data, appropriate detection information (information associated with necessary pixels) can be obtained.

[0671] According to the second, third, seventh and eighth embodiments described above, in a state where the ADC / pixel selector 3 has selected a detection element using the ROI 21 or the AROI 22, the calculation unit 8 detects an object from the detection signal acquired by the array sensor 2, and gives an instruction to the signal processing unit 30 to regenerate region information based on the detection of the object using the ADC / pixel selector 3 as region information for acquiring a detection signal in the next frame from the array sensor 2 (see Fig. 9 , Fig.14 , Fig.29 and Fig.32 as well as Fig.10 .

[0672] By detecting an object also from an image signal frame containing information associated with only part of the pixels, the ROI 21 or AROI 22 can be corrected according to the change in the position of the object. In this way, the area acquired by the ADC / pixel selector 3 changes according to the movement of the object (e.g., a person) in the next frame image. In other words, even if the position of the target object changes within the image of each frame, pixels can be selected and read at the position after the change in each frame. Therefore, even as the frame progresses, the state of achieving appropriate detection information (pixel information) while reducing the amount of data can continue.

[0673] According to the second, third, seventh and eighth embodiments, in a state where the ADC / pixel selector 3 has selected a detection element using the ROI 21 or the AROI 22, the calculation unit 8 detects an object from the detection signal acquired by the array sensor 2. In a case where the target object is not detected, the calculation unit 8 gives an instruction to acquire the detection signal in the next frame from the array sensor 2 in a state where the ADC / pixel selector 3 has not selected a detection element using the ROI 21 or the AROI 22 (see Fig. 9 , Fig.14 , Fig.29 and Fig.32 Steps S205, S201 and S202).

[0674] Specifically, when the target object is no longer detected in a frame containing only information associated with part of the detection elements from the array sensor 2 , the calculation unit 8 returns the acquisition of the detection signal by the acquisition unit to a normal state.

[0675] In this way, the state returns to the state where the object is detected from the image signal containing all effective pixels in one frame, and the detection of the target object is also achievable for the entire captured image. In other words, a state for monitoring the entire image can be achieved.

[0676] According to the second, third, fourth, seventh and eighth embodiments described above, the calculation unit 8 calculates the bounding box 20 surrounding the area of ​​the object detected from the detection signal using the array sensor 2, and generates the ROI 21 or AROI 22 as the area information based on the bounding box 20 (see Fig. 9 , Fig.14 , Fig.16 , Fig.29 and Fig.32 ).

[0677] An ROI 21 or AROI 22 corresponding to the location of the target object in the image may be generated by generating a bounding box 20 using object detection, and generating an ROI 21 or AROI 22 based on the generated bounding box 20. In this way, appropriate readout pixels in the next frame may be selected.

[0678] According to the second, fourth, seventh and eighth embodiments, the calculation unit 8 generates the ROI 21 by extending the bounding box 20 (see Fig. 9 , Fig.16 , Fig.29 and Fig.32 ).

[0679] The bounding box 20 surrounds the object area in the current frame. However, the location of the object may change in subsequent frames. Therefore, the ROI 21 is generated by extending the bounding box 20.

[0680] In this way, the information acquired from the next frame (the signal of the AD-converted pixel) is more likely to contain the object. In other words, the pixels containing the necessary information can be continuously read out as long as possible.

[0681] According to the description of the second embodiment, the calculation unit 8 determines the area of ​​the detection object in units of detection elements, and generates area information based on the determination (see Fig.11 ).

[0682] Specifically, the ROI 21 is generated based on the semantic segmentation. In this case, the ROI 21 having a shape other than a rectangle is also generated.

[0683] Depending on the object, rectangular clipping may lose information. For example, a truck with a protrusion, a cyclist, etc. may have a protruding portion when clipped into a rectangle. If the ROI 21 is shaped to cover the protruding portion, the size of the ROI 21 is uselessly increased. As a result, the data reduction effect is reduced. Therefore, the necessary area is selectable at the pixel level. In this way, the necessary information can be obtained with a minimum amount of data.

[0684] The ROI 21 based on semantic segmentation described herein is also useful in the case of setting a region with a low compression rate in the fourth embodiment.

[0685] According to the second, third, fourth, seventh and eighth embodiments, the computing unit 8 detects an object in a frame (key frame) at the object detection key frame recording timing in the detection signal obtained from the array sensor 2, and generates region information based on the object detection (see Fig. 9 , Fig.14 , Fig.16 , Fig.29 and Fig.32 .

[0686] This object detection is performed not on all frames but only on key frames. Therefore, the processing load imposed on the calculation unit 8 does not increase excessively. In addition, by selecting key frames based on an appropriate selection algorithm, a state for achieving appropriate object detection can be maintained.

[0687] In this case, the key frame is also a frame obtained for every predetermined time interval, or a frame obtained at a time based on a command from an external device.

[0688] For example, by setting key frames according to the type, purpose, etc. of the device including the sensor apparatus 1, it is possible to perform object detection from all pixels in a frame at a time required by the device or application and reduce the amount of data in subsequent frames.

[0689] According to the second, third, fourth, seventh and eighth embodiments, the calculation unit 8 identifies the class of the object detected in the detection signal obtained from the array sensor 2, determines whether the identified class is the target class, and generates region information (ROI 21 or AROI 22) corresponding to the object of the target class (see Fig. 9 , Fig.14 , Fig.16 , Fig.29 and Fig.32 , and then execute the following steps (steps S204 and S205).

[0690] Accurate region information for acquiring information associated with an object of a detection target can be created by generating the ROI 21 or the AROI 22 based on determination of the presence of a target class for defining an object region of the target class.

[0691] According to the third and eighth embodiments described above, the calculation unit 8 recognizes the class of the object detected in the detection signal obtained from the array sensor 2, and generates the region information (AROI 22) corresponding to the object using the template corresponding to the recognized class (see Fig.14 S210 and S211 in Fig.32 S262 and S211 in ).

[0692] By using a template corresponding to a category, an AROI 22 suitable for a different important region for each category can be generated.

[0693] Particularly in the case where the array sensor 2 is composed of imaging elements, photoelectric conversion consumes the largest amount of power. In this case, it is desirable to reduce the pixels to be photoelectrically converted as much as possible. By reducing the pixels to be photoelectrically converted according to the template, a reduction in the amount of effective data can be achieved without affecting the accuracy of detection. Particularly in the case of images, it is not important to generate a beautiful image for the viewer, but to generate an image in which the object is accurately recognized by the processor 11. An image generated by specifying pixels to be photoelectrically converted and converted into digital data using a template is an image suitable for effective object detection achieved with a small amount of data.

[0694] Furthermore, the template indicates an acquisition area in which a detection signal is acquired for each class.

[0695] For example, assume that the template indicates the detection elements from which detection information is to be acquired among the detection elements of the array sensor for each category such as "person" and "car" (see Fig.12 and Fig.13 ).

[0696] By using a template indicating pixels to be read corresponding to a category, it is possible to read appropriate information for each category from the array sensor 2. Specifically, Fig.12 and Fig.13 As in the example of , by increasing the density in the component (face or number plate portion), information corresponding to the particularly needed portion can be centrally acquired for each category.

[0697] According to the fourth embodiment described above, the signal processing unit 30 uses the logic unit 5 to perform compression processing for compressing the detection signal received from the array sensor 2. The logic unit 5 performs compression processing at a compression rate different for each region based on the region information received from the calculation unit 8 (see Fig.16 ).

[0698] In this manner, the signal processing unit 30 (logic unit 5) achieves data compression without reducing important information by setting different compression rates for important areas and less important areas in a frame.

[0699] Furthermore, the logic unit 5 performs compression processing at a low compression rate in an area specified by the area information, and performs compression processing at a high compression rate in another area (see Fig.16 ).

[0700] The signal processing unit 30 (logic unit 5) performs compression processing at a low compression rate in the area specified by the ROI 21, and reduces the amount of data at a high compression rate in other areas in the next frame and subsequent frames where the object has been detected. The ROI 21 is generated according to the object detection. Therefore, the area indicated by the ROI 21 is also an important area for the object detection performed by the processor 11. A low compression rate is set for this area to avoid information reduction. In this way, a decrease in detection accuracy is prevented. On the other hand, the area other than the area indicated by the ROI 21 is an area that has little influence on the object detection. Therefore, this area is compressed at a high compression rate to effectively reduce the amount of data.

[0701] As described in the fifth and ninth embodiments, the sensor device 1 includes: an array sensor 2, including a plurality of detection elements arranged one-dimensionally or two-dimensionally; a signal processing unit 30, which acquires a detection signal obtained by the array sensor 2 and performs signal processing on the detection signal; and a computing unit 8, which detects an object from the detection signal obtained by the array sensor 2 and gives an instruction to change the frame rate of the detection signal received from the array sensor 2 based on the detection of the object.

[0702] As in the embodiment, for detecting an object from an image, an image signal with a high frame rate is not always required. For example, for detecting a person, even if the frame rate is low, it will not cause a problem in a frame that does not contain a person. On the contrary, the frame rate is increased during a period of time when a human being appears. In this case, the amount of information increases, thereby allowing the detection of an object (person) and the increase of information that can be identified based on the detection.

[0703] Specifically, by changing the frame rate according to the detection of the object, the amount of data is adaptively increased when data is needed and reduced when data is not needed. Therefore, the amount of processed data or the amount of transmitted data can be reduced without reducing the object detection performance.

[0704] Note that an instruction to change the frame rate of the detection signal received from the array sensor can be given based on object detection from the detection signal obtained from the array sensor, not only in the form of an image signal, but also in the form of an acoustic wave detection signal, a tactile detection signal, etc. In this way, even in the case of using an acoustic wave sensor array or a contact sensor array, the amount of data is adaptively increased when data is needed and reduced when data is not needed. Therefore, the effect of reducing the amount of processed data or the amount of transmitted data without reducing the object detection performance can be produced.

[0705] In the case where the array sensor is an imaging element array, the frame refers to a frame of an image. Even in the case of an acoustic wave detection element or a tactile sensor element, the frame has the same meaning and indicates a unit of data read from a plurality of detection elements of the array sensor in one reading cycle regardless of the type of the array sensor. The frame rate corresponds to the density per unit time of the frame.

[0706] In the case of the fifth and ninth embodiments, the interface unit 7 (output unit) that outputs the detection signal processed by the signal processing unit 30 to the external device is configured to output the detection signal whose frame rate is changed based on the result of the object detection to the processor 11, etc.

[0707] Therefore, the amount of data to be transmitted is greatly reduced. As a result, a reduction in communication cost and a reduction in transmission time are achieved. In addition, information required for target object detection is contained. Therefore, the accuracy of object detection by the processor 11 and the like is not reduced. In addition, the processing load imposed on the processor 11 is also reduced by the reduction in the amount of data.

[0708] Furthermore, processing delays caused by heavy calculation loads on the processor 11 side and increases in power consumption of the entire system can be avoided.

[0709] According to the fifth and ninth embodiments, the calculation unit 8 stores at least the setting value of the frame rate of the first mode (idle mode) and the setting value of the frame rate of the second mode (normal mode), and performs control based on the setting value of the first mode or the second mode according to the result of object detection (see Fig.18 and Fig.35 ).

[0710] In this way, control as active sampling can be achieved by a simple process of selecting a setting value according to the result of object detection, that is, the presence or absence of an object of the target class.

[0711] Preferably, in the fifth and ninth embodiments, one or both of the setting value of the frame rate of the first mode and the setting value of the frame rate of the second mode can be rewritten from an external device. For example, the setting value can be variable according to the use application, processing power, use purpose of the application, etc. of the external processor.

[0712] The frame rate setting according to the purpose of the processor 11 or the application of the processor 11 can be realized by rewriting the setting value from the processor 11 or the like.

[0713] According to the examples in the fifth and ninth embodiments described above, the calculation unit 8 changes the frame rate by giving instructions for the readout interval of the detection signal from the array sensor 2 (see Fig.18 and Fig.35 ).

[0714] For example, the calculation unit 8 (key frame selection unit 81) instructs the array sensor 2 and the ADC / pixel selector 3 to switch between the idle mode and the normal mode to change the interval of the image signal readout performed by the array sensor 2 and the ADC / pixel selector 3 for frame rate switching.

[0715] In this case, the interval of photoelectric conversion and readout itself performed by the array sensor 2 is increased in the idle mode to reduce the frame rate. The array sensor 2 consumes a large amount of power for photoelectric conversion. Therefore, the increase of the readout interval of the array sensor 2 is quite effective in reducing power consumption.

[0716] According to the description of the fifth and ninth embodiments, the calculation unit 8 may instruct the signal processing unit 30 (logic unit 5) to change the frame rate to achieve the frame rate change.

[0717] Specifically, the switching of the frame rate is performed by frame rate conversion in the signal processing process.

[0718] The frame rate conversion may be implemented by the logic unit 5. For example, the frame rate may be reduced by performing frame thinning. In this case, in a state where the array sensor 2 continues to perform readout at a high frame rate, the effect of reducing the power consumption of the array sensor 2 is not produced. However, even in this case, the effect of reducing the amount of data transmitted to the processor 11 can be produced.

[0719] According to the fifth and ninth embodiments described above, the calculation unit 8 identifies the class of the object detected in the detection signal obtained from the array sensor 2, determines whether the identified class is the target class, and gives an instruction for frame rate change according to the determination result (see Fig.18 S304, S305 and S301 in, and Fig.35 S304, S352 and S301 in it).

[0720] The presence of a target class is determined, and the frame rate is increased by selecting a normal mode based on the presence of an object of the target class. In the case where no object of the target class is detected, the frame rate is reduced by selecting an idle mode. In this way, accurate detection can be achieved based on the presence of an object corresponding to the detection target.

[0721] For example, for usage purposes such as monitoring people, detailed monitoring is achieved by increasing the frame rate based on detecting people as a target category, while for other purposes, the idle mode is selected to achieve reduction in power consumption or reduction in data volume.

[0722] According to various embodiments, the detection elements of the array sensor 2 are constituted by imaging elements. Specifically, the detection signal generated by the array sensor 2 is an image signal generated by image capture (photoelectric conversion).

[0723] Therefore, in object detection using captured images, it is possible to appropriately reduce the amount of data while maintaining the accuracy of object detection, and reduce the processing load and transmission cost by reducing the amount of data.

[0724] According to the embodiment, it is assumed that the sensor device 1 has an integrated type sensing module configuration having an AI chip or a DRAM chip constituting the calculation unit 8 .

[0725] On the other hand, an example may be adopted having a configuration in which an AI chip or a DRAM chip constituting the calculation unit 8 is provided outside the array sensor 2. In this case, the externally provided calculation unit performs the readout and signal processing control described in the respective embodiments.

[0726] Furthermore, an example may be adopted in which the array sensor 2 and the AI ​​chip constituting the calculation unit 8 are unified and an external DRAM chip is used.

[0727] According to the sixth embodiment, a threshold setting unit 85 is provided, which sets thresholds for all or some parameters used for image processing performed by the logic unit 5 (image processing unit) or image capture processing associated with image capture performed by the array sensor 2, so as to implement processing using parameters based on the threshold settings.

[0728] For example, an image signal having the minimum necessary quality, etc. (e.g., minimum necessary resolution) for processing such as object detection can be output by setting (changing) parameters using threshold values. Therefore, while reducing the amount of data of the image signal to be output, a decrease in the performance, accuracy, etc. of processing in a subsequent stage (e.g., object detection) can be avoided.

[0729] In addition, power consumption can be reduced and processing speed can be increased accordingly.

[0730] In addition, according to the sixth embodiment described Fig. 22 In a processing example, the computing unit 8 identifies the category of an object detected in an image signal obtained from the array sensor 2 by image capture, selects a parameter set for image processing based on the identified category, executes processing settings of the logic unit 5 using the selected parameters, and sets (adjusts) the selected parameters according to a threshold value. Therefore, part or all of the parameters included in the parameter set applicable to each category are further adjusted and set according to the threshold value. Therefore, after adapting to the category, the amount of data is further adjusted to a necessary and sufficient amount while maintaining detection accuracy, for example, object detection. By further adjusting the parameter set suitable for the category, the amount of data can be further reduced. Therefore, the reduction in power consumption and the improvement in processing speed can be further promoted.

[0731] However, other than Fig. 22Configurations other than the examples depicted in . All or some of the setting parameters may be changed simply based on the threshold value regardless of the category. In this case, a reduction in the amount of image signal data, a reduction in power consumption, and an increase in processing speed may be achieved while maintaining the performance and accuracy required for object detection, etc.

[0732] According to the sixth embodiment, the threshold setting unit 85 sets a threshold corresponding to the category of the object detected from the image signal. For example, a threshold considered to be appropriate is set according to categories such as "person", "car" and "sign".

[0733] The relationship between the resolution required for an image for the purpose of object detection, etc. and the detection accuracy is different for each category. Therefore, a threshold is set according to the category, and the resolution, etc. of the image signal output from the logic unit 5 is changed. In this way, an output based on the minimum necessary resolution, etc. can be achieved according to the category. Specifically, data reduction, power consumption reduction, processing speed increase, etc. are achievable while optimizing parameters such as resolution according to the category and maintaining the object detection accuracy, etc. at a desired level.

[0734] According to the sixth embodiment, the threshold value setting unit 85 sets the threshold value based on the learning process performed on the image signal. For example, the threshold value is obtained by performing a learning process as local learning on the sensor device 1 side. The threshold value and the ideal value of the parameter corresponding to the threshold value are determined by performing learning on the image signal. In this way, parameter setting can be adaptively achieved according to the image capturing environment, the captured image content, the detection target object, etc.

[0735] Furthermore, a threshold value suitable for the image accuracy required for the sensor device 1 can be calculated by performing this local learning within the sensor device 1 or the terminal apparatus 100 including the sensor device 1 .

[0736] Furthermore, optimization of the resolution and the like of the image signal to be output is achieved by setting a threshold using local learning for each category to further adapt parameter settings for the category.

[0737] According to the sixth embodiment, the threshold value setting unit 85 sets the threshold value so that a predetermined ratio is obtained as the confidence level (confidence level of object detection).

[0738] The confidence level required to accurately detect objects from an image is different for each purpose, target, device / application type, season, region, etc. of detection.

[0739] For example, in the case where only 80% confidence is required, it is sufficient if the parameters are set according to the threshold to obtain 80% or higher confidence. In addition, in the case where 95% or higher confidence is required, it is sufficient if the parameters are set based on an increased threshold.

[0740] Therefore, by setting the threshold (and setting parameters) based on the confidence required for object detection, the desired quality of the image signal and corresponding data reduction, power consumption reduction, processing speed improvement, etc. can be achieved as an adaptation according to the execution of the threshold setting processing.

[0741] According to the sixth embodiment described above, the threshold setting unit 85 is provided in a device having the same housing as the logic unit 5 (see Fig.21 , Fig.25 and Fig.26 ). For example, the threshold setting unit 85 is provided in a unit used as the sensor device 1, or is provided in the terminal device 100 including the sensor device 1. In addition, the threshold setting unit 85 sets the threshold and the parameter corresponding to the threshold by performing local learning. This means that the threshold is set by learning the state suitable for the device as the sensor device 1 and the terminal device 100. In this way, it is possible to realize the appropriate threshold setting for the output required by the sensor device 1 or the terminal device 100.

[0742] According to the seventh and eighth embodiments, the calculation unit 8 sets the active area RA ( 20 ) of the detection signal acquired from the array sensor 2 based on the information associated with the previous area information (the object detection area bounding box 20 based on which the area information is generated, or the ROI 21 or the AROI 22 each of which is the area information itself). Fig.29 and Fig.32 (S251 in).

[0743] In addition, the calculation unit 8 detects the object from the detection signal in the active area RA, and gives an instruction to the signal processing unit 30 to generate the ROI 21 or AROI 22 based on the detection of the object as region information associated with the acquisition of the detection signal obtained from the array sensor 2 or associated with the signal processing of the detection signal.

[0744] In this way, the processing load of the object detection for setting the ROI 21 or the AROI 22 is significantly reduced. Specifically, the processing of step S203 is reduced. Therefore, the effects of reduced processing load, increased speed, and reduced power consumption can be achieved.

[0745] According to the seventh and eighth embodiments, the calculation unit 8 sets the active area RA so that the active area RA contains the detection area for object detection based on which the previous ROI 21 or AROI 22, ie, the bounding box 20, is generated.

[0746] In some cases, continued processing generates an area in the image that does not contain the bounding box 20 at all. This area can be designated as a non-active area DA where the target object is not detectable, while another area can be designated as an active area RA, i.e., an area where the object is detectable.

[0747] The active area RA can be easily and appropriately set based on a plurality of previous bounding boxes 20. Furthermore, the active area RA suitable for an image capturing environment, an image capturing direction, and the like can be set.

[0748] According to the seventh and eighth embodiments, in a state where the ADC / pixel selector 3 has selected a detection element using the ROI 21 or the AROI 22, the calculation unit 8 detects an object from the detection signal acquired by the array sensor 2, and in a case where the target object is not detected, instructs the ADC / pixel selector 3 to acquire the detection signal of the active area in the next frame from the array sensor (see Fig.29 and Fig.32 S205, S201 and S252).

[0749] Specifically, when the target object is no longer detected in a frame in which information associated with only part of the detection elements is acquired from the array sensor 2 , the calculation unit 8 returns the acquisition of the detection signal by the acquisition unit to a normal state.

[0750] In this way, the state returns to the state in which the object is detected from the image signal of the active area in one frame, and detection of the target object can be achieved again within the necessary range of the captured image. Thus, a state for monitoring the entire image is actually generated.

[0751] According to the seventh and eighth embodiments, the calculation unit 8 generates region information based on object detection of the detection signal of the active area RA in the key frame of the detection signal obtained from the array sensor 2 (see Fig.29 and Fig.32 S201 and S252 in it).

[0752] This category identification is performed not on all frames but only on key frames. Therefore, the processing load imposed on the calculation unit 8 does not increase excessively. In addition, by selecting key frames based on an appropriate selection algorithm, a state for achieving appropriate object detection can be maintained.

[0753] In this case, it is also possible to designate a frame obtained for each predetermined time interval as a key frame, or designate a key frame at a time based on a command from the outside such as the processor 11 .

[0754] According to the eighth embodiment, the calculation unit 8 identifies the category of the object detected in the detection signal obtained from the array sensor 2, and generates the region information (AROI 22) corresponding to the object using the template corresponding to the identified category. In this case, the AROI 22 is used, in which parameters such as resolution are calculated and recorded based on a threshold value (see Fig.32 S262 and S211 in ).

[0755] For example, by setting (changing) parameters in the acquisition region indicated by the AROI 22 using a threshold value, an image signal having minimum necessary quality or the like (eg, minimum necessary resolution) for processing of object detection or the like can be output.

[0756] Furthermore, an image generated by specifying pixels to be photoelectrically converted and converted into digital data using a template is an image suitable for effective object detection with a small amount of data.

[0757] Therefore, by using a template and setting parameters such as resolution using a threshold, it is possible to avoid a reduction in the performance, accuracy, etc. of subsequent processing (e.g., object detection) while reducing the amount of data of the image signal to be output. In addition, power consumption can be reduced and processing speed can be increased accordingly.

[0758] Furthermore, the template indicates an acquisition area for which a detection signal of each category, for example, “person” and “car” is acquired. Therefore, information associated with a particularly necessary part can also be acquired collectively for each category.

[0759] According to the ninth embodiment, there is provided a threshold value setting unit 85 which sets a threshold value of a frame rate according to a class identified by an object detected in a detection signal obtained from the array sensor 2, thereby executing processing using a frame rate set based on the threshold value (see Fig.35 ).

[0760] By setting (changing) the frame rate using a threshold value, a frame rate suitable for the category of the detection target can be applied. Specifically, by reducing the frame rate while preventing the performance of object detection in the detection target category from being reduced, a reduction in the amount of data of the image signal, a reduction in power consumption, and an increase in processing speed can be achieved.

[0761] According to the ninth embodiment, the threshold setting unit 85 sets the threshold to maintain the frame rate of object tracking from the image.

[0762] In this way, data reduction according to categories, power consumption reduction, processing speed increase, and the like can be achieved while maintaining the accuracy of object detection performed together with object tracking from images.

[0763] According to the ninth embodiment, the calculation unit 8 uses the frame rate set by the threshold setting unit 85 as the frame rate in the second mode (normal mode) for increasing the frame rate. In this way, when the frame rate is increased, the frame rate relatively reduced according to the category is used.

[0764] The first, second, third, fourth, fifth, sixth, seventh, eighth and ninth embodiments may be appropriately combined.

[0765] These combinations can increase the effects of the various embodiments. Specifically, the effects of reducing the amount of image signal data, reducing power consumption, and increasing processing speed can be further increased while maintaining the accuracy of processing such as detecting an object from an image.

[0766] Note that the advantageous effects to be produced are not limited to the advantageous effects described in this specification presented only by way of example. In addition, other advantageous effects can be provided.

[0767] Note that the present technology can adopt the following configurations.

[0768] (1) A sensor device comprising:

[0769] An array sensor comprising a plurality of visible light or invisible light imaging elements arranged in one or two dimensions;

[0770] an image processing unit that performs image processing on an image signal obtained by image capturing using the array sensor; and

[0771] A threshold setting unit sets thresholds for parameter setting for all or some parameters used in image capture processing associated with image capture performed by the array sensor or image processing performed by the image processing unit to implement processing using threshold-based parameter setting.

[0772] (2) The sensor device according to (1) above,

[0773] The threshold setting unit sets a threshold corresponding to the category of the object detected from the image signal.

[0774] (3) The sensor device according to (1) or (2) above,

[0775] Here, the threshold value setting unit sets the threshold value based on a learning process performed on the image signal.

[0776] (4) The sensor device according to any one of (1) to (3) above,

[0777] The threshold setting unit sets the threshold so as to obtain a predetermined ratio as the confidence level of the object detection.

[0778] (5) The sensor device according to any one of (1) to (4) above,

[0779] The threshold setting unit is provided in a device having a housing which is the same as that of the image processing unit.

[0780] (6) A sensor device comprising:

[0781] An array sensor includes a plurality of detection elements arranged in one or two dimensions;

[0782] a signal processing unit that acquires a detection signal obtained by the array sensor and performs signal processing on the detection signal; and

[0783] a computing unit that detects an object from a detection signal obtained by the array sensor and gives an instruction to the signal processing unit to generate region information based on the detection of the object as region information associated with acquisition of the detection signal obtained from the array sensor or associated with signal processing of the detection signal,

[0784] The calculation unit sets an active region of a detection signal acquired from the array sensor based on information associated with previous region information, and detects the object using the detection signal of the active region.

[0785] (7) The sensor device according to (6) above,

[0786] Here, the calculation unit sets the active area for the plurality of pieces of area information generated in a predetermined period in the past so that the active area includes the detection area detected by the object detection based on the generated corresponding area information.

[0787] (8) The sensor device according to (6) or (7) above,

[0788] The signal processing unit includes an acquisition unit for selectively acquiring detection signals from the detection elements of the array sensor.

[0789] The acquisition unit acquires the detection signal of the detection element selected based on the area information received from the calculation unit as a frame of detection signal, and

[0790] When the acquisition unit has selected the detection element based on the area information and the target object is not detected from the detection signal acquired from the array sensor through object detection, the calculation unit instructs the acquisition unit to acquire the detection signal of the active area from the array sensor in a subsequent frame.

[0791] (9) The sensor device according to any one of (6) to (8) above,

[0792] The calculation unit detects an object from detection signals of an active area in a target frame designated as a key frame among detection signals obtained from the array sensor, and generates area information based on the detection of the object.

[0793] (10) The sensor device according to any one of (6) to (9) above,

[0794] wherein the computing unit identifies the category of the object detected in the detection signal obtained from the array sensor, and generates area information corresponding to the object using a template, the template indicating an acquisition area of ​​the detection signal corresponding to the identified category, and

[0795] The calculation unit includes a threshold setting unit, which sets thresholds for all or some parameters used for signal processing performed by the signal processing unit or for acquiring detection signals using an array sensor, and sets parameters for processing performed on an acquisition area indicated by a template based on the threshold settings.

[0796] (11) A sensor device comprising:

[0797] An array sensor includes a plurality of detection elements arranged in one or two dimensions;

[0798] a signal processing unit that acquires a detection signal obtained by the array sensor and performs signal processing on the detection signal; and

[0799] A threshold setting unit sets a threshold of a frame rate according to a category identified for an object detected from a detection signal obtained by the array sensor, so that processing using a frame rate set based on the threshold is performed according to the category identified for an object detected from a detection signal obtained by the array sensor.

[0800] (12) The sensor device according to (11) above,

[0801] The threshold setting unit sets the threshold to maintain a frame rate for object tracking from the image.

[0802] (13) The sensor device according to (11) or (12) above,

[0803] wherein, according to a result of the object detection, control is performed based on a setting value of a frame rate of the first mode or the second mode, and

[0804] The setting value of the frame rate set by the threshold setting unit is used as the setting value of the frame rate in one of the first mode and the second mode.

[0805] (14) The sensor device according to any one of (1) to (13) above, further comprising:

[0806] The output unit outputs the image signal that has undergone the image processing by the image processing unit to an external device.

[0807] (15) According to the sensor device of (14) above,

[0808] The output unit further sends information associated with a category identifier of the image signal to be output.

[0809] (16) The sensor device according to any one of (1) to (15) above,

[0810] In which, the image processing unit performs at least one of color correction, gamma correction, color grayscale processing, gain processing, contour emphasis processing, data compression processing, frame rate conversion, resolution conversion, aspect ratio conversion, contrast adjustment processing, clarity adjustment processing, grayscale adjustment processing or sampling rate change processing as image processing of the image signal.

[0811] (17) A signal processing method performed by a sensor device, comprising:

[0812] An array sensor comprising a plurality of visible light or invisible light imaging elements arranged in one or two dimensions, and

[0813] an image processing unit that performs image processing on an image signal obtained by image capturing using the array sensor,

[0814] Signal processing methods include:

[0815] Thresholds for parameter setting are set for all or some parameters used in image capture processing associated with image capture performed by an array sensor or image processing performed by an image processing unit to implement processing using parameters that change based on the thresholds.

[0816] (18) A signal processing method performed by a sensor device, comprising:

[0817] An array sensor includes a plurality of detection elements arranged in one or two dimensions, and

[0818] a signal processing unit, which acquires the detection signal obtained by the array sensor and performs signal processing on the detection signal,

[0819] Signal processing methods include:

[0820] performing the following processing: detecting an object from a detection signal obtained by the array sensor, and giving an instruction to a signal processing unit to generate region information based on the detection of the object as region information associated with acquisition of the detection signal obtained from the array sensor or associated with signal processing of the detection signal;

[0821] performing a process of setting an active area of ​​a detection signal acquired from an array sensor based on previous area information; and

[0822] The object is detected using the detection signal of the active area as a detection signal obtained by the array sensor.

[0823] (19) A signal processing method performed by a sensor device, comprising:

[0824] An array sensor comprises a plurality of detection elements arranged in one or two dimensions, and

[0825] a signal processing unit, which acquires the detection signal obtained by the array sensor and performs signal processing on the detection signal,

[0826] Signal processing methods include:

[0827] A threshold value of the frame rate is set according to a category identified for an object detected from a detection signal obtained by the array sensor, so that processing using the frame rate set based on the threshold value is performed according to the category identified for an object detected from a detection signal obtained by the array sensor.

[0828] [reference numerals list]

[0829] 1: Sensor device

[0830] 2: Array sensor

[0831] 3: ADC / Pixel Selector

[0832] 4: Buffer

[0833] 5: Logic Unit

[0834] 6: Memory

[0835] 7: Interface unit

[0836] 8: Computational unit

[0837] 11: Processor

[0838] 12: External Sensors

[0839] 20: Bounding Box

[0840] 21: ROI

[0841] 22: Advanced ROI (AROI)

[0842] 23: Candidate area

[0843] 30: Signal processing unit

[0844] 81: Keyframe selection unit

[0845] 82: Object Region Recognition Unit

[0846] 83: Category recognition unit

[0847] 84: Parameter selection unit

[0848] 85: Threshold setting unit

[0849] 100: terminal device.

Claims

1. A sensor device, comprising: An array sensor comprising a plurality of visible light or invisible light imaging elements arranged in one dimension, or a plurality of visible light or invisible light imaging elements arranged in two dimensions; an image processing unit that performs image processing on an image signal obtained by image capturing using the array sensor; as well as a threshold setting unit that sets a threshold value for parameter setting for all or some parameters used in image capture processing associated with the image capture performed by the array sensor or used in the image processing performed by the image processing unit to achieve processing using the parameters set based on the threshold value, Wherein, the computing unit of the sensor device: Compute a bounding box around the area of ​​objects that are classified into the target category, and wherein the calculation unit: calculates pixels in an appearance area of ​​the array sensor, wherein the bounding box of the target category has appeared in a previous predetermined cycle; records the calculated pixel area and the category name as an active area; obtains image data in the active area that is analog-to-digital converted according to the area of ​​interest calculated based on the bounding box; And wherein the threshold value setting unit sets a threshold value corresponding to each of categories of objects detected from the image signal of an image.

2. The sensor device according to claim 1, in, The threshold value setting unit sets the threshold value based on a learning process performed on the image signal.

3. The sensor device according to claim 1, in, The threshold setting unit sets the threshold so as to obtain a predetermined ratio as a confidence level of the object detection.

4. The sensor device according to claim 1, in, The threshold setting unit is provided in a device having a housing which is the same as that of the image processing unit.

5. The sensor device according to claim 1, further comprising: An output unit outputs the image signal that has undergone the image processing by the image processing unit to an external device.

6. The sensor device according to claim 5, in, The output unit also transmits information associated with a category identification of the image signal to be output.

7. The sensor device according to claim 1, in, The image processing unit performs at least one of color correction, gamma correction, color grayscale processing, gain processing, contour emphasis processing, data compression processing, frame rate conversion, resolution conversion, aspect ratio conversion, contrast adjustment processing, clarity adjustment processing, grayscale adjustment processing, and sampling rate change processing as the image processing for the image signal.

8. A signal processing method performed by a sensor device, the sensor device comprising: An array sensor comprising a plurality of visible light or invisible light imaging elements arranged in one dimension, or a plurality of visible light or invisible light imaging elements arranged in two dimensions, and an image processing unit that performs image processing on an image signal obtained by image capturing using the array sensor, The signal processing method comprises: setting a threshold value for parameter setting for all or some of the parameters used in image capture processing associated with the image capture performed by the array sensor or the image processing performed by the image processing unit to achieve processing using parameters that are changed based on the threshold value, wherein the threshold setting unit sets a threshold value corresponding to each of the categories of objects detected from the image signal of an image; Wherein, the computing unit of the sensor device: computing a bounding box around the area of ​​the object classified into the category of the target category, And wherein, the computing unit: calculates pixels in an appearance area of ​​the array sensor, wherein the bounding box of the target category has appeared in a previous predetermined cycle; records the calculated pixel area and category name as an active area; and obtains image data in the active area that has been converted from analog to digital based on an area of ​​interest calculated based on the bounding box.

Images