Sensor device, sensor-equipped device, and sensor device processing method

Abstract

The sensor device is provided with an array sensor having a plurality of detection elements arranged in one or two dimensions, a signal processing unit for processing a detection signal from the array sensor, and a computation unit for performing object detection based on the detection signal from the array sensor, for controlling operation of the signal processing unit based on the object detection, and for performing switching processing for changing processing content based on device information input from a sensor-equipped device in which the sensor device is installed.

Description

TECHNICAL FIELD

[0001] The present technology relates to a sensor device, a sensor-equipped device, and a processing method of a sensor device, and particularly relates to a technology accompanying input and output between a sensor device and a sensor-equipped device. BACKGROUND

[0002] The following PTL 1 discloses a technology in which an image sensor includes an interface for a plurality of sensors such as an acceleration sensor and senses movement of a device equipped with the image sensor (for example, a smartphone or the like), thereby enabling imaging even in a state in which an application processor is not activated.

[0003] [CITATION LIST]

[0004] [Patent Literature]

[0005] [Patent Literature 1]

[0006] JP 2017-228975 A SUMMARY

[0007] [TECHNICAL PROBLEM]

[0008] The technology disclosed in PTL 1 is useful because the image sensor has a function of performing imaging using input of an external sensor as a trigger, but it is not possible to change an operation parameter of the image sensor to an optimal setting for an environment and an application according to a remaining capacity of a battery of a device equipped with the image sensor, a communication situation, or the like.

[0009] Therefore, the present disclosure proposes a technology in which a sensor device can autonomously optimize processing according to a state and operation of a device.

[0010] [SOLUTION TO PROBLEM]

[0011] A sensor device according to the present technology includes an array sensor in which a plurality of detection elements are arranged one-dimensionally or two-dimensionally, a signal processing unit that performs signal processing on a detection signal obtained by the array sensor, and an arithmetic operation unit that performs object detection from the detection signal obtained by the array sensor, performs operation control of the signal processing unit based on the object detection, and performs switching processing for changing a processing content based on device information input from a device equipped with a sensor on which the sensor device is mounted.

[0012] That is, the signal processing unit performs signal processing on a detection signal obtained by the array sensor and outputs the processed signal, and controls the processing operation in the signal processing unit based on the object detection. Further, based on device information of a sensor-equipped device in which the sensor device is installed, switching is made between processing contents in the signal processing unit and the arithmetic unit. For example, execution / non-execution of the processing itself, the type of processing to be executed, the parameter setting of the processing, and the like can be switched.

[0013] Note that the object detected from the detection signal of the array sensor is an object to be detected, and any object can be the object to be detected mentioned here. For example, all objects such as a person, an animal, a moving object (a car, a bicycle, a flying vehicle, or the like), a natural object (a vegetable, a plant, or the like), an industrial product / component, a building, a facility, a mountain, the sea, a river, a star, the sun, a cloud, and the like can be the target object.

[0014] Further, as the detection element of the array sensor, an imaging element for visible light or invisible light, a sound wave detection element that detects a sound wave, a touch sensor element that detects a touch, and the like are assumed.

[0015] The signal processing for the detection signal obtained by the array sensor can include all kinds of processing from the reading processing of the detection signal obtained by the array sensor to the processing for output from the sensor device.

[0016] In the sensor device according to the present technology, it is assumed that the arithmetic operation unit performs processing to generate sensor operation information based on the object detection or the processing operation of the signal processing unit and transmits the generated sensor operation information to the sensor-equipped device.

[0017] The operation state in the signal processing unit and the object detection state include information such as a confidence rate, a category, and an object region of the object detection, a frame rate, a processing parameter, an image quality parameter, a resolution, and various other information. For example, the information is transmitted to the device.

[0018] In the sensor device according to the present technology, it is assumed that the arithmetic operation unit performs switching processing based on the registration information about the sensor-equipped device registered in advance or the registration information as the operation switching condition.

[0019] Since various situations differ depending on the type of the device and the type of application in the device, the switching processing is performed based on the registration information set in advance.

[0020] In the sensor device according to the present technology, it is assumed that the arithmetic operation unit obtains power information as the device information.

[0021] For example, assume that the remaining capacity of a battery of the device, a battery voltage, a battery temperature, a state of a power supply circuit, and the like are taken as the power supply information.

[0022] In the sensor device according to the present technology, it is conceivable that the arithmetic operation unit obtains communication state information as the device information.

[0023] For example, assume that a communication connection method, an execution throughput, whether or not connection can be made, and the like are taken as the communication state information.

[0024] In the sensor device according to the present technology, it is conceivable that the arithmetic operation unit obtains information on hardware or applications of a device equipped with a sensor as the device information.

[0025] For example, assume that information on a hardware configuration such as the presence or absence of a micro control unit (MCU), the capability of the MCU, a temperature in the equipment, sensor information, and a memory size, information on a target of an application, information of necessary metadata, and the like are taken as the information on hardware or applications of the device.

[0026] In the sensor device according to the present technology, it is conceivable that the arithmetic operation unit performs processing to perform class recognition on an object detected from a detection signal obtained by an array sensor and selects a parameter for signal processing of a signal processing unit based on the recognized class, as the processing related to the switching processing.

[0027] That is, a signal processing parameter for an image signal (for example, a signal obtained by an array sensor) can be set based on class recognition of a detected object.

[0028] Note that the processing related to the switching processing is processing that switches execution / non-execution, processing whose contents are switched by the switching processing, processing of a parameter, processing that is selectively executed together with other processing by the switching processing, and the like, as processing that is a certain kind of switching object.

[0029] In addition, a class is a kind of an object recognized using image recognition. For example, a to-be-detected object such as "person", "car", "airplane", "ship", "truck", "bird", "cat", "dog", "deer", "frog", and "horse" is classified.

[0030] In the sensor device according to the present technology, it is conceivable that the arithmetic operation unit sets a threshold value of all or some parameters used for signal processing of a signal processing unit or detection processing of an array sensor based on the device information to perform processing using the parameters set based on the threshold value, as the processing related to the switching processing.

[0031] The threshold value can be set, and a parameter used in an image processing unit, an array sensor, and the like can be changed based on the threshold value.

[0032] Examples of parameters related to detection processing of the array sensor include an exposure time in the array sensor, a timing of reading from the array sensor, a frame rate, and the like.

[0033] Examples of parameters related to signal processing include a resolution of an image, a number of tones, and the like.

[0034] In the sensor device according to the present technology, it is conceivable that, as the processing related to the switching processing, the arithmetic operation unit executes processing for performing object detection on the detection signal obtained by the array sensor and giving an instruction of region information generated based on the object detection to the signal processing unit as the signal processing regarding the region information or the detection signal obtained from the array sensor.

[0035] That is, the signal processing unit executes signal processing on the detection signal obtained by the array sensor and outputs the processed signal from the output unit, and sets the region information regarding the detection signal obtained from the array sensor or the signal processing in the signal processing unit based on the object detection.

[0036] In the sensor device according to the present technology, it is conceivable that, as the processing related to the switching processing, the arithmetic operation unit executes processing to perform object detection on the detection signal obtained by the array sensor, gives an instruction of region information generated based on the object detection to the signal processing unit as the signal processing regarding the region information or the detection signal obtained from the array sensor, performs class recognition on the object detected from the detection signal obtained by the array sensor, and generates region information corresponding to the object using a template corresponding to the recognized class.

[0037] For example, templates of region information corresponding to classes such as "person", "car", and the like are prepared, and the templates are selected and used in accordance with the class recognition. For example, the template indicates, according to the class, a detection element in the array sensor for which detection information needs to be obtained.

[0038] In the sensor device according to the present technology, it is conceivable that the arithmetic operation unit executes processing as the processing related to the switching processing, the processing for setting a threshold value of a parameter for all or some of the parameters used for the signal processing of the signal processing unit or the detection processing of the array sensor, and setting a processed parameter based on the threshold value for the region information indicated by the template.

[0039] The threshold value is set, and it is possible to change the processed parameter of the region indicated by the template based on the threshold value.

[0040] In the sensor device according to the present technology, it is conceivable that the arithmetic operation unit executes processing for performing object detection from the detection signal obtained by the array sensor and giving an instruction of region information generated based on the detection of the object to the signal processing unit as signal processing of the region information or the detection signal related to the acquisition of the detection signal from the array sensor, as processing related to the switching processing, and the signal processing unit performs compression processing of the detection signal obtained by the array sensor at a compression ratio different for each region based on the region information received from the arithmetic operation unit.

[0041] That is, the signal processing unit sets the compression rate of each region by using the region information.

[0042] In the sensor device according to the present technology, it is conceivable that the arithmetic operation unit executes the following processing: setting an effective region of the detection signal obtained from the array sensor based on information on past region information, performing object detection from the detection signal of the effective region, and giving an instruction of region information generated based on the detection of the object to the signal processing unit as signal processing of the region information or the detection signal related to the acquisition of the detection signal from the array sensor, as processing related to the switching processing.

[0043] That is, the object detection for generating the region information is performed based on information from the region set as the effective region, not from all regions of the array sensor.

[0044] Note that the information related to the region information is information of the object detection region that is a source of the region information, the region information itself, or the like.

[0045] In the sensor device according to the present technology, as processing related to the switching processing, it is conceivable that the arithmetic operation unit executes processing to perform object detection from the detection signal obtained by the array sensor and give an instruction for making the frame rate of the detection signal obtained by the array sensor variable based on the detection of the object.

[0046] That is, the frame rate of the detection signal obtained by the array sensor is changed according to the object detection result.

[0047] In the sensor device according to the present technology, it is conceivable that the arithmetic operation unit sets a threshold value of the frame rate according to a category identified for an object detected from the detection signal obtained by the array sensor to perform processing using the frame rate set based on the threshold value as processing related to the switching processing.

[0048] For example, the frame rate suitable for object detection and capable of reducing the amount of data can be set according to the category.

[0049] The sensor-equipped device according to the present technology includes a sensor device and a control unit capable of communicating with the sensor device, wherein the sensor device includes: an array sensor in which a plurality of detection elements are arranged one-dimensionally or two-dimensionally; a signal processing unit that performs signal processing on a detection signal obtained by the array sensor; and an arithmetic operation unit that performs object detection based on the detection signal obtained by the array sensor, performs operation control of the signal processing unit based on the object detection, and performs switching processing of operation setting of the signal processing unit based on device information input from the control unit.

[0050] That is, the control unit inputs the device information of the sensor-equipped device to the arithmetic operation unit in the sensor device.

[0051] In the sensor-equipped device according to the present technology, the arithmetic operation unit performs processing to generate sensor operation information based on a state of processing operation of the signal processing unit or a state of object detection and transmit the generated sensor operation information to the control unit, and the control unit controls device operation based on the sensor operation information.

[0052] That is, the arithmetic operation unit in the sensor device outputs various information to the control unit of the sensor-equipped device.

[0053] The processing method of the sensor device according to the present technology includes: performing object detection from a detection signal obtained by an array sensor in the sensor device; performing operation control of a signal processing unit based on the object detection; and performing switching processing for changing processing content based on device information input from a sensor-equipped device on which the sensor device is installed.

[0054] That is, the sensor device itself can switch processing according to information of the sensor-equipped device. BRIEF DESCRIPTION OF DRAWINGS

[0055] Figure 1 is a diagram illustrating a sensor-equipped device according to an embodiment of the present technology.

[0056] Figure 2 is a diagram illustrating a sensor-equipped device and a sensor device according to an embodiment.

[0057] Figure 3 is a block diagram of a sensor-equipped device and a sensor device according to an embodiment.

[0058] Figure 4 is a diagram illustrating threshold setting for image adaptive processing according to an embodiment.

[0059] Figure 5 is a diagram illustrating a frame on which region cropping is performed according to an embodiment.

[0060] Figure 6 is a diagram illustrating region cropping analysis and a region of interest (ROI) according to an embodiment.

[0061] Figure 7 is a diagram illustrating an advanced ROI (AROI) according to an embodiment.

[0062] Figure 8 is a diagram illustrating an advanced ROI according to an embodiment.

[0063] Figure 9 is a diagram illustrating an advanced ROI and a threshold according to an embodiment.

[0064] Figure 10 is a diagram illustrating intelligent compression according to an embodiment.

[0065] Figure 11 is a diagram illustrating effective region cropping according to an embodiment.

[0066] Figure 12 is a diagram illustrating effective region cropping according to an embodiment.

[0067] Figure 13 is a diagram illustrating active sampling according to an embodiment.

[0068] Figure 14 is a diagram illustrating active sampling according to an embodiment.

[0069] Figure 15 is a diagram illustrating active sampling according to an embodiment.

[0070] Figure 16 is a flowchart illustrating a first processing example according to an embodiment.

[0071] Figure 17 is a flowchart illustrating a second processing example according to an embodiment.

[0072] Figure 18 is a flowchart illustrating a third processing example according to an embodiment.

[0073] Figure 19 is a block diagram illustrating another configuration example applicable to the embodiment.

[0074] Figure 20 is a block diagram illustrating yet another configuration example applicable to the embodiment. DETAILED DESCRIPTION

[0075] Hereinafter, the embodiment will be described in the following order.

[0076] <1. Configuration of sensor-equipped device and sensor apparatus>

[0077] <2. Various processes>

[0078] [2-1: Classification image adaptation]

[0079] [2-2: Image adaptation by threshold setting]

[0080] [2-3: Region cropping]

[0081] [2-4: Region cropping using AROI]

[0082] [2-5: Region cropping using threshold setting and AROI]

[0083] [2-6: Intelligent compression]

[0084] [2-7: Active area cropping]

[0085] [2-8: Active sampling]

[0086] [2-9: Active sampling by threshold setting]

[0087] <3. Overview of operations based on input and output information>

[0088] <4. Processing examples>

[0089] [4-1: First processing example]

[0090] [4-2: Second processing example]

[0091] [4-3: Third processing example]

[0092] <5. Another configuration example of a sensor device>

[0093] <6. Conclusion and examples applied to each process>

[0094] Further, in the embodiments described below, a sensor device 1 that functions as an image sensor having an array of imaging elements and outputs an image signal as a detection signal will be described as an example. Specifically, the sensor device 1 according to the embodiments has an object detection function based on image analysis, and is a device that can be referred to as an intelligent array sensor (IAS).

[0095] <1. Configuration of a sensor-equipped device and a sensor device>

[0096] Figure 1An example of a sensor-equipped device 100 according to the present embodiment is shown. The sensor-equipped device 100 is a device equipped with the above-described sensor apparatus called an IAS, and examples thereof include a self-propelled robot 101, a small flight vehicle 102 such as a drone, a communication equipment 103 called an agent device or an IoT equipment, a surveillance camera 104, and the like, as shown in the drawing. In addition to these, examples thereof also include wearable devices such as glasses type, head-mounted earphone type, and wristwatch type, various home electric appliances, and the like.

[0097] These will be collectively referred to as the sensor-equipped device 100. Note that the sensor-equipped device 100 is also simply referred to as "device 100".

[0098] Figure 2 The sensor apparatus 1 and the device-side processing unit 50 in the sensor-equipped device 100 are shown.

[0099] As the device-side processing unit 50, a processor, a microcomputer, or the like serving as a control unit in the sensor-equipped device 100 is assumed. The device-side processing unit 50 can be constituted of a plurality of processors. In addition, a sensor, a logic circuit, an actuator driving circuit, or the like can be assumed as the device-side processing unit 50, instead of an algorithm operation processing device such as a processor or a microcomputer.

[0100] Figure 3 The sensor apparatus 1 as an IAS is shown in detail in the middle, and can perform image capturing by being mounted on the device 100, and perform object detection and various algorithm operation processing related to the object detection from the image. Note that the object detection in the sensor apparatus 1 can be fixed as processing for improving processing efficiency and power saving, instead of the object detection performed by the device 100. For example, in the case of the device 100 in which the device-side processing unit 50 performs the object detection, the object detection is performed in the sensor apparatus 1 so as to improve the processing efficiency of the device-side processing unit 50 and improve the detection accuracy. An example of such processing will be described later.

[0101] Naturally, in the case of the device 100 in which the object detection is not performed by the device-side processing unit 50, or even in the case of performing the object detection, the result of the object detection performed in the sensor apparatus 1 can be used for any processing of the device 100.

[0102] An interface for mutually inputting and outputting various data is formed between the sensor apparatus 1 and the device-side processing unit 50.

[0103] Generally, in a device equipped with a sensor device, a sensing signal (an image signal, a detection signal, etc.) obtained by the sensor device is naturally supplied to a processing unit on the device side, but the interface mentioned here is not an interface for such a sensing signal, and is an interface for, for example, device information or sensor operation information other than the sensing signal as shown in the figure.

[0104] From the perspective of the sensor device 1, device information is input from the device-side processing unit 50. Examples of the device information include power information, communication status information, hardware / application information, and the like of the device 100.

[0105] In addition, the sensor device 1 outputs sensor operation information to the device-side processing unit 50. The sensor operation information is information indicating a state of a signal processing operation of a sensing signal in the sensor device 1 and information based on object detection.

[0106] Examples of the information indicating a state of a signal processing operation include various information such as a frame rate of an image, a processing parameter, an image quality parameter, a resolution, and other parameters related to signal processing.

[0107] Examples of the information based on object detection include information such as a confidence rate of object detection, a level, and an object region.

[0108] Figure 3 A detailed configuration example of the sensor device 1 is mainly shown.

[0109] In Figure 3 , the processor 11 and the external sensor 12 are also shown as devices that perform data communication with the sensor device 1. In this example, it is assumed that the processor 11 is equivalent to the device-side processing unit 50 in Figure 2 .

[0110] The sensor device 1 includes an image sensor device, a storage region such as a dynamic random access memory (DRAM), and as hardware, a component as an artificial intelligence (AI) function processor. The integrated device is formed of three components constituting a three-layer structure, a so-called planar configuration is formed of one layer, or a laminated structure is constituted of two layers (for example, the DRAM and the AI function processor are the same layer), and the like.

[0111] As shown in Figure 3 , the sensor device 1 includes an array sensor 2, an analog-digital converter (ADC) / pixel selector 3, a buffer 4, a logic unit 5, a memory 6, an interface (I / F) unit 7, and an arithmetic operation unit 8.

[0112] The ADC / pixel selector 3, the buffer 4, and the logic unit 5 function as a signal processing unit 30 that performs signal processing to output a detection signal obtained by the array sensor 2 to the outside.

[0113] The array sensor 2 is configured so that the detection elements are imaging elements of visible light or invisible light, and the plurality of imaging elements are arranged one-dimensionally or two-dimensionally. For example, the array sensor is configured so that a large number of imaging elements are two-dimensionally arranged in a row direction and a column direction, and a two-dimensional image signal is output by photoelectric conversion in each imaging element.

[0114] The ADC / pixel selector 3 converts an electric signal that has been subjected to photoelectric conversion by the array sensor 2 into digital data and outputs an image signal as digital data.

[0115] Note that the ADC / pixel selector 3 has a pixel selection function for the pixels (imaging elements) of the array sensor 2, and thus, can read a photoelectric conversion signal only for the pixels selected in the array sensor 2, convert the read signal into digital data, and output the digital data. That is, the ADC / pixel selector 3 can output digital data of a photoelectric conversion signal only for the selected pixels, although normally digital data of a photoelectric conversion signal for all effective pixels constituting an image of one frame is output.

[0116] Although the image signal is read by the ADC / pixel selector 3 in units of frames, the image signal of the frame is temporarily stored in the buffer 4, read at an appropriate timing, and supplied for processing by the logic unit 5.

[0117] The logic unit 5 performs various types of necessary signal processing (image processing) on each input frame image signal.

[0118] For example, it is assumed that the logic unit 5 performs image quality adjustment by processing such as color correction, gamma correction, color gradation processing, gain processing, contour emphasis processing, contrast adjustment processing, sharpness adjustment processing, and gradation level adjustment processing.

[0119] In addition, it is also assumed that the logic unit 5 performs processing for changing the data size, such as data compression processing, resolution conversion, frame rate conversion, aspect ratio conversion, and sampling rate change.

[0120] For each processing by the logic unit 5, parameters used for each processing are set. For example, there are setting values such as color and brightness correction coefficients, gain values, compression rates, frame rates, resolutions, processing target regions, and sampling rates. The logic unit 5 performs necessary processing using the parameters set for each processing. In the present embodiment, the arithmetic operation unit 8 can set these parameters.

[0121] The image signal processed by the logic unit 5 is stored in the memory 6.

[0122] The image signal stored in the memory 6 is sent and output to an external processor 11 or the like at a necessary timing through the interface unit 7.

[0123] Note that a DRAM, a static random access memory (SRAM), a magnetoresistive random access memory (MRAM), or the like can be envisaged as the memory 6.

[0124] Note that the MRAM is a memory that stores data using magnetism, and is known to use a tunnel magnetoresistance (TMR) element instead of a magnetic core. The TMR element has an extremely thin insulating layer that includes several atoms inserted between magnetic substances, and its resistance varies according to the magnetization direction of the magnetic layer. The magnetization direction of the TMR element does not change even when not powered, and thus can be used as a nonvolatile memory. Since a write current needs to be increased as miniaturization progresses, a spin-torque transfer (STT)-MRAM using a spin current with uniform spin flow for writing without using a magnetic field is known for miniaturization of a memory cell.

[0125] Naturally, storage elements other than this can be envisaged as specific examples of the memory 6.

[0126] The external processor 11 of the sensor device 1 is able to perform necessary object detection and the like by performing image analysis and image recognition processing on the image signal sent from the sensor device 1. Alternatively, the processor 11 can perform signal processing for storage, communication, display, and the like of the image signal.

[0127] The external processor 11 can also refer to detection information of the external sensor 12.

[0128] Note that the processor 11 is installed on the device 100 together with the sensor device 1, but it is conceivable that the processor 11 is connected to the sensor device 1 in a wired or wireless manner.

[0129] The arithmetic operation unit 8 is configured as, for example, one AI processor. The arithmetic operation unit 8 includes a key frame selection unit 81, an object region recognition unit 82, a category recognition unit 83, a parameter selection unit 84, a threshold setting unit 85, a switching determination unit 86, and an operation information generation unit 87 as executable algorithm operation functions as illustrated. Further, these algorithm operation functions can be configured by a plurality of processors.

[0130] The arithmetic operation unit 8 can also communicate with the processor 11 through the interface unit 7. For example, the arithmetic operation unit 8 can input device information from the processor 11 through the interface unit 7 as illustrated in Figure 2 In addition, the arithmetic operation unit 8 can output sensor operation information to the processor 11 through the interface unit 7.

[0131] The key frame selection unit 81 in the arithmetic operation unit 8 performs processing for selecting a key frame as a moving image in frames of an image signal in response to a predetermined algorithm or an instruction.

[0132] The object region recognition unit 82 performs detection of a region of an object serving as a detection candidate in an image (frame) and region recognition processing of a target object of detection on a frame of an image signal that has undergone photoelectric conversion of the array sensor 2 and is read by the ADC / pixel selector 3.

[0133] An object detected from an image refers to an object that can be a detection target for a purpose of recognition from an image. Although the object that becomes a detection target is determined in accordance with a detection purpose, processing capacity, application type, and the like of the sensor device 1 and the processor 11, any object can be a detection target as mentioned here. Some examples of possible detection target objects include an animal, a moving object (a car, a bicycle, a flying vehicle, and the like), a natural object (a vegetable, a plant, and the like), an industrial product / part, a building, a facility, a mountain, a sea, a river, a star, the sun, a cloud, and the like.

[0134] The category recognition unit 83 performs category classification on an object detected by the object region recognition unit 82.

[0135] A category is a kind of an object that is recognized using image recognition. For example, detection target objects such as “a person”, “a car”, “an airplane”, “a ship”, “a truck”, “a bird”, “a cat”, “a dog”, “a deer”, “a frog”, and “a horse” are classified.

[0136] The parameter selection unit 84 stores parameters for signal processing corresponding to a category and selects one or more corresponding parameters by using a category of a detection object, a region thereof, and the like that are recognized by the category recognition unit 83. Then, the parameter selection unit 84 sets one or more parameters in the logic unit 5.

[0137] The threshold setting unit 85 has a function of a DNN engine and performs processing for setting a threshold of a parameter for image processing of the logic unit 5 or imaging processing (processing of the array sensor 2 and the ADC / pixel selector 3) related to imaging of the array sensor 2, on all or a part of the parameters.

[0138] Further, the threshold setting unit 85 causes all or some of the logic unit 5, the array sensor 2, and the ADC / pixel selector 3 to perform processing using a parameter that is changed based on a threshold.

[0139] Specifically, the threshold setting unit 85 changes a parameter, for example, for image processing in the logic unit 5 and sets the changed parameter in the logic unit 5.

[0140] Alternatively, the threshold setting unit 85 changes parameters for exposure operation and imaging processing (such as reading processing and AD conversion processing of the ADC / pixel selector 3) in the array sensor 2 based on the threshold value, and sets the changed parameters in the array sensor 2 and the ADC / pixel selector 3.

[0141] The switching determination unit 86 makes switching determination of the processing content of the sensor device 1 based on the device information input from the processor 11, the registration information (described later) set in advance, and the like, and makes switching control of the processing content as necessary. For example, the switching determination unit 86 performs switching control of execution / non-execution of the processing itself, the kind of the processing executed, parameter setting of the processing, and the like.

[0142] The operation information generation unit 87 generates sensor operation information based on the processing operation of the signal processing unit 30 or the object detection of the arithmetic operation unit 8, and performs processing for transmitting the generated sensor operation information to the processor 11. Examples of the sensor operation include, for example, information of the confidence rate of the object detection, the class, and the object region, the frame rate, the processing parameter, the image quality parameter, the resolution, and different other information.

[0143] These functions of the arithmetic operation unit 8 are processing that is not generally performed in the array sensor, and in the array sensor in the present embodiment, object detection, class recognition, color processing, and the like are performed. Therefore, the image signal provided to the processor 11 is suitable for the device 100.

[0144] Note that, in addition to outputting the image signal to the processor 11, the interface unit 7 can output the above-described sensor operation information together with the image signal as, for example, metadata, or can output the sensor operation information independently of the image signal. Further, for example, only the class information and the like can also be output.

[0145] Further, for example, it is also conceivable that the processor 11 gives an instruction to the interface unit 7 to give necessary information, and the interface unit 7 transfers the corresponding information to the arithmetic operation unit 8 and the like.

[0146] <2. Various processing>

[0147] Here, various processing that can be performed by the sensor device 1 including the arithmetic operation unit 8 is described. In the following, as processing examples, classification image adaptation, image adaptation based on threshold setting, region cropping, region cropping using AROI, region cropping using threshold setting and AROI, smart compression, effective region cropping, active sampling, and active sampling based on threshold setting will be described in order.

[0148] Further, each process is an example of a process related to a switching process based on the function of the switching determination section 86. The process related to switching is a process that is a target of some kind of switching, such as a process in which execution / non-execution of itself is switched, a process in which contents and parameters of the process are switched by the switching process, or a process that is selectively executed with other processes by the switching process.

[0149] [2-1: Classification image adaptation]

[0150] Classification image adaptation is a process for setting a processing parameter of the signal processing unit 30 in accordance with a class of an object.

[0151] The key frame selection unit 81 executes a process for selecting a key frame in a frame of an image captured at a timing corresponding to a key frame selection algorithm.

[0152] The sensor device 1 selects a key frame from an image signal in units of frames of pixel array output signals as an array sensor 2 and executes image recognition to recognize a class of an object to be imaged. The key frame is selected in accordance with a key frame selection algorithm, whereby a still image (any frame) is selected.

[0153] An example of the key frame selection algorithm will be described.

[0154] First, there is a method for selecting one frame for each specified time interval. For example, one frame is set as a key frame at intervals of 30 seconds. Naturally, 30 seconds is an example.

[0155] Further, it is also conceivable to select a key frame at a timing according to a command from outside the sensor device 1 (the processor 11 or the like). For example, a key frame is selected in response to an instruction received from a device or equipment in which the sensor device 1 is installed. For example, in a case where the sensor device 1 is installed on a car and the car is stopped at a parking lot, however, a key frame can be selected at a timing at which the car starts traveling.

[0156] Further, the method of selecting a key frame can be changed according to the situation. For example, in a case where the sensor device 1 is installed on a car, the interval of key frames is changed during stop of the car, during normal travel, and during high-speed travel.

[0157] When a key frame is selected, the object region recognition unit 82 detects the position of a candidate of an object in the key frame.

[0158] That is, the object region recognition unit 82 searches for a candidate of an object to be detected in an image of a key frame and obtains the position (position coordinates in the image) of one or a plurality of candidates.

[0159] The class recognition unit 83 executes classification of a detected object into a class. That is, class recognition is executed for each candidate of an object to classify the candidate.

[0160] As described above, the category is a kind of the object recognized using the image recognition. For example, the category recognition such as "person" and "flower" is performed on the detected object.

[0161] The parameter selection unit 84 performs the parameter control corresponding to the category obtained as a result of the category recognition.

[0162] For example, the parameter selection unit 84 selects a parameter set based on the category of the object, the number of the objects, the area of the object, and the like.

[0163] For example, in a case where one category exists in the image, the parameter selection unit 84 selects a parameter set corresponding to the category. For example, in a case where "person" exists in the recognized category, the parameter selection unit 84 selects a parameter set suitable for the image of the person.

[0164] In a case where a plurality of types of categories of objects exist in the screen, the following examples can be conceived.

[0165] For example, it can be conceived that a parameter set corresponding to a category in which the number of objects is the largest is selected.

[0166] Alternatively, in a case where a plurality of types of categories of objects exist in the screen, it can be conceived that a parameter set corresponding to a category of an object having the largest area is selected.

[0167] Alternatively, in a case where a plurality of types of categories of objects exist in the screen, it can be conceived that a parameter set corresponding to a category in which the total area of each category is the largest is selected.

[0168] Alternatively, in a case where a plurality of types of categories of objects exist in the screen, it can be conceived that a category having the highest priority is obtained from the number of objects and the total area (or the maximum value) for each category, and a parameter set corresponding to the category is selected.

[0169] Naturally, various other methods of selecting a parameter set exist, and in any case, a parameter set corresponding to a category of a dominant object in the screen or a category of an object to be preferentially detected can be selected.

[0170] In addition, the parameter selection unit 84 performs processing for setting the selected parameter set in the logic unit 5.

[0171] Accordingly, the logic unit 5 subsequently performs various image processing on the image signal of the frame input sequentially using the set parameter set.

[0172] The processed image signal, the set parameter, the information of the recognized category, and the like are temporarily stored in the memory 6.

[0173] The sensor device 1 can output all or at least any one of information such as an image signal (still image, moving image), category recognition information (category, number of objects, etc.), and a parameter set used, in response to a request of the processor 11.

[0174] That is, any piece of information temporarily stored in the memory 6 is read by the interface unit 7 and transmitted in response to a request of the processor 11.

[0175] Through the above processing, an image signal having undergone parameter setting according to the presence of the category of the object included in the image is supplied to the processor 11. The image signal is an image signal having undergone image processing so as to obtain an image quality suitable for the category or an image signal having undergone image processing suitable for the category of the object.

[0176] [2-2: Image adaptation by threshold setting]

[0177] Image adaptation based on threshold setting is a processing example in which the idea of changing a parameter corresponding to a threshold setting is added to the above-described classification image adaptation processing.

[0178] As an example of the parameter mentioned here, assume a parameter used in image processing in the logic unit 5, and set (adjust and change) a parameter of image processing used in the logic unit 5, for example, so as to satisfy a threshold value set in the sensor device 1.

[0179] Further, as the parameter, assume also a parameter for signal reading in the ADC / pixel selector 3 and imaging processing such as exposure operation in the array sensor 2. A control parameter or the like of imaging processing operation of the ADC / pixel selector 3 and the array sensor 2 is set (adjusted and changed), for example, so as to satisfy a threshold value set in the sensor device 1.

[0180] In the above-described classification image adaptation, the parameter used by the logic unit 5 is selected according to the category recognition, but it is also possible to set (adjust and change) the selected parameter based on a threshold value.

[0181] Alternatively, the parameter is not necessarily limited to a parameter selected based on the category recognition, and it is conceivable that the parameter is set based on a threshold value as long as the parameter is used in the logic unit 5, the ADC / pixel selector 3, and the array sensor 2.

[0182] Specific examples of the parameter related to imaging processing and the parameter related to image processing which are automatically set in this way based on a threshold value will be described.

[0183] For example, the parameter related to image processing is exemplified as follows.

[0184] • Image aspect ratio

[0185] • Resolution

[0186] • Number of tones (number of colors or number of bits)

[0187] • Contrast adjustment value

[0188] • Sharpness adjustment value

[0189] • Gray scale adjustment value

[0190] • Gamma correction value

[0191] • Sampling rate conversion ratio

[0192] The parameters of the image aspect ratio and the resolution are also reflected in the ROI 21 described later.

[0193] The number of tones, the contrast adjustment value, the sharpness adjustment value, the gray scale adjustment value, the gamma correction value, and the resolution are parameters related to image quality.

[0194] The sampling rate conversion ratio is a parameter of the temporal resolution.

[0195] In addition, the parameters related to the imaging processing are as follows.

[0196] • Sampling rate

[0197] • Resolution (for example, resolution set at the time point of reading the ADC / pixel selector 3)

[0198] • Shutter speed (exposure time) of the array sensor 2

[0199] Of course, the parameters automatically set based on the threshold value also include parameters other than the above-described parameters.

[0200] This setting of the threshold value based on the parameters is performed so that, in a case where the processor 11 performs object detection based on learning using a deep neural network (DNN), a reduction in the amount of data, an improvement in processing speed, low power consumption, and the like are achieved while ensuring the actual accuracy of the output of the object detection.

[0201] That is, the amount of imaging data is reduced by changing parameters such as the resolution and the number of colors, but this also allows the accuracy of the object detection to be maintained at a required level.

[0202] In Figure 4 , the idea of the parameter setting based on the threshold value is described.

[0203] For example, in a case where the sensor device 1 images a person, it is assumed that an output image has information of all the pixels (all the effective pixels) of the array sensor 2, and full-color image data is output at a frame rate of, for example, 60 fps (frames per second).

[0204] Further, for example, in a case where the processor 11 performs object detection on such image data, assuming that a person can be correctly detected at a rate of 98%, a confidence rate CR = 0.98 is set. The confidence rate is a rate of certainty that an object can be correctly distinguished and detected.

[0205] On the other hand, assuming that in a case where image data of which the output resolution is slightly reduced, the number of tones is slightly reduced, and the frame rate is set to 30 fps, the confidence rate CR is set to 0.92. Further, assuming that in a case where image data of which the output resolution is more reduced, the number of tones is more reduced, and the frame rate is set to 15 fps, the confidence rate CR is set to 0.81.

[0206] Further, assuming that in a case where image data of which the resolution is drastically reduced, the number of tones is drastically reduced, and the frame rate is set to 10 fps is output, the confidence rate CR is set to 0.58.

[0207] The above is only an example for explanation, but the confidence rate fluctuates by changing parameters related to imaging or image quality, such as resolution, the number of colors, and the time resolution of image data to be analyzed in this way. That is, the accuracy of image analysis and object detection varies.

[0208] Incidentally, the confidence rate for object detection is high, but in reality, the highest rate is not always required.

[0209] For example, in a case where it is desired to roughly detect the number of people from an image obtained by photographing a park as shown in A, a high accuracy is not required. For example, in a case where a detection result indicating several people, about 10 people, about 20 people, or the like is obtained, a confidence rate CR of about 0.6 can be sufficient. Figure 5 A shown in A, a high accuracy is not required. For example, in a case where a detection result indicating several people, about 10 people, about 20 people, or the like is obtained, a confidence rate CR of about 0.6 can be sufficient.

[0210] On the other hand, in a case where it is desired to strictly monitor the intrusion of a person with a monitoring camera or the like, a confidence rate CR of about 0.95 can be required.

[0211] In addition, a confidence rate CR of 0.70 can be set during the day, but a confidence rate CR of about 0.90 can be desired to be set at night.

[0212] That is, the confidence rate CR required for the accuracy of object detection differs depending on various factors such as the purpose for detection, the target, the type of equipment / application, the time, and the region.

[0213] Further, the confidence rate also fluctuates depending on the analysis capability and the degree of learning of the processor 11, and also fluctuates depending on the object and the category to be detected.

[0214] According to these circumstances, for example, by determining a threshold based on an appropriate required confidence rate and changing the parameters accordingly, an image signal that satisfies the requirements of object detection or the like can be output.

[0215] No, in Figure 4 In the example of the present embodiment, it is assumed that a confidence rate CR of 0.80 or higher is required.

[0216] In this case, a threshold having 0.80 or more is calculated as a parameter of the confidence rate CR, and the parameters used in the logic unit 5 and the like are set. Specifically, parameters that are higher than the threshold but have a relatively small amount of data are set.

[0217] For example, parameters such as the resolution, the number of tones, and the frame rate in which the confidence rate CR is 0.81 as shown in the drawing are set. Then, compared to a case in which the parameters are set so that the confidence rate CR is set to, for example, 0.98 and an image signal is output, the amount of data can be greatly reduced and the required object detection accuracy can be maintained.

[0218] Note that, although the "threshold value" can be regarded as a value as a requirement of the confidence rate, as for the threshold value calculated for parameter adjustment, the threshold value can also be regarded as a value of the parameter for obtaining the confidence rate as a requirement "threshold value".

[0219] That is, as for the processing of "setting a threshold value of the parameter and performing processing using a parameter set based on the threshold value", a processing method as described in [1] and [2] below is assumed.

[0220] [1] A threshold value such as an index value of the confidence rate appropriate for the use mode and the use environment is calculated, and a parameter actually used as a parameter value for obtaining an index value that exceeds the threshold value is set. That is, the threshold value of the parameter is set from the viewpoint of the index value of the object detection.

[0221] [2] A threshold value for obtaining a value required as an index value (such as a confidence rate) of a parameter is calculated, and a parameter actually used is set based on the threshold value. That is, the threshold value of the parameter is set from the viewpoint of the value of the parameter itself.

[0222] In the present embodiment, for example, a threshold value is set based on the confidence rate as in [1] or [2] described above, and a parameter actually used is set to a parameter appropriate for reducing the amount of image data as much as possible. Such a parameter is calculated in real time (for example, periodically during imaging) to dynamically change the parameter.

[0223] For example, depending on the application, the target class, and the imaging environment of the sensor device 1, by calculating an appropriate threshold value and corresponding parameters through DNN processing and changing these parameters, improvement of the speed appropriate for the application, reduction of the power consumption, and improvement of the accuracy, and the like are performed.

[0224] Specifically, for parameter adjustment, a threshold value based on the confidence of object detection is set, and a set value of a parameter that is as close to the threshold value as possible and is not lower than the threshold value is calculated.

[0225] Further, it is appropriate to set a threshold value and a corresponding parameter for each category. For example, the detection accuracy of the quality of an image signal and the required accuracy differ depending on the category of an object such as "face" and "road sign", and thus the threshold value is appropriately set and the parameter is changed depending on the category.

[0226] Further, by using the maximum value of the confidence rate to set the threshold value and the corresponding parameter, the threshold value and the corresponding parameter are changed depending on the remaining capacity of the battery, and the threshold value and the corresponding parameter are changed so that object tracking at the time of object detection can be maintained.

[0227] Therefore, signal processing can be performed depending on the purpose of object detection, the operation situation, the environment, and the like.

[0228] [2-3: Region Cropping]

[0229] Region cropping is an example of a process of detecting an object by using only a specific region in the entire image as a processing target.

[0230] With respect to an image signal detected by the array sensor 2, it is generally conceivable to transmit the information of all pixels of each frame to the processor 11 to perform image recognition.

[0231] However, when the information of all pixels of all frames is transmitted to the processor 11 and object detection is performed by the processor 11, the amount of transmitted information significantly increases, and the transmission time is also required, particularly as the resolution of the image captured by the array sensor 2 becomes higher. Further, in the case of cloud transmission, the increase in the amount of communication greatly affects the communication cost and time. Further, the load of the storage amount in the processor 11 and the cloud also increases, and the analysis processing load and the processing time also increase, which leads to a concern that the object detection performance can deteriorate.

[0232] As a result, once a necessary subject is recognized in the image of a certain frame, the image signal is obtained and transmitted at the pixel level of the region of the essential subject from the next frame and subsequent frames, and the pixels of other regions are prevented from existing as information, thereby improving the processing efficiency.

[0233] The outline is shown in Figure 6

[0234] The outline is shown in Figure 6 The image of the certain frame F1 is shown in A. In the case where a "person" is set as the object to be detected, the region of the person is detected in the image of the frame F1. In addition, the region in which the person has been detected is set as a region of interest (ROI) 21 as a region of interest.​

[0235] In the subsequent frames F2, F3,..., and Fn, only the pixels in the region set as the ROI 21 are read from the array sensor 2. The image is an image including only the information of the portion of the ROI 21, as shown in Figure 6 B.

[0236] Further, the analysis in the operation and processing unit 8 is performed based on the image signal including such partial pixel information, or the image is sent to the processor 11 and analyzed.

[0237] Specifically, as schematically shown in Figure 6 A, a certain frame Fl that is a ratio of one frame to N frames of the image signal obtained by the array sensor 2 is set as an image including information on all the effective pixels. Further, the operation and processing unit 8 scans the entire screen to detect the presence or absence and the position of the object. Then, the ROI 21 is set.

[0238] When the subsequent frame F2 is obtained, as shown in Figure 6 B, an image signal that has undergone AD conversion only in the pixels of the ROI 21 set as the target region is obtained. Note that the squares divided by the grid in the figure represent the pixels.

[0239] In this way, for example, the object is detected by performing the full-screen scan only in one frame out of every N frames, and as shown in Figure 6 C, the image analysis is performed only on the detected region of the object in the previous frame in the subsequent frames F2, F3, F4,....

[0240] By performing this processing, the amount of analysis data and the amount of communication data are reduced without degrading the accuracy of the object detection targeted by the application, and reduction of the power consumption of the sensor device 1 and increase in the speed of the image analysis related to the object detection of the entire system in which the sensor device 1 is installed are performed.

[0241] [2-4: Region clipping using AROI]

[0242] An advanced ROI (also referred to as "AROI") is a ROI that is set using a template set according to a category.

[0243] The array sensor 2 (image sensor) consumes the largest amount of power in photoelectric conversion. For this reason, in order to reduce the power consumption, it is desirable to reduce the number of pixels to be subjected to photoelectric conversion as much as possible.

[0244] Further, the image signal obtained by the array sensor 2 is used for image analysis and is not seen by a person, and thus the image signal does not need to be human- visible and recognizable or a clear image. In other words, it is important that the image is able to perform object detection with high accuracy.

[0245] For example, in the above-described region cropping, a class recognition is performed on the detected object, but when the class recognition is performed in this way, a minimum region for recognition corresponding to the class can be set as the ROI. Thus, the AROI 22 is set as shown in Figure 7 and Figure 8 .

[0246] Figure 7 An AROI 22 generated using a template of the class of "person" corresponding to the image region of a person is shown. The grid in the figure is a pixel, and the dark pixels are the pixels designated by the AROI.

[0247] For example, in the template corresponding to the class of "person", the face portion is configured as a high-density necessary pixel, and the body part is configured as a low-density necessary pixel to cover the entirety.

[0248] Further, Figure 8 An AROI 22 generated using a template corresponding to the class of "car" is shown. This example shows an adaptation to a rear image of a car. For example, the portion where the number plate is positioned is configured as a high-density necessary pixel, and the necessary pixels are arranged in a low density in other portions to be able to cover the entirety.

[0249] In fact, it is also conceivable to subdivide the class of "person" and subdivide the template into "side-view person", "person facing forward", "person sitting", and the like, or for the class of "car", to subdivide the template into "side-view image", "front image", "rear image", and the like.

[0250] The template is selected in this way according to the class, and the template is enlarged / reduced according to the size of the region in the actual frame to generate the AROI 22.

[0251] The AROI 22 is generated using a template set according to the level, and thus it is possible to obtain information that enables accurate execution of object detection according to the level even when the number of pixels to be subjected to photoelectric conversion is significantly reduced.

[0252] [2-5: Region cropping using threshold setting and AROI]

[0253] Next, a processing example in which the efficiency of the region cropping using the AROI 22 is further improved will be described.

[0254] In the case of using the AROI 22 using a template, the parameter is set based on the set threshold value of the object (class), portion, and the like of the detection target. That is, adopting the above-described threshold idea, the threshold value is determined based on the correct answer rate of the object detection calculated by the DNN, and the parameter is set.

[0255] For example, the resolution distribution of the region of interest in the AROI 22 is determined in accordance with a threshold set using a confidence rate.

[0256] Figure 9 Examples are schematically shown. It is conceivable that a person is set as a target category and a face is set as a target category.

[0257] It is assumed that a relationship of first resolution > second resolution > third resolution is established.

[0258] It is assumed that a confidence rate CR for face detection is 0.95 at the first resolution, 0.86 at the second resolution, and 0.66 at the third resolution.

[0259] It is assumed that a confidence rate CR for person (body) detection is 0.98 at the first resolution, 0.81 at the second resolution, and 0.65 at the third resolution.

[0260] In a case where a threshold thF for face detection is set to 0.85, the second resolution is selected as a parameter suitable for making the amount of image data as small as possible, and image processing is performed on the pixels in the template.

[0261] Further, in a case where a threshold thP for person detection is set to 0.80, the second resolution is selected as a parameter suitable for making the amount of image data as small as possible, and image processing is performed on the pixels in the template.

[0262] In either case, the second resolution is appropriate, but it is also assumed in some cases that the first resolution is set when the threshold thF for face detection is 0.94, or the third resolution is set when the threshold thP for person detection is 0.60.

[0263] That is, in a case where the AROI 22 is used, a threshold is set for each object category to set a parameter such as image processing and reading processing for the pixels in the AROI 22.

[0264] For the AROI 22, a parameter setting corresponding to, for example, a confidence rate is also performed, and a parameter such as resolution is set so that it is possible to improve the efficiency of imaging processing and image processing while maintaining the accuracy of object detection.

[0265] [2-6: Intelligent compression]

[0266] Intelligent compression is to specify an object to be detected, to compress the object at a low compression rate, and to compress a region other than the object at a high compression rate.

[0267] A specific example is shown in Figure 10

[0268] ​Figure 10 A illustrates a state in which ROI 21 is generated as a region corresponding to each car in a case where a "car" category is detected as a target category from an image of any frame.

[0269] Figure 10 B illustrates an image signal in which a region of ROI 21 is compressed at a low compression rate and other regions are compressed at a high compression rate.

[0270] With this configuration, the amount of analysis data and the amount of communication data are reduced without degrading the accuracy of object detection for which object detection is targeted.

[0271] Furthermore, reduction of power consumption of the sensor device 1 and increase of speed of image analysis related to object detection of the entire system having the sensor device 1 mounted thereon can also be achieved.

[0272] [2-7: Effective region cropping]

[0273] In the above-described region cropping, an example in which ROI 21 is set for a to-be-detected object and only pixels in a region set as ROI 21 are read from the array sensor 2 has been described.

[0274] Here, a case in which a region set as ROI 21 can be concentrated in a specific region in an image will be mainly described.

[0275] Figure 11 A illustrates an example of an image of a monitoring camera in a building, for example. It is assumed that ROI 21 is set using a person as a detection target. In the drawing, a position in an image of a bounding box 20 that is a source of ROI 21 set in a predetermined period in the past is illustrated. The bounding box 20 is a region set based on ROI 21. For example, the bounding box 20 is set as a pixel range obtained by slightly expanding ROI 21 or the like.

[0276] For example, in this case, the set position of the bounding box 20 (and ROI 21) is a region close to a floor in the image in a predetermined period in the past.

[0277] In other words, no person appears in a region near the ceiling of the image, and thus it can be said that a person detection process can not be performed in the image region near the ceiling.

[0278] Therefore, for example, as Figure 11 B illustrates a case in which a region of "person" as a detection target, that is, a region in which the bounding box 20 has been set in a predetermined period in the past, is set as an active region RA, and a region of "person" as a detection target, that is, a region in which the bounding box 20 has not been set in a predetermined period in the past, is set as a passive region DA.

[0279] Figure 12 A shows an example of an image of a monitoring camera that monitors a car as a detection target, for example, on an expressway, and shows the position of a bounding box 20 set in a predetermined period of time in the past.

[0280] Also in this case, the car appears near the road surface, and thus, as in Figure 12 B, an active region RA and an inactive region DA are set.

[0281] The active region RA is set as in Figure 11 B and Figure 12 B, and the object is detected from the detection signal in the active region RA among the imaging pixels by the array sensor 2. Further, similarly to the above-described region cropping processing, the instruction for the ROI 21 generated based on the detection of the object is given to the signal processing unit 30 as a region related to the acquisition of the detection signal or the signal processing of the detection signal.

[0282] That is, for the object detection key frame, the object detection is performed by partially performing the photoelectric conversion based on the history information of the object detection instead of the full-screen scanning.

[0283] Note that the object detection key frame is a frame in which information is acquired in all the effective pixel regions of the array sensor 2 for object detection in the region cropping processing. It is the effective region cropping processing that information is acquired only in the pixel region of the effective region RA in the key frame.

[0284] When the effective region cropping processing is applied, the object detection of the arithmetic operation unit 8 can be performed only in the effective region RA instead of in all the effective pixel regions of one frame. Further, the active region RA is a region in which the object detection of the target category can be performed. In other words, a region in which it is almost impossible to detect the target of the target category other than the active region RA.

[0285] Therefore, it is possible to improve the processing efficiency, reduce the power consumption, and the like by reducing the number of read pixels of the object detection key frame and reducing the detection range.

[0286] [2-8: Active Sampling]

[0287] The active sampling indicates a processing for dynamically changing the frame rate in accordance with the presence or absence of the object. It can be said that it is a compression of the amount of data in the time axis direction corresponding to the presence or absence of the object. Further, it is also possible to achieve a reduction in the power consumption of the sensor device 1.

[0288] The active sampling will be described using Figure 13 [2-8-1: Active Sampling]

[0289] Now, a person is detected from the captured image by setting the target category to "person". For example, assume a case where the outside of a building is imaged by a monitoring camera through an entrance.

[0290] Figure 13 A shows a state in which a person is not included in the captured image. In this case, the frame rate is set to a low rate, for example, 1 fps.

[0291] Figure 13 B shows a state in which a person is detected in the captured image. In this case, the frame rate is changed to a high rate, for example, 100 fps.

[0292] That is, by limiting the detection target to reduce the frame rate in a particularly unnecessary case (when no person is detected) and to increase the frame rate and make the amount of information intensive in a necessary case (when a person is detected) and to dynamically change the frame rate.

[0293] In a case where such active sampling is performed, the arithmetic operation unit 8 (key frame selection unit 81) sets the moving image capture of the ADC / pixel selector 3, for example, in accordance with the setting of the idle mode that is stored in advance in the arithmetic operation unit 8.

[0294] For example, the setting of the idle mode and the setting of the normal mode are stored in the parameter selection unit 84 in the arithmetic operation unit 8.

[0295] The active sampling is provided with the idle mode and the normal mode, and the idle mode is a mode until an object of a target category is confirmed in the imaging screen.

[0296] In the idle mode, the moving image is captured at a frame rate lower than the frame rate in the normal mode.

[0297] It is conceivable that the idle mode is started in response to a command received from the outside of the sensor device 1. Further, the idle mode can respond to a command at a time interval obtained from data of the idle mode received from the outside of the sensor device 1. For example, in a case where an instruction of 60 seconds has been given, the object detection key frame recording timing is set at an interval of 60 seconds.

[0298] The normal mode is a normal moving image shooting mode. For example, the normal mode responds to a command at a time interval obtained from data of the normal mode received from the outside of the sensor device 1.

[0299] In the normal mode, the moving image is generally captured at a higher frame rate than the frame rate in the idle mode, and the normal mode is a mode in which imaging is performed at an interval of 0.01 seconds (100 fps) in a case where an instruction of, for example, 0.01 seconds has been given.

[0300] Accordingly, the arithmetic operation unit 8 gives an instruction of the idle mode to the ADC / pixel selector 3, and thus, when the idle mode is set to 1fsp, a moving image is captured at an interval of, for example, 1 second.

[0301] In addition, the setting of the idle mode and the setting of the normal mode do not necessarily have to be stored in the arithmetic processing device 8, but can be stored in an external memory of the arithmetic processing device 8.

[0302] Of course, the frame rate in the idle mode and the normal mode is an example of the frame rate.

[0303] The arithmetic operation unit 8 (object region recognition unit 82) detects a position of a candidate of an object in the obtained image.

[0304] The arithmetic operation unit 8 (category recognition unit 83) performs category classification on the object detected as the candidate.

[0305] Further, the arithmetic operation unit 8 confirms whether or not a target category exists in the category obtained as a result of the category recognition.

[0306] When the target category does not exist, the arithmetic operation unit 8 obtains an image of the next frame in the idle mode, and similarly detects a position of a candidate of an object and performs category recognition. In this case, for example, when imaging is performed at 1fps, these processes are performed on the image after 1 second.

[0307] For example, in a case where "person" is set as the target category and "person" exists as a recognized category, the arithmetic operation unit 8 sets moving image capture in the ADC / pixel selector 3 according to the setting of the normal mode stored, and instructs the ADC / pixel selector 3 to perform imaging in the normal mode.

[0308] Thereby, when the setting of the normal mode is 100fsp, moving image capture is performed at an interval of, for example, 0.01 second.

[0309] The arithmetic operation unit 8 performs the object detection process in this way in a state where the mode is switched to the normal mode.

[0310] In addition, the normal mode continues as long as the target category exists in the captured image, and the mode is switched to the idle mode when the target category does not exist.

[0311] When the process of the active sampling is performed in this way, particularly in a period where the target category does not exist, data amount compression is performed by reducing the frame rate, and thus power consumption is reduced.

[0312] Note that the arithmetic operation unit 8 makes the frame rate variable by instructing the ADC / pixel selector 3 to change the frame rate, but the arithmetic operation unit 8 can instruct the logic unit 5 to change the frame rate.

[0313] For example, reading of the array sensor 2 is always performed at 100 fps, and in the case of the idle mode, the indication logic unit 5 performs frame refinement. Thus, the amount of data related to transmission to the processor 11 can be reduced.

[0314] [2-9: Active sampling by threshold setting]

[0315] A method in which the time resolution is determined based on the correct answer rate of object detection calculated by a DNN will be added to the description of the example of the above-mentioned active sampling method.

[0316] That is, processing for dynamically changing the frame rate based on the average movement amount of the target class per unit time is performed.

[0317] In the above-mentioned active sampling, the normal mode and the idle mode are prepared and switched, but in addition to this processing, the frame rate in the normal mode is set according to the target class.

[0318] Figure 14 A shows an example of an image in the case where the sensor device 1 is used for a monitoring camera that captures an image of the state on a highway. When the target class is a car, a bounding box 20 is shown. The dotted arrow indicates the moving direction of a certain car.

[0319] Figure 14 B shows the movement amount of the imaged car as a change in the position (pixel position) of the bounding box 20 on the image in consecutive frames. When considering such movement amounts of a large number of cars, the average movement amount is assumed to be 1152 pixels / sec.

[0320] In this case, the frame rate at which object tracking can be maintained is calculated to be 5 fps Figure 14 C).

[0321] Next, Figure 15 A shows an example of an image in the case where the sensor device 1 is used for a monitoring camera in a building. When the target class is a person, a bounding box 20 is shown. The arrow indicates the moving direction of a certain person.

[0322] Figure 15 B shows the movement amount of the imaged person as a change in the position (pixel position) of the bounding box 20 on the image in consecutive frames. When considering such movement amounts of a large number of persons, the average movement amount is assumed to be 192 pixels / sec.

[0323] In this case, the frame rate at which object tracking can be maintained is calculated to be 5 fps Figure 15 C).

[0324] For example, as described above, the frame rate at which the object tracking can be maintained differs between the case where the target category is a car and the case where the target category is a person.

[0325] Then, when the frame rate at which the object tracking can be maintained is obtained and a threshold value (lower limit of the allowable frame rate) thereof is obtained by the DNN according to the target category, it is possible to maintain the accuracy of the object detection while tracking the object and keeping the amount of data as small as possible.

[0326] Note that the frame rate is determined by setting the reading timing of the array sensor 2 and setting the sampling rate of the ADC / pixel selector 3.

[0327] In this case, the operation processing unit 8 calculates the threshold value (frame rate as the threshold value) at which the target tracking is maintained while the frame rate of the target category is changed at the time of the object detection.

[0328] Thereafter, in the operation processing unit 8, the calculated threshold value, the target category, and the information of the threshold value calculation strategy used for the calculation of the threshold value are recorded in association with each other. For example, these records are recorded in a recording region within the operation processing unit 8, in a predetermined region of the memory 6, or are transferred to the processor 11 and recorded therein.

[0329] Therefore, for example, a parameter based on the threshold value corresponding to the target category, that is, a value of the frame rate as low as possible at which the object tracking can be maintained, is set. That is, the frame rate in the normal mode corresponding to the target category can be set. Then, the processing of the active sampling described above is performed.

[0330] When the processing of the active sampling is performed as described above, the data amount compression is performed by reducing the frame rate in the period in which the target category is not present, and thus the power consumption is reduced.

[0331] Further, even when the normal mode is set, the processing is performed at the frame rate adapted according to the target category, and thus a very low frame rate (5 fps or the like described above) is set according to the category. Thereby, the data amount compression and the reduction of the power consumption are performed even in the normal mode.

[0332] <3. Overview of the operation based on the input and output information>

[0333] As described above, the sensor device 1 inputs and outputs various information to and from the device-side processing unit 50 (for example, the processor 11). Hereinafter, the processing for inputting and outputting the information between the sensor device 1 and the processor 11 will be described. Figure 2

[0334] First, the processing in which the sensor device 1 inputs the device information will be described.

[0335] ​Examples of the device information include power information, communication status information, hardware / application information, and the like of the device 100.

[0336] As the processing of the sensor device 1 of this information, the following processing can be considered.

[0337] For example, in a case where the power information indicating that the device 100 has a small battery remaining capacity has been received from the processor 11, the arithmetic operation unit of the sensor device 1 performs, for example, the following processing.

[0338] • In the above processing of "image adaptation by threshold setting", the threshold setting is performed by reducing the confidence rate to adjust the parameter.

[0339] • The frame rate in the above "active sampling" is reduced.

[0340] These are examples, and when the sensor device 1 consuming the same battery in the device 100 performs the processing for reducing the power consumption, the accuracy of the object detection can be slightly reduced, but the operation time of the device 100 can be extended.

[0341] Further, for example, in a case where the communication status information indicating that the communication situation is poor has been received from the processor 11, the arithmetic operation unit of the sensor device 1 performs, for example, the following processing.

[0342] • The frame rate of capturing an image is reduced.

[0343] • The compression rate of capturing an image is increased.

[0344] • The resolution of capturing an image is reduced.

[0345] These are examples, and in a case where the device 100 transmits the image data captured by the sensor device 1 to the outside, the data amount can be effectively reduced due to the deterioration of the communication situation. Further, the reduction of the frame rate and the like also leads to the reduction of the power consumption of the sensor device 1, and it is also possible to prevent the device 100 from unnecessarily consuming energy by performing optimal streaming based on the communication situation.

[0346] Further, for example, the information indicating that the processing load is large has been received from the processor 11, the arithmetic operation unit of the sensor device 1 performs, for example, the following processing.

[0347] • The resolution of capturing an image is reduced.

[0348] • Switching from a state in which normal image imaging processing is executed to a state in which any one or two or more of "classification image adaptation", "image adaptation by threshold setting", "region cropping", "region cropping using AROI", "region cropping using threshold setting and AROI", "intelligent compression", "effective region cropping", "active sampling", and "active sampling by threshold setting" is executed. Alternatively, these are selectively switched among them instead of executing them.

[0349] These are examples, and when the sensor device 1 activates processing capable of reducing the load of the processor 11, it is also possible to reduce the amount of calculation of the device 100 as a whole and to facilitate smooth processing.

[0350] As in the above examples, the sensor device 1 executes adaptive processing corresponding to the state of the device, and thus executes advantageous processing for the sensor device 1 itself and the device 100 as a whole.

[0351] Specific examples of device information are listed below.

[0352] For example, the following information exists as power information that is one of the device information.

[0353] • Battery remaining capacity

[0354] • Battery voltage

[0355] • Battery temperature

[0356] For example, the following information exists as communication state information that is one of the device information.

[0357] • Connection method

[0358] a. Gateway / LAN / WAN

[0359] b. WiFi (registered trademark)

[0360] c. Bluetooth (registered trademark) / beacon

[0361] d. Cellular (registered trademark)

[0362] e. Satellite

[0363] f. Ethernet (registered trademark)

[0364] g. Low Power Wide Area (LPWA) communication

[0365] • Effective throughput

[0366] • Whether connection is possible

[0367] As the hardware / application information as one of the device information, first, for example, there is the following information as the information related to the hardware.

[0368] • Presence or absence of MCU

[0369] • MCU performance (floating point operations per second: FROPS)

[0370] • Temperature in the equipment

[0371] • Other sensor information

[0372] • Memory size

[0373] • Device type

[0374] a. Drone

[0375] b. Robot

[0376] c. Camera panoramic filter

[0377] d. Wearable device

[0378] e. Other

[0379] • Device name, device ID

[0380] • Difference between fixed device and mobile device

[0381] • Difference between active state and standby state

[0382] For example, in the hardware / application information, there is the following information as the information on the application.

[0383] Target class corresponding to the application

[0384] • Metadata expected to be received from the sensor device 1 (attribute feature amount, individual feature amount, and trend data, etc.)

[0385] A detailed example of the metadata is as follows.

[0386] First, the attribute feature amount and the individual feature amount include data of the feature amount of the attribute and the individual for each ID number.

[0387] The ID number is automatically numbered by the sensor device 1 for each target individual designated by the ID class. Note that the ID number is a code written in numbers, alphabets, and symbols, by which the type and the individual can be identified.

[0388] The attribute feature amount of each ID is data indicating the target attribute of the type and is estimated by the sensor device 1.

[0389] For example, the following example is assumed as the type name and the attribute feature amount.

[0390] • height, gender, age, etc. as attribute features for the type name "person"

[0391] • color, type such as normal car / large car / bus, etc. as attribute features for the type name "car"

[0392] • type of component as attribute features for the type name "component"

[0393] The individual feature quantity of each ID is data that encodes a feature of an individual capable of identifying the ID, and is calculated by the sensor device 1.

[0394] For example, the following examples are assumed as the type name and individual feature quantity.

[0395] • as individual feature quantity for the type name "person", a feature quantity in which a feature such as an appearance feature (clothes feature, presence or absence of glasses) or a walking style of movement of a person is encoded.

[0396] • as individual feature quantity for the type name "car", number of cars, type, number of passengers, or appearance feature quantity

[0397] • as individual feature quantity for the type name "part" printed on a part, number, bar code, etc.

[0398] The stream data as metadata includes the ID number each time (interval can be freely set) and the position (coordinate information) of the ID in the screen.

[0399] Next, the output of the sensor operation information by the sensor device 1 will be described.

[0400] For example, information such as a confidence rate, a class, and an object region of object detection, and information such as a frame rate, a processing parameter, an image quality parameter, and a resolution are sequentially sent to the processor 11. In this case, the following examples are conceivable.

[0401] For example, in the case of a movable device 100 such as a self-propelled robot 101 or a flying vehicle 102, the sensor device 1 sends sensor operation information indicating a case where an object to be detected is small in the image and confidence rate cannot be increased to the processor 11. Specifically, the arithmetic operation unit 8 detects a case where a value of confidence and information of an object region of a target class (number of pixels of a target, etc.) of a target are far away by the function of the operation information generation unit 87, and generates and sends sensor operation information indicating the above case. Alternatively, the value of confidence and the information of the object region of the target class can be sent as sensor operation information, and the processor 11 can determine the case.

[0402] In response thereto, the processor 11 can control the device operation so as to, for example, approach the target object. As a result, the confidence of the target object is improved.

[0403] Further, the arithmetic operation unit 8 transmits sensor operation information indicating a case where the brightness of the image is low and the confidence rate of the target object does not increase to the processor 11. Specifically, the arithmetic operation unit 8 detects a situation where the detection accuracy is lowered due to darkness from the value of the average brightness of the captured image and the confidence, and generates and transmits sensor operation information indicating the situation. Alternatively, the value of the confidence rate, the average brightness value, and the like can be transmitted as the sensor operation information, and the processor 11 can determine the situation.

[0404] In response thereto, the processor 11 controls the device operation so as to, for example, turn on the illumination. As a result, the confidence of the target object is improved.

[0405] Further, the arithmetic operation unit 8 transmits sensor operation information indicating a case where the target object is not in the image to the processor 11. Specifically, the arithmetic operation unit 8 generates and transmits sensor operation information indicating the situation from the fact that the state of the object of the object class continues to be unable to be detected, and the like. Alternatively, information indicating that the target class cannot be detected can be continuously transmitted as the sensor operation information, and the processor 11 can determine the situation.

[0406] For example, in a case where the device 100 is the monitoring camera 104, the processor 11 controls the device operation so as to, for example, pan or tilt the imaging direction in response thereto. As a result, the target object can be detected. In other words, an appropriate operation can be adaptively performed as a monitoring operation.

[0407] Further, the arithmetic operation unit 8 transmits sensor operation information indicating a case where the target object moves to the left in the image and disappears from the screen to the processor 11. Specifically, the arithmetic operation unit 8 detects the moving direction from the object region of the object of the target class at each time, and generates and transmits sensor operation information indicating the state of the fact that the object cannot be detected, and the like. Alternatively, information on the object region at each time and information indicating that the target class cannot be detected can be continuously transmitted as the sensor operation information, and the processor 11 can determine the situation.

[0408] For example, in a case where the device 100 is the monitoring camera 104, the processor 11 controls the device operation so as to, for example, pan the imaging direction to the left in response thereto. As a result, the target object can be imaged and detected again. It can be said that an appropriate tracking operation can be performed as a monitoring operation.

[0409] As in the above example, the sensor device 1 transmits the sensor operation information to the processor 11 to obtain a state suitable for sensing of the sensor device 1 or to obtain a favorable result of the device 100 as a whole.

[0410] Specific examples of the sensor operation information are as follows.

[0411] Object detection information: for example, confidence rate, class, object area, etc.

[0412] • Frame rate (frames per second: FPS)

[0413] • Image quality parameter

[0414] Incidentally, in order for the sensor device 1 to perform the processing corresponding to the device information as described above and transmit the sensor operation information and perform the processing corresponding to the condition of the device 100, it is necessary to register the information of the device 100 installed in advance and the conditions for determining the operation switching.

[0415] Therefore, in the sensor device 1, the hardware / application information and the switching condition information are registered in, for example, the memory 6 or the like, for the switching determination unit 86 and the operation information generation unit 87 that mainly process the arithmetic operation unit 8.

[0416] Although an example of the adjustment of the hardware / application information has been described above, when the hardware / application information is registered, the operation information generation unit 87 of the sensor device 1 can generate necessary or appropriate sensor operation information according to the device 100 and transmit the generated sensor operation information to, for example, the processor 11.

[0417] In addition, when the switching condition information is registered, the switching determination unit 86 can appropriately perform the operation switching control of the sensor device 1 according to the device information.

[0418] Examples of the switching condition information will be described. The state of the operation setting of each of the following "first operation setting" to "eleventh operation setting" will be described.

[0419] • Regarding the battery remaining capacity

[0420] Less than 20%: First operation setting

[0421] 20% or more and less than 90%: Second operation setting

[0422] 90% or more: Third operation setting

[0423] • Regarding the battery voltage

[0424] Less than 3.2V: Fourth operation setting

[0425] 3.2V or more: Fifth operation setting

[0426] • regarding battery temperature

[0427] less than 10°C: sixth operation setting

[0428] 10°C or more: seventh operation setting

[0429] • communication method

[0430] in the case of WiFi: eighth operation setting

[0431] in the case of Bluetooth: ninth operation setting

[0432] • regarding effective throughput

[0433] less than 800 Mbps: tenth operation setting

[0434] 800 Mbps or more: eleventh operation setting

[0435] For example, the switching conditions of the operation settings are recorded in this way, and therefore the sensor device 1 can independently switch the operation settings in accordance with the type of the device 100, the operation purpose, and the like.

[0436] <4. Processing example>

[0437] Hereinafter, a processing example of the arithmetic operation unit 8 of the sensor device 1 will be described, specifically, a processing example based on the functions of the switching determination unit 86 and the operation information generation unit 87.

[0438] [4-1: First processing example]

[0439] Figure 16 A flowchart of the processing of the arithmetic operation unit 8 as the first processing example is shown. This is a processing example in which the arithmetic operation unit 8 performs processing in accordance with the functions of the switching determination unit 86 to be input by the device 100.

[0440] In step S101, the arithmetic operation unit 8 confirms whether or not the necessary information is registered. The necessary information referred to here is, for example, the registration information described above, and is, for example, hardware / application information and switching condition information.

[0441] When the necessary information has not been registered, the arithmetic operation unit 8 proceeds to step S102 and performs information registration processing. For example, the arithmetic operation unit 8 communicates with the processor 11 to obtain and register the necessary information. Regarding the switching condition information, for example, a default setting value can be registered.

[0442] Note that the switching condition information can be updated in accordance with the operation of the user, the time determination of the arithmetic operation unit 8, and the like.

[0443] When the information is registered, the processing of step S103 and the subsequent processing are executed.

[0444] In step S103, the arithmetic operation unit 8 determines whether it is the operation termination timing of the sensor device 1. For example, in the case where the power of the device 100 itself is turned off, or in the case where an instruction for turning off the operation of the sensor device 1 is given from the processor 11 of the device 100 for some reason, the processing is terminated in step S108.

[0445] During the operation continuation period, the arithmetic operation unit 8 proceeds to step S104, and determines whether it is the timing for confirming the device information. For example, this is the device information that is periodically obtained from the processor 11.

[0446] Note that even when the processor 11 is not particularly synchronized with the arithmetic operation unit 8, the processor 11 can transmit the device information at an asynchronous timing and store the device information in a predetermined area of the memory 6.

[0447] When it is not the confirmation timing, the arithmetic operation unit 8 returns to step S103.

[0448] The arithmetic operation unit 8 proceeds to step S105 to confirm the memory 6 and can obtain the device information at each confirmation timing.

[0449] Note that the device information mentioned here is information that fluctuates sequentially, such as power information or communication state information. Although the hardware / application information is mainly registered information that is obtained in step S122, the hardware / application information can be included in the information obtained in step S105, as long as there is information that fluctuates sequentially as hardware and application information.

[0450] Note that a processing mode can be employed in which the arithmetic operation unit 8 requests the processor 11 to transmit the device information when it is the confirmation timing, and the processor 11 transmits the device information accordingly.

[0451] In the case where the device information has been obtained, the arithmetic operation unit 8 determines whether the device information corresponding to the switching condition has changed in step S106.

[0452] For example, as an example of the above-described battery remaining capacity, in the case where the last battery remaining capacity was 91% and the present battery remaining capacity is 88%, it is judged that there is a change corresponding to the switching condition.

[0453] When the change corresponding to the switching condition has not been made, the arithmetic operation unit 8 returns to step S103.

[0454] In the case where the change corresponding to the switching condition has been made, the arithmetic operation unit 8 proceeds to step S107 and executes control for switching the operation setting.

[0455] For example, adaptive control such as reduction of the threshold value corresponding to the confidence rate or reduction of the frame rate is executed.

[0456] [4-2: Second processing example]

[0457] As Figure 17 As a second processing example in the above-described example, the processing of the arithmetic operation unit 8 and the processing of the processor 11 are described. This is an example in which the arithmetic operation unit 8 generates sensor operation information using the function output of the operation information generation unit 87 and the processor 11 controls the device operation accordingly.

[0458] In step S121, the arithmetic unit 8 confirms whether or not the required information is registered. In this case, the necessary information is, for example, the hardware / application information in the above-described registration information.

[0459] In the case where the necessary information is not registered, the arithmetic processing unit 8 proceeds to step S122, and executes information registration processing. For example, the arithmetic operation unit 8 communicates with the processor 11 to acquire and register the information required for the hardware and the application of the device 100.

[0460] When the information is registered, the processing of step S123 and the subsequent processing are performed.

[0461] In step S123, the arithmetic operation unit 8 determines whether or not it is the timing to output the sensor operation information. For example, when the sensor operation information is output periodically, the timing is waited for.

[0462] If it is the output timing, the arithmetic processing unit 8 proceeds to step S124, generates the sensor operation information, and outputs the generated sensor operation information to the processor 11.

[0463] In step S125, the arithmetic operation unit 8 determines whether or not it is the operation termination timing of the sensor device 1. For example, in the case where the power of the device 100 itself is turned off, or in the case where an instruction for turning off the operation of the sensor device 1 is given from the processor 11 of the device 100 for some reason, the processing is terminated in step S126.

[0464] During the period in which the operation is continued, the arithmetic operation unit 8 returns to step S123. Thereby, for example, the sensor operation information is transmitted to the processor 11 periodically.

[0465] The processor 11 waits for reception of the sensor operation information in step S201.

[0466] In addition, generation of the operation termination trigger is monitored in step S204. In the case where the trigger for turning off the power has been generated, the processing for turning off the power is executed in step S205 to terminate the operation.

[0467] During the operation cycle, each time sensor operation information is received, the processor 11 proceeds to step S202, and determines whether sensor operation information corresponding to an operation change condition of the device 100 has been recognized. For example, the processor 11 determines whether a request for movement, translation, or the like has been made, whether an operation state of the sensor device 1 in which movement, translation, or the like is estimated to be required has been observed as sensor operation information, and determines whether the operation needs to be changed.

[0468] Further, in the case of determining whether the operation needs to be changed, the processor 11 proceeds to step S203 to perform operation control of the device 100. For example, the self-propelled actuator is controlled to perform a predetermined movement or to perform translation / tilt.

[0469] [4-3: Third Processing Example]

[0470] Also in the third processing example of Figure 18 , the processing of the arithmetic operation unit 8 and the processing of the processor 11 are described. This is an example in which the arithmetic operation unit 8 performs switching control using the function of the switching determination unit 86 and outputs sensor operation information using the function of the operation information generation unit 87, and the processor 11 controls the device operation accordingly. That is, the third processing example is a processing example that combines the first and second processing examples. The same processing as in Figure 16 and Figure 17 is indicated by the same step number. In addition, assuming that the registration information has been registered in the sensor device 1, the subsequent processing is described.

[0471] The arithmetic operation unit 8 determines whether it is a timing to output sensor operation information in step S123.

[0472] In the case of not being the output time, the arithmetic operation unit 8 proceeds to step S104.

[0473] If it is the output timing, the arithmetic operation unit 8 proceeds to step S124, generates sensor operation information, and outputs the generated sensor operation information to the processor 11. Then, the arithmetic operation unit 8 proceeds to step S104.

[0474] In step S104, the arithmetic operation unit 8 determines whether it is a timing to confirm device information. When it is not the confirmation timing, the arithmetic operation unit 8 proceeds to step S130.

[0475] The arithmetic operation unit 8 proceeds to step S105 to obtain device information from step S104 at each confirmation timing.

[0476] When the operation processing unit 8 has obtained the device information, in step S106, the operation processing unit 8 determines whether the device information corresponding to the switching condition has changed.

[0477] When the device information corresponding to the switching condition has not changed, the operation processing unit 8 proceeds to step S130.

[0478] When the device information corresponding to the switching condition has changed, the operation processing unit 8 proceeds to step S107 and executes control for switching the operation setting.

[0479] In step S130, the operation processing unit 8 determines whether it is the operation termination timing of the sensor device 1. When it is the operation termination timing, the processing is terminated in step S131.

[0480] During the period in which the operation is continued, the operation processing unit 8 returns to step S123.

[0481] The processor 11 waits for reception of the sensor operation information in step S201. When no sensor operation information is received, the processor 11 proceeds to step S210.

[0482] Each time sensor operation information is received, the processor 11 proceeds to step S202 and determines whether sensor operation information corresponding to the operation change condition of the device 100 has been recognized.

[0483] When it is determined that the operation does not need to be changed, the processor 11 proceeds to step S210.

[0484] On the other hand, when it is determined that the operation needs to be changed, the processor 11 proceeds to step S203 and executes operation control of the device 100. For example, the self-propelled actuator is controlled to perform a predetermined movement or to perform panning / tilting. Then, the processor 11 proceeds to step S210.

[0485] In step S210, the processor 11 confirms whether it is the timing to output the device information. When it is not the timing to output, the processor 11 proceeds to step S204.

[0486] When it is the timing to output, the processor 11 proceeds to step S211 and outputs the device information. Specifically, the processor 11 transmits information that fluctuates in sequence, such as power information or communication state information.

[0487] In step S204, the processor 11 monitors generation of an operation termination trigger. When a trigger for turning off the power has been generated, processing for turning off the power is executed in step S205 to terminate the operation.

[0488] As in the above first, second, and third processing examples, the sensor device 1 and the processor 11 transmit and receive the device information and the sensor operation information, respectively, to thereby perform the adaptive processing.

[0489] <5. Another configuration example of the sensor device>

[0490] The configuration example of the sensor device 1 is not limited to Figure 3 and other examples can be conceived.

[0491] Figure 19 A configuration example in which the arithmetic operation unit 8 is provided separately from the sensor device 1 in the device 100 is shown. The arithmetic operation unit 8 is provided in the device 100 as a chip separate from the sensor device 1, and can communicate with the sensor device 1 and the processor 11 through the interface unit 7.

[0492] Further, the arithmetic operation unit 8 is provided with the functions of the switching determination unit 86 and the operation information generation unit 87, and thus can perform the same processing as in Figure 1 the case.

[0493] Figure 20 In the configuration example in

[0494] Although not shown in the drawing, in a configuration in which the sensor device 1 and the arithmetic operation unit 8 are provided separately as in Figure 19 , the switching determination unit 86 and the operation information generation unit 87 can be constituted by a separate processor or the like.

[0495] Further, it is also conceivable to arrange the key frame selection unit 81, the object region recognition unit 82, the category recognition unit 83, the parameter selection unit 84, the threshold setting unit 85, and the like outside the sensor device 1 or outside the algorithm operation unit 8.

[0496] Further, the arithmetic operation unit 8 does not need to have all the functions shown in the drawing. For example, some of the key frame selection unit 81, the object region recognition unit 82, the category recognition unit 83, the parameter selection unit 84, and the threshold setting unit 85 can not be provided. In addition, only one of the switching determination unit 86 and the operation information generation unit 87 can also be provided.

[0497] <6. Conclusion and examples applied to each processing>

[0498] In the above embodiments, the following effects are obtained.

[0499] The sensor device 1 of the present embodiment includes an array sensor 2, a signal processing unit 30, and an arithmetic operation unit 8 (switching determination unit 86). In the array sensor 2, a plurality of detection elements are arranged one-dimensionally or two-dimensionally. The signal processing unit 30 performs signal processing on a detection signal obtained by the array sensor 2. The arithmetic operation unit 8 performs object detection based on the detection signal obtained by the array sensor 2, performs operation control of the signal processing unit based on the object detection, and performs switching processing for changing the processing content based on device information input from a sensor device 100 in which the sensor device 1 itself is installed.

[0500] The processing in the signal processing unit 30 is controlled by the arithmetic operation unit 8 based on the object detection, and thus a signal processing operation suitable for the object to be detected is performed. For example, parameter setting corresponding to the object to be detected, designation of a sampling region, and efficient processing control having a high power saving effect according to a frame rate or the like are performed.

[0501] Then, switching of the operation setting is performed according to the device information of the device 100, so that a processing operation that is appropriate for the condition of the device 100 can be performed and switching is performed according to the condition such as a power saving priority or a reliability priority. Thereby, the sensor device 1 can be autonomously optimized as the device-installed sensor device 1.

[0502] In the embodiment, a description has been given of an example in which the arithmetic operation unit 8 (operation information generation unit 87) performs processing for generating sensor operation information based on the object detection or a processing operation of the signal processing unit 30 and transmits the generated sensor operation information to the device 100 (for example, the processor 11).

[0503] Information and the like on the processing of the signal processing unit 30 and the object detection in the arithmetic operation unit 8 are transmitted to the device 100, and thus the device 100 can perform a state appropriate for the operation of the sensor device 1. For example, a state appropriate for sensing of the sensor device 1 is obtained by approaching an imaging target or changing an imaging direction. Thereby, the sensor device 1a also autonomously performs optimization associated with the device 100 on which the sensor device 1 is installed.

[0504] Further, the processor 11 can also reduce the calculation cost of processing by using metadata output by the object detection function of the sensor device 1. Therefore, it is useful for reducing power consumption and data amount. Further, the application (processor 11) can perform optimal operation by using the metadata output by the sensor device 1, and also can improve the performance and efficiency of the entire movement of the device 100 equipped with the sensor.

[0505] In the embodiment, a description has been given of an example in which the arithmetic operation unit 8 (switching determination unit 86) executes the switching process based on the registration information (hardware / application information) registered in advance on the device 100 or the registration information (switching condition information) as the operation switching condition.

[0506] The appropriate processing operation of the sensor device 1 varies depending on the type, processing capability, operation function, and the like of the device 100 (the processor 11), the application of the operation function, and the like. In addition, the condition for switching the operation state also varies. As a result, the necessary information is registered in advance, and the switching process is executed accordingly. Thus, the sensor device 1 can be generally used for various types of devices 100, and can be optimized to be suitable for the device 100.

[0507] In the embodiment, a description has been given of an example in which the arithmetic operation unit 8 obtains the power supply information as the device information.

[0508] By obtaining the power supply information of the device 100 as the device information, the sensor device 1 can execute the switching process for executing the processing corresponding to the power supply state. Thus, the power saving of the device 100, the increase in the operation time, and the like can be achieved.

[0509] In the embodiment, a description has been given of an example in which the arithmetic operation unit 8 obtains the communication state information as the device information.

[0510] By obtaining the communication state information as the device information, the sensor device 1 can execute the switching process for executing the processing corresponding to the communication condition. For example, the unnecessary energy consumption can be reduced by reducing the frame rate due to the deterioration of the communication condition, or the packet loss and the like can be prevented from occurring in the transmission from the device 100 to the external device.

[0511] In the embodiment, a description has been given of an example in which the arithmetic operation unit 8 obtains the information on the hardware or the application as the device information.

[0512] By obtaining the information on the hardware of the device 100, the sensor device 1 can execute the processing corresponding to the configuration and the function of the device 100. In addition, the metadata output according to the application can be set by obtaining the information on the application. Thus, the processing of the signal processing unit can be appropriately adapted and the switching process can be executed according to the hardware configuration and the application.

[0513] In the embodiment, a description has been given of an example in which the arithmetic operation unit 8 executes the processing for executing the class recognition of the detection object from the detection signal obtained by the array sensor 2 and selecting the parameter for the signal processing of the signal processing unit 30 based on the recognized class as the processing related to the switching process.

[0514] That is, although the logic unit 5 performs image processing on the image signal obtained by the array sensor 2 as a process of "classification image adaptation", the parameter of the image processing is set based on the class recognition of the object detected in the image signal.

[0515] In the case where object detection from an image is performed, an image with high quality seen by a person is not necessarily an image with high recognition accuracy. Further, the desired image quality varies according to the class of the object to be recognized. That is, an image on which image processing is performed by ordinary parameter setting to achieve high image quality at the time of visual recognition does not necessarily have an image quality suitable for object detection. In addition, the desired image processing parameter varies according to the class of the object to be recognized.

[0516] As a result, a parameter set is held in advance for each class, and the parameter set to be used is selected according to the class recognition of the object detected in the captured image. Thereby, image processing suitable for detecting the target object is performed. According to the image having undergone such image processing, an improvement in object detection accuracy can be achieved.

[0517] Further, the desired image quality adjustment for object detection is different from the image quality adjustment for making a person feel beautiful, and therefore, for example, a blur filter for prioritizing beauty is not used. For this reason, the parameter set often results in a low processing load.

[0518] Further, the data amount is generally reduced according to the parameters corresponding to the class (for example, parameters related to gradation variation and compression). In this case, it is also possible to avoid an increase in processing time lag and power consumption of the entire system due to high load on the processor 11 side.

[0519] In addition, as the switching processing based on the device information, it is assumed that the parameter set corresponding to the class is switched. For example, a plurality of parameter sets corresponding to the class "person" are prepared according to the device information, and therefore, the parameter set corresponding to the class is also switched in the signal processing unit 30 according to the state of the device 100.

[0520] In addition, as the switching processing based on the device information, it is possible to switch whether or not classification image adaptation is performed.

[0521] Thereby, it is possible to optimize the operation of the sensor device 1 according to the device information.

[0522] In addition, it is possible to output the information of the class in the object detection, the confidence rate, the information of the object region, the information on whether or not the target class has been detected, and the like to the device 100.

[0523] In the embodiment, the description has been given of the example in which the arithmetic operation unit 8 sets a threshold value to all or some of the parameters for the signal processing of the signal processing unit 30 or the detection processing of the array sensor 2 based on the device information to perform the processing using the parameters set based on the threshold value as the description of the processing related to the switching processing.

[0524] That is, as described in "Image adaptation by threshold setting", the arithmetic operation unit 8 includes a threshold setting unit 85 that sets a threshold value to all or some of the parameters of the parameters used for the image processing of the logic unit 5 or the imaging processing related to the imaging of the array sensor 2 and performs the processing using the parameters set based on the threshold value.

[0525] For example, by using the threshold setting (changing) of the parameters, it is possible to output the image signal with the minimum quality (for example, the minimum resolution required) required for the processing such as the object detection. Therefore, it is also possible to reduce the amount of data in the image signal to be output while not reducing the performance, accuracy, and the like of the subsequent processing (object detection and the like).

[0526] Further, by setting the threshold value according to the device information, it is possible to realize the reduction of the power consumption according to the state of the device (for example, the battery state and the like), the increase of the processing speed of the device 100 as needed, and the like.

[0527] In the embodiment, the description has been given of the example in which the arithmetic operation unit 8 performs the object detection from the detection signal obtained by the array sensor 2 and performs the processing of giving the instruction of the region information generated based on the object detection to the signal processing unit 30 as the signal processing of the region information or the detection signal related to the obtaining of the detection signal obtained by the array sensor 2 as the description of the processing related to the switching processing.

[0528] As described in "Region clipping", the arithmetic operation unit 8 of the sensor device 1 performs the object detection from the detection signal obtained by the array sensor 2 and gives the instruction of the region information (ROI 21 and AROI 22) generated based on the object detection to the signal processing unit 30 as the region information related to the obtaining of the detection signal obtained by the array sensor 2 or the signal processing of the detection signal.

[0529] That is, the signal processing unit 30 performs the signal processing of the detection signal obtained by the array sensor 2 and outputs the processed signal from the interface unit 7, but the region information related to the obtaining of the detection signal obtained by the array sensor 2 or the signal processing of the detection signal in the signal processing unit 30 is set based on the object detection.

[0530] In a case where subject detection from an image is performed as in the embodiment, information on all pixels in each frame is not always required. For example, in a case where a person is detected, only detection information of a region in which the person appears in the frame is required. Therefore, the arithmetic operation unit 8 generates the ROI 21 and the AROI 22 based on the object detection, and performs the processing of the signal processing unit 30 using the ROI 21 and the AROI 22, that is, the detection signal and the compression processing in the logic unit 5 are obtained from the array sensor 2 by the ADC / pixel selector 3.

[0531] Therefore, reduction of the amount of data of the processing target and improvement of the processing speed can be realized, and an image signal with which the detection accuracy is not reduced can be obtained.

[0532] Then, whether or not such region clipping is performed can be switched according to the device information.

[0533] Further, it is also conceivable to switch the setting range of the ROI 21 according to the device information while performing the region clipping.

[0534] Further, it is also conceivable to switch the size and the resolution of the key frame to be subjected to the full-screen scanning according to the device information.

[0535] Thereby, the operation of the sensor device 1 can be optimized according to the device information.

[0536] Note that the present application is not limited to an image signal, and object detection is also performed on a detection signal obtained by the array sensor 2 as a sound wave detection signal, a touch detection signal, or the like, and the signal processing unit 30 can also receive an instruction of region information generated based on the object detection as region information on the acquisition of the detection signal from the array sensor 2 or the signal processing of the detection signal.

[0537] Therefore, also in a case where a sound wave sensor array and a touch sensor array are used, reduction of the amount of data of the processing target, improvement of the processing speed, and the like can be realized adaptively according to the device information, and the effect of obtaining a detection signal with which the detection accuracy is not reduced can be obtained.

[0538] In the embodiment, a description has been given of an example in which the arithmetic operation unit 8 performs the following processing: detecting an object from a detection signal obtained by the array sensor 2, giving an instruction of region information generated based on the object detection to the signal processing unit 30 as region information on the acquisition of the detection signal from the array sensor 2 or the signal processing of the detection signal, performing class recognition on the object detected from the detection signal obtained by the array sensor 2, and generating region information corresponding to the object using a template corresponding to the recognized class, as the processing related to the switching processing.

[0539] As described in "Region clipping using AROI", the arithmetic operation unit 8 performs class recognition on the object detected from the detection signal obtained by the array sensor 2, and generates region information (AROI 22) corresponding to the object using a template corresponding to the recognized class.

[0540] By using the template corresponding to the class, an AROI 22 that is different for each class and suitable for an important region can be generated.

[0541] In particular, in the case where the array sensor 2 is constituted by an imaging element, power consumption in photoelectric conversion is the largest. In this case, it is desirable to reduce the number of pixels to be subjected to photoelectric conversion as much as possible. By reducing the pixels to be subjected to photoelectric conversion according to the template, the amount of data can be effectively reduced without affecting the detection accuracy. Specifically, it is important that the image is an image in which the processor 11 can accurately recognize the object, rather than an image that is not visible to people but feels beautiful. The image in which the pixels subjected to photoelectric conversion and converted to digital data using the template is suitable for efficient object detection with a small amount of data.

[0542] In addition, the template indicates a region in which a detection signal is obtained for each class.

[0543] For example, the template indicates detection elements (see FIG. 6) in which detection elements of the array sensor should obtain their detection information according to the class such as "person" and "car". Figure 7 and Figure 8

[0544] By using the template for specifying the pixels corresponding to the class to be read, appropriate information can be read from the array sensor 2 for each class. Specifically, as shown in the examples of Figure 7 and Figure 8 , a part (a part of a face or a part of a number plate) is made dense, and thus information on a particularly necessary part can also be obtained densely for each class.

[0545] According to the device information, it is possible to switch whether to perform region clipping using this AROI 22.

[0546] Alternatively, it is also conceivable to prepare a plurality of templates for each class, and select a template for a certain class according to the device information.

[0547] Thereby, it is possible to optimize the operation of the sensor device 1 according to the device information.

[0548] ​In the embodiment, a description has been given of an example in which the arithmetic operation unit 8 performs processing for setting a threshold value for all or some of the parameters used for signal processing of the signal processing unit 30 or detection processing of the array sensor 2, and setting a parameter for processing of the region information indicated by the template based on the threshold value, as processing related to switching processing.

[0549] As described in "Region cropping using threshold setting and AROI", the arithmetic operation unit 8 performs class recognition on an object detected from a detection signal obtained by the array sensor 2, and generates region information (AROI 22) corresponding to the object using a template corresponding to the recognized class. In this case, the AROI 22 in which a parameter such as resolution calculated based on a threshold value is recorded is used.

[0550] By setting (changing) the parameter of the obtained region indicated by the AROI 22 using the threshold value, it is possible to output an image signal at a minimum required quality (for example, a minimum required resolution) for processing such as object detection, for example.

[0551] Further, the image of the pixels subjected to photoelectric conversion and converted to digital data using the template is suitable for efficient object detection with a small amount of data.

[0552] Therefore, by using the template and setting a parameter such as resolution using a threshold value, it is also possible to reduce the amount of data in the image signal to be output while not reducing the performance, accuracy, or the like of subsequent processing (object detection, or the like). Therefore, it is also possible to achieve a reduction in power consumption and an increase in processing speed.

[0553] Further, the template indicates a region in which a detection signal is obtained for each class (such as "person" or "car"), and therefore it is also possible to concentrate on obtaining information on a particularly necessary part for each class.

[0554] According to the device information, it is possible to switch whether to perform region cropping using the threshold setting and the AROI 22.

[0555] By this, it is possible to optimize the operation of the sensor device 1 according to the device information.

[0556] In the embodiment, a description has been given of an example in which the arithmetic operation unit 8 performs object detection from a detection signal obtained by the array sensor 2, gives the signal processing unit 30 the instruction of region information generated based on the detection of the object as region information obtained with respect to the detection signal obtained by the array sensor 2 or as signal processing of the detection signal related to switching processing, and the signal processing unit 30 performs compression processing using a different compression rate for each region on the detection signal obtained by the array sensor 2 based on the region information received from the arithmetic operation unit 8.

[0557] That is, as described in "intelligent compression", in the signal processing unit 30, the logic unit 5 performs compression processing on the detection signal obtained by the array sensor 2, but the logic unit 5 performs compression processing using a compression rate different for each region based on the region information received from the arithmetic operation unit 8 (see Figure 10 ).

[0558] Therefore, the signal processing unit 30 (logic unit 5) can perform data compression so as to not reduce important information by making the compression ratio different between important regions and less important regions in a frame.

[0559] For example, the logic unit 5 performs compression processing at a low compression rate in the region designated by the region information, and performs compression processing at a high compression rate in other regions.

[0560] The signal processing unit 30 (logic unit 5) performs compression processing at a low compression rate in the region designated by the ROI 21, and from the next frame in which the object is detected, reduces the data amount at a high compression ratio in other regions. Because the ROI 21 is generated from object detection, the region indicated by the ROI 21 is an important region for object detection in the processor 11 and has a low compression ratio so that information is not reduced. Thereby, the detection accuracy does not decrease. On the other hand, regions other than the region indicated by the ROI 21 are regions that do not greatly affect object detection, and therefore compression can be performed at a high compression rate and the data amount can be efficiently reduced.

[0561] Then, whether or not to perform compression processing such as this intelligent compression can be switched according to the device information.

[0562] Alternatively, even when intelligent compression is always performed, switching processing can be performed so as to change the compression rate according to the device information.

[0563] Thereby, the operation of the sensor device 1 can be optimized according to the device information.

[0564] In the embodiment, a description has been given of the example in which the arithmetic operation unit 8 performs processing to set an effective region for the detection signal obtained from the array sensor 2 based on information about past region information, perform object detection from the detection signal of the effective region, and give an instruction of the region information generated based on the detection of the object to the signal processing unit 30 as the region information or the detection signal regarding the detection signal obtained from the array sensor 2 as signal processing, as processing related to switching processing.

[0565] As described in "effective region clipping", the arithmetic operation unit 8 sets an effective region RA for the detection signal obtained from the array sensor 2 based on information related to past region information (the bounding box 20 of the object detection region as a source of region information, the ROI 21 and the AROI 22 as region information itself).

[0566] Further, object detection is performed from the detection signal of the active region RA, and the signal processing unit 30 is given an instruction for the ROI 21 and the AROI 22 generated based on the detection of the object as region information on the signal processing of the detection signal obtained from the array sensor 2 or the detection signal.

[0567] Thereby, the processing load of the object detection for setting the ROI 21 and the AROI 22 is significantly reduced. Therefore, an effect of reducing the processing load, increasing the speed, and reducing the power consumption can be obtained.

[0568] Then, whether to perform such effective region clipping can be switched according to the device information.

[0569] Thereby, the operation of the sensor device 1 can be optimized according to the device information.

[0570] In the present embodiment, a description has been given as an example of processing involving switching processing in which the arithmetic operation unit 8 performs processing for performing object detection from the detection signal obtained by the array sensor 2 and giving an instruction for making the frame rate of the detection signal obtained by the array sensor 2 variable based on the detection of the object.

[0571] As described in "active sampling", the arithmetic operation unit 8 performs object detection from the detection signal obtained by the array sensor 2 and gives an instruction for making the frame rate of the detection signal obtained by the array sensor 2 variable based on the detection of the object.

[0572] In the case where object detection from an image is performed, it is not always necessary to have an image signal with a high frame rate. For example, in the case where a person is detected, whether the frame rate in a frame where there is no person is low or not is irrelevant. In contrast, the frame rate is increased during a period when a person appears, and thus the amount of information becomes rich, and it is also possible to increase the number of detected objects (persons) and information that can be recognized in association with object detection.

[0573] That is, by changing the frame rate according to the detection of the object, it is possible to adaptively increase the amount of data when necessary, reduce the amount of data when not necessary, and reduce the amount of data of processing and the amount of transfer without reducing the performance of object detection.

[0574] Then, whether to perform such active sampling can be switched according to the device information.

[0575] Optionally, in a case where active sampling is performed, the sampling rate can be switched between the normal mode and the idle mode according to the device information.

[0576] Thereby, the operation of the sensor device 1 can be optimized according to the device information.

[0577] Note that the present application is not limited to image signals, and also performs object detection on detection signals obtained by the array sensor 2 as sound wave detection signals, touch detection signals, and the like, and can give an instruction for making the frame rate of the detection signals obtained by the array sensor 2 variable based on the detection of the object. Thus, even in a case where a sound wave sensor array and a touch sensor array are used, the following effects are obtained: the data amount can be adaptively increased when needed, the data amount can be reduced when not needed, and the amount of data processed and transmitted is reduced without reducing the object detection performance.

[0578] A frame is an image frame in a case where the array sensor 2 is an imaging element array, but has the same meaning in a case of a sound detection element or a touch sensor element. A frame is a unit of data read from a plurality of detection elements of the array sensor 2 in one read cycle, regardless of the type of the array sensor 2. The frame rate is the density of such frames in a unit time.

[0579] In the embodiment, a description has been given of the following example in which the arithmetic operation unit 8 performs, as the processing related to the switching processing, processing of setting a threshold value of the frame rate according to the category identified for the object detected from the detection signals obtained by the array sensor 2 by using the frame rate based on the threshold value setting.

[0580] As described in “Active sampling by threshold value setting”, the threshold value of the frame rate is set according to the category identified for the object detected from the detection signals obtained by the array sensor 2, and processing using the frame rate based on the threshold value setting is performed (see Figure 14 and Figure 15 ).

[0581] The frame rate suitable for the category of the detection target can be applied by setting (changing) the frame rate using the threshold value. Specifically, the reduction of the data amount of the image signals, the reduction of the power consumption, and the increase of the processing speed can be realized by reducing the frame rate without degrading the object detection performance of one category of the detection target.

[0582] Then, whether such active sampling is performed or not can be switched by the threshold value setting according to the device information.

[0583] Optionally, in a case where active sampling is performed by the threshold value setting, the sampling rate can be switched according to the device information.

[0584] Thereby, the operation of the sensor device 1 can be optimized in accordance with the device information.

[0585] The technology of the present disclosure is not limited to the configuration examples in the embodiments, and various modifications are assumed.

[0586] The configuration of the sensor device 1 is not limited to those shown in Figure 3 , Figure 19 and Figure 20 .

[0587] The array sensor 2 is not limited to pixels that receive visible light, and can be provided with a plurality of imaging elements for non-visible light.

[0588] Note that the advantageous effects described in this specification are merely illustrative and not limiting, and other advantageous effects can be obtained.

[0589] Note that the present technology can also employ the following configurations.

[0590] (1) A sensor device comprising:

[0591] an array sensor in which a plurality of detection elements are arranged one-dimensionally or two-dimensionally;

[0592] a signal processing unit that performs signal processing on a detection signal obtained by the array sensor; and

[0593] an arithmetic operation unit that performs object detection based on the detection signal obtained by the array sensor, performs operation control of the signal processing unit based on the object detection, and performs switching processing for changing the processing content based on device information input from a sensor-equipped device in which the sensor device is installed.

[0594] (2) The sensor device according to (1), wherein the arithmetic operation unit performs processing to generate sensor operation information based on the object detection or the processing operation of the signal processing unit and transmit the generated sensor operation information to the sensor-equipped device.

[0595] (3) The sensor device according to (1) or (2), wherein the arithmetic operation unit performs the switching processing based on registration information of the sensor-equipped device registered in advance or registration information as an operation switching condition.

[0596] (4) The sensor device according to any one of (1) to (3), wherein the arithmetic operation unit obtains power information as the device information.

[0597] (5) The sensor device according to any one of (1) to (4), wherein the arithmetic operation unit obtains communication state information as the device information.

[0598] (6) The sensor apparatus according to any one of (1) to (5), wherein the arithmetic operation unit obtains information on hardware or an application of the device equipped with the sensor as the device information.

[0599] (7) The sensor apparatus according to any one of (1) to (6), wherein the arithmetic operation unit performs processing to perform class recognition on the object detected from the detection signal obtained by the array sensor, and selects a parameter for signal processing of the signal processing unit based on the recognized class, as the processing related to the switching processing.

[0600] (8) The sensor apparatus according to any one of (1) to (7), wherein the arithmetic operation unit sets a threshold value of all or some of the parameters used for signal processing of the signal processing unit or detection processing of the array sensor based on the device information and performs processing using the parameters set based on the threshold value as the processing related to the switching processing.

[0601] (9) The sensor apparatus according to any one of (1) to (8), wherein the arithmetic operation unit performs processing to perform object detection on the detection signal obtained by the array sensor and give an instruction of region information generated based on the object detection to the signal processing unit as the region information on the detection signal obtained from the array sensor or the signal processing of the detection signal, as the processing related to the switching processing.

[0602] (10) The sensor apparatus according to any one of (1) to (9), wherein the arithmetic operation unit performs processing to perform object detection on the detection signal obtained by the array sensor, give an instruction of region information generated based on the object detection to the signal processing unit as the region information on the detection signal obtained from the array sensor or the signal processing of the detection signal, perform class recognition on the object detected from the detection signal obtained by the array sensor, and generate region information corresponding to the object using a template corresponding to the recognized class, as the processing related to the switching processing.

[0603] (11) The sensor apparatus according to (10), wherein the arithmetic operation unit performs processing as the processing related to the switching processing to set a threshold value of all or some of the parameters used for signal processing of the signal processing unit or detection processing of the array sensor, and set parameters for processing of the region information indicated by the template based on the threshold value.

[0604] (12) The sensor apparatus according to any one of (1) to (11), wherein the arithmetic operation unit executes processing for instructing the signal processing unit of region information generated based on the detection of the object as the region information on the signal processing of the detection signal obtained from the array sensor or the detection signal, as the processing related to the switching processing, and

[0605] The signal processing unit executes compression processing on the detection signal obtained by the array sensor at a compression ratio different for each region based on the region information received from the arithmetic operation unit.

[0606] (13) The sensor apparatus according to any one of (1) to (12), wherein the arithmetic operation unit executes processing for setting an effective region of the detection signal obtained from the array sensor based on information on past region information, executing the detection of the object from the detection signal of the effective region, and instructing the signal processing unit of region information generated based on the detection of the object as the region information on the signal processing of the detection signal obtained from the array sensor or the detection signal, as the processing related to the switching processing.

[0607] (14) The sensor apparatus according to any one of (1) to (13), wherein the arithmetic operation unit executes processing as the processing related to the switching processing for executing the detection of the object from the detection signal obtained by the array sensor and instructing the frame rate of the detection signal obtained by the array sensor to be variable based on the detection of the object.

[0608] (15) The sensor apparatus according to (14), wherein the arithmetic operation unit sets a threshold value of the frame rate in accordance with a category identified for the object detected from the detection signal obtained by the array sensor to execute processing using the frame rate set based on the threshold value as the processing related to the switching processing.

[0609] (16) An apparatus equipped with a sensor, comprising:

[0610] a sensor apparatus; and

[0611] a control unit capable of communicating with the sensor apparatus,

[0612] wherein the sensor apparatus includes:

[0613] an array sensor in which a plurality of detection elements are arranged one-dimensionally or two-dimensionally,

[0614] a signal processing unit that executes signal processing on a detection signal obtained by the array sensor, and

[0615] The arithmetic operation unit performs object detection on the detection signal obtained by the array sensor, performs operation control of the signal processing unit based on the object detection, and performs switching processing of operation setting of the signal processing unit based on the device information input from the control unit.

[0616] (17) The sensor apparatus according to (16), wherein the arithmetic operation unit performs processing to generate sensor operation information based on a state of the processing operation of the signal processing unit or a state of the object detection and transmit the generated sensor operation information to the control unit, and

[0617] The control unit controls the device operation based on the sensor operation information.

[0618] (18) A processing method of a sensor apparatus including an array sensor in which a plurality of detection elements are arranged one-dimensionally or two-dimensionally and a signal processing unit that performs signal processing on a detection signal obtained by the array sensor, the processing method including:

[0619] performing object detection from a detection signal obtained by the array sensor, performing operation control of the signal processing unit based on the object detection, and performing switching processing for changing a processing content based on device information input from a device equipped with the sensor to which the sensor apparatus is installed.

[0620] [REFERENCE NUMBER LIST]

[0621] 1 Sensor apparatus

[0622] 2 Array sensor

[0623] 3 ADC / pixel selector

[0624] 4 Buffer

[0625] 5 Logic unit

[0626] 6 Memory

[0627] 7 Interface unit

[0628] 8 Arithmetic operation unit

[0629] 11 Processor

[0630] 12 External sensor

[0631] 30 Signal processing unit

[0632] 50 Device-side processing unit

[0633] 81 Key frame selection unit

[0634] 82 Object region recognition unit

[0635] 83 category recognition unit

[0636] 84 parameter selection unit

[0637] 85 threshold setting unit

[0638] 86 switching determination unit

[0639] 87 operation information generation unit

[0640] 100 sensor-equipped device

Claims

1. A sensor device comprising: an array sensor in which a plurality of detection elements are arranged one-dimensionally or two-dimensionally; a signal processing unit that performs signal processing on a detection signal obtained by the array sensor; and an arithmetic operation unit that performs object detection on the detection signal obtained by the array sensor, performs operation control of the signal processing unit based on object detection, and performs switching processing for changing processing content including at least one of execution / non-execution of processing, type of processing to be executed, and parameter setting of processing, based on device information indicating a state and operation of a sensor-equipped device in which the sensor device is installed, which is received from the sensor-equipped device, to optimize processing of the sensor device, wherein the arithmetic operation unit performs class recognition on an object detected from the detection signal obtained by the array sensor, and selects a parameter for the signal processing based on a class of the recognized object, as processing related to the switching processing, the class of the object indicating a kind of the recognized object using image recognition of the object.

2. The sensor device of claim 1, wherein, The arithmetic operation unit performs processing of generating sensor operation information based on the object detection or processing operation of the signal processing unit and transmitting the generated sensor operation information to the sensor-equipped device.

3. The sensor device of claim 1, wherein, The arithmetic operation unit performs the switching processing based on registration information of the sensor-equipped device registered in advance or registration information as an operation switching condition.

4. The sensor device of claim 1, wherein, The arithmetic operation unit acquires power information as the device information.

5. The sensor device of claim 1, wherein, The arithmetic operation unit acquires communication state information as the device information.

6. The sensor device of claim 1, wherein, The arithmetic operation unit acquires information on hardware or applications of the sensor-equipped device as the device information.

7. The sensor device of claim 1, wherein, The arithmetic operation unit sets a threshold value for all or some of the parameters used for the signal processing of the detection signal by the signal processing unit or detection processing of the array sensor based on the device information, and performs the signal processing using parameters set based on the threshold value, as processing related to the switching processing.

8. The sensor device of claim 1, wherein, The arithmetic operation unit performs processing for performing object detection on the detection signal obtained by the array sensor, and gives an instruction to the signal processing unit for region information generated based on the detection of the object as region information on acquisition of the detection signal from the array sensor or signal processing of the detection signal, as processing related to the switching processing.

9. The sensor device of claim 1, wherein, The arithmetic operation unit performs object detection on the detection signal obtained by the array sensor, and gives an instruction to the signal processing unit for region information generated based on the detection of the object as region information on acquisition of the detection signal from the array sensor or signal processing of the detection signal, and generates region information corresponding to the object using a template corresponding to a class of the object, as processing related to the switching processing.

10. The sensor device of claim 9, wherein, The arithmetic operation unit executes processing for setting a threshold value of all or some of the parameters for signal processing of the detection signal by the signal processing unit or for detection processing of the array sensor, and setting a parameter for processing of region information indicated by the template based on the threshold value, as processing related to the switching processing.

11. The sensor device of claim 1, wherein, The arithmetic operation unit executes object detection on the detection signal obtained by the array sensor and gives an instruction of region information generated based on the detection of the object to the signal processing unit, as region information on acquisition of the detection signal from the array sensor or signal processing of the detection signal, as processing related to the switching processing, and The signal processing unit executes compression processing on the detection signal obtained by the array sensor at a compression ratio different for each region based on the region information received from the arithmetic operation unit.

12. The sensor device of claim 1, wherein, The arithmetic operation unit executes processing for setting an effective region of the detection signal acquired from the array sensor based on information on past region information, executing object detection on the detection signal from the effective region, and giving an instruction of region information generated based on the detection of the object to the signal processing unit, as region information on acquisition of the detection signal from the array sensor or signal processing of the detection signal, as processing related to the switching processing.

13. The sensor device of claim 1, wherein, The arithmetic operation unit executes object detection on the detection signal obtained by the array sensor and gives an instruction for changing a frame rate of the detection signal obtained by the array sensor based on the object detection, as processing related to the switching processing.

14. The sensor device of claim 13, wherein, The arithmetic operation unit sets a threshold value of the frame rate according to a category of the object detected from the detection signal obtained by the array sensor and executes the signal processing using a frame rate set based on the threshold value, as processing related to the switching processing.

15. A sensor-equipped device comprising: the sensor device according to claim 1; and a control unit capable of communicating with the sensor device.

16. The sensor-equipped device of claim 15, wherein, The arithmetic operation unit executes processing for generating sensor operation information based on a state of signal processing by the signal processing unit or a state of the object detection and transmitting the generated sensor operation information to the control unit, and The control unit controls device operation based on the sensor operation information.

17. A processing method of a sensor device including an array sensor in which a plurality of detection elements are arranged one-dimensionally or two-dimensionally, a signal processing unit that executes signal processing on a detection signal obtained by the array sensor, and an arithmetic operation unit, the processing method being executed by the arithmetic operation unit, the processing method comprising: performing object detection on the detection signal obtained from the array sensor, performing operation control of the signal processing unit based on the object detection, and performing switching processing for changing processing content including at least one of execution / non-execution of processing, type of processing to be executed, and parameter setting of processing, based on device information indicating a state and operation of a sensor-equipped device in which the sensor device is installed, which is received from the sensor-equipped device, to optimize processing of the sensor device, performing class recognition on an object detected from the detection signal obtained from the array sensor, the class of the object indicating a kind of the object recognized using image recognition of the object, and selecting, as processing related to the switching processing, a parameter for signal processing of the detected signal by the signal processing unit, based on the class of the object.

Images