Imaging device, imaging control method, and computer-readable non-transitory storage medium

Abstract

An imaging device according to the present invention comprises a motion extraction unit, a region determination unit, and a mode control unit. The motion extraction unit extracts motion of a subject on the basis of image information obtained in a motion detection mode. The region determination unit acquires, as a motion region, an image region in which the motion of the subject was extracted. The mode control unit determines, on the basis of the size of the motion region, the resolution of image information acquired in a feature detection mode. The mode control unit switches to a specific processing mode in response to detection of a specific feature of interest from the image information obtained in the feature detection mode.

Description

Imaging device, imaging control method, and computer-readable non-transitory storage medium

[0001] The present invention relates to an imaging device, an imaging control method, and a computer-readable non-transitory storage medium.

[0002] In the field of surveillance cameras and the like, it is known that video recording is performed in three modes: motion detection, feature detection, and imaging. For example, a possible usage scenario is that a camera in a store detects a customer (motion detection mode), and if the customer exhibits suspicious features (feature detection mode), imaging for recording or display begins (imaging mode). Note that instead of imaging for recording or display, other processing such as issuing an alert may also be performed. Hereinafter, processing such as imaging or issuing an alert performed upon feature detection will be referred to as a specific processing, and the operating mode for performing the specific processing will be referred to as a specific processing mode.

[0003] Japanese Patent Application Laid-Open No. 2021-150814

[0004] Always capturing images at high resolution results in high power consumption. Patent Document 1 proposes setting the resolution of the motion detection mode and feature detection mode lower than that of the imaging mode. Patent Document 1 also proposes setting the resolution of the feature detection mode higher than that of the motion detection mode, but because the resolution is set uniformly across the board, power consumption may be wasted depending on the situation. For example, when a subject is close, the resolution is unnecessarily high compared to the size of the subject. Conversely, when a subject is far away, the resolution is likely to be insufficient compared to the size of the subject.

[0005] Therefore, the present disclosure proposes an imaging device, an imaging control method, and a computer-readable non-transitory storage medium that are capable of reducing power consumption.

[0006] According to the present disclosure, there is provided an imaging device including: a motion extraction unit that extracts motion of a subject based on image information obtained in a motion detection mode; an area determination unit that acquires an image area from which the motion of the subject is extracted as a motion area; and a mode control unit that determines a resolution of image information to be acquired in a feature detection mode based on the size of the motion area and switches to a specific processing mode in response to detection of a specific feature of interest from the image information obtained in the feature detection mode. Also, according to the present disclosure, there is provided an imaging control method in which information processing of the imaging device is executed by a computer, and a computer-readable non-transitory storage medium storing a program for causing a computer to realize the information processing of the imaging device.

[0007] 1 is a diagram illustrating an example of a system configuration of an imaging device. FIG. 2 is a diagram illustrating an example of a configuration of an image sensor. FIG. 3 is a diagram illustrating an example of a transition state of an operation mode. FIG. 4 is a diagram illustrating an example of a processing flow related to an imaging control method. FIG. 5 is a diagram illustrating resolution settings for each operation mode. FIG. 6 is a diagram illustrating resolution settings for each operation mode. FIG. 7 is a diagram illustrating an example of image information acquired in feature detection mode. FIG. 8 is a diagram illustrating an example of image information acquired in feature detection mode. FIG. 9 is a diagram illustrating an example of generation of logical pixels. FIG. 10 is a diagram illustrating a motion detection method. FIG. 11 is a diagram illustrating a motion detection method. FIG. 12 is a diagram illustrating an example of an arrangement of subjects in a readout area. FIG. 13 is a diagram illustrating an example of an arrangement of subjects in a readout area. FIG. 14 is a diagram illustrating an example of row scanning control and column scanning control in motion detection mode. FIG. 15 is a diagram illustrating an example of row scanning control and column scanning control in feature detection mode. FIG. 16 is a diagram illustrating a modified example of a processing flow related to an imaging control method. FIG. 17 is a diagram illustrating an example of an external appearance of an information processing system of the present disclosure. FIG. 18 is a diagram illustrating an example of an external appearance of an information processing system of the present disclosure. FIG. 19 is a diagram illustrating an example of a hardware configuration of an information processing system of the present disclosure.

[0008] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In the following embodiments, the same components are designated by the same reference numerals, and redundant description will be omitted.

[0009] The description will be given in the following order: [1. System configuration example of an imaging device] [2. Configuration example of an image sensor] [3. Imaging control method] [4. Resolution setting for each operation mode] [5. Motion detection method] [6. Adjustment of resolution and frame rate] [7. Modification example] [8. Information processing system] [9. Effects]

[0010] 1 is a diagram showing an example of the system configuration of an imaging device 100. The imaging device 100 is used, for example, as a surveillance camera. The imaging device 100 is installed at a fixed position in a store and monitors the behavior of customers.

[0011] The imaging device 100 includes an image sensor 10, a control unit 1, a storage unit 2, a display unit 3, an operation unit 4, and a communication unit 5. These components are electrically connected to one another via a bus 6.

[0012] The image sensor 10 has a solid-state imaging element (pixel array unit 11) such as a CCD (Charge Coupled Device) sensor or a CMOS (Complemented Metal Oxide Semiconductor) sensor. The image sensor 10 has an optical system including a plurality of lenses, an aperture, a shutter, etc. The image sensor 10 outputs acquired image information to the storage unit 2 and the display unit 3 as necessary. The detailed configuration of the image sensor 10 will be described later with reference to FIG. 2 .

[0013] The control unit 1 is configured by, for example, a CPU (Central Processing Unit) etc. The control unit 1 executes various calculations based on various programs stored in the storage unit 2, and performs overall control of each unit of the imaging device 100.

[0014] The storage unit 2 includes a non-volatile memory that stores various programs required for processing by the control unit 1 and image information acquired by the image sensor 10, and a volatile memory that is used as a work area for the control unit 1. The various programs may be read from a portable non-transitory recording medium such as an optical disk or semiconductor memory, or may be downloaded from a server device on a network.

[0015] The display unit 3 is configured by, for example, a liquid crystal display, an EL (Electro-Luminescence) display, etc. Based on the control of the control unit 1, the display unit 3 displays image information acquired by the image sensor 10 on the screen as necessary.

[0016] The operation unit 4 is a push button type, proximity type, or other type of operation unit, and detects operations by the user and outputs the results to the control unit 1.

[0017] The communication unit 5 communicates with external devices via a wired or wireless connection. Examples of external devices that communicate with the imaging device 100 include mobile phones (including smartphones), PCs (personal computers), and server devices on a network.

[0018] 1, the image sensor 10, the control unit 1, the storage unit 2, the display unit 3, and the operation unit 4 are provided within the same device, but these components may be distributed across multiple devices. In this case, for example, the image sensor 10 is provided in the imaging device 100, and the other components (the control unit 1, the storage unit 2, the operation unit 4, etc.) are provided in separate devices.

[0019] That is, the imaging device 100 is required to include at least the image sensor 10. The device separate from the imaging device 100 may be a dedicated device or a general-purpose device. If the separate device is a general-purpose device, the separate device may be a mobile phone (including a smartphone), a PC, a server device on a network, or the like.

[0020] 2. Configuration Example of Image Sensor FIG. 2 is a diagram showing a configuration example of the image sensor 10. As shown in FIG.

[0021] The image sensor 10 includes a pixel array unit 11, an ADC (Analog to Digital Converter) 12, a row scanning circuit 13, a column scanning circuit 14, and a sensor control unit 15. The sensor control unit 15 includes a mode control unit 16, a timing control unit 17, a motion detection unit 18, a feature detection unit 19, an image processing unit 20, a switch 21, and a readout area control unit 22.

[0022] The imaging device 100 has three possible operating modes: a motion detection mode, a feature detection mode, and an imaging mode. The three modes are transitioned under the control of the mode control unit 16. The imaging mode is an example of a specific processing mode implemented during feature detection. The mode control unit 16 switches to the specific processing mode in response to detection of a specific feature of interest from image information obtained in the feature detection mode. The specific processing mode is, for example, an imaging mode with a higher resolution than the feature detection mode, or a mode that outputs an alert. In the following description, the imaging mode will be described as a processing mode during feature detection.

[0023] The motion detection mode is a mode in which image information with a resolution relatively lower than that used in the feature detection mode and the imaging mode is acquired, and the motion of a subject is detected based on this low-resolution image information. In the following description, the image information acquired in the motion detection mode will be referred to as low-resolution image information for convenience.

[0024] The feature detection mode is a mode in which image information having a resolution relatively higher than that of the image information used in the motion detection mode but a resolution relatively lower than that of the image information used in the imaging mode is acquired, and the features of the subject are detected based on this image information of resolution. In the following description, the image information acquired in the feature detection mode will be referred to as medium-resolution image information for convenience.

[0025] The imaging mode is a mode in which image information with a relatively higher resolution than the image information used in the motion detection mode and the feature detection mode is acquired and imaging for recording or display is performed. In the following description, the image information acquired in the imaging mode will be referred to as high-resolution image information for convenience.

[0026] Note that only high-resolution image information acquired in the imaging mode is used for recording or display. Low-resolution image information acquired in the motion detection mode is used only for motion detection and is not used for storage or display. Similarly, medium-resolution image information acquired in the feature detection mode is used only for feature detection and is not used for storage or display.

[0027] The pixel array unit 11 has a plurality of pixels arranged in a two-dimensional lattice. Each pixel performs photoelectric conversion on light incident through an optical system such as a lens, and generates charge information according to the intensity of the light. The pixel array unit 11 performs photoelectric conversion on the plurality of pixels to generate image information as an analog signal, and outputs the generated image information as an analog signal to the ADC 12.

[0028] In the motion detection mode, for example, a plurality of pixels (e.g., 16×16=256 pixels, 32×32=1024 pixels, etc.) in the pixel array unit 11 are considered to be one block, and one block is treated as the smallest unit of image information.

[0029] Hereinafter, each pixel in the pixel array unit 11 may be referred to as a physical pixel PX (see FIG. 6), and each block formed by a plurality of physical pixels PX may be referred to as a logical pixel RP (see FIG. 6). The greater the number of pixels (physical pixels PX) included in one block (logical pixel RP), the lower the resolution of the acquired image.

[0030] In motion detection mode, charge information is read out for each block (logical pixel RP) to obtain image information. Methods for reading charge information in block units include binning and random sampling. Binning artificially combines multiple pixels (physical pixels PX) and treats them as a single pixel (logical pixel RP). Random sampling treats the charge information of any one pixel within a block as the charge information of the block. The resolution of image information in motion detection mode is, for example, 40 x 30 px, 32 x 20 px, 16 x 5 px, etc.

[0031] In the feature detection mode, as in the motion detection mode, multiple pixels (e.g., 2 x 2 = 4 pixels, 4 x 4 = 16 pixels, etc.) in the pixel array unit 11 are considered to be one block. The number of pixels included in one block in the feature detection mode is set to be smaller than the number of pixels included in one block in the motion detection mode. Therefore, the resolution of the image information acquired in the feature detection mode is higher than the resolution of the image information acquired in the motion detection mode.

[0032] In the feature detection mode, similar to the motion detection mode, charge information is read out for each block (logical pixel RP) to obtain image information. The resolution of the image information in the feature detection mode corresponds to, for example, VGA (Video Graphics Array) (640 x 480 px), QVGA (Quarter VGA) (320 x 240 px), QQVGA (Quarter QVGA) (160 x 120 px), etc.

[0033] In the imaging mode, for example, charge information is read out for each pixel (physical pixel PX) to acquire image information. The resolution of the image information in the imaging mode corresponds to, for example, FHD (Full High Definition) (1920 x 1080), HD (High Definition) (1280 x 720), VGA (640 x 480), etc. In the imaging mode, charge information is read out in units of pixels (physical pixels PX) rather than in units of blocks (logical pixels RP). Therefore, the resolution of the image acquired in the imaging mode is higher than the resolution of the image acquired in the motion detection mode and the feature detection mode.

[0034] The row scanning circuit 13, the column scanning circuit 14, and the ADC 12 constitute a pixel circuit unit that drives the pixel array unit 11. The pixel circuit unit drives each pixel in the pixel array unit 11 to cause the pixel array unit 11 to acquire image information in the form of an analog signal. A mode signal indicating which of three modes the current operation mode is is input to the row scanning circuit 13 from the mode control unit 16. The pixel circuit unit reads out charge information in units of pixels (physical pixels PX) or blocks (logical pixels RP) based on the mode signal.

[0035] For example, the row scanning circuit 13 scans the pixel array unit 11 in the row direction according to the unit of readout of charge information. The column scanning circuit 14 controls the ADC 12 to output digital image information signals to the switch 21. The ADC 12 converts the analog image information signals input from the pixel array unit 11 into digital image information signals in synchronization with a clock signal. The ADC 12 outputs the generated digital image information signals to the switch 21.

[0036] In the motion detection mode, the pixel circuit unit controls the pixel array unit 11 to read out charge information for each block and generate low-resolution image information. In the feature detection mode, the pixel circuit unit controls the pixel array unit 11 to read out charge information for each block and generate medium-resolution image information. In the imaging mode, the pixel circuit unit controls the pixel array unit 11 to read out charge information for each pixel and generate high-resolution image information.

[0037] The ADC 12 A / D converts low-resolution image information in the motion detection mode, A / D converts medium-resolution image information in the feature detection mode, and A / D converts high-resolution image information in the imaging mode. The amount of data that needs to be A / D converted is smallest in the motion detection mode, next smallest in the feature detection mode, and largest in the imaging mode. Therefore, the power consumption of the ADC 12 is smallest in the motion detection mode, next smallest in the feature detection mode, and largest in the imaging mode. As a result, the power consumption of the ADC 12 is reduced compared to when high-resolution image information is constantly acquired and A / D converted.

[0038] The sensor control unit 15, for example, based on a program, comprehensively controls each unit of the image sensor 10. Note that specific processing by the sensor control unit 15 (particularly the mode control unit 16, the motion detection unit 18, the feature detection unit 19, and the readout area control unit 22) will be described in detail later with reference to Figures 4 to 13.

[0039] The timing control unit 17 controls the operation timing of the row scanning circuit 13, the ADC 12, and the column scanning circuit 14. A vertical synchronization signal of a predetermined frequency (e.g., 30 Hz) is input to the timing control unit 17. The timing control unit 17 generates timing signals indicating the operation timing of each of the row scanning circuit 13, the ADC 12, and the column scanning circuit 14 in synchronization with the vertical synchronization signal, and outputs the corresponding timing signals to each unit.

[0040] The switch 21 switches the output destination of the image information output from the ADC 12 based on a mode signal input from the mode control unit 16. For example, in the motion detection mode, the switch 21 outputs low-resolution image information from the ADC 12 to the motion detection unit 18. In the feature detection mode, the switch 21 outputs medium-resolution image information from the ADC 12 to the feature detection unit 19. In the imaging mode, the switch 21 outputs high-resolution image information from the ADC 12 to the image processing unit 20.

[0041] In the motion detection mode, the motion detection unit 18 acquires low-resolution image information from the ADC 12 via the switch 21. The motion detection unit 18 determines whether or not motion of the subject has been detected based on the acquired low-resolution image information. The motion detection unit 18 outputs the determination result to the mode control unit 16.

[0042] For example, the motion detection unit 18 includes a motion extraction unit 18a, a motion determination unit 18b, and a region determination unit 18c. The motion extraction unit 18a extracts the motion of the subject based on low-resolution image information obtained in the motion detection mode and outputs the extracted motion information (motion information) to the motion determination unit 18b. The motion determination unit 18b determines whether motion has been detected based on the motion information and outputs the determination result to the mode control unit 16. The region determination unit 18c acquires the image region from which the motion of the subject has been extracted as a motion region MA (see FIG. 5 ). The region determination unit 18c outputs the size (region size) of the motion region MA to the mode control unit 16, reads out coordinate information (region coordinates) of the motion region MA, and outputs it to the region control unit 22.

[0043] In the feature detection mode, the readout region control unit 22 sets a group of pixels included in the pixel array unit 11 as a readout region RA (see FIG. 5 ). For example, the readout region control unit 22 sets an image region that includes a motion region MA in its center as the readout region RA. The readout region control unit 22 controls the timing control unit 17 to selectively read out image signals from the readout region RA that includes the motion region MA.

[0044] The feature detection unit 19 detects specific features of interest from the medium-resolution image information obtained in the feature detection mode. The features to be detected are set arbitrarily depending on the purpose of the imaging. For example, if the purpose of the imaging is to monitor suspicious individuals, the features to be detected include facial features, facial expressions, and movements that characterize suspicious behavior. If the purpose of the imaging is to collect customer data, the features to be detected include the age, gender, and purchasing behavior of customers.

[0045] For example, in the feature detection mode, the feature detection unit 19 acquires medium-resolution image information from the ADC 12 via the switch 21. Based on the acquired medium-resolution image information, the feature detection unit 19 determines whether a specific feature of interest has been detected. The feature detection unit 19 outputs the determination result to the mode control unit 16.

[0046] For example, the feature detection unit 19 includes a feature extraction unit 19 a and a feature determination unit 19 b. The feature extraction unit 19 a extracts features of the subject based on medium-resolution image information and outputs the extracted feature information to the feature determination unit 19 b. The feature determination unit 19 b determines whether a specific feature of interest has been detected based on the feature information and outputs the determination result to the mode control unit 16.

[0047] In the imaging mode, the image processing unit 20 acquires high-resolution image information from the ADC 12 via the switch 21. The image processing unit 20 performs necessary image processing on the acquired high-resolution image information (for example, various image processing such as various corrections for black level and white spots (defective pixels), rearrangement of pixel order, color processing, gamma correction processing, demosaic processing, white balance processing, and filter processing). The image processing unit 20 outputs the high-resolution image information after image processing to the storage unit 2 and the display unit 3.

[0048] The mode control unit 16 controls transitions among three modes, namely, the motion detection mode, the feature detection mode, and the imaging mode, based on the determination results input from the motion detection unit 18 and the determination results input from the feature detection unit 19. When a mode transition occurs, the mode control unit 16 outputs a mode signal indicating the destination mode (indicating the current mode) to the row scanning circuit 13 and the switch 21.

[0049] FIG. 3 is a diagram showing an example of a transition state of an operation mode.

[0050] When subject motion is detected in the motion detection mode, the mode control unit 16 transitions the mode from the motion detection mode to the feature detection mode. The mode control unit 16 determines the resolution of the image information to be acquired in the feature detection mode based on the size of the motion area MA detected in the motion detection mode. The mode control unit 16 can increase the image resolution in the feature detection mode as the motion area MA becomes smaller, so that the target feature can be detected with high accuracy.

[0051] The mode control unit 16 switches from the feature detection mode to the imaging mode in response to the detection of a specific feature of interest from the image information obtained in the feature detection mode. The mode control unit 16 switches from the feature detection mode to the motion detection mode in response to the failure to detect a specific feature of interest in the feature detection mode. The mode control unit 16 transitions from the imaging mode to the motion detection mode when a specified number of images have been obtained in the imaging mode.

[0052] In this embodiment, the image sensor 10 is in an active state regardless of the mode, but the control unit 1, storage unit 2, display unit 3, communication unit 5, etc. (hereinafter referred to as the control unit 1, etc.) are in a sleep state in some modes. In other words, the control unit 1, etc. are in a sleep state in the motion detection mode and feature detection mode, and are in an active state only in the imaging mode. However, when there is an input from the user via the operation unit 4, the control unit 1, etc. are exceptionally in an active state regardless of the mode.

[0053] When the mode is shifted from the feature detection mode to the imaging mode, the mode control unit 16 wakes up the control unit 1 etc. from a sleep state. At this time, the mode control unit 16 also wakes up a reference clock generation circuit (not shown) that supplies a reference clock to the control unit 1 etc. When the mode is shifted from the imaging mode to the motion detection mode, the mode control unit 16 changes the control unit 1 etc. from an active state to a sleep state. At this time, the mode control unit 16 also stops the reference clock generation circuit (not shown) that supplies a reference clock to the control unit 1 etc. In this embodiment, by activating the control unit 1 etc. and the reference clock generation circuit only at the necessary timing (imaging mode), it is possible to reduce power consumption of the entire imaging device 100.

[0054] 3. Imaging Control Method FIG. 4 is a diagram showing an example of a processing flow relating to an imaging control method.

[0055] The mode control unit 16 sets the operation mode to a motion detection mode (step ST1). The pixel array unit 11 outputs low-resolution image information in which multiple physical pixels PX are grouped into one logical pixel RP (block). The image information is input to the motion detection unit 18 via the switch 21. The motion extraction unit 18a extracts the difference in image information between frames as an inter-frame difference (motion detection operation: step ST2). The motion determination unit 18b determines whether or not there is motion of the subject based on the inter-frame difference (step ST3).

[0056] For example, the motion extractor 18a compares the inter-frame difference with a predetermined threshold and determines that subject motion has been detected if the inter-frame difference exceeds the threshold (step ST3: Yes), whereas the motion extractor 18a determines that subject motion has not been detected if the inter-frame difference is equal to or less than the threshold (step ST3: No).

[0057] If no subject movement is detected (step ST3: No), the process returns to step ST2, and the above-described process is repeated until subject movement is detected.

[0058] If subject motion is detected (step ST3: Yes), the region determination unit 18c acquires the image region from which the subject motion is extracted as a motion region MA. The readout region control unit 22 sets a partial pixel group included in the pixel array unit 11 as a readout region RA (step ST4). The mode control unit 16 selects the resolution of the image information to be acquired in the feature detection mode based on the size of the motion region MA (step ST5). The mode control unit 16 outputs a mode signal indicating that the current operating mode is the feature detection mode to the row scanning circuit 13 and the switch 21, thereby transitioning the operating mode from the motion detection mode to the feature detection mode (step ST6).

[0059] The readout region control unit 22 controls the pixel circuit unit to selectively read out image signals of the readout region RA from the pixel array unit 11, and acquires medium-resolution image information. The switch 21 switches the output destination of the medium-resolution image information output from the ADC 12 from the motion detection unit 18 to the feature detection unit 19. The feature extraction unit 19a extracts feature information of the subject (feature point group information, edge information, distance information, polarization information, spectral information, etc.) from the medium-resolution image information (feature detection operation: step ST7).

[0060] The feature determination unit 19b determines whether a specific feature of interest has been detected based on the feature information of the subject (step ST8). For example, the feature determination unit 19b compares the extracted feature information with a recognition model, and if the feature information and the recognition model match, determines that a specific feature has been detected (step ST8: Yes). If the feature information and the recognition model do not match, the feature determination unit 19b determines that a specific feature has not been detected (step ST8: No). The feature determination unit 19b outputs the determination result of whether a specific feature has been detected to the mode control unit 16.

[0061] The recognition model is a model corresponding to a subject (e.g., a person, an animal, a car, etc.) that is assumed to be the image capture target, and is stored in advance in the feature determination unit 19 b. For example, if a person is assumed to be the image capture target, the feature determination unit 19 b compares the extracted feature information with a recognition model that represents the entire body of the person and a recognition model that represents the face of the person, and determines whether a specific feature has been detected (person detection, face detection).

[0062] When an animal such as a cat or a dog is the subject of imaging, the feature determination unit 19b compares the extracted feature information with a recognition model showing the whole body of the animal and a recognition model showing the face of the animal to determine whether a specific feature has been detected (animal detection, animal face detection).When a car is the subject of imaging, the feature determination unit 19b compares the extracted feature information with a recognition model showing the whole car and a recognition model showing part of the car to determine whether a specific feature has been detected (car detection, part detection).

[0063] When multiple types of subjects are to be imaged, the feature determination unit 19b compares the feature information with the corresponding recognition models to determine whether or not a specific feature of interest has been detected. For example, when a person and a car are to be imaged, the feature determination unit 19b compares the extracted feature information with a recognition model showing the whole body of the person, a recognition model showing the face of the person, a recognition model showing the whole car, and a recognition model showing each part of the car to determine whether or not a specific feature has been detected.

[0064] The user may be able to select a desired imaging target from a plurality of types of imaging targets prepared in advance via the operation unit 4. For example, suppose that various types of imaging targets, such as people, animals, and cars, are prepared in advance, and the user selects a person as the desired imaging target from these. In this case, the feature determination unit 19b compares the feature information with a recognition model representing the entire body of a person and a recognition model representing the face of a person, and determines whether a specific feature has been detected.

[0065] If the specific feature of interest is not detected in the feature detection mode (step ST8: No), the process returns to step ST1. The mode control unit 16 outputs a mode signal indicating that the current operation mode is the motion detection mode to the row scanning circuit 13 and the switch 21, and transitions the operation mode from the feature detection mode to the motion detection mode.

[0066] If a specific feature of interest is detected in the feature detection mode (step ST8: Yes), the mode control unit 16 outputs a mode signal indicating that the current operation mode is the imaging mode to the row scanning circuit 13 and the switch 21, and transitions the operation mode from the feature detection mode to the imaging mode (step ST9). The mode control unit 16 also wakes up the control unit 1 and the like from a sleep state and also wakes up the reference clock generation circuit that supplies a reference clock to the control unit 1 and the like.

[0067] In the imaging mode, the pixel array unit 11 outputs high-resolution image information based on the charge information of each pixel. The switch 21 switches the output destination of the high-resolution image information output from the ADC 12 from the feature detection unit 19 to the image processing unit 20. The image processing unit 20 performs various image processing such as demosaic processing on the high-resolution image information, and outputs the processed high-resolution image information to the storage unit 2 and the display unit 3 (imaging operation: step ST10).

[0068] When acquisition of high-resolution image information starts in the imaging mode, the mode control unit 16 determines whether the number of pieces of image information acquired in the imaging mode has reached a specified number (step ST11). The number of pieces of image information acquired in the imaging mode may be set appropriately to one, two, three, etc. The number of pieces of image information acquired in the imaging mode may be changeable by the user through input from the operation unit 4.

[0069] If the number of acquired image information does not reach the specified number (step ST11: No), the process returns to step ST10, and the above-described process is repeated until the number of image information reaches the specified number.

[0070] If the number of acquired image information reaches the specified number (step ST11: Yes), the process returns to step ST1. The mode control unit 16 outputs a mode signal indicating that the current operation mode is the motion detection mode to the row scanning circuit 13 and the switch 21, and transitions the current operation mode from the imaging mode to the motion detection mode. The mode control unit 16 also switches the control unit 1 and the like from the active state to the sleep state, and stops the reference clock generation circuit that supplies the reference clock to the control unit 1 and the like.

[0071] 4. Resolution Settings for Each Operation Mode FIGS. 5 and 6 are diagrams for explaining resolution settings for each operation mode.

[0072] The mode control unit 16 varies the resolution of the image output from the pixel array unit 11 among the motion detection mode, feature detection mode, and imaging mode. In the motion detection mode, the resolution is the lowest because only the presence or absence of motion of the subject SB is to be detected. In the feature detection mode, the resolution is set higher than in the motion detection mode because the features of the subject SB are to be detected. In the imaging mode, the resolution is set highest because image information for recording or display is to be acquired.

[0073] For example, the resolution of the imaging mode is HD, 4K, or 8K, which is suitable for visual inspection of the image or for advanced signal processing. The resolution of the feature detection mode is sufficient for recognizing the object using a recognition device such as AI, such as VGA, QVGA, CIF, or QCIF. The resolution of the motion detection mode is determined by the size at which the moving part within the angle of view can be identified and recognized in the feature detection mode. Therefore, for example, a range of several to several tens of pixels on one side is treated as one block (logical pixel RP).

[0074] The motion extraction unit 18a extracts the motion of subject SB based on image information obtained in motion detection mode. In the example of Fig. 5, two people (subject SB1 and subject SB2) are captured within the field of view. Subject SB1 in the foreground appears large, while subject SB2 in the background appears small. The area determination unit 18c acquires the image area from which the motion of subject SB1 has been extracted as motion area MA1. The area determination unit 18c acquires the image area from which the motion of subject SB2 has been extracted as motion area MA2. The motion area MA2 is smaller than the motion area MA1.

[0075] The mode control unit 16 determines the resolution of the image information acquired in the feature detection mode based on the size of the motion area MA. For example, if the resolution of a small subject SB2 located far away is not increased, the fine features cannot be accurately detected. Therefore, the mode control unit 16 increases the resolution of the image information acquired in the feature detection mode as the size of the motion area MA decreases. In the example of Figure 5, the resolution of the small motion area MA2 corresponding to the distant subject SB2 is set higher than the resolution of the large motion area MA2 corresponding to the closer subject SB1.

[0076] The mode control unit 16 may determine the resolution of the image information acquired in the feature detection mode based on the distance to the subject SB captured in the motion area MA. The mode control unit 16 may increase the resolution of the image information acquired in the feature detection mode as the distance to the subject SB increases. For example, in a situation where objects closer to the bottom of the imaging area are captured and objects farther away to the top are captured, the resolution of the bottom may be set low and the resolution of the top may be set high. This can improve the accuracy of feature detection and power efficiency.

[0077] 7 and 8 are diagrams showing examples of image information acquired in the feature detection mode.

[0078] The size of a block (logical pixel RP), which is the smallest unit of image information, is determined according to the resolution determined by the mode control unit 16. The readout area control unit 22 defines the number of pixels (physical pixels PX) set according to the resolution as the logical pixel RP, which is the smallest unit of image information.

[0079] The readout area control unit 22 determines the range of the readout area RA so that the number of logical pixels RP included in the readout area RA is constant. For example, the number of logical pixels RP included in the readout area RA is determined based on the network size of a mathematical model used for feature detection, such as a convolutional neural network (CNN). The readout area control unit 22 sets an image area including a predetermined fixed number of logical pixels RP as the readout area RA.

[0080] The feature detection unit 19 generates an image in which pixel values ​​in the readout area RA outside the movement area MA are filled with dummy data as a detection target image TG. Dummy data refers to data indicating a specific value (e.g., 0) set by the system. In the example of FIG. 7, the movement area MA is detected as a distorted shape that matches the shape of the subject SB1 (see FIG. 5). The feature detection unit 19 fills the data outside the movement area MA with dummy data to generate a rectangular detection target image TG.

[0081] The feature detection unit 19 detects specific features of interest from the detection target image TG. For example, the feature extraction unit 19a performs CNN calculations on a preset feature map and an image signal to extract features. At this time, the power consumption of feature detection is reduced by skipping the calculation of dummy data. The feature determination unit 19b determines the degree of match with a preset determination target object such as a person, hand, or car, and outputs a determination signal indicating whether or not there is a match.

[0082] The number of logical pixels RP included in the readout area RA can be varied depending on the target for feature detection. For example, when detecting the presence or absence of a person, a square area with 100 pixels on a side can be set as the readout area RA. When detecting the shape of a hand, a square area with 200 pixels on a side can be set as the readout area RA. The number of pixels on one side refers to the number of logical pixels RP.

[0083] FIG. 9 is a diagram showing an example of generating logical pixels RP.

[0084] In the example of Fig. 9, physical pixels PX are integrated by binning. The circled areas in Fig. 9 are transistors Tr that are turned ON when adjusting the resolution. By turning on multiple transistors Tr, the charge information of multiple physical pixels PX is aggregated into one floating diffusion FD. The charge information aggregated into the floating diffusion FD is read out as the charge information of one logical pixel RP.

[0085] The highest resolution is achieved when charge information is read out from each physical pixel PX individually without turning on the transistor Tr (1 logical pixel = 1 physical pixel). When charge information from multiple physical pixels PX is aggregated, the resolution decreases by the number of connected physical pixels PX. In Figure 9, a state in which four physical pixels PX are connected and the resolution is reduced to 1 / 4 is shown as an example of "medium resolution," and a state in which 16 physical pixels PX are connected and the resolution is reduced to 1 / 16 is shown as an example of "low resolution."

[0086] The method for integrating the charge information of the physical pixels PX is not limited to the method shown in Fig. 9. Analog signals may be added in a column circuit such as the ADC 12, or digital signals may be added after A / D conversion by the ADC 12. This method also makes it possible to integrate the charge information of multiple physical pixels PX.

[0087] 5. Motion Detection Method FIGS. 10 and 11 are diagrams for explaining a motion detection method.

[0088] The motion extraction unit 18a stores, as data, image signals of, for example, a previous frame in motion detection mode. The motion extraction unit 18a compares the signal amounts of the image signals of the current frame and the previous frame. The motion extraction unit 18a calculates the difference in signal amount for each region and outputs an image indicating the difference (difference image) as the comparison result.

[0089] The motion determination unit 18b scans the difference image and determines that motion has occurred when the difference exceeds a preset amount. For example, the motion determination unit 18b sets a certain threshold for the difference and extracts logical pixels RP whose difference exceeds the threshold as motion detection pixels MP. The motion determination unit 18b determines that motion has occurred between frames when the number of motion detection pixels MP exceeds a certain number. The motion determination unit 18b outputs an image indicating the motion detection pixels MP as a motion detection frame.

[0090] The region determination unit 19c calculates the position and size of the motion region MA from the coordinate information of the motion detection pixels MP. The region determination unit 19c acquires a group of adjacent motion detection pixels MP as the motion region MA. If there are discrete logical pixel groups in which motion is detected, each of the logical pixel groups is acquired as a motion region MA. The mode control unit 16 determines the imaging region and resolution in the feature detection mode based on the position and size of the motion region MA.

[0091] 11, the region determination unit 19c creates a contracted image by addition processing, and determines the portion of the image that exceeds the threshold value after addition as the motion region MA of the large subject (nearby subject). The region determination unit 19c masks the motion detection frame with the motion detection frame of the large subject to generate a motion detection frame of the small subject (distant subject). The mode control unit 16 acquires the position and size of the motion region MA for each subject from the motion detection frame of each individual subject.

[0092] The resolution of feature detection (size of logical pixels RP) is set based on the size of the motion area MA. The readout area control unit 22 sets a small area containing the motion area MA and having a certain number of logical pixels RP as the readout area RA. The feature detection unit 19 applies the image information of the readout area RA to a mathematical model such as CNN to extract features of the subject SB. This method reduces the input data size of the mathematical model, thereby reducing the amount of calculation, and aligns the positions of the subjects, making it easier to train the mathematical model.

[0093] 12 and 13 are diagrams showing examples of the arrangement of subjects SB within the readout area RA.

[0094] 12 shows an example in which the readout area RA is the entire area of ​​the pixel array unit 11, and the feature detection resolution is constant regardless of the size of the motion area MA. In this example, the range of feature detection is wide, resulting in a large amount of calculation. Furthermore, it is necessary to train images of subjects SB of various sizes and positions, which increases the network size of the mathematical model.

[0095] 13 shows an example in which the readout area RA is limited to a portion of the pixel array unit 11, and the feature detection resolution is changed according to the size of the motion area MA. In this example, the range of feature detection is limited to the readout area RA. Since the amount of calculation required for feature detection is reduced, power consumption is reduced.

[0096] In addition, in the example of FIG. 13 , the positions of the subjects SB are generally aligned near the center of the readout area RA. This reduces the variation in learning data, reducing the number of feature maps and CNN circuits and achieving low power consumption. For example, if the positions of the subjects SB are varied, it is necessary to train images in which the subjects SB are positioned in various positions. If the position of the subjects SB is limited to the center, it is sufficient to prepare only images for learning in which the subjects SB are located near the center.

[0097] 6. Adjustment of Resolution and Frame Rate FIGS. 14 to 16 are diagrams showing examples of adjustment of resolution and frame rate.

[0098] 14 is a diagram showing an example of row scanning control and column scanning control in motion detection mode. Row scanning refers to scanning of pixels in the row direction (vertical direction) of the pixel array unit 11. Column scanning refers to scanning of pixels in the column direction (horizontal direction) of the pixel array unit 11. In motion detection mode, multiple pixels arranged in the row and column directions are treated as one block, and scanning is performed for each block. In motion detection mode, the charge information of all blocks becomes valid data.

[0099] 15 and 16 are diagrams showing examples of row and column scan control in feature detection mode. Fig. 15 shows an example where the motion area MA is relatively large and therefore the resolution is relatively low (medium resolution). Fig. 16 shows an example where the motion area MA is relatively small and therefore the resolution is relatively high (high resolution).

[0100] In feature detection mode, scanning is performed block by block, as in motion detection mode. However, the block size is smaller than in motion detection mode, and therefore the resolution is higher than in motion detection mode. In feature detection mode, a portion of the image area including the motion area MA is set as a readout area RA. The readout area RA is set as a data area of ​​a fixed size, and dummy data is assigned to blocks other than the motion area MA. The dummy data is not valid data, and only the charge information in the motion area MA is valid data.

[0101] In the example of Fig. 15, the pixel addition unit is large and the sensitivity is high. Therefore, the exposure time (accumulation period) from shutter scanning to readout scanning may be shortened (see the right side of Fig. 15). In the example of Fig. 16, the pixel addition unit is small and the sensitivity is low. Therefore, the accumulation period may be lengthened (see the right side of Fig. 16). For example, the higher the resolution of the image information acquired in feature detection mode, the longer the accumulation period from shutter scanning to readout scanning may be set by the timing control unit 17.

[0102] If the subject SB is close, the distance it moves within the angle of view is long, making it more likely to blur. On the other hand, if the subject SB is farther away, the distance it moves within the angle of view is short, making it less likely to blur. Therefore, the timing control unit 17 can shorten the accumulation period and increase the frame rate as the distance to the subject SB captured in the moving area MA becomes shorter.

[0103] 7. Modifications FIG. 17 is a diagram showing a modification of the processing flow relating to the imaging control method.

[0104] This modified example differs from the example in Fig. 4 in that priority levels for feature detection are set for multiple motion areas MA. Only steps ST0, ST4A, and ST12 differ from the processing flow in Fig. 4. Only the different steps will be described below.

[0105] In this modification, first, in step ST0, the priority of the motion areas MA for which feature detection is to be performed is set. The priority is set by reading information on priority criteria stored in the storage unit 2. For example, the order in which the motion areas MA are read out may be determined based on the size or position of the motion areas MA. The criteria for proceeding to motion detection may be changed depending on the position of the motion area MA (number of frames, less movement in the center, more movement in the periphery, etc.).

[0106] In step ST4A, the region determination unit 18c acquires each of the image regions from which the movement of the subject has been extracted as a movement region MA. The readout region control unit 22 sets a readout region RA for each movement region MA. The readout region control unit 22 assigns a priority to each readout region RA.

[0107] The feature detection unit 19 performs feature detection on each motion area MA in order of priority. The feature detection unit 19 determines whether feature detection has been performed on all motion areas MA in step ST12. If feature detection has been performed on all motion areas MA (step ST12: Yes), the process returns to step ST0. If there is a motion area MA for which feature detection has not been performed (step ST12: No), the process returns to step ST4A, and the processes of steps ST4A to ST8 are repeated until feature detection on all motion areas MA is completed.

[0108] 18 is a diagram illustrating an example of the appearance of an information processing system 1001 according to the present disclosure. As shown in Fig. 18, the information processing system 1001 according to this embodiment is configured as a head-mounted display (HMD). An example of the appearance of the head-mounted display (HMD) according to this embodiment will be described with reference to Fig. 18.

[0109] In this example, the HMD 1001 is composed of an output mechanism unit 1011 and a wearing mechanism unit 1012. The wearing mechanism unit 1012 includes a wearing band 1013 that wraps around the head when worn by the user, securing the device in place. Note that the band does not have to wrap around the head as long as it is secured to the head.

[0110] The output mechanism unit 1011 includes a housing 1014 shaped to cover the left and right eyes when the user wears the HMD 1001, and includes a display panel inside that faces the eyes when worn. The housing 1014 may further include lenses that are positioned between the display panel (display unit 2005 ( FIG. 20 )) and the user's eyes when the HMD 1001 is worn, and that expand the user's field of view. The display panel may be divided into left and right halves, and each of the divided regions may display a stereo image corresponding to the parallax between the eyes, and stereoscopic vision may be realized by such a display.

[0111] The HMD 1001 may further include speakers or earphones at positions corresponding to the user's ears when worn. In this example, the HMD 1001 includes a camera 1015 on the front surface of the housing 1014, which captures a video of the surrounding real space in a field of view corresponding to the user's line of sight.

[0112] The camera 1015 includes, for example, an image sensor such as a CCD (Charge Coupled Device) sensor or a CMOS (Complementary Metal Oxide Semiconductor) sensor, a light detection device such as a distance sensor, and an optical system such as an imaging lens. For example, in FIG. 18 , the camera 1015 is configured as a stereo camera that captures images of the space in front of it from left and right viewpoints corresponding to the user's left and right eyes. Note that the camera 1015 is not limited to this, and may be a monocular camera or a multi-lens camera with three or more lenses. Furthermore, a combination of multiple types of sensors may also be used. For hand tracking applications, the camera 1015 may be configured to capture images of the space below the information processing system. For eye tracking or face tracking applications, the camera 1015 may be configured to capture images of the user's eyes or face.

[0113] The HMD 1001 also includes a sensor 2008 ( FIG. 20 ). The sensor may include at least one of various sensors for deriving the movement, posture, position, etc. of the HMD 1001, such as an acceleration sensor, a gyro sensor, an angular velocity sensor, and a geomagnetic sensor.

[0114] The HMD 1001 may be connected to other processing devices via wireless communication, or may be connected via a wired connection such as a USB (Universal Serial Bus).

[0115] In this case, the HMD 1001 may be configured to run an online application such as a game in which multiple users can participate via a network. In this case, the HMD 1001 performs predetermined processing on the image captured by the camera 1015, and generates and displays a display image within the field of view of the camera 1015.

[0116] The content of the displayed image is not particularly limited, and may vary depending on the functions the user desires from the system and the content of the application that has been started.

[0117] For example, the HMD 1001 may perform some processing on the image captured by the camera 1015, or may superimpose a virtual object that interacts with the image of a real object. Alternatively, the HMD 1001 may render a virtual world in a field of view corresponding to the user's field of view based on the captured image or measurements by a motion sensor included in the sensor group of the HMD 1001.

[0118] Representative examples of these modes include virtual reality (VR), augmented reality (AR), and mixed reality (MR). In addition, a see-through mode (VST: VideoSeeThrough) in which the real world can be seen through the screen of the HMD 1001 may be realized by directly displaying an image captured by the camera 1015 as a display image.

[0119] Fig. 19 is a diagram illustrating an example of the appearance of an information processing system 1101 according to the present disclosure. As shown in Fig. 19, the information processing system 1101 according to this embodiment is configured as a glasses-type HMD.

[0120] The HMD main body 1111 is worn on the user's head when in use. The HMD main body 1111 has a front part 1112, a right temple part 1113 provided on the right side of the front part 1112, a left temple part 1114 provided on the left side of the front part 1112, and a glass part 1115 attached to the underside of the front part 1112. Note that although the glass is shown as a single unit in Fig. 19, it may alternatively have two separate glasses, one for each eye, or may be configured to cover only one eye.

[0121] The display unit 1103 is a see-through type display unit and is provided on the surface of the glass unit 1115. The display unit 1103 performs AR display of a virtual object in accordance with the control of the processing circuit 2001. Note that the display unit 1103 may be a non-see-through type display unit. In this case, AR display is performed by displaying on the display unit 1103 an image in which a virtual object is superimposed on an image currently captured by the camera 1104.

[0122] The camera 1104 includes, for example, a photodetector such as an image sensor, such as a CCD (Charge Coupled Device) sensor or a CMOS (Complemented Metal Oxide Semiconductor) sensor, or a distance sensor, and an optical system such as an imaging lens. The camera 1104 is provided facing outward on the outer surface of the front unit 1112, captures an image of an object in real space, and outputs the image information obtained by the capture to the processing circuit 2001. In FIG. 19 , for example, two cameras 1104 are provided laterally at a predetermined interval on the front unit 1112. Note that the camera 1104 is not limited to this, and may be a monocular camera or a multi-lens camera with three or more lenses. Furthermore, a combination of multiple types of sensors may be used. In hand tracking applications, the camera 1104 may be provided to capture an image of the space below the information processing system. In eye-tracking or face-tracking applications, camera 1104 may be configured to capture images of the user's eyes or face.

[0123] The glasses-type HMD 1101 also includes a sensor 2008 ( FIG. 20 ). The sensor may include at least one of various sensors for deriving the movement, posture, position, etc. of the HMD 1101, such as an acceleration sensor, a gyro sensor, an angular velocity sensor, or a geomagnetic sensor.

[0124] Next, an example of the hardware configuration of the information processing system (HMD 1001 or glasses-type HMD 1101) will be described with reference to Fig. 20. As shown in Fig. 20, the hardware of the information processing system is composed of a processing circuit 2001, a memory 2002, a camera 2003, a display unit 2005, an input unit 2006, an output unit 2007, a sensor 2008, a communication interface (IF) 2009, an external network 2010, and a secondary storage device 2011, which are mutually connected via a bus 2012 and can transmit and receive data and programs.

[0125] The processing circuit 2001 operates based on programs stored in the memory 2002 or the secondary storage device 2011 and controls the overall operation of the information processing systems 1001 and 1101. The processing circuit is, for example, a processor, and realizes functions corresponding to each program by reading and executing each program from the memory 2002. The processor may include, for example, any one or more of a multi-core processor, a controller, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or an equivalent discrete logic circuit or integrated logic circuit. The processing circuit may be realized as multiple chips.

[0126] The memory 2002 may include any type of memory for storing data and executable software instructions, implemented, for example, by random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electronically erasable programmable read-only memory (EEPROM), or semiconductor memory devices such as flash memory, hard disks, optical disks, etc.

[0127] The camera 2003 corresponds to the camera 1015 in FIG. 18 and the camera 1104 in FIG. 19, and includes a photodetector such as an image sensor or distance measuring sensor, such as a CCD (Charge Coupled Device) sensor or a CMOS (Complemented Metal Oxide Semiconductor) sensor, and an optical system such as an imaging lens.

[0128] The display unit 2005 is a display panel provided inside the housing, and is made up of a display device such as an LCD (Liquid Crystal Display) or an organic EL (ElectroLuminescence).

[0129] The input unit 2006, which is not shown in Figures 18 and 19, is composed of input devices such as a keyboard, mouse, touch panel, microphone, and controller through which the user inputs operation commands, and supplies various input signals to the processing circuit 2001.

[0130] The output unit 2007 is composed of an audio output device such as a speaker, a force feedback device, an odor feedback device, etc., and is controlled by the processing circuit 2001 to output the processing results as sound, force feedback, or odor.

[0131] The sensor 2008 may include at least one of various sensors, such as an acceleration sensor, a gyro sensor, an angular velocity sensor, and a geomagnetic sensor, for detecting the movement, posture, and position of the HMDs 1001 and 1101. The sensor 2008 may also include a biosensor that senses a person's biometric information and a pressure sensor that detects input.

[0132] The communication interface 2009 is an interface for connecting the information processing systems 1001 and 1101 to an external network 2010. It communicates with a smartphone or an external device other than a smartphone (for example, a personal computer (PC) or a server device on a network) via a wired or wireless connection. For example, the processing circuit 2001 receives data from other devices and transmits data generated by the processing circuit 2001 to other devices via the communication interface 2009.

[0133] The above describes an example of an information processing system to which the technology according to the present disclosure can be applied. The technology according to the present disclosure can be applied to the camera 2003 in the above-described configuration.

[0134] 1 can be applied to a camera 2003, and more specifically, the image sensor 10 can be applied to the light detection device 2004, and the sensor control unit 15 can be applied to the parts of the camera 2003 excluding the light detection device 2004. The display unit 3 of the image capture device 100 can be applied to the display unit 2005 of an information processing system.

[0135] In addition, at least some of the functions of the sensor control unit 15 may be performed by the processing circuit 2001. For example, as a non-limiting example, the functions of the feature detection unit 19 and the motion detection unit 18 of the sensor control unit 15 may be provided within the processing circuit 2001 in the information processing system, and may be performed by the processing circuit 2001 of the information processing system.

[0136] In this way, by having the processing circuit 2001 perform at least some of the functions of the sensor control unit 15, the size of the camera 2003 can be reduced, the camera can be made less noticeable, and the design freedom of the information processing system can be increased.

[0137] Furthermore, if the processing speed of the processing circuit 2001 of the information processing system is faster than the processing speed of the camera 2003, the speed of the entire information processing system can be increased.

[0138] [9. Effects] The imaging device 100 has a motion extraction unit 18a, a region determination unit 18c, and a mode control unit 16. The motion extraction unit 18a extracts the movement of the subject SB based on image information obtained in the motion detection mode. The region determination unit 18c acquires the image region from which the movement of the subject SB is extracted as a movement region MA. The mode control unit 16 determines the resolution of the image information acquired in the feature detection mode based on the size of the movement region MA. The mode control unit 16 switches to the specific processing mode in response to detection of a specific feature of interest from the image information obtained in the feature detection mode. In the imaging control method of the present disclosure, the processing of the imaging device 100 is executed by a computer. The computer-readable non-transitory storage medium of the present disclosure stores a program that causes a computer to realize the processing of the imaging device 100.

[0139] According to this configuration, the resolution of the feature detection mode is adjusted according to the size of the moving area MA, which makes it possible to reduce power consumption without impairing the accuracy of feature detection.

[0140] The mode control unit 16 increases the resolution of the image information acquired in the feature detection mode as the size of the moving area MA decreases.

[0141] According to this configuration, even if the size of the motion area MA is small, highly accurate feature detection can be performed.

[0142] The mode control unit 16 determines the resolution of the image information acquired in the feature detection mode based on the distance to the subject SB captured in the moving area MA.

[0143] According to this configuration, the characteristics of the distant subject SB can be detected with high accuracy.

[0144] The imaging device 100 has a readout region control unit 22. In the feature detection mode, the readout region control unit 22 sets a partial pixel group included in the pixel array unit 11 as a readout region RA. The readout region control unit 22 selectively reads out image signals of the readout region RA that includes the motion region MA.

[0145] According to this configuration, the range of feature detection is limited to the readout area RA, and the amount of calculation required for feature detection is reduced, resulting in reduced power consumption.

[0146] The readout area control unit 22 sets an image area that includes the movement area MA in the center as the readout area RA.

[0147] This configuration makes it easy to train the mathematical model used for feature detection. For example, if the positions of the subjects SB are varied, it is necessary to train images in which the subjects SB are positioned in various positions. If the motion area MA is fixed to the center of the image, only images in which the subjects SB are positioned in the center can be used for training.

[0148] The readout area control unit 22 defines a number of pixels set according to the resolution as a logical pixel RP, which is the smallest unit of image information. The readout area control unit 22 sets an image area including a predetermined fixed number of logical pixels RP as a readout area RA.

[0149] This configuration allows the size of the mathematical model used for feature detection to be reduced. For example, if the input data size varies, it is necessary to train images of various sizes. Therefore, the size of the mathematical model must be increased to match the maximum size of the training data. If the input data size is constant, the size of the mathematical model can be set small to match the size of the input data.

[0150] The imaging device 100 has a feature detection unit 19. The feature detection unit 19 generates an image in which pixel values ​​in a readout area RA outside the movement area MA are filled with dummy data as a detection target image TG. The feature detection unit 19 detects a specific feature of interest from the detection target image TG.

[0151] This configuration facilitates calculations for feature detection.

[0152] The imaging apparatus 100 includes a timing control unit 17. The timing control unit 17 lengthens the accumulation period from shutter scanning to readout scanning as the resolution of the image information acquired in the feature detection mode increases.

[0153] This configuration compensates for the decrease in sensitivity that accompanies an increase in resolution.

[0154] The timing control unit 17 increases the frame rate as the distance to the subject SB captured in the moving area MA decreases.

[0155] This configuration reduces image blurring caused by the nearby subject SB moving significantly within the angle of view.

[0156] The resolution of the image information acquired in the feature detection mode is higher than the resolution of the image information acquired in the motion detection mode.

[0157] According to this configuration, feature detection can be performed with high accuracy.

[0158] The mode control unit 16 switches to the motion detection mode in response to the fact that a specific feature of interest is not detected in the feature detection mode.

[0159] According to this configuration, motion detection continues until a specific feature of interest is detected.

[0160] The specific processing mode is an imaging mode with a higher resolution than the feature detection mode, or a mode that outputs an alert.

[0161] According to this configuration, in response to the detection of a particular feature of interest, an image is taken for recording or display, or an alert is output.

[0162] The effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0163] [Additional Notes] The present technology may also have the following configurations. (1) An imaging device including: a motion extraction unit that extracts motion of a subject based on image information obtained in a motion detection mode; an area determination unit that acquires an image area from which the motion of the subject is extracted as a motion area; and a mode control unit that determines a resolution of image information to be acquired in a feature detection mode based on a size of the motion area, and switches to a specific processing mode in response to detection of a specific feature of interest from the image information obtained in the feature detection mode. (2) The imaging device described in (1) above, in which the mode control unit increases the resolution of the image information to be acquired in the feature detection mode as the size of the motion area decreases. (3) The imaging device described in (1) or (2) above, in which the mode control unit determines the resolution of the image information to be acquired in the feature detection mode based on a distance to the subject appearing in the motion area. (4) The imaging device according to any one of (1) to (3), further comprising a readout region control unit that, in the feature detection mode, sets a group of pixels included in a pixel array unit as a readout region and selectively reads out image signals of the readout region that includes the movement region. (5) The imaging device according to (4), further comprising: a readout region control unit that sets an image region that includes the movement region in a center thereof as the readout region. (6) The imaging device according to (4) or (5), further comprising: a readout region control unit that defines a number of pixels set according to the resolution as a logical pixel that is the smallest unit of the image information, and sets an image region that includes a predetermined fixed number of the logical pixels as the readout region. (7) The imaging device according to (6), further comprising: a feature detection unit that generates an image in which pixel values ​​of the readout region outside the movement region are filled with dummy data as a detection target image, and detects the specific feature of interest from the detection target image. (8) The imaging device according to any one of (1) to (7) above, further comprising a timing control section that extends the accumulation period from shutter scanning to readout scanning as the resolution of the image information acquired in the feature detection mode increases.(9) The imaging device according to (8) above, wherein the timing control unit increases the frame rate as the distance to the subject captured in the motion area decreases. (10) The imaging device according to any one of (1) to (9) above, wherein the resolution of image information acquired in the feature detection mode is higher than the resolution of image information acquired in the motion detection mode. (11) The imaging device according to any one of (1) to (10) above, wherein the mode control unit switches to the motion detection mode in response to the fact that the specific feature of interest is not detected in the feature detection mode. (12) The imaging device according to any one of (1) to (11) above, wherein the specific processing mode is an imaging mode with a higher resolution than the feature detection mode, or a mode that outputs an alert. (13) An imaging control method executed by a computer, comprising: extracting a motion of a subject based on image information obtained in a motion detection mode, acquiring an image region from which the motion of the subject has been extracted as a motion region, determining a resolution of image information to be acquired in a feature detection mode based on the size of the motion region, and switching to a specific processing mode in response to detection of a specific feature of interest from the image information obtained in the feature detection mode. (14) A computer-readable non-transitory storage medium storing a program that causes a computer to extract a motion of a subject based on image information obtained in a motion detection mode, acquiring an image region from which the motion of the subject has been extracted as a motion region, determining a resolution of image information to be acquired in a feature detection mode based on the size of the motion region, and switching to a specific processing mode in response to detection of a specific feature of interest from the image information obtained in the feature detection mode.

[0164] REFERENCE SIGNS LIST 11 pixel array unit 16 mode control unit 17 timing control unit 18a motion extraction unit 18c area determination unit 19 feature detection unit 22 readout area control unit 100 imaging device MA motion area RA readout area RP logical pixel SB subject TG detection target image

Claims

1. An imaging device comprising: a motion extraction unit that extracts the motion of a subject based on image information obtained in a motion detection mode; a region determination unit that acquires, as a motion region, an image region in which the motion of the subject is extracted; and a mode control unit that determines the resolution of image information to be acquired in a feature detection mode based on the size of the motion region, and performs a switch to a specific processing mode in response to the detection of a specific feature of interest from the image information obtained in the feature detection mode.

2. The imaging device according to claim 1, wherein the mode control unit increases the resolution of the image information to be acquired in the feature detection mode as the size of the motion region becomes smaller.

3. The imaging device according to claim 1, wherein the mode control unit determines the resolution of the image information to be acquired in the feature detection mode based on the distance to the subject captured in the motion region.

4. The imaging device according to claim 1, further comprising a readout region control unit that, in the feature detection mode, sets a partial pixel group included in a pixel array unit as a readout region, and selectively reads out an image signal of the readout region that encloses the motion region.

5. The imaging device according to claim 4, wherein the readout region control unit sets, as the readout region, an image region that encloses the motion region at a central part.

6. The imaging device according to claim 4, wherein the readout region control unit defines a set number of pixels corresponding to the resolution as logical pixels that are the minimum unit of the image information, and sets, as the readout region, an image region that includes a pre-fixed number of the logical pixels.

7. The imaging device according to claim 6, further comprising a feature detection unit that generates, as a detection target image, an image in which pixel values of the readout region outside the motion region are filled with dummy data, and detects a specific feature of interest from the detection target image.

8. The imaging device according to claim 1, further comprising a timing control unit that increases an accumulation period from a shutter scan to a readout scan as the resolution of the image information to be acquired in the feature detection mode becomes higher.

9. The imaging device according to claim 8, wherein the timing control unit increases a frame rate as the distance to the subject captured in the motion region becomes shorter.

10. The imaging device according to claim 1, wherein the resolution of the image information to be acquired in the feature detection mode is higher than the resolution of the image information to be acquired in the motion detection mode.

11. The imaging device according to claim 1, wherein the mode control unit switches to the motion detection mode in response to the fact that a specific feature to be targeted is not detected in the feature detection mode.

12. The imaging device according to claim 1, wherein the specific processing mode is an imaging mode with a higher resolution than the feature detection mode or a mode for outputting an alert.

13. An imaging control method executed by a computer, comprising: extracting the motion of a subject based on the image information obtained in the motion detection mode; acquiring, as a motion area, the image area in which the motion of the subject is extracted; determining the resolution of the image information acquired in the feature detection mode based on the size of the motion area; and switching to the specific processing mode in response to the fact that a specific feature to be targeted is detected from the image information obtained in the feature detection mode.

14. A computer-readable non-transitory storage medium storing a program that causes a computer to: extract the motion of a subject based on the image information obtained in the motion detection mode; acquire, as a motion area, the image area in which the motion of the subject is extracted; determine the resolution of the image information acquired in the feature detection mode based on the size of the motion area; and switch to the specific processing mode in response to the fact that a specific feature to be targeted is detected from the image information obtained in the feature detection mode.

Images