Image sensor, image processing method, and program

Abstract

This image sensor comprises: an image acquisition unit that acquires a plurality of images; a vector calculation unit that calculates vectors representing the movement of a subject from the plurality of images; a vector integration unit that integrates the vectors; and an output unit that outputs one of the plurality of images and information on the integrated vectors.

Description

[Technical Field]

[0001] The present technology relates to an image sensor, an image processing method, and a program, and in particular to a technology for outputting image data on which information is superimposed. [Background technology]

[0002] Image sensors capable of capturing images at a high frame rate are known. For example, a configuration such as that shown in Patent Document 1 can achieve a high frame rate. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2019-022020 Summary of the Invention [Problem to be solved by the invention]

[0004] To perform image processing (such as image recognition processing) on high-frame-rate image data output from an image sensor, a processing unit with high processing capabilities is required, and devices equipped with such a processing unit are likely to be expensive. Furthermore, it is possible to sacrifice the gradation of the image output from the image sensor by adopting equipment with a processing unit that does not have high processing power downstream of the image sensor, but in that case, there is a risk that appropriate image processing will not be possible.

[0005] Therefore, the purpose of this technology is to output information from an image sensor so that image processing similar to that performed when a high frame rate image is output can be performed in a subsequent stage without reducing the gradation of the image output from the image sensor. [Means for solving the problem]

[0006] The image sensor according to the present technology includes an image acquisition unit that acquires a plurality of images, a vector calculation unit that calculates a vector representing the movement of a subject from the plurality of images, a vector integration unit that integrates the plurality of vectors, and an output unit that outputs one of the plurality of images and information about the integrated vector. As a result, for example, information about one of the multiple images and vector information representing the movement of the subject are output.

[0007] The image sensor may further include an effective area setting unit that sets an effective area in the image, and the vector calculation unit may calculate the vector for the effective area. As a result, vectors are calculated for the valid area, and vectors are not calculated for areas other than the valid area.

[0008] The effective area setting unit in the image sensor may set the effective area so that the vector is calculated for each predetermined number of pixels in the image. For example, if an effective area is set to consist of four pixels (two pixels vertically and two pixels horizontally), vectors are calculated for 1 / 4 of the total number of pixels.Also, if an effective area is set to consist of 64 pixels (eight pixels vertically and eight pixels horizontally), vectors are calculated for 1 / 64 of the total number of pixels.

[0009] The effective area setting section in the image sensor may set a partial range area in the image as the effective area. The partial range region is, for example, a region made up of a plurality of continuous pixels, and is, for example, a region in which a specific subject (such as a ball) is captured.

[0010] The valid area setting unit in the image sensor may determine whether or not to continue each valid area as a valid area each time the vector is integrated. For example, when integrating vectors, it is possible to estimate how the effective area has moved, but the calculated vector may not be correct. In such cases, not integrating vectors prevents the accuracy of the integrated vector from decreasing.

[0011] The image sensor may further include a reliability calculation unit that calculates the reliability of the vector. For example, by outputting the reliability together with the vector, it is possible to change how the vector is handled in image recognition processing in a downstream application or device.

[0012] The image sensor may further include a counting unit that counts the number of integrations of the vector, and the reliability calculation unit may calculate the reliability based on the count of the number of integrations. For example, if a decision is made each time a vector is calculated as to whether or not to multiply a newly calculated vector with a previously multiplied vector, it is necessary to decide whether or not the multiplication of the vectors is appropriate. If it is decided not to multiply a vector, not only will the vector not be multiplied, but subsequent vector calculations and multiplications may also not be performed. In this case, the number of vectors multiplied, that is, the number of vectors multiplied, indicates the reliability of the multiplied vectors.

[0013] The effective area setting section in the image sensor may change the effective area based on a predetermined condition. The valid area is the area in which vectors are calculated, but it is expected that the valid area will be changed as appropriate depending on predetermined conditions, allowing appropriate processing to be performed in subsequent applications and devices.

[0014] In the image sensor described above, the predetermined condition may be the magnitude of the vector. Objects with large motion vectors have a wide range of predicted positions after movement, so the position of such objects needs to be identified more carefully.

[0015] The image sensor may further include a base image selection unit that selects the image to be output from the plurality of images as a base image. By selecting the base image, it is possible to selectively output a vector representing a movement before the base image or a vector representing a movement after the base image.

[0016] The base image selection unit in the image sensor described above may select the image captured earliest in time series from the plurality of images as the base image, and the vector accumulation unit may accumulate a vector representing the movement of the subject from the base image to an image captured after the base image. As a result, a vector indicating a certain image data and its subsequent movement is output.

[0017] The base image selection unit in the image sensor described above may select the image among the plurality of images that was captured latest in the time series as the base image, and the vector accumulation unit may accumulate vectors representing the movement of the subject from an image captured after the base image to the base image. As a result, a vector indicating the image data and the preceding motion is output.

[0018] The base image selection unit in the image sensor described above may select an image other than the image captured earliest and latest in the chronological order from among the plurality of images as the base image, and the vector accumulation unit may accumulate vectors representing the movement of the subject from an image captured after the base image to the base image, and accumulate vectors representing the movement of the subject from the base image to an image captured after the base image. As a result, a vector indicating the movement of a certain image data and the movement before and after the certain image data is output.

[0019] The vector calculation section in the image sensor may calculate an acceleration vector from the plurality of images, and the output section may output the acceleration vector. For example, if the subject makes circular movements or undergoes sudden acceleration or deceleration, the subject's movement may not be accurately captured using only a vector indicating the amount of movement.

[0020] The image sensor described above may have a stacked structure including a first layer having a light receiving unit that outputs a pixel signal based on the amount of received light through photoelectric conversion, and a second layer having the image acquisition unit, the vector calculation unit, the vector integration unit, and the output unit. In other words, it is an image sensor that is composed of one chip equipped with each part.

[0021] The image processing method according to the present technology is executed by an image processing device that acquires a plurality of images, calculates a vector representing the movement of a subject from the plurality of images, integrates the plurality of vectors, and outputs one of the plurality of images and information about the integrated vector.

[0022] The program related to the present technology is executed by a computer to acquire multiple images, calculate a vector representing the movement of a subject from the multiple images, add up the multiple vectors, and output information about one image of the multiple images and the added vector. [Brief explanation of the drawings]

[0023] [Figure 1] FIG. 1 illustrates an example of the configuration of an imaging apparatus according to an embodiment. [Figure 2] FIG. 1 is a diagram illustrating an example of the configuration of an image sensor. [Figure 3] FIG. 2 is a diagram illustrating an example of the configuration of a light receiving unit. [Figure 4] 10A and 10B are diagrams illustrating an example in which the density at which target pixels are provided varies for each region in an image. [Figure 5] FIG. 10 is a schematic diagram for explaining the movement of a subject. [Figure 6]10 is an example of a calculated difference vector. [Figure 7] This is an example of a motion vector calculated by accumulating differential vectors. [Figure 8] 10 is an example of a motion vector calculated for each of a plurality of target pixels set in a base image. [Figure 9] This is an example of a case where the calculated difference vector is significantly different from the previous difference vectors. [Figure 10] This is an example in which the difference vectors are not multiplied. [Figure 11] 10 is an example of a base image and a motion vector representing the previous motion. [Figure 12] 10 is an example of a base image and a motion vector representing the movement before and after the base image. [Figure 13] 10 shows an example of a base image, a motion vector, and an acceleration vector. [Figure 14] FIG. 1 is a functional block diagram of an image sensor. [Figure 15] 10 is a flowchart of a first example of processing. [Figure 16] 10 is a flowchart of a second example of the process. [Figure 17] 10 is a flowchart of a third example of the process. [Figure 18] This is an example of target pixels that are uniformly set in the first image. [Figure 19] 10 is an example of target pixels irregularly arranged relative to the base image. DETAILED DESCRIPTION OF THE INVENTION

[0024] The embodiments will be described below in the following order. <1. Configuration of imaging device> 2. Image sensor configuration <3. About motion vectors> <4. Selection of base image> <5.Acceleration Vector> <6. Image sensor control block> <7. Processing example> <7-1. Example 1> <7-2. Second Example> <7-3. Third Example> <8. Variations> <9. Summary> <10. This Technology>

[0025] <1. Configuration of imaging device> The configuration of an imaging device 100 including an image sensor 1 according to an embodiment will be described with reference to FIG. The imaging device 100 includes an optical system 101, an image sensor 1, a camera signal processing unit 102, a recording control unit 103, a communication unit 104, a camera control unit 105, a memory unit 106, and a driver unit 107. Note that the configuration shown in Fig. 1 is merely an example, and some of the components may not be included, or components other than those shown in Fig. 1 may be included.

[0026] The optical system 101 includes lenses such as a zoom lens and a focus lens, an aperture mechanism, etc. The optical system 101 guides light (incident light) from a subject and focuses the light on the image sensor 1.

[0027] The image sensor 1 is configured as, for example, a CMOS (Complementary Metal Oxide Semiconductor) type sensor, receives incident light from an optical system 101, and obtains an image signal through photoelectric conversion. The image sensor 1 calculates a motion vector Vm for each area set on an image, and outputs the motion vector Vm and the image signal to a camera signal processing unit 102 and a camera control unit 105 in the subsequent stage. The motion vector Vm represents the movement of a subject captured in that area. The image sensor 1 includes various units required for performing these outputs. The configuration of the image sensor 1 will be described in detail later with reference to FIG. 2.

[0028] The camera signal processing unit 102 is configured as an image processor, for example, using a DSP (Digital Signal Processor), etc. The camera signal processing unit 102 performs various types of signal processing on the digital signal (image signal) from the image sensor 1. For example, as camera processes, the camera signal processing unit 102 performs preprocessing, synchronization processing, YC generation processing, resolution conversion processing, file formation processing, etc.

[0029] In the pre-processing, the captured image signal from the image sensor 1 is subjected to clamping processing for clamping the R, G, and B black levels to a predetermined level, correction processing between the R, G, and B color channels, and the like. In the synchronization process, a color separation process is performed so that the image data for each pixel contains all the color components R, G, and B. For example, in the case of an image sensor that uses a Bayer color filter, a demosaic process is performed as the color separation process. In the YC generation process, a luminance (Y) signal and a color (C) signal are generated (separated) from R, G, and B image data. In the resolution conversion process, the image data that has been subjected to various signal processes is subjected to the resolution conversion process.

[0030] In the file creation process, for example, image data that has undergone the various processes described above is subjected to compression encoding for recording or communication, formatting, generation and addition of metadata, etc., to create a file for recording or communication. For example, still image files can be generated in formats such as JPEG (Joint Photographic Experts Group), TIFF (Tagged Image File Format), and GIF (Graphics Interchange Format).It is also possible to generate image files in the MP4 format used for recording MPEG-4 compliant video and audio. It is also possible to generate an image file as a RAW file (RAW image data).

[0031] The recording control unit 103 performs recording and reproduction on a recording medium such as a nonvolatile memory, for example, and performs processing to record image files such as moving image data and still image data, thumbnail images, and the like on the recording medium. The recording control unit 103 may take a variety of actual forms. For example, the recording control unit 103 may be configured as a flash memory built into the imaging device 100 and a write / read circuit for the flash memory. The recording control unit 103 may also take the form of a card recording / playback unit that performs recording / playback access to a recording medium that can be attached to or detached from the imaging device 100, such as a memory card (such as a portable flash memory). The recording control unit 103 may also be realized as an HDD (Hard Disk Drive) built into the imaging device 100.

[0032] The communication unit 104 performs data communication and network communication with external devices via wire or wirelessly. For example, the captured image data (still image files and moving image files) is transmitted and output to an external display device, recording device, playback device, or the like. The communication unit 104 also serves as a network communication unit, and can communicate via various networks such as the Internet, a home network, and a LAN (Local Area Network), and can transmit and receive various data to and from servers, terminals, and the like on the network.

[0033] The camera control unit 105 is configured by a microcomputer (arithmetic processing device) equipped with a CPU (Central Processing Unit).

[0034] The memory unit 106 stores information and the like used for processing by the camera control unit 105. The illustrated memory unit 106 comprehensively represents, for example, a read only memory (ROM), a random access memory (RAM), a flash memory, and the like. The memory unit 106 may be a memory area built into a microcomputer chip that serves as the camera control unit 105, or may be configured as a separate memory chip. The camera control unit 105 controls the entire imaging device 100 by executing a program stored in the ROM, flash memory, or the like of the memory unit 106 . For example, the camera control unit 105 controls the shutter speed of the image sensor 1, instructs various signal processing in the camera signal processing unit 102, controls the operation of each necessary part regarding the imaging operation and recording operation in response to user operations, playback operation of recorded image files, operation of the optical system 101 such as zoom, focus, and aperture adjustment in the lens barrel, and user interface operation.

[0035] The RAM in the memory unit 106 is used as a work area for the CPU of the camera control unit 105 to process various data, and is used to temporarily store data, programs, and the like. The ROM and flash memory (non-volatile memory) in memory unit 106 are used to store the OS (Operating System) that the CPU uses to control each part, content files such as image files, application programs for various operations, firmware, various setting information, etc. The various types of setting information include exposure settings, shutter speed settings, and mode settings as setting information related to imaging operations, white balance settings, color settings, and settings related to image effects as setting information related to image processing, and custom key settings and display settings as setting information related to operability.

[0036] The driver unit 107 includes, for example, a motor driver for a zoom lens drive motor, a motor driver for a focus lens drive motor, a motor driver for a diaphragm mechanism motor, and the like. These motor drivers apply drive currents to the corresponding drivers in response to instructions from the camera control unit 105, and cause the focus lens and zoom lens to move, and the diaphragm blades of the diaphragm mechanism to open and close.

[0037] In the imaging device 100, for example, software for the processing of the present disclosure can be installed via network communication by the communication unit 104 or a removable recording medium. Alternatively, the software may be stored in advance in a ROM, memory unit 106, or the like.

[0038] In addition to the above, the imaging device 100 also includes various units necessary for imaging operations, such as a display unit, controls, and a sensor unit.

[0039] 2. Image sensor configuration The configuration of the image sensor 1 will be described with reference to FIG. The image sensor 1 is composed of an imaging processing block 2 and a signal processing block 3.

[0040] The imaging processing block 2 includes, for example, a light receiving section 21, a light receiving processing section 22, an output section 23, and a light receiving control section 24.

[0041] The light receiving section 21 has a pixel configuration arranged in a two-dimensional array. The light receiving section 21 is driven by a light receiving processing section 22. Each pixel receives incident light from the optical system 101 and includes various components such as a PD (Photodiode) 21a for obtaining an image signal through photoelectric conversion, a color filter, a microlens, and a circuit configuration related to readout. The light receiving unit 21 outputs the image signal as an analog signal to a light receiving processing unit 22 at a subsequent stage.

[0042] The light reception processing unit 22 is controlled by a light reception control unit 24. The light reception processing unit 22 executes various processes such as readout control of light reception signals for each pixel row, processing related to readout timing, and reset control of accumulated charges. The light receiving processing unit 22 also performs AD (Analog to Digital) conversion processing to convert the received light signal output from the light receiving unit 21 as an analog signal into a digital signal. The light receiving processing unit 22 outputs an image signal as a digital signal to the output unit and the signal processing block 3 at the subsequent stage.

[0043] The output unit 23 outputs the image signal as a digital signal supplied from the light receiving processing unit 22 to the outside of the image sensor 1. Specifically, the output unit 23 outputs the image signal to the camera signal processing unit 102 and the camera control unit 105 in FIG.

[0044] The output unit 23 is also capable of outputting to each subsequent unit the processed image data and metadata supplied from the signal processing block 3. Various types of metadata are conceivable, but in this example, a motion vector Vm that represents the movement of the subject for each area (effective area, described later) set on the image is taken as an example.

[0045] The light-receiving control unit 24 acquires information such as exposure time and frame rate according to the shooting mode, and controls the light-receiving unit 21 via the light-receiving processing unit 22. The light-receiving control unit 24 also acquires cut-out range information and controls the cut-out process of the light-receiving processing unit 22 by specifying the range to be cut out from the image signal output by the light-receiving unit 21. In order to execute these various processes, the light-receiving control unit 24 performs communication processing to acquire information from outside the image sensor 1. Alternatively, the light-receiving control unit 24 may be able to acquire information necessary for control by utilizing a communication function provided in the signal processing block 3.

[0046] The signal processing block 3 includes a control unit 31, a memory unit 32, a communication unit 33, and a bus .

[0047] The control unit 31 is configured by, for example, a CPU or a DSP, and may include both in some cases. The control unit 31 executes a program stored in the memory unit 32 to perform various processes on the image signal output from the imaging processing block 2 .

[0048] The various processes performed by the control unit 31 include, for example, calculation of a defocus amount. The calculated defocus amount may be output to the driver unit 107 via the light receiving control unit 24 and the camera control unit 105, or may be output to the driver unit 107 via the communication unit 33 and the camera control unit 105.

[0049] Furthermore, the control unit 31 in this example processes the image signal output from the light receiving processing unit 22, thereby performing the above-mentioned processes such as calculation of the motion vector Vm, integration processes, and the like. This information is output as metadata from the output unit 23 to the outside. The various processes for the motion vector Vm will be described in detail later.

[0050] The memory unit 32 is configured with SRAM (Static Random Access Memory), DRAM (Dynamic RAM), etc., and stores programs executed by the control unit 31 and information used by the control unit 31 for processing (e.g., image signals output from the light receiving processing unit 22).

[0051] The communication unit 33 is capable of executing, for example, a process of receiving a program executed by the control unit 31, a process of outputting a processing result of the control unit 31, and the like.

[0052] The bus 34 is provided to interconnect the control unit 31, memory unit 32, and communication unit 33. Furthermore, an image signal from the light receiving processing unit 22 of the imaging processing block 2 can be output to the control unit 31 and memory unit 32 via the bus 34. The processing result of the control unit 31 can be output to the output unit 23 of the imaging processing block 2 via the bus 34.

[0053] The image sensor 1 is, for example, a so-called one-chip complementary metal oxide semiconductor (COMS) type image sensor having a first layer formed as an upper chip and a second layer formed as a lower chip. This makes it possible to prevent the image sensor 1 from becoming larger.

[0054] The image sensor 1 does not necessarily have to be configured from one chip, but may be configured from multiple chips.

[0055] For example, a light receiving unit 21 is arranged on the first layer, and for example, a light receiving processing unit 22, an output unit 23, a light receiving control unit 24, a control unit 31, a memory unit 32, and a communication unit 33 are arranged on the second layer. The first and second layers are electrically connected by one or more control lines, making it possible to adopt the configuration shown in FIG.

[0056] In the image sensor 1 of this example, the frame rate of the image signal as an analog signal output from the light receiving unit 21 is different from the frame rate of the image signal output from the output unit 23. Specifically, the frame rate of the image signal output from the light receiving unit 21 is set to be higher frequency than the frame rate of the image signal output from the output unit 23. For example, the frame rate of the image signal output from the light receiving unit 21 is set to 1000 fps, and the frame rate of the image signal output from the output unit 23 is set to 30 fps. These figures are merely examples, and various configurations are possible, such as the light receiving unit 21 being set to 60 fps and the output unit 23 being set to 30 fps.

[0057] <3. About motion vectors> The control unit 31 of the signal processing block 3 of the image sensor 1 processes the image signal output from the light receiving processing unit 22 to calculate the motion vector Vm. Here, the motion vector Vm will be described.

[0058] The motion vector Vm is information on the direction of movement and the amount of movement that represents the movement of a subject captured at a certain pixel. The motion vector Vm is calculated for each pixel, for example. The pixels for which the motion vector Vm is calculated may be all pixels included in the light receiving unit 21, or may be one pixel selected for each predetermined region. When the motion vector Vm is calculated for one pixel selected for each predetermined region, the motion vector Vm is regarded as information representative of the movement of the subject captured in the predetermined region.

[0059] Some examples of the predetermined region will be described below.

[0060] 3 shows an example of the light receiving unit 21, in which each pixel has a color filter in a Bayer array. In the configuration shown in FIG. 3, it is conceivable to calculate one motion vector Vm for each region 41 consisting of two G (Green) pixels, one B (Blue) pixel, and one R (Red) pixel.

[0061] When calculating one motion vector Vm from the region 41, the motion vector Vm may be calculated based on one color signal (e.g., R signal). In this case, the motion vector Vm may be calculated based on the pixel signal of a pixel equipped with one color filter (e.g., R) so that demosaicing is not required.

[0062] According to this method, the motion vector Vm is calculated for 25% of all pixels, which means that the amount of calculation is less than when the motion vector Vm is calculated for all pixels. In the following description, the pixel for which the motion vector Vm is calculated will be referred to as the "target pixel 42."

[0063] Another example of the predetermined area is shown in Fig. 4. Fig. 4 shows an image of a soccer game, but it is conceivable that a partial range area within the entire image area is designated as the predetermined area. For example, suppose that a pixel region in which a soccer ball is captured is designated as a region 41 for calculating a motion vector Vm. In this case, for example, one pixel selected from the region 41 (for example, a pixel near the center of the region) is set as a target pixel 42 to calculate a motion vector Vm, and motion vectors Vm are not calculated for pixels in other regions.

[0064] Alternatively, for the soccer ball imaging area, which is the predetermined area, one target pixel 42 may be set for every four pixels to calculate the motion vector Vm, as described above, and for other areas, one target pixel 42 may be set for every 64 pixels, which is eight pixels vertically and horizontally, to calculate the motion vector Vm (see FIG. 4). In this case, the motion vector Vm is calculated more densely for the predetermined region (the region where the soccer ball is captured) than for other regions (for example, the background region).

[0065] When changing the predetermined area depending on the subject appearing in the image, the predetermined area may be determined based on information for identifying the subject. For example, if a soccer ball is white, the predetermined area is set to a white pixel area, thereby setting the soccer ball as the predetermined area. Alternatively, if the object is a tennis ball, the predetermined area may be set to a yellow pixel area. The predetermined area may also be determined based on information about the shape and size.

[0066] The motion vector Vm is calculated by adding (accumulating) a plurality of pieces of vector information. In the following description, a case will be considered in which image data captured at 1000 fps is output from the output unit 23 at 30 fps.

[0067] In this case, one image is output from approximately 33 images. As a result, the output image is information that has been thinned out in the time axis direction, but instead, the motion vector Vm calculated from the 33 images is output as metadata to the outside of the image sensor 1. In other words, instead of outputting the 32 images that were not selected as images to be output, the motion vector Vm is output from the image sensor 1 as metadata.

[0068] By outputting the motion vector Vm from the image sensor 1, it becomes possible to estimate the position of an object with higher accuracy in subsequent processing than when only the selected image is output. Furthermore, the amount of output data can be reduced compared to outputting all 33 images.

[0069] Here, a method for calculating the motion vector Vm from 33 images will be described.

[0070] As described above, one motion vector Vm is calculated for one target pixel 42. The motion vector Vm is calculated by detecting the movement of the subject imaged at the target pixel 42.

[0071] When calculating the motion vector Vm, first, information about the motion of the subject captured at the target pixel 42 is calculated based on two images. The information about the motion of the subject calculated from the two images can be expressed as a vector. In the following description, the vector that represents the difference in the position information of the subject calculated from the two images will be referred to as the "difference vector Vd."

[0072] The difference vector Vd can be calculated using, for example, the gradient method (LK method) or the block matching method.

[0073] The motion vector Vm is obtained by accumulating the difference vectors Vd. When outputting image data at 30 fps from image data captured at 1000 fps, one motion vector Vm is calculated by accumulating 32 difference vectors Vd for one output image.

[0074] A specific description will be given with reference to FIGS. 5 to 10. FIG. 5 is a schematic diagram for explaining the movement of a subject based on two images. As shown in the figure, when a subject captured at target pixel 42a in a first image is subsequently captured at target pixel 42b in a second image, the difference vector Vd is calculated as a vector as shown in Fig. 6. The calculated difference vector Vd is referred to as difference vector Vd1.

[0075] Similarly, if the difference vector Vd calculated from the second and third images is called the difference vector Vd2, and the difference vector Vd calculated from the third and fourth images is called the difference vector Vd4, the motion vector Vm will be as shown in Figure 7.

[0076] When such a motion vector Vm is calculated for each target pixel 42 evenly distributed across the entire image, metadata such as that shown in Fig. 8 is generated. As shown in the figure, information on the motion vector Vm is superimposed and displayed for each target pixel 42. Note that the information on the motion vector Vm as metadata output from the image sensor 1 only needs to be linked to information on the direction and magnitude of the vector as information on the motion vector Vm and information on the corresponding pixel position, and does not need to be superimposed on the image.

[0077] Here, although an example of the motion vector Vm obtained by accumulating all the calculated difference vectors Vd is shown in FIGS. 7 and 8, it is not necessarily necessary to accumulate all the difference vectors Vd. For example, if the calculated difference vector Vd is abnormal data, the difference vector may not be added.

[0078] In particular, in image data captured at 1000 fps or the like, the capture time of two adjacent images in the time axis direction is very short, so it is assumed that the difference vectors Vd adjacent in the time axis direction are similar. Specifically, as shown in Fig. 9, the difference vector Vd1 calculated from the first and second images and the difference vector Vd2 calculated from the second and third images are considered to be similar in direction and magnitude. Similarly, the difference vector Vd2 calculated from the second and third images and the difference vector Vd3 calculated from the third and fourth images are considered to be similar.

[0079] However, if the difference vector Vd4 calculated from the fourth and fifth images is significantly different from the difference vector Vd3, there is a high possibility that the subject detected in the fifth image is an erroneous detection. In such a case, the motion vector Vm may be calculated without accumulating the difference vector Vd4, as shown in Fig. 10. Furthermore, subsequent accumulation of difference vectors is not performed.

[0080] In such a case, information is added to indicate that the motion vector Vm was calculated using only four images, in other words, that the motion vector Vm was calculated by accumulating only three difference vectors Vd, and the motion vector Vm is output as metadata from the image sensor 1.

[0081] Therefore, for example, when a motion vector Vm is calculated for each target pixel 42 and output to a downstream processing unit or application, information on the direction and magnitude of the motion vector Vm, the number of difference vectors Vd accumulated to calculate the motion vector Vm (or the number of images used), and information on the pixel position of the target pixel 42 are linked and output. Hereinafter, the number of accumulated difference vectors Vd will be referred to as the "accumulation number." Alternatively, instead of the magnitude information of the motion vector Vm, a numerical value obtained by dividing the magnitude of the motion vector Vm by the number of accumulated difference vectors Vd may be output. In other words, the motion vector Vm may be normalized and output. This simplifies the processing executed in the subsequent stages and reduces the processing load.

[0082] Although an example in which the calculated difference vector Vd4 is not integrated has been described in FIG. 10, various types of difference vectors Vd may be determined not to be integrated.

[0083] For example, if the inner product of two difference vectors Vd adjacent in the time axis direction (e.g., difference vectors Vd3 and Vd4) is equal to or greater than a threshold, it may be determined not to perform the integration. This prevents an inappropriate difference vector Vd calculated based on an erroneously recognized subject from being integrated when the magnitude of the difference vector Vd becomes significantly large, as shown in FIG.

[0084] Alternatively, if the difference in magnitude or direction between two difference vectors Vd adjacent in the time axis direction is greater than a threshold, it may be determined that the integration is not performed.

[0085] The threshold used to determine whether to integrate the difference vector Vd may be variable depending on the shooting environment. For example, the threshold may be changed depending on whether the shooting environment is bright or dark, or whether the noise level is high or low. Specifically, the threshold is changed so that the likelihood of determining not to integrate the difference vector Vd is reduced in a shooting environment with high noise. Also, the threshold is changed so that the likelihood of determining not to integrate the difference vector Vd is increased in a shooting environment with low noise. This allows the determination process as to whether or not to integrate the difference vector Vd to be performed appropriately according to the shooting environment.

[0086] In the above example, a determination as to whether or not to accumulate the difference vector Vd is made based on two difference vectors Vd adjacent to each other in the time axis direction. However, the determination may also be made based on three difference vectors Vd adjacent to each other in the time axis direction.

[0087] 10, for example, the difference vector Vd4 is significantly different from the difference vector Vd3, so the decision on whether to accumulate the difference vector Vd4 is put on hold. If the difference vector Vd5 calculated from the fifth and sixth images is similar to the difference vector Vd4, it is highly likely that the subject's movement has actually changed significantly, so it is decided to continue accumulating the difference vectors Vd. On the other hand, if the difference vector Vd5 is significantly different from the difference vector Vd4, it is determined that there is a high possibility that the subject has not been correctly recognized, and the difference vectors Vd4 and Vd5 are not accumulated, and the subsequent difference vectors Vd are not accumulated either, and the vector calculated by accumulating only the difference vectors Vd1, Vd2, and Vd3 is used as the motion vector Vm.

[0088] Alternatively, it may be determined whether or not to perform the integration based on the motion vector Vm of the target pixel 42 in the vicinity on the image plane. For example, even if the difference vector Vd4 is significantly different from the difference vector Vd3, if a similar change is observed in the nearby target pixel 42, it may be determined that the subject movement has been correctly recognized, and the accumulation of the difference vector Vd may continue.

[0089] Alternatively, whether or not to add up the difference vector Vd may be determined depending on the magnitude of the difference vector Vd. For example, if the difference vector Vd is large, it is estimated that the subject has moved significantly in a short period of time. In a scene where such movement is not realistic, it may be determined that the difference vector Vd is to be integrated if the difference vector Vd is smaller than a certain threshold value. Such thresholds may also vary depending on the type of sport, for example. For example, the threshold may be set to a larger value when the subject is a motorsport vehicle than when the subject is a marathon runner, since the subject is more likely to move significantly. In this way, by appropriately changing the threshold value depending on the type of subject, the shooting environment, the scene, or the type of event, it is possible to appropriately estimate the direction and amount of movement of the subject.

[0090] The above-mentioned accumulated number can be treated as the reliability of the motion vector Vm. For example, if the difference vector Vd1 and the difference vector Vd2 are significantly different and the motion vector Vm is calculated using only the difference vector Vd1, the cumulative number will be 1. In this case, it is not clear whether the motion vector Vm was calculated as a result of the difference vector Vd1 correctly recognizing the subject, and therefore the reliability of the motion vector Vm is estimated to be low.

[0091] On the other hand, the motion vector Vm obtained by adding 32 difference vectors Vd, from difference vector Vd1 to difference vector Vd32, means that similar difference vectors Vd have been calculated across 33 images. Therefore, it is estimated that the subject captured in target pixel 42 is accurately linked to the motion vector Vm. Therefore, such a motion vector Vm is considered to be highly reliable.

[0092] That is, the number of accumulations of the difference vector Vd can be treated as the reliability of the motion vector Vm. Information on the reliability of the motion vector Vm is output from the image sensor 1 as metadata.

[0093] While the difference vector Vd for the target pixel 42 is being accumulated, the target pixel 42 is set as a valid pixel, and the area 41 represented by the valid pixels is set as a valid area. 3, if one target pixel 42 is set for each region 41 of two pixels vertically and horizontally, when the first difference vector Vd is calculated from 33 images, all of the target pixels 42 are considered to be valid pixels. Therefore, the regions 41 for each target pixel 42 are all considered to be valid regions.

[0094] As the accumulation of the difference vector Vd progresses, some target pixels 42 will not be accumulated with the difference vector Vd. These target pixels 42 will no longer be treated as valid pixels. Therefore, the area 41 for each such target pixel 42 will also be excluded from the valid area. That is, as the accumulation of the difference vector Vd proceeds, the effective area decreases, and therefore the processing load related to the calculation and accumulation of the difference vector Vd is reduced.

[0095] <4. Selection of base image> For example, when 1000 fps image data is output as 30 fps image data, the image output from the image sensor 1 is one image selected from 33 images. This selected image is called the "base image." There are various possibilities for selecting an image from the 33 images as the base image.

[0096] 8 shows an example in which the first captured image of 33 images is selected as the base image. That is, the motion vector Vm as metadata is data representing the subsequent movement of the subject captured in the output image. This is suitable for later applications when processing is required based on the subject captured in the image and its subsequent movement.

[0097] Alternatively, the last image captured among the 33 images may be selected as the base image and output from the image sensor 1. In this case, the image sensor 1 outputs a motion vector Vm as metadata as shown in FIG. That is, the motion vector Vm shown in FIG. 11 is data representing the previous motion of the subject captured in the output base image. This is suitable for later applications when processing is required based on the subject captured in the image and its preceding movement.

[0098] Furthermore, the middle image (for example, the 17th image) of the 33 images may be selected as the base image and output from the image sensor 1. In this case, the image sensor 1 outputs the motion vectors Vmb and Vma as metadata as shown in FIG. Here, the motion vector Vmb is the previous motion vector of the subject captured in the base image, and the motion vector Vma is the subsequent motion vector of the subject captured in the base image, i.e., data representing the before-and-after motion of the subject in the base image. This is suitable for later applications when processing is required based on the subject captured in the image and its movements before and after. Furthermore, if an occlusion occurs in a subject, the forward and backward movement of the image is expressed, making it possible to grasp the direction of movement of the subject. Therefore, appropriate image recognition processing can be performed in subsequent processing.

[0099] <5.Acceleration Vector> In the above example, the difference vector Vd continues to be accumulated for a subject moving in a substantially uniform linear motion while a series of images (for example, 33 images) is being captured. However, for example, in the case of objects such as a wheel that rotates at high speed, an object that moves in a parabolic trajectory, or a hand that is rotating, such as when rotating an arm around a shoulder, even if these are set as areas to be calculated for the motion vector Vm, the direction of the difference vector Vd may change in a short period of time, and therefore the difference vector Vd may not be accumulated, and it may not be possible to calculate a reliable motion vector Vm (or motion vectors Vmb, Vma, etc.).

[0100] Therefore, an example in which the acceleration vector Va is output as metadata in addition to the motion vector Vm will be described.

[0101] Specifically, an image in which the subject is a vehicle, as shown in FIG. 13, will be used for explanation. For the target pixel 42 set in the body portion of the vehicle, the motion vector Vmb and the motion vector Vma are set to be substantially the same vector, and the magnitude of the acceleration vector Va is set to be zero.

[0102] On the other hand, for the target pixel 42 set in the wheel portion of the vehicle, the motion vector Vmb indicating the movement before the timing of capturing the image and the motion vector Vma indicating the movement after the timing of capturing the image are vectors that are approximately the same in magnitude but different in direction. In FIG. 13, the acceleration vector Va calculated from the change in the motion vectors Vmb and Vma is indicated by a dashed arrow.

[0103] Calculating the acceleration vector Va allows subsequent applications to more accurately grasp the movement of the subject, making it possible to omit image recognition processing as appropriate and reducing the processing load. This is because if the movement of the subject can be accurately grasped, it becomes possible to identify the area in which the subject appears without performing image recognition processing.

[0104] Furthermore, by using the information on the acceleration vector Va in subsequent applications, it is possible to correctly recognize that the movement of the subject before and after the output image (base image) is circular, thereby enabling accurate display, for example, when displaying the movement of the subject.

[0105] Specifically, when a circle drawn by a subject moving their hand in the air is to be accurately reproduced from the movement of the subject's arm (or finger) in a subsequent application, by using information on the acceleration vector Va, a smooth, non-polygonal circle can be faithfully reproduced.

[0106] <6. Image sensor control block> FIG. 14 shows a control block of the image sensor 1 for realizing the various examples described above. In the image sensor 1, for example, a functional configuration as shown in FIG. 14 is constructed in the control unit 31 of the signal processing block 3 by a program or the like.

[0107] The image sensor 1 includes, for example, an image acquisition unit 51, a base image selection unit 52, a vector calculation unit 53, a vector integration unit 54, a valid area setting unit 55, a counting unit 56, and a reliability calculation unit 57.

[0108] The image acquisition unit 51 acquires, for example, images captured at 1000 fps from the light reception processing unit 22.

[0109] The base image selection unit 52 performs a process of selecting an image to be output from the image sensor 1. For example, in the above example, one image is selected from 33 images in order to output images at 30 fps. As a selection example, the first image captured among the 33 images may be selected as the base image, the last image captured may be selected as the base image, or any other image may be selected as the base image.

[0110] The vector calculation unit 53 calculates a difference vector Vd from, for example, two images. In calculating the difference vector Vd, various methods such as a gradient method or a block matching method can be used. Note that information about the valid area is input to the vector calculation unit 53. Therefore, while the valid area exists, the difference vector Vd is calculated, but if all the target pixels 42 are excluded from the valid area, the difference vector Vd is not calculated.

[0111] The vector accumulator 54 determines whether or not to accumulate the calculated difference vector Vd, and appropriately accumulates the difference vector Vd with the motion vector Vm up to that point. As a result, one image and the motion vector Vm are output from the image sensor 1 every 33 images. The vector accumulator 54 is provided with information about the maximum number of accumulations of the difference vector Vd (32 in the above example). Therefore, the vector accumulator 54 accumulates the difference vector Vd until the number of accumulations of the difference vector Vd reaches the maximum value.

[0112] The valid area setting unit 55 manages the target pixel 42 (or its area 41) for which the difference vector Vd is to be accumulated. That is, if the calculated difference vector Vd is estimated to be noise or an abnormal value, the calculation of the difference vector Vd is stopped and the pixel is excluded from the valid area, thereby managing the valid area. The valid area can be managed, for example, by using a valid flag that indicates whether or not the pixel is a valid area.

[0113] The counting unit 56 manages the number of accumulations of the difference vector Vd. The number of accumulations of the difference vector Vd can be used as a measure of the reliability of the motion vector Vm. The number of accumulations can also be used to normalize the motion vector Vm.

[0114] The reliability calculation unit 57 calculates the reliability of the motion vector Vm using the above-mentioned integrated number, etc. The calculated reliability is output from the image sensor 1 as metadata. The reliability of the motion vector Vm can be determined by other means than the number of accumulations of the difference vector Vd. For example, when the difference vector Vd is calculated using the gradient method, it is conceivable to calculate the reliability of the motion vector Vm using an error value. Furthermore, when the difference vector Vd is calculated using the block matching method, it is conceivable to calculate the reliability of the motion vector Vm using the degree of match for each block.

[0115] <7. Processing example> The processes executed in the signal processing block 3 of the image sensor 1 to realize the various functions described above will be described with reference to FIGS.

[0116] <7-1. Example 1> The first example will be described with reference to FIG. The first example is a case where, for example, the first image (first image) or the last image (33 images in the above example) among a plurality of images is predetermined as the base image. That is, this is a case where only one of the motion vector Vm representing the movement before the base image and the motion vector Vm representing the movement after the base image is output. That is, this is a case where the motion vector Vm shown in FIG. 8 or FIG. 11 is output.

[0117] The series of processes shown in Fig. 15 are executed to output one base image. Therefore, when outputting a base image at 30 fps, the series of processes shown in Fig. 15 are executed 30 times per second. This also applies to Figs. 16 and 17, which will be described later.

[0118] In step S101, the image sensor 1 reads an image at an initial point in time. This is, for example, processing for reading image data of the first of 33 images.

[0119] Next, in step S102, the image sensor 1 performs an initial setting process for the valid area of the read image. In the initial setting process for the valid area, a target pixel 42 is set for each predetermined area 41, and the valid flag is set to ON.

[0120] As described above, the predetermined area 41 may be set uniformly in the image area by setting it every few pixels, or it may be set based on a subject identified by a specified color, shape, or size.

[0121] In the process of step S102, the valid areas are set, and the target pixels 42 for calculating the motion vector Vm for each valid area are also set.

[0122] In step S103, the image sensor 1 reads the next image. For example, when the process of step S103 is executed for the first time, the image data of the second sheet is read.

[0123] Next, in step S104, the image sensor 1 calculates the difference vector Vd for the target pixel 42 set for each valid area. As a result, the difference vector Vd is calculated based on, for example, the first image data and the second image data.

[0124] In step S105, the image sensor 1 calculates whether the difference vector calculated in the immediately preceding step S104 is similar to the difference vector Vd calculated previously (i.e., the difference vector Vd calculated in the immediately preceding step S104), i.e., whether the vector difference is equal to or smaller than a certain value, and sets an area where the difference is equal to or smaller than the certain value as a valid area. In other words, areas where the difference vectors Vd are not similar are excluded from the valid area.

[0125] Note that when the process of step S105 is executed for the first time after the process of step S102 is executed, the process of step S105 is omitted because the difference vector Vd calculated in the previous step S104 does not exist.

[0126] In step S106, the image sensor 1 accumulates the difference vector Vd for each effective area. That is, the accumulation of the difference vector Vd is not performed for areas excluded from the effective area in the previous step S105.

[0127] In step S107, the image sensor 1 updates the reliability information. For example, if information on the integrated number is used as the reliability information, the reliability information is updated by updating the integrated number.

[0128] In step S108, the image sensor 1 determines whether or not an effective area exists. If all areas are excluded from the effective area in step S105, there is no point in repeatedly executing the processes from step S103 to step S107, and the process proceeds to step S110.

[0129] On the other hand, if the valid area still exists, the image sensor 1 determines in step S109 whether the number of integrations has reached the maximum number. In the above example, since there are 32 difference vectors Vd calculated from 33 images, the accumulation process is performed a maximum of 32 times. When the number of accumulations reaches 32, the determination in step S109 is "Yes," and the process proceeds to step S110.

[0130] If the number of accumulations has not reached the maximum number, the image sensor 1 executes the processes from step S103 to step S107 to calculate the difference vector Vd and accumulate the motion vector Vm, and then executes the determination processes of step S108 and step S109.

[0131] Then, when it is no longer necessary to perform the processes from step S103 to step S107, the image sensor 1 outputs the data and metadata of the base image to each subsequent unit in step S110. The metadata output includes information specifying the position of the target pixel 42 in the base image, the motion vector Vm, the number of integrations, and reliability information.

[0132] The output process of step S110 may be timed so that image data is output at regular intervals from the image sensor 1. That is, for example, if the valid area soon disappears, the process of step S110 may be executed after a certain waiting time has elapsed.

[0133] <7-2. Second Example> The second example will be described with reference to FIG. The second example is an example in which a base image can be selected from a plurality of images (33 images in the above example). That is, there may be cases in which both a motion vector Vmb representing the movement before the base image and a motion vector Vma representing the movement after the base image are output. That is, this example also corresponds to the case in which the motion vectors Vmb and Vma shown in FIG. 12 are output.

[0134] Of the processes shown in FIG. 16, detailed description of the processes already explained using FIG. 15 will be omitted to avoid duplication.

[0135] In step S121, the image sensor 1 acquires the number of the image to be output, that is, information for specifying the base image.

[0136] In step S122, the image sensor 1 sets a motion vector to be integrated. Specifically, the image sensor 1 selects either the motion vector Vmb representing the motion before the base image or the motion vector Vma representing the motion after the base image. If the base image is the first image, the motion vector Vma is selected as the motion vector to be integrated, otherwise the motion vector Vmb is selected.

[0137] Next, the image sensor 1 executes the processes of steps S101 and S102 to read an image at the initial point in time and to initialize the effective area.

[0138] Next, the image sensor 1 executes the processes of steps S103 to S107 to calculate the difference vector Vd, set the valid area, update the motion vector Vm by accumulating the difference vector Vd, and update the reliability information.

[0139] In steps S108 and S109, the image sensor 1 performs a determination process as to whether or not each process is to be repeatedly executed.

[0140] If it is determined that a valid area exists and the accumulated number has not reached the maximum, that is, if it is determined that the processes from step S103 to step S107 need to be executed again, the image sensor 1 determines in step S123 whether the last image read is the image to be output. That is, this is the process of determining whether the last image read is the base image.

[0141] If it is determined that the image is the base image, the calculation of the motion vector Vmb is completed. In this case, the image sensor 1 proceeds to step S124 and performs processing to switch the motion vector to be integrated from the motion vector Vmb to the motion vector Vma. The subsequent accumulation process in step S106 is performed on the motion vector Vma. Furthermore, when the reliability information in step S107 is calculated for each of the motion vectors Vmb and Vma, the reliability information is also updated for the motion vector Vma.

[0142] Then, when it is no longer necessary to execute the processes from step S103 to step S107, the image sensor 1 outputs the data and metadata of the origin image to each subsequent unit in step S110.

[0143] <7-3. Third Example> The third example will be described with reference to FIG. The third example is an example in which not only the motion vector Vm but also the acceleration vector Va is output. Note that the example shown in Fig. 17 shows a case in which the acceleration vector Va is additionally output in addition to the first example shown in Fig. 15, but by combining it with the second example shown in Fig. 16, it is also possible to output the motion vectors Vmb, Vma and the acceleration vector Va as shown in Fig. 13.

[0144] Of the processes shown in FIG. 17, detailed description of the processes already explained using FIG. 15 will be omitted to avoid duplication.

[0145] The image sensor 1 executes the processes of steps S101 and S102 to read an image at the initial time point and to initialize the effective area.

[0146] Next, the image sensor 1 executes the processes of steps S103 to S107 to calculate the difference vector Vd, set the valid area, update the motion vector Vm by accumulating the difference vector Vd, and update the reliability information.

[0147] Next, in step S141, the image sensor 1 calculates the acceleration vector Va. Note that in the first step S141 after the process of step S102 is executed, only one difference vector Vd is calculated, so the acceleration vector Va is not calculated. In other words, the process of step S141 is not executed.

[0148] In steps S108 and S109, the image sensor 1 performs a determination process as to whether or not each process is to be repeatedly executed. If it is determined that it is not necessary to execute each process, the image sensor 1 outputs the data and metadata of the base image to each subsequent unit in step S110.

[0149] The process of step S141 may be executed at any timing after the calculation of the difference vector Vd in step S104 is completed and before the determination process of step S108 is executed.

[0150] Furthermore, the process of step S141 may be executed only once or multiple times during the execution of the series of processes shown in FIG. When step S141 is executed multiple times, multiple acceleration vectors Va are calculated for one target pixel 42. In this case, the information on the acceleration vector Va output to a subsequent stage in step S110 may be all the calculated acceleration vectors Va, or may be the average or median of the calculated acceleration vectors Va. Alternatively, an acceleration vector Va calculated from a difference vector Vd calculated near the base image may be output.

[0151] <8. Variations> In the above example, the target pixel 42 is selected for each predetermined pixel, but the target pixels 42 selected at this time may all have the same color filter. Furthermore, when calculating the difference vector Vd, the pixel to which the subject has moved may be identified from among pixels having the same color filter. This eliminates the need for demosaicing when using a gradient method or block matching method, thereby reducing the processing load.

[0152] Furthermore, if a white (W) pixel is provided at regular intervals, the target pixel 42 may be selected from the white pixels that are set to have high sensitivity. This makes it possible to calculate the difference vector Vd with high accuracy.

[0153] In the third processing example, an example was described in which the acceleration vector Va is calculated along with the motion vector Vm and output as metadata, but it is also possible to calculate and output only the acceleration vector Va without calculating the motion vector Vm (i.e., accumulating the difference vector Vd). In the subsequent stage, when only the acceleration vector Va is required, it is possible to reduce the processing load on the image sensor 1. Also, it becomes possible to output only the acceleration vector Va at an even higher fps.

[0154] 4, an example has been described in which a pixel region in which a soccer ball is captured is selected as the predetermined region. That is, an example has been described in which a motion vector Vm is calculated only for the soccer ball region, and an example in which high-density information is output only for that region. Such a predetermined area may be changeable during the execution of the series of processes shown in FIGS. For example, when a series of processes is performed on 33 images, since the first to twentieth images do not show a soccer ball, a target pixel 42 is provided at each predetermined pixel, thereby calculating the motion vector Vm evenly across the entire image, and since the 21st to 33rd images show a soccer ball as the subject in the images, the target pixels 42 may be densely set to the soccer ball imaging area as the predetermined area.

[0155] In this case, in the series of processes shown in Figures 15, 16 and 17, after the image reading process in step S103, it is determined whether or not there is an area in the image that contains a subject that meets the conditions, and if there is, the valid area is reset.

[0156] There are several possible conditions for detecting a subject that will cause the effective area to be reset, such as whether an image area containing a subject of a predetermined color, shape, or size exists, or whether a subject whose moving speed (for example, the magnitude of the difference vector Vd or the magnitude of the motion vector Vm) is equal to or greater than a predetermined value exists.

[0157] In the above examples, the target pixels 42 are arranged evenly or regularly on the output base image, but other cases are also possible. For example, if the valid area (target pixels 42) set by the processing of step S102 in Figure 16 is evenly arranged on the image (see Figure 18), the target pixels 42 on the base image output from the image sensor 1 may become irregular as shown in Figure 19.

[0158] <9. Summary> As explained in each of the above examples, the image sensor 1 includes an image acquisition unit 51 that acquires multiple images, a vector calculation unit 53 that calculates a vector (difference vector Vd) representing the movement of the subject from the multiple images, a vector accumulation unit 54 that accumulates the multiple vectors, and an output unit 23 that outputs information about one of the multiple images and the accumulated vector (motion vector Vm). As a result, for example, information on one of the multiple images and information on a vector (motion vector Vm) representing the movement of the subject for each pixel are output. Therefore, the position of an object (subject) can be estimated with higher accuracy than when only image data (base image data) is output. For example, by thinning images captured at 1000 fps and outputting them at 30 fps, downstream image recognition applications or devices do not need to be configured for high-frame-rate image processing. Then, by converting the thinned-out image information into information called a motion vector Vm and outputting it, the position of a subject (such as a person or object) can be identified with high accuracy, just as if image recognition processing were performed on each high-frame-rate image. In other words, after performing image recognition processing on one image output at 30 fps, the results of the image recognition processing can be carried over to the following few frames. This enables image recognition with low latency, thereby speeding up processing. Furthermore, by outputting thinned image data, the amount of output data can be reduced.

[0159] As explained in the control block of the image sensor 1 (see FIG. 14), the image sensor 1 may include an effective area setting unit 55 that sets an effective area (target pixels 42) in an image. In this case, the vector calculation unit 53 may calculate the vector (difference vector Vd) for the valid area. As a result, the difference vector Vd is calculated for the valid area, and the difference vector Vd is not calculated for areas other than the valid area. Therefore, the processing load for calculating the difference vector Vd is reduced.

[0160] As explained with reference to FIG. 3 in the explanation of the motion vector, the valid area setting unit 55 may set the valid area (target pixels 42) so that a vector (motion vector Vm) is calculated for each predetermined number of pixels in the image. For example, if an effective area is set to consist of four pixels, two by two, the motion vector Vm is calculated for 1 / 4 of the total pixels.Also, if an effective area is set to consist of 64 pixels, eight by eight, the motion vector Vm is calculated for 1 / 64 of the total pixels. This makes it possible to significantly reduce the processing load required to calculate the motion vector Vm.

[0161] As explained with reference to FIG. 4 in the description of the motion vector, the valid area setting unit 55 may set a partial range area in the image as the valid area. The partial range region is, for example, a region made up of a plurality of continuous pixels, and is, for example, a region in which a specific subject (such as a ball) is captured. By setting such a partial range area as the valid area, it becomes possible to calculate the motion vector Vm for a specific subject. Also, by setting the area for calculating the motion vector Vm small, it is possible to reduce the processing load on the image sensor 1 (or the processing load on each subsequent section) and to calculate and integrate vectors densely in the time axis direction. This configuration is suitable for cases where only some of the subjects move quickly. That is, by setting only the area where a fast-moving subject (such as a ball in a sporting event or an athlete in a track and field event) is captured as the valid area for calculating the motion vector Vm, it is possible to output information about the valid area densely. Outputting information about the valid area densely may mean, for example, information dense in the time axis direction or information dense in the planar direction (for example, calculating the motion vector Vm for all pixels in the valid area).

[0162] As explained in step S105 of FIG. 15 and the like, the valid area setting unit 55 may determine whether or not to continue each valid area as a valid area each time a vector (difference vector Vd) is integrated. For example, when accumulating the difference vector Vd, it is conceivable to estimate how the effective area has moved, but the calculated difference vector Vd may not be correct. In such a case, by performing a continuation determination for each valid area, it is possible to prevent an inappropriate accumulation of the difference vector Vd.

[0163] As explained in the control block of the image sensor 1 (see FIG. 14), the image sensor 1 may include a reliability calculation unit 57 that calculates the reliability of a vector (motion vector Vm). For example, by outputting the reliability together with the motion vector Vm, it is possible to change how the motion vector Vm is handled in the image recognition process in a subsequent application or device. This allows downstream applications and devices to perform appropriate processing, such as not using information about motion vectors Vm with low reliability.

[0164] As explained in the control block of the image sensor 1 (see FIG. 14), the image sensor 1 may include a counting unit 56 that counts the number of integrations of the vector (difference vector Vd). In this case, the reliability calculation unit 57 may calculate the reliability based on the count of the integrated number. For example, if a determination is made each time a difference vector Vd is calculated as to whether or not to multiply a newly calculated difference vector Vd with a motion vector Vm, it is necessary to determine whether or not the multiplication of the difference vector Vd is appropriate. If it is determined that the difference vector Vd should not be multiplied, not only will the difference vector Vd not be multiplied, but subsequent calculations and multiplications of the difference vector Vd may also not be performed. In this case, the number of accumulated difference vectors Vd, that is, the accumulated number of difference vectors Vd, indicates the reliability of the accumulated motion vector Vm. According to this configuration, by calculating the reliability of the motion vector Vm based on the accumulated number of the difference vector Vd, it becomes possible to execute appropriate processing in, for example, a downstream application or device. In particular, calculating the accumulated number for each valid area assigns the reliability of the motion vector Vm for each object. In other words, it is possible to determine whether the estimated position of the object is highly reliable. This makes it possible to execute various processes in the downstream application or device. For example, consider the case where a foreign object flies into the path of a drone while it is flying. In this case, the reliability of the object's (subject's) motion vector Vm is taken into consideration, and if the reliability is high, maneuvering is performed to avoid the object. On the other hand, if the reliability of the motion vector Vm is low, no maneuvering is performed to avoid the object, and instead, observation accuracy can be improved by increasing the frequency of acquiring the object's motion vector Vm. By using the integrated number as the reliability, the reliability can be easily calculated, and an increase in the calculation cost for calculating the reliability can be prevented.

[0165] As explained in the modified example, the valid area setting section 55 may change the valid area (or the target pixel 42) based on a predetermined condition. The valid area is the area for which the motion vector Vm is calculated, but it is expected that the valid area will be changed as appropriate depending on predetermined conditions, allowing appropriate processing to be performed in subsequent applications and devices. For example, if a foreign object flies into the path of a drone while it is flying, it is possible to limit the effective area to the area where the foreign object is captured in order to calculate the vector of the object with higher accuracy. If, as a result of calculating the motion vector Vm of the object with high accuracy, it is determined in processing subsequent to the image sensor 1 that the object may collide with the drone, it is possible to have the drone take evasive action. That is, by providing a configuration that allows the effective area to be changed as needed, it becomes possible to appropriately perform subsequent processing.

[0166] As explained in the modified example, the predetermined condition may be the magnitude of the vector (motion vector Vm or difference vector Vd). An object with a large motion vector Vm or difference vector Vd has a wide range of predicted positions after movement, and the position of such an object may need to be specified more carefully. Therefore, for example, when there is an object whose motion vector Vm or difference vector Vd is large, it is conceivable to calculate vectors spatially and temporally densely for the imaging region of the object. By performing such processing, processing to deal with unexpected subject movements can be executed in subsequent applications or devices. The predetermined condition can be specified not only by the magnitude of the vector but also by the color, size, shape, etc. of the subject. For example, if yellow is specified to obtain a dense motion vector Vm for a yellow ball, when the ball is outside the field of view, the motion vector Vm can be calculated evenly across all pixel areas, and when the ball enters the field of view, the pixel area in which the ball is imaged can be set as the valid area.

[0167] As explained in the control block of the image sensor 1 (see FIG. 14), the image sensor 1 may include a base image selection unit 52 that selects an image to be output from a plurality of images as a base image. By selecting the base image, it is possible to selectively output a motion vector Vmb that represents a motion before the base image or a motion vector Vma that represents a motion after the base image. This makes it possible to output appropriate data according to downstream applications and devices.

[0168] As explained with reference to FIG. 8 regarding the selection of the base image, the base image selection unit 52 may select, as the base image, the image captured earliest in time series among the plurality of images. In this case, the vector accumulating section 54 may accumulate a difference vector Vd that represents the movement of the subject from the base image to an image captured after the base image. As a result, a motion vector Vm indicating the image data and the subsequent motion is output. Therefore, it is possible to appropriately perform processing according to the subject captured in the image and its subsequent movement.

[0169] As explained with reference to FIG. 11 regarding the selection of the base image, the base image selection unit 52 may select, as the base image, the image captured latest in time series among the plurality of images. In this case, the vector accumulation section 54 may accumulate a difference vector Vd that represents the movement of the subject from an image captured after the base image to the base image. As a result, a motion vector Vm indicating the motion of a certain image data and the preceding image data is output. Therefore, it is possible to appropriately perform processing according to the subject captured in the image and the movement leading up to that subject.

[0170] As explained with reference to Figure 12 regarding the selection of the base image, the base image selection unit 52 may select as the base image an image other than the image captured earliest and latest in the time series among the multiple images. In this case, the vector accumulation unit 54 may accumulate a difference vector Vd representing the movement of the subject from an image captured after the base image to the base image, and accumulate a difference vector Vd representing the movement of the subject from the base image to an image captured after the base image. As a result, a motion vector Vm indicating the motion of a certain image data and its preceding and following points is output. Therefore, for example, if an image captured at exactly the midpoint of a plurality of images is selected as the base image, the magnitude of the vector will be small, making it possible to calculate the forward and backward movement with high accuracy. Also, in an imaging environment where occlusion occurs, there is a high possibility that at least one of the vectors representing the subject's immediate movement and the vector representing the subject's immediate movement will be output, increasing the possibility of properly calculating the subject's movement direction, speed, etc.

[0171] As described with respect to the acceleration vector with reference to FIG. 13, the vector calculation unit 53 may calculate the acceleration vector Va from a plurality of images, and the output unit 23 may output the acceleration vector Va. For example, when a subject makes circular motion or undergoes sudden acceleration or deceleration, the subject's movement may not be accurately captured using only the motion vector Vm that indicates the amount of movement. According to this configuration, the acceleration vector Va is output, so that the movement of the subject can be estimated more accurately, and appropriate processing can be performed in subsequent applications or devices. Furthermore, since the movement of the subject can be accurately captured, it is possible to reduce the number of times the image recognition process is executed.

[0172] As explained in the configuration of the image sensor, the image sensor 1 may have a stacked structure including a first layer having a light receiving section 21 that outputs a pixel signal based on the amount of received light through photoelectric conversion, and a second layer having an image acquisition section 51, a vector calculation section 53, a vector integration section 54, and an output section 23. In other words, it is an image sensor that is composed of one chip equipped with each part. This makes it possible to add to the image sensor 1 a function for calculating and integrating the difference vector Vd, while preventing the image sensor 1 from becoming large.

[0173] The image processing method performed by the image sensor 1 involves acquiring multiple images, calculating a vector (difference vector Vd) representing the movement of the subject from the multiple images, accumulating the multiple difference vectors Vd, and outputting information about one of the multiple images and the motion vector Vm.

[0174] The program executed by the control unit 31 of the image sensor 1 acquires multiple images, calculates a vector (difference vector Vd) representing the movement of the subject from the multiple images, accumulates the multiple difference vectors Vd, and outputs information about one of the multiple images and the motion vector Vm.

[0175] The effects described in this specification are merely examples and are not limiting, and other effects may also be present. Furthermore, the above examples can be combined in any way as long as the combination is not impossible.

[0176] <10. This Technology> This technology can also be configured as follows. (1) an image acquisition unit that acquires a plurality of images; a vector calculation unit that calculates a vector representing a movement of a subject from the plurality of images; a vector multiplication unit that multiplies a plurality of the vectors; an output unit that outputs one image out of the plurality of images and information about the integrated vectors; Image sensor. (2) an effective area setting unit that sets an effective area in the image; The vector calculation unit calculates the vector for the effective area. The image sensor according to (1) above. (3) The valid area setting unit sets the valid area so that the vector is calculated for each predetermined pixel in the image. The image sensor according to (2) above. (4) The valid area setting unit sets a partial range area in the image as the valid area. The image sensor according to (2) above. (5) The valid area setting unit determines whether or not to continue each valid area as a valid area each time the vector is integrated. The image sensor according to any one of (2) to (4) above. (6) A reliability calculation unit for calculating the reliability of the vector The image sensor according to any one of (1) to (5) above. (7) a counting unit that counts the number of integrations of the vector; The reliability calculation unit calculates the reliability based on the count of the integrated number. The image sensor according to (6) above. (8) The effective area setting unit changes the effective area based on a predetermined condition. The image sensor according to any one of (2) to (5) above. (9) The predetermined condition is the magnitude of the vector. The image sensor according to (8) above. (10) a base image selection unit that selects the image to be output from the plurality of images as a base image; The image sensor according to any one of (1) to (9) above. (11) the base image selection unit selects an image captured earliest in time series from among the plurality of images as the base image; The vector accumulator accumulates a vector representing a movement of the subject from the base image to an image captured after the base image. The image sensor according to (10) above. (12) the base image selection unit selects, as the base image, an image captured latest in time series from among the plurality of images; The vector accumulator accumulates vectors representing the movement of the subject from an image captured after the base image to the base image. The image sensor according to (10) above. (13) the base image selection unit selects, as the base image, an image other than the image captured earliest and latest in time series from among the plurality of images; The vector accumulator accumulates vectors representing the movement of the subject from an image captured after the base image to the base image, and accumulates vectors representing the movement of the subject from the base image to an image captured after the base image. The image sensor according to (10) above. (14) the vector calculation unit calculates an acceleration vector from the plurality of images; The output unit outputs the acceleration vector. The image sensor according to any one of (1) to (13) above. (15) The laminated structure includes a first layer having a light receiving unit that outputs a pixel signal based on the amount of received light by photoelectric conversion, and a second layer having the image acquisition unit, the vector calculation unit, the vector integration unit, and the output unit. The image sensor according to any one of (1) to (14) above. (16) Acquire multiple images, calculating a vector representing the movement of the subject from the plurality of images; multiplying a plurality of said vectors; One of the plurality of images and information on the integrated vector are output. An image processing method executed by an image processing device. (17) Acquire multiple images, calculating a vector representing the movement of the subject from the plurality of images; multiplying a plurality of said vectors; One of the plurality of images and information on the integrated vector are output. A program that a computer runs. [Explanation of symbols]

[0177] 1 Image sensor 21 Light receiving part 23 Output section 31 Control Unit 42, 42a, 42b Target pixels 51 Image acquisition unit 52 Base image selection section 53 Vector calculation unit 54 Vector multiplication unit 55 Effective area setting section 56 Counting section 57 Reliability calculation unit Vm, Vmb, Vma motion vectors Vd, Vd1, Vd2, Vd3, Vd4, Vd5, Vd32 difference vectors Va acceleration vector

Claims

1. an image acquisition unit that acquires a plurality of images; a vector calculation unit that calculates a vector representing a movement of a subject from the plurality of images; a vector multiplication unit that multiplies a plurality of the vectors; an output unit that outputs one image of the plurality of images and information about the integrated vector; a reliability calculation unit that calculates the reliability of the integrated vector; a counting unit that counts the number of integrations of the vector, The reliability calculation unit uses the count of the integrated number as the reliability. Image sensor.

2. an effective area setting unit that sets an effective area in the image; The vector calculation unit calculates the vector for the effective area. The image sensor of claim 1 .

3. The valid area setting unit sets the valid area so that the vector is calculated for each predetermined pixel in the image.

3. The image sensor according to claim 2.

4. The valid area setting unit sets a partial range area in the image as the valid area.

3. The image sensor according to claim 2.

5. The valid area setting unit determines whether or not to continue each valid area as a valid area each time the vector is integrated.

3. The image sensor according to claim 2.

6. The effective area setting unit changes the effective area based on a predetermined condition.

3. The image sensor according to claim 2.

7. The predetermined condition is the magnitude of the vector.

7. The image sensor according to claim 6.

8. a base image selection unit that selects the image to be output from the plurality of images as a base image; The image sensor of claim 1 .

9. the base image selection unit selects an image captured earliest in time series from among the plurality of images as the base image; The vector accumulator accumulates a vector representing a movement of the subject from the base image to an image captured after the base image.

9. The image sensor according to claim 8.

10. the base image selection unit selects, as the base image, an image captured latest in time series from among the plurality of images; The vector accumulator accumulates vectors representing the movement of the subject from an image captured before the base image to the base image.

9. The image sensor according to claim 8.

11. the base image selection unit selects, as the base image, an image other than the image captured earliest and latest in time series from among the plurality of images; The vector accumulator accumulates vectors representing the movement of the subject from an image captured before the base image to the base image, and accumulates vectors representing the movement of the subject from the base image to an image captured after the base image.

9. The image sensor according to claim 8.

12. the vector calculation unit calculates an acceleration vector from the plurality of images; The output unit outputs the acceleration vector. The image sensor of claim 1 .

13. The laminated structure includes a first layer having a light receiving unit that outputs a pixel signal based on the amount of received light by photoelectric conversion, and a second layer having the image acquisition unit, the vector calculation unit, the vector integration unit, and the output unit. The image sensor of claim 1 .

14. Acquire multiple images, calculating a vector representing the movement of the subject from the plurality of images; multiplying a plurality of said vectors; Counting the number of integrations of the vector; outputting information on one of the plurality of images and the integrated vector; The count of the number of integrations is used as the reliability of the integrated vector. An image processing method executed by an image processing device.

15. Acquire multiple images, calculating a vector representing the movement of the subject from the plurality of images; multiplying a plurality of said vectors; Counting the number of integrations of the vector; outputting information on one of the plurality of images and the integrated vector; The count of the number of integrations is used as the reliability of the integrated vector. A program that a computer runs.

Images