Information processing device, information processing method, and information processing program
The information processing device improves image recognition efficiency by subsampling high-resolution images and integrating feature amounts across frames, enabling simultaneous and accurate detection of distant objects.
Patent Information
- Application Number
- JP2022511953
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2020-03-31
- Filing Date
- 2021-03-22
- Publication Date
- 2024-11-13
- Estimated Expiration
- 2041-03-22
AI Technical Summary
Conventional image recognition techniques using high-resolution images require significant computational resources, making it difficult to achieve simultaneous recognition processing.
An information processing device that sets different pixel positions for acquiring sampling pixels in time-series imaging information, performs subsampling on high-resolution images, and integrates feature amounts across frames to improve recognition processing efficiency.
Enhances the simultaneity of recognition results while utilizing the resolution of high-resolution images, allowing for rapid and accurate detection of distant objects.
Smart Images

Figure 0007582302000001 
Figure 0007582302000002 
Figure 0007582302000003
Abstract
Description
[Technical field]
[0001] The present disclosure relates to an information processing device, an information processing method, and an information processing program. [Background technology]
[0002] In recent years, with the increasing resolution of imaging devices such as digital still cameras, digital video cameras, and compact cameras mounted on multi-function mobile phones (smartphones), information processing devices equipped with an image recognition function for recognizing specific objects contained in captured images have been developed. [Prior art documents] [Patent documents]
[0003] [Patent Document 1] JP 2017-112409 A Summary of the Invention [Problem to be solved by the invention]
[0004] In the image recognition function, it is possible to improve the detection performance of an object by using a captured image with a higher resolution. However, in the conventional technology, image recognition using a captured image with a higher resolution requires a large amount of calculations related to the image recognition process, and it is difficult to improve the concurrency of the recognition process for the captured image.
[0005] An object of the present disclosure is to provide an information processing device, an information processing method, and an information processing program capable of improving the characteristics of recognition processing using captured images. [Means for solving the problem]
[0006] The information processing device according to the present disclosure includes a setting unit that sets pixel positions for acquiring sampling pixels for each divided area into which imaging information composed of pixels is divided, a calculation unit that calculates features of a sampling image composed of the sampling pixels, and a recognition unit that performs recognition processing based on the features of the sampling image and outputs a result of the recognition processing, and the setting unit sets different pixel positions between first imaging information and second imaging information acquired following the first imaging information in the chronological order among the imaging information. [Brief description of the drawings]
[0007] [Figure 1] FIG. 2 is a block diagram showing an example of a basic configuration of an information processing device applicable to each embodiment. [Figure 2A] FIG. 1 is a diagram illustrating an example of recognition processing using a DNN. [Figure 2B] FIG. 1 is a diagram illustrating an example of recognition processing using a DNN. [Figure 3A] FIG. 1 is a diagram illustrating a first example of a classification process using a DNN when time-series information is used. [Figure 3B] FIG. 1 is a diagram illustrating a first example of a classification process using a DNN when time-series information is used. [Figure 4A] FIG. 13 is a diagram illustrating a second example of a classification process using a DNN when time-series information is used. [Figure 4B] FIG. 13 is a diagram illustrating a second example of a classification process using a DNN when time-series information is used. [Diagram 5] FIG. 2 is a block diagram illustrating an example of a hardware configuration of an imaging apparatus as an information processing apparatus applicable to each embodiment. [Figure 6A] FIG. 13 is a diagram showing an example in which an imaging section is generated by a stacked CIS having a two-layer structure. [Figure 6B] FIG. 12 is a diagram showing an example in which the imaging unit 1200 is generated using a stacked CIS having a three-layer structure. [Figure 7] FIG. 2 is a block diagram showing an example of a configuration of an imaging unit applicable to each embodiment. [Figure 8A]FIG. 11 is a diagram for explaining the resolution of an image used in the recognition process. [Figure 8B] FIG. 11 is a diagram for explaining the resolution of an image used in the recognition process. [Figure 9] 1 is a block diagram showing a configuration of an example of an information processing device according to a first embodiment of the present disclosure. [Figure 10] FIG. 2 is a schematic diagram for explaining the recognition process according to the first embodiment. [Figure 11] FIG. 4 is a schematic diagram for explaining a sampling process according to the first embodiment. [Figure 12A] FIG. 2 is a diagram for more specifically explaining the recognition process according to the first embodiment. [Figure 12B] FIG. 2 is a diagram for more specifically explaining the recognition process according to the first embodiment. [Figure 12C] FIG. 2 is a diagram for more specifically explaining the recognition process according to the first embodiment. [Figure 12D] FIG. 2 is a diagram for more specifically explaining the recognition process according to the first embodiment. [Figure 12E] FIG. 2 is a diagram for more specifically explaining the recognition process according to the first embodiment. [Figure 13A] FIG. 4 is a schematic diagram for explaining a subsampling process in the recognition process according to the first embodiment. [Figure 13B] FIG. 4 is a schematic diagram for explaining a subsampling process in the recognition process according to the first embodiment. [Figure 14] 5 is an example time chart showing an example of control of a readout and recognition process according to the first embodiment. [Figure 15] FIG. 2 is a functional block diagram illustrating an example of functions of a preprocessing unit and a recognition unit according to the first embodiment in more detail. [Figure 16] FIG. 1 is a schematic diagram for explaining a first example of a method for reducing the amount of calculations according to existing technology. [Figure 17] FIG. 11 is a schematic diagram for explaining a second example of a method for reducing the amount of calculation according to the existing technology. [Figure 18] 10A to 10C are schematic diagrams showing examples in which 1 / N×1 / N reduction, 1 / N×1 / N cropping, and the recognition process according to the first embodiment are compared with the recognition process using the original image as is. [Figure 19A] FIG. 1 is a diagram illustrating a recognition process using a reduced image according to existing technology. [Figure 19B] FIG. 2 is a diagram illustrating an outline of a recognition process according to the first embodiment. [Figure 20] 10 is a time chart showing an example of control of readout and recognition processing in a case where image data of a captured image is used without being reduced according to existing technology. [Figure 21] 11 is a time chart showing an example of control of readout and recognition processing in the case of "1 / N x 1 / N reduction" according to existing technology. [Figure 22] 4 is a schematic diagram for explaining a delay in a recognition result according to the first embodiment and a delay in a recognition result according to the existing technology; FIG. [Figure 23A] FIG. 2 is a schematic diagram for generally explaining the relationship between the recognition process and the braking of the vehicle. [Figure 23B] FIG. 2 is a schematic diagram for generally explaining the relationship between the recognition process and the braking of the vehicle. [Figure 24] 1A to 1C are diagrams for explaining an example of an improvement made to the recognition process of the existing technology by the recognition process according to the first embodiment; [Figure 25A] FIG. 2 is a schematic diagram showing a first example of an effective application scene of the recognition processing according to the first embodiment. [Figure 25B] FIG. 4 is a schematic diagram showing a second example of an effective application scene of the recognition process according to the first embodiment. [Figure 26A] 11 is a schematic diagram showing an example of performing subsampling using sampling pixels selected by thinning out every other pixel in a divided region in both the row and column directions. FIG. [Figure 26B] 13 is a schematic diagram showing an example of subsampling using sampling pixels selected by thinning out every third pixel in the row and column directions of a divided region. FIG. [Figure 27]FIG. 13 is a schematic diagram for explaining sub-sampling according to a fourth modified example of the first embodiment. [Figure 28] FIG. 13 is a schematic diagram for explaining sub-sampling according to a fifth modified example of the first embodiment. [Figure 29] FIG. 13 is a schematic diagram showing an example of exposure control according to a sixth modified example of the first embodiment. [Diagram 30] FIG. 11 is a block diagram showing a configuration of an example of an information processing device according to a second embodiment. [Diagram 31] 10 is a time chart illustrating an example of control of a readout and recognition process according to a second embodiment. [Diagram 32] FIG. 11 is a schematic diagram for explaining a first application example of the second embodiment. [Diagram 33] FIG. 11 is a schematic diagram for explaining a second application example of the second embodiment. [Diagram 34] FIG. 11 is a schematic diagram for explaining a third application example of the second embodiment. [Diagram 35] FIG. 13 is a block diagram showing a configuration of an example of an information processing device according to a third embodiment. [Diagram 36] FIG. 13 is a schematic diagram for explaining a subsampling process in the recognition process according to the fourth embodiment. [Figure 37] FIG. 1 is a diagram showing an example of use of an information processing device according to a first embodiment and each of its modified examples, a second embodiment, a third embodiment, and a fourth embodiment. [Figure 38] 1 is a block diagram showing an example of a schematic configuration of a vehicle control system; [Figure 39] 4 is an explanatory diagram showing an example of the installation positions of an outside-vehicle information detection unit and an imaging unit; FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0008] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In the following embodiments, the same components are denoted by the same reference numerals, and duplicated descriptions will be omitted.
[0009] Hereinafter, embodiments of the present disclosure will be described in the following order. 1. Technologies Applicable to Each Embodiment 1-0. Overview of Recognition Processing Applicable to Each Embodiment 1-1. Hardware configuration applicable to each embodiment 1-1-1. Configuration examples of an imaging unit applicable to each embodiment 1-1-2. Resolution of captured images 2. First embodiment 2-1. Configuration according to the first embodiment 2-1-1. Overview of configuration applicable to the first embodiment 2-1-2. Example of Recognition Processing According to the First Embodiment 2-1-3. Subsampling Process According to the First Embodiment 2-1-4. More specific configuration example according to the first embodiment 2-2. Advantages of the First Embodiment 2-2-1. Overview of existing technologies 2-2-2. Comparison of the disclosed technology with existing technologies 2-2-2-1.Comparing recognition results 2-2-2-2. Comparison of recognition processing control 2-2-2-3. Comparison of recognition processing delays 2-2-2-4. Example of Improvement by Recognition Processing According to the First Embodiment 2-2-2-5. Effective application scenes of the recognition processing according to the first embodiment 2-3. Modification of the first embodiment 2-3-1. First modified example 2-3-2. Second modified example 2-3-3.Third modified example 2-3-4.Fourth modified example 2-3-5. Fifth modified example 2-3-6. Sixth Modification 2-3-7. Seventh Variation 3. Second embodiment 3-1. Configuration example according to the second embodiment 3-2. Application example of the second embodiment 3-2-1. First application example 3-2-2. Second application example 3-2-3.Third application example 4. Third embodiment 5. Fourth embodiment 6. Fifth embodiment 6-1. Application examples of the technology disclosed herein 6-2. Application examples for moving objects
[0010] [1. Technologies Applicable to Each Embodiment] First, for ease of understanding, a brief description will be given of techniques applicable to each embodiment.
[0011] (1-0. Overview of Recognition Processing Applicable to Each Embodiment) Fig. 1 is a block diagram showing a basic configuration example of an information processing device applicable to each embodiment. In Fig. 1, the information processing device 1a includes a sensor unit 10a and a recognition processing unit 20a. Although not shown in the figure, the sensor unit 10a includes an imaging means (camera) and an imaging control unit that controls the imaging means.
[0012] The sensor unit 10a captures images under the control of the imaging control unit, and supplies image data of the captured images acquired by imaging to the recognition processing unit 20a. The recognition processing unit 20a performs recognition processing on the image data using a deep neural network (DNN). More specifically, the recognition processing unit 20a includes a recognition model that has been learned in advance using predetermined teacher data by machine learning, and performs recognition processing using the DNN on the image data supplied from the sensor unit 10a based on the recognition model. The recognition processing unit 20a outputs the recognition result obtained by the recognition processing to, for example, the outside of the information processing device 1a.
[0013] 2A and 2B are diagrams that show an example of recognition processing by a DNN. In this example, as shown in FIG. 2A, one image is input to the DNN. In the DNN, recognition processing is performed on the input image, and the recognition result is output.
[0014] The process of FIG. 2A will be described in more detail with reference to FIG. 2B. As shown in FIG. 2B, the DNN executes feature extraction processing and recognition processing. In the DNN, feature amounts are extracted from an input image by feature extraction processing. This feature extraction processing is performed, for example, using a CNN (Convolutional Neural Network) of the DNN. In addition, in the DNN, recognition processing is performed on the extracted feature amounts to obtain a recognition result.
[0015] In the DNN, the recognition process can be performed using time-series information. FIG. 3A and FIG. 3B are diagrams that show a first example of the recognition process by the DNN when the time-series information is used. In the examples of FIG. 3A and FIG. 3B, the recognition process by the DNN is performed using a fixed number of past information on the time series. In the example of FIG. 3A, an image [T] at time T, an image [T-1] at time T-1 before time T, and an image [T-2] at time T-2 before time T-1 are input to the DNN. In the DNN, the recognition process is performed on each of the input images [T], [T-1], and [T-2], and the recognition result [T] at time T is obtained.
[0016] FIG. 3B is a diagram for explaining the process of FIG. 3A in more detail. As shown in FIG. 3B, in the DNN, the feature extraction process described with reference to FIG. 2B above is executed one-to-one for each of the input images [T], [T-1], and [T-2], and features corresponding to the images [T], [T-1], and [T-2] are extracted. In the DNN, the features obtained based on these images [T], [T-1], and [T-2] are integrated, and a discrimination process is executed on the integrated features to obtain a recognition result [T] at time T. The features obtained based on the images [T], [T-1], and [T-2] can be said to be intermediate data for obtaining the integrated features used in the recognition process.
[0017] 4A and 4B are diagrams that show a second example of a recognition process by a DNN when time-series information is used. In the example of Fig. 4A, an image [T] at time T is input to a DNN whose internal state has been updated to a state at time T-1, and a recognition result [T] at time T is obtained.
[0018] FIG. 4B is a diagram for explaining the process of FIG. 4A in more detail. As shown in FIG. 4B, in the DNN, the feature extraction process described above with reference to FIG. 2B is executed for the image [T] input at time T, and the feature corresponding to the image [T] is extracted. In the DNN, the internal state is updated by the image before time T, and the feature related to the updated internal state is stored. The feature related to the stored internal information and the feature in the image [T] are integrated, and the identification process is executed on the integrated feature. In this case, the feature related to the stored internal information and the feature in the image [T] can be said to be intermediate data for obtaining the integrated feature used in the recognition process.
[0019] The classification process shown in Figures 4A and 4B is executed using a DNN whose internal state has been updated using the immediately preceding recognition result, and is a recursive process. A DNN that performs recursive processing in this way is called an RNN (Recurrent Neural Network). Classification processing using an RNN is generally used for video image recognition, and it is possible to improve classification accuracy by sequentially updating the internal state of the DNN using frame images that are updated in a time series, for example.
[0020] (1-1. Hardware Configuration Applicable to Each Embodiment) Fig. 5 is a block diagram showing an example of a hardware configuration of an information processing device applicable to each embodiment. In Fig. 5, the information processing device 1 includes an imaging unit 1200, a memory 1202, a DSP (Digital Signal Processor) 1203, an interface (I / F) 1204, a CPU (Central Processing Unit) 1205, a ROM (Read Only Memory) 1206, and a RAM (Random Access Memory) 1207, which are communicably connected to each other via a bus 1210. The information processing device 1 may further include an input device for receiving user operations, a display device for displaying information to the user, and a storage device for non-volatilely storing data.
[0021] The CPU 1205 operates using the RAM 1207 as a work memory according to a program pre-stored in the ROM 1206, and controls the overall operation of the information processing device 1. The interface 1204 communicates with the outside of the information processing device 1 by wired or wireless communication. For example, when the information processing device 1 is used for in-vehicle applications, the information processing device 1 can communicate with a braking control system of the vehicle in which the information processing device 1 is mounted, via the interface 1204.
[0022] The imaging unit 1200 captures moving images at a predetermined frame period and outputs pixel data for forming frame images. More specifically, the imaging unit 1200 includes a plurality of photoelectric conversion elements that convert received light into pixel signals, which are electrical signals, by photoelectric conversion, and a drive circuit that drives each photoelectric conversion element. In the imaging unit 1200, the plurality of photoelectric conversion elements are arranged in a matrix to form a pixel array.
[0023] For example, the sensor unit 10a in FIG. 1 includes an imaging unit 1200, and outputs pixel data output from the imaging unit 1200 within one frame period as image data for one frame.
[0024] Here, each photoelectric conversion element corresponds to a pixel in image data, and the pixel array unit has photoelectric conversion elements arranged in a matrix, the number of which corresponds to, for example, 1920 pixels x 1080 pixels in terms of the number of pixels in rows and columns. Note that one frame of image is formed by pixel signals from the photoelectric conversion elements in the number corresponding to 1920 pixels x 1080 pixels.
[0025] The optical unit 1201 includes a lens, an autofocus mechanism, and the like, and irradiates light incident on the lens onto a pixel array unit of the imaging unit 1200. The imaging unit 1200 generates a pixel signal for each photoelectric conversion element in response to the light irradiated onto the pixel array unit via the optical unit 1201. The imaging unit 1200 converts the pixel signal, which is an analog signal, into pixel data, which is a digital signal, and outputs the pixel data. The pixel data output from the imaging unit 1200 is stored in a memory 1202. The memory 1202 is, for example, a frame memory, and is capable of storing pixel data for at least one frame.
[0026] The DSP 1203 performs a predetermined image processing on the pixel data stored in the memory 1202. The DSP 1203 also includes a recognition model that has been learned in advance, and performs the above-mentioned recognition processing using the DNN on the image data stored in the memory 1202 based on the recognition model. The recognition result that is the result of the recognition processing by the DSP 1203 is temporarily stored in, for example, a memory provided in the DSP 1203 or a RAM 1207, and is output to the outside from the interface 1204. Not limited to this, when the information processing device 1 includes a storage device, the recognition result may be stored in the storage device.
[0027] However, the functions of the DSP 1203 may be realized by the CPU 1205. Also, a GPU (Graphics Processing Unit) may be used instead of the DSP 1203.
[0028] The imaging unit 1200 may be a CMOS image sensor (CIS) in which each unit included in the imaging unit 1200 is integrally formed using a CMOS (Complementary Metal Oxide Semiconductor). The imaging unit 1200 may be formed on a single substrate. The imaging unit 1200 may be a stacked CIS in which a plurality of semiconductor chips are stacked and integrally formed. The imaging unit 1200 is not limited to this example, and may be another type of optical sensor, such as an infrared optical sensor that captures images using infrared light.
[0029] As an example, the imaging unit 1200 can be formed by a two-layer stacked CIS in which semiconductor chips are stacked in two layers. FIG. 6A is a diagram showing an example in which the imaging unit 1200 is formed by a two-layer stacked CIS. In the structure of FIG. 6A, a pixel unit 2020a is formed in a semiconductor chip of a first layer, and a memory+logic unit 2020b is formed in a semiconductor chip of a second layer. The pixel unit 2020a includes at least a pixel array unit in the imaging unit 1200. The memory+logic unit 2020b includes, for example, a drive circuit for driving the pixel array unit. The memory+logic unit 2020b can further include a memory 1202.
[0030] As shown on the right side of FIG. 6A, the imaging unit 1200 is configured as one solid-state imaging element by bonding the semiconductor chip of the first layer and the semiconductor chip of the second layer together while being in electrical contact with each other.
[0031] As another example, the imaging unit 1200 can be formed by a three-layer structure in which semiconductor chips are stacked in three layers. FIG. 6B is a diagram showing an example in which the imaging unit 1200 is formed by a stacked CIS with a three-layer structure. In the structure of FIG. 6B, the pixel unit 2020a is formed in the semiconductor chip of the first layer, the memory unit 2020c is formed in the semiconductor chip of the second layer, and the logic unit 2020d is formed in the semiconductor chip of the third layer. In this case, the logic unit 2020d includes, for example, a drive circuit for driving the pixel array unit. In addition, the memory unit 2020c can include a frame memory and the memory 1202.
[0032] As shown on the right side of Figure 6B, the imaging unit 1200 is constructed as a single solid-state imaging element by bonding the first layer semiconductor chip, the second layer semiconductor chip, and the third layer semiconductor chip together while maintaining electrical contact.
[0033] In the configurations of FIGS. 6A and 6B, the memory+logic unit 2020b may include components equivalent to the DSP 1203, the interface 1204, the CPU 1205, the ROM 1206, and the RAM 1207 shown in FIG.
[0034] (1-1-1. Configuration Examples of an Imaging Unit Applicable to Each Embodiment) Fig. 7 is a block diagram showing an example of the configuration of an imaging unit 1200 applicable to each embodiment. In Fig. 7, the imaging unit 1200 includes a pixel array unit 1001, a vertical scanning unit 1002, an AD (Analog to Digital) conversion unit 1003, a pixel signal line 1006, a vertical signal line VSL, a control unit 1100, and a signal processing unit 1101. In Fig. 7, the control unit 1100 and the signal processing unit 1101 can also be realized by, for example, the CPU 1205 and the DSP 1203 shown in Fig. 5.
[0035] The pixel array section 1001 includes a plurality of pixel circuits 1000, each of which includes a photoelectric conversion element, for example a photodiode, for performing photoelectric conversion on received light, and a circuit for reading out electric charges from the photoelectric conversion element. In the pixel array section 1001, the plurality of pixel circuits 1000 are arranged in a matrix array in the horizontal direction (row direction) and the vertical direction (column direction). In the pixel array section 1001, the row direction arrangement of the pixel circuits 1000 is called a line. For example, when one frame of image is formed with 1920 pixels x 1080 lines, the pixel array section 1001 includes at least 1080 lines, each including at least 1920 pixel circuits 1000. One frame of image (image data) is formed by pixel signals read out from the pixel circuits 1000 included in the frame.
[0036] In addition, pixel signal lines 1006 are connected to the pixel array unit 1001 for each row and column of each pixel circuit 1000, and vertical signal lines VSL are connected to each column. An end of the pixel signal line 1006 that is not connected to the pixel array unit 1001 is connected to a vertical scanning unit 1002. The vertical scanning unit 1002 transmits control signals such as drive pulses for reading pixel signals from pixels to the pixel array unit 1001 via the pixel signal line 1006 under the control of a control unit 1100 described later. An end of the vertical signal line VSL that is not connected to the pixel array unit 1001 is connected to an AD conversion unit 1003. The pixel signals read from the pixels are transmitted to the AD conversion unit 1003 via the vertical signal line VSL.
[0037] The following provides an overview of control for reading out pixel signals from the pixel circuit 1000. Reading out pixel signals from the pixel circuit 1000 is performed by transferring charges accumulated in a photoelectric conversion element upon exposure to a floating diffusion layer (FD) and converting the transferred charges in the floating diffusion layer into a voltage. The voltage into which the charges are converted in the floating diffusion layer is output as a pixel signal to a vertical signal line VSL via an amplifier.
[0038] More specifically, in the pixel circuit 1000, during exposure, the photoelectric conversion element and the floating diffusion layer are turned off (open) to accumulate charges generated in response to the incident light by photoelectric conversion in the photoelectric conversion element. After the exposure is completed, the floating diffusion layer and the vertical signal line VSL are connected in response to a selection signal supplied via the pixel signal line 1006. Furthermore, the floating diffusion layer is connected to a supply line of a power supply voltage VDD or a black level voltage for a short period of time in response to a reset pulse supplied via the pixel signal line 1006 to reset the floating diffusion layer. A voltage (referred to as voltage A) of a reset level for the floating diffusion layer is output to the vertical signal line VSL. Thereafter, a transfer pulse supplied via the pixel signal line 1006 turns the photoelectric conversion element and the floating diffusion layer to an on (closed) state, and the charges accumulated in the photoelectric conversion element are transferred to the floating diffusion layer. A voltage (referred to as voltage B) corresponding to the amount of charge in the floating diffusion layer is output to the vertical signal line VSL.
[0039] The AD conversion unit 1003 includes an AD converter 1007 provided for each vertical signal line VSL, a reference signal generation unit 1004, and a horizontal scanning unit 1005. The AD converter 1007 is a column AD converter that performs AD conversion processing for each column of the pixel array unit 1001. The AD converter 1007 performs AD conversion processing on pixel signals supplied from the pixel circuits 1000 via the vertical signal lines VSL, and generates two digital values (values corresponding to voltage A and voltage B, respectively) for correlated double sampling (CDS) processing for noise reduction.
[0040] The AD converter 1007 supplies the two generated digital values to a signal processing unit 1101. The signal processing unit 1101 performs CDS processing based on the two digital values supplied from the AD converter 1007, and generates pixel data, which is a pixel signal based on a digital signal.
[0041] The reference signal generating unit 1004 generates a ramp signal as a reference signal, which is used by each AD converter 1007 to convert a pixel signal into two digital values, based on a control signal input from the control unit 1100. The ramp signal is a signal whose level (voltage value) decreases with time at a constant slope, or a signal whose level decreases in a step-like manner. The reference signal generating unit 1004 supplies the generated ramp signal to each AD converter 1007. The reference signal generating unit 1004 is configured using, for example, a DAC (Digital to Analog Converter) or the like.
[0042] When a ramp signal, the voltage of which drops stepwise according to a predetermined slope, is supplied from the reference signal generating unit 1004, the counter starts counting according to the clock signal. The comparator compares the voltage of the pixel signal supplied from the vertical signal line VSL with the voltage of the ramp signal, and stops counting by the counter at the timing when the voltage of the ramp signal crosses the voltage of the pixel signal. The AD converter 1007 converts the pixel signal, which is an analog signal, into a digital value by outputting a value corresponding to the count value at the time when the counting is stopped.
[0043] The AD converter 1007 supplies the two generated digital values to the signal processing unit 1101. The signal processing unit 1101 performs CDS processing based on the two digital values supplied from the AD converter 1007, and generates a pixel signal (pixel data) using a digital signal. The pixel data generated by the signal processing unit 1101 is stored in a frame memory (not shown), and when one frame's worth of pixel data has been stored in the frame memory, it is output from the imaging unit 1200 as one frame's image data.
[0044] Under the control of the control unit 1100, the horizontal scanning unit 1005 performs selective scanning to select each AD converter 1007 in a predetermined order, thereby causing each digital value temporarily held by each AD converter 1007 to be sequentially output to the signal processing unit 1101. The horizontal scanning unit 1005 is configured using, for example, a shift register, an address decoder, etc.
[0045] The control unit 1100 performs drive control of the vertical scanning unit 1002, the AD conversion unit 1003, the reference signal generation unit 1004, the horizontal scanning unit 1005, etc., in accordance with the imaging control signal supplied from the sensor control unit 11. The control unit 1100 generates various drive signals that serve as references for the operations of the vertical scanning unit 1002, the AD conversion unit 1003, the reference signal generation unit 1004, and the horizontal scanning unit 1005. The control unit 1100 generates control signals for the vertical scanning unit 1002 to supply to each pixel circuit 1000 via the pixel signal line 1006, based on, for example, a vertical synchronization signal or an external trigger signal included in the imaging control signal and a horizontal synchronization signal. The control unit 1100 supplies the generated control signals to the vertical scanning unit 1002.
[0046] Furthermore, the control unit 1100 passes, for example, information indicating an analog gain included in an imaging control signal supplied from the CPU 1205 to the AD conversion unit 1003. The AD conversion unit 1003 controls the gain of a pixel signal input to each AD converter 1007 included in the AD conversion unit 1003 via a vertical signal line VSL in accordance with the information indicating the analog gain.
[0047] The vertical scanning unit 1002 supplies various signals including a drive pulse to each pixel circuit 1000 for each line to the pixel signal line 1006 of a selected pixel row of the pixel array unit 1001 based on a control signal supplied from the control unit 1100, and causes each pixel circuit 1000 to output a pixel signal to a vertical signal line VSL. The vertical scanning unit 1002 is configured using, for example, a shift register, an address decoder, etc. Furthermore, the vertical scanning unit 1002 controls the exposure in each pixel circuit 1000 according to information indicating the exposure supplied from the control unit 1100.
[0048] The imaging unit 1200 configured in this manner is a column AD type CMOS (Complementary Metal Oxide Semiconductor) image sensor in which the AD converters 1007 are arranged for each column.
[0049] (1-1-2. About the resolution of captured images) Next, the resolution of the image used in the recognition process will be described with reference to Fig. 8A and Fig. 8B. Fig. 8A and Fig. 8B are diagrams that show examples of captured images 30a and 30b when the same imaging range is captured using a low-resolution imaging device and a high-resolution imaging device, respectively. The imaging range shown in Fig. 8A and Fig. 8B includes a "person" in the center at a position some distance away from the imaging device. Consider the case where the "person" is recognized as the target object by the recognition process.
[0050] In the low-resolution example of Fig. 8A, it is difficult to recognize the "person" in the captured image 30a, and the recognition performance of the "person" by the recognition process is extremely low. On the other hand, in the high-resolution example of Fig. 8B, it is easy to recognize the "person" in the captured image 30b, and the recognized "person" is obtained as the recognition result 40, which is higher in recognition performance than the low-resolution example of Fig. 8A.
[0051] On the other hand, the recognition process for a high-resolution image requires more calculations and more time than the recognition process for a low-resolution image. Therefore, it is difficult to improve the simultaneity between the recognition result and the captured image. In contrast, the recognition process for a low-resolution image requires less calculations and can be completed in a short time, making it relatively easy to improve the simultaneity with the captured image.
[0052] As an example, consider a case where recognition processing is performed based on an image captured by an on-vehicle imaging device. In this case, since it is necessary to recognize distant objects (for example, an oncoming vehicle traveling in the opposite direction to the traveling direction of the vehicle in the oncoming lane) with high simultaneity, it is considered to perform recognition processing on a low-resolution image. However, as described with reference to FIG. 8A, when a low-resolution image is used, it is difficult to recognize distant objects. Also, when a high-resolution image is used, it is relatively easy to recognize distant objects, but it is difficult to increase the simultaneity of the captured image, and there is a possibility that it is not possible to respond to an emergency situation.
[0053] In each embodiment of the present disclosure, in order to easily and quickly recognize a distant object, a recognition process is performed on a sampled image obtained by thinning out pixels of a high-resolution captured image through subsampling according to a predetermined rule. For a captured image acquired in a next frame, pixels different from those subsampling performed for the immediately preceding captured image are sampled, and the recognition process is performed on the sampled image obtained through the sampled pixels.
[0054] In the second captured image, which is captured next in time series after the first captured image, the operation of performing recognition processing on a sampled image in which pixels different from those in the first captured image are sampled is repeatedly executed on a frame-by-frame basis. This makes it possible to obtain recognition results at high speed while using high-resolution captured images. In addition, by sequentially integrating the feature amount extracted during the recognition processing with the feature amount extracted in the recognition processing for the next sampled image, it is possible to obtain recognition results with higher accuracy.
[0055] [2. First embodiment] Next, a first embodiment of the present disclosure will be described.
[0056] (2-1. Configuration according to the first embodiment) (2-1-1. Overview of Configuration Applicable to First Embodiment) Fig. 9 is a block diagram showing an example of a configuration of an information processing device according to the first embodiment of the present disclosure. In Fig. 9, the information processing device 1b includes a sensor unit 10b and a recognition processing unit 20b. Although not shown, the sensor unit 10b includes an imaging means (camera) and an imaging control unit that controls the imaging means, similar to the sensor unit 10a described using Fig. 1. The imaging means performs imaging at a high resolution (for example, 1920 pixels x 1080 pixels). The sensor unit 10b supplies image data of the captured image captured by the imaging means to the recognition processing unit 20b.
[0057] The recognition processing unit 20b includes a preprocessing unit 210 and a recognition unit 220. Image data supplied from the sensor unit 10b to the recognition processing unit 20b is input to the preprocessing unit 210. The preprocessing unit 210 performs subsampling on the input image data by thinning out pixels according to a predetermined rule. A sampled image obtained by subsampling the image data is input to the recognition unit 220.
[0058] The recognition unit 220 performs recognition processing on image data using DNN, similar to the recognition processing unit 20a in Fig. 1. More specifically, the recognition processing unit 20a includes a recognition model that has been trained in advance using predetermined teacher data by machine learning, and performs recognition processing using DNN on the image data supplied from the sensor unit 10a based on the recognition model. At this time, as the teacher data, a sampling image that has been subsampled in the same manner as the pre-processing unit 210 is used.
[0059] The recognition unit 220 outputs the recognition result from the recognition process to, for example, the outside of the information processing device 1b.
[0060] (2-1-2. Example of Recognition Processing According to the First Embodiment) 10 is a schematic diagram for explaining the recognition process according to the first embodiment. Image data 32 generally indicates one frame of image data of an image captured by the sensor unit 10b. The image data 32 includes a plurality of pixels 300 arranged in a matrix. The image data 32 is input to a pre-processing unit 210 in the recognition processing unit 20b. The pre-processing unit 210 performs sub-sampling on the image data 32 by thinning out the image data according to a predetermined rule (step S10).
[0061] The sampled image based on the sub-sampled sampling pixels is input to the recognition unit 220. The recognition unit 220 extracts features of the input sampled image by using a DNN (step S11). Here, the recognition unit 220 extracts the features by using a CNN of the DNN.
[0062] The recognition unit 220 stores the feature amount extracted in step S11 in a storage unit (for example, RAM 1207) not shown. At this time, if the feature amount extracted in the immediately previous frame has already been stored in the storage unit, the recognition unit 220 recursively uses the feature amount stored in the memory and integrates it with the extracted feature amount (step S12). The recognition unit 220 stores the feature amounts extracted up to the immediately previous frame in the storage unit, and accumulates and integrates them. That is, the process in this step S12 corresponds to a process using an RNN of DNNs.
[0063] The recognition unit 220 executes recognition processing based on the feature amounts accumulated and integrated in step S12 (step S13).
[0064] Here, the sub-sampling process by the pre-processing unit 210 in step S10 will be described in more detail. FIG. 11 is a schematic diagram for explaining the sampling process according to the first embodiment. In FIG. 11, section (a) shows a schematic example of image data 32. As described above, the image data 32 includes a plurality of pixels 300 arranged in a matrix. The pre-processing unit 210 divides the image data 32 into divided regions 35 including two or more pixels 300. In the example of FIG. 11, the divided region 35 is an area having a size of 4 pixels by 4 pixels, and includes 16 pixels 300.
[0065] The pre-processing unit 210 sets pixel positions for selecting sampling pixels by sub-sampling from each pixel 300 included in the divided region 35 for this divided region 35. In addition, the pre-processing unit 210 sets different pixel positions for each frame as pixel positions for selecting sampling pixels.
[0066] Section (b) of Fig. 11 shows an example of pixel positions set for a divided region 35 in a certain frame. In this example, pixel positions are set so that every other pixel 300 is selected in each of the row and column directions in the divided region 35, and pixels 300sa1, 300sa2, 300sa3, and 300sa4 at each of the set pixel positions are selected as sampling pixels. In this way, the pre-processing unit 210 performs sub-sampling with the divided region 35 as a unit.
[0067] The pre-processing unit 210 generates an image consisting of the pixels 300sa1-300sa4 selected as sampling pixels in a certain frame as a sampling image consisting of sampling pixels. Section (c) of Fig. 11 shows an example of a sampling image 36 generated from the pixels 300sa1-300sa4 selected as sampling pixels in section (b) of Fig. 11. The pre-processing unit 210 inputs this sampling image 36 to the recognition unit 220. The recognition unit 220 executes recognition processing on this sampling image 36.
[0068] The recognition process according to the first embodiment will be described in more detail with reference to Fig. 12A to Fig. 12E. As described above, the pre-processing unit 210 sets a different pixel position for each frame as the pixel position for selecting a sampling pixel. The recognition unit 220 performs the recognition process for each frame based on a sampling image made up of the pixels 300 at each of the set pixel positions. Fig. 12A to Fig. 12E show each recognition process for the image data 32a to 32d, 32a' of each of frames #1 to #5 captured sequentially in time series by the sensor unit 10b.
[0069] 12A to 12E, the images based on the image data 32a to 32d, 32a' each include human objects 41 and 42. Object 41 is located at a relatively short distance (referred to as a medium distance) from sensor unit 10b. On the other hand, object 42 is located at a distance (referred to as a long distance) farther away than the medium distance from sensor unit 10b, and its size in the image is smaller than object 41.
[0070] 12A, the pre-processing unit 210 performs sub-sampling on each divided region 35 of the image data 32a of frame #1, for example, with the pixel position of the upper left corner as the base point. More specifically, the pre-processing unit 210 performs sub-sampling in which every other pixel 300 selected in the row and column directions in each divided region 35 of the image data 32a with the pixel position of the upper left corner as the base point is selected as pixels 300sa1 to 300sa4, which are sampled pixels (step S10a).
[0071] The pre-processing unit 210 generates a sampled image 36φ1 of the first phase from the sub-sampled pixels 300sa1 to 300sa4, as shown in section (b). The generated sampled image 36φ1 is input to the recognition unit 220.
[0072] The recognition unit 220 extracts the feature amount 50a of the input sampling image 36φ1 using DNN (step S11). The recognition unit 220 stores and accumulates the feature amount 50a extracted in step S11 in the storage unit (step S12). When a feature amount has already been accumulated in the storage unit, the recognition unit 220 can accumulate the feature amount 50a in the storage unit and integrate it with the feature amount already accumulated. Section (b) of FIG. 12A shows how the first feature amount 50a is stored in an empty storage unit as the process of step S12.
[0073] The recognition unit 220 executes a recognition process based on the feature amount 50a stored in the storage unit (step S13). In the example of Fig. 12A, as shown in section (b) as step S13, the object 41 located at a middle distance is recognized and obtained as a recognition result 60. On the other hand, the object 42 located at a long distance is not recognized.
[0074] In section (a) of Fig. 12B, the pre-processing unit 210 performs sub-sampling for each divided region 35 of the image data 32b of frame #2, by setting each pixel position shifted by one pixel in the horizontal direction from the pixel position set for each divided region 35 of the image data 32a of frame #1 shown in Fig. 12A as the pixel position of the sampled pixel (step S10b). That is, each sampled pixel selected in step S10b is each pixel 300 at a pixel position adjacent to the right in Fig. 12A with respect to the pixel position of each sampled pixel selected in step S10a.
[0075] The pre-processing unit 210 generates a sampled image 36φ2 of the second phase by each sampling pixel sub-sampled in step S10b as shown in section (b). The generated sampled image 36φ2 is input to the recognition unit 220.
[0076] The recognition unit 220 extracts the feature amount 50b of the input sampling image 36φ2 using DNN (step S11). The recognition unit 220 stores and accumulates the feature amount 50b extracted in step S11 in the storage unit (step S12). In this example, as shown as step S12 in section (b), the feature amount 50a extracted from the sampling image 36φ1 of the first phase has already been accumulated in the storage unit. Therefore, the recognition unit 220 accumulates the feature amount 50b in the accumulation unit and integrates the feature amount 50b with the accumulated feature amount 50a.
[0077] The recognition unit 220 executes a recognition process based on the integrated feature amount of the feature amount 50a and the feature amount 50b (step S13). In the example of Fig. 12B, as shown in section (b) as step S13, the object 41 located at a middle distance is recognized and obtained as the recognition result 60, but the object 42 located at a long distance is not recognized at this point.
[0078] In section (a) of Fig. 12C, the pre-processing unit 210 performs sub-sampling for each divided region 35 of the image data 32c of frame #3, in which the pixel position of each sampled pixel is set to a pixel position shifted by one pixel in the column direction from the pixel position set for each divided region 35 of the image data 32a of frame #1 shown in Fig. 12A (step S10c). That is, each sampled pixel selected in step S10c is each pixel 300 at a pixel position adjacent below in the figure to the pixel position of each sampled image selected in step S10a in Fig. 12A.
[0079] The pre-processing unit 210 generates a sampled image 36φ3 of the third phase by each sampling sub-sampled in step S10c as shown in section (b). The generated sampled image 36φ3 is input to the recognition unit 220.
[0080] The recognition unit 220 extracts the feature amount 50c of the input sampled image 36φ3 using the DNN (step S11). The recognition unit 220 stores and accumulates the feature amount 50c extracted in step S11 in the storage unit (step S12). In this example, as shown as step S12 in section (b), the feature amounts 50a and 50b extracted from the sampled images 36φ1 and 36φ2 of the first and second phases, respectively, have already been accumulated in the storage unit. Therefore, the recognition unit 220 accumulates the feature amount 50c in the storage unit and integrates the feature amount 50c with the accumulated feature amounts 50a and 50b.
[0081] The recognition unit 220 executes a recognition process based on the feature amount obtained by integrating the feature amounts 50a and 50b with the feature amount 50c (step S13). In the example of Fig. 12C, as shown in section (b) as step S13, the object 41 located at a middle distance is recognized and obtained as the recognition result 60, but the object 42 located at a long distance is not recognized at this point.
[0082] In section (a) of Fig. 12D, the pre-processing unit 210 performs sub-sampling for each divided region 35 of the image data 32d of frame #4, by setting each pixel position shifted by one pixel in the horizontal direction from the pixel position set for each divided region 35 of the image data 32c of frame #3 shown in Fig. 12C as the pixel position of the sampled pixel (step S10d). That is, each sampled pixel selected in step S10d is each pixel 300 at a pixel position adjacent to the right in Fig. 12C with respect to the pixel position of each sampled image selected in step S10c.
[0083] The pre-processing unit 210 generates a sampled image 36φ4 of the fourth phase by each sampling sub-sampled in step S10d as shown in section (b). The generated sampled image 36φ4 is input to the recognition unit 220.
[0084] The recognition unit 220 extracts a feature amount 50d of the input sampling image 36φ4 using DNN (step S11). The recognition unit 220 stores and accumulates the feature amount 50d extracted in step S11 in the accumulation unit (step S12). In this example, as shown as step S12 in section (b), the feature amounts 50a to 50c extracted from the sampling images 36φ1 to 36φ3 of the first to third phases have already been accumulated in the accumulation unit. Therefore, the recognition unit 220 accumulates the feature amount 50d in the accumulation unit and integrates the feature amount 50d with the accumulated feature amounts 50a to 50c.
[0085] The recognition unit 220 executes a recognition process based on the integrated feature amount of the feature amounts 50a to 50c and the feature amount 50d (step S13). In the example of Fig. 12D, as shown in section (b) as step S13, the object 41 located at a middle distance is recognized and obtained as a recognition result 60, and the object 42 located at a long distance is further recognized and obtained as a recognition result 61.
[0086] 12A to 12D, all pixel positions of the 16 pixels 300 included in each divided region 35 are selected as pixel positions of the sampling pixels. In other words, the pre-processing unit 210 selects the pixel positions of all pixels 300 included in one frame as pixel positions of the sampling pixels. The period from the time when the pixel position of the sampling pixel is first selected for each divided region 35 or one frame to the time when the pixel positions of all pixels 300 included in each divided region 35 or one frame are selected as pixel positions of the sampling pixels is defined as one cycle. That is, the pre-processing unit 210 circulates through each pixel position of each divided region 35 at a constant cycle and sets all pixel positions in the divided region 35 as pixel positions for acquiring the sampling pixels.
[0087] When the sub-sampling and recognition process for one period is completed, the sub-sampling and recognition process for the next period is started.
[0088] That is, in section (a) of Fig. 12E, the pre-processing unit 210 performs sub-sampling on each divided region 35 of the image data 32a' of frame #1', starting from the pixel position of the upper left corner, in the same manner as in the example of Fig. 12A (step S10a'). As shown in section (b), the pre-processing unit 210 generates a sampled image 36φ1' of the first phase by each sampling performed in step S10a'. The generated sampled image 36φ1' is input to the recognition unit 220.
[0089] The recognition unit 220 extracts a feature amount 50a' of the input sampling image 36φ1' using DNN (step S11). The recognition unit 220 stores and accumulates the feature amount 50a' extracted in step S11 in the accumulation unit (step S12). In this example, as shown as step S12 in section (b), the feature amounts 50a to 50d extracted from the sampling images 36φ1 to 36φ4 of the first to fourth phases in the previous period have already been accumulated in the accumulation unit. Therefore, the recognition unit 220 accumulates the feature amount 50a' in the accumulation unit and integrates the feature amount 50a' with the accumulated feature amounts 50a to 50d.
[0090] Alternatively, the recognition unit 220 may reset the storage unit for each period of pixel position selection of the sampling pixels. The storage unit can be reset, for example, by deleting the feature amounts 50a to 50d for one period stored in the storage unit from the storage unit.
[0091] The recognition unit 220 can also store a constant amount of feature amounts in the storage unit. For example, the recognition unit 220 stores feature amounts for one period, that is, feature amounts for four frames, in the storage unit. In this case, when a new feature amount 50a' is extracted, the recognition unit 220 deletes, for example, the oldest feature amount 50d from among the feature amounts 50a to 50d stored in the storage unit, and stores and accumulates the new feature amount 50a' in the storage unit. The recognition unit 220 executes the recognition process based on the accumulated amount obtained by integrating the feature amounts 50a to 50c remaining after the feature amount 50d has been deleted and the new feature amount 50a'.
[0092] The recognition unit 220 executes a recognition process based on the feature amount obtained by integrating the feature amounts 50a to 50d already stored in the storage unit and the newly extracted feature amount 50a' (step S13). In the example of Fig. 12E, as shown in section (b) as step S13, the object 41 located at a middle distance is recognized and obtained as a recognition result 60, and the object 42 located at a long distance is further recognized and obtained as a recognition result 61.
[0093] 11, the sampled image 36 is a thinned image obtained by thinning out pixels from the original image data 32. In the example of Fig. 11, the sampled image 36 is image data obtained by reducing the image data 32 by half in both the row and column directions, and is a reduced image having a pixel count of 1 / 4 that of the original image data 32. Therefore, the recognition unit 220 can perform recognition processing on the sampled image 36 at higher speed than recognition processing using all the pixels 300 included in the original image data 32.
[0094] Furthermore, the pixel positions of the pixels 300 to be set as sampling pixels for generating the sampling image 36 are selected by shifting them by one pixel for each frame within the divided region 35. Therefore, it is possible to obtain the sampling image 36 whose phase is shifted by one pixel for each frame. At this time, the pixel positions of all the pixels 300 included in the divided region 35 are selected as the pixel positions of the pixels 300 to be set as sampling pixels.
[0095] In this way, the pixel positions of the pixels 300 that generate the sampled images 36 are selected, and the feature amounts calculated from each sampled image 36 are accumulated and integrated. This allows the pixels 300 at all pixel positions included in the image data 32 to be involved in the recognition process, making it possible to easily recognize even distant objects, for example.
[0096] In the above description, the preprocessing unit 210 sets pixel positions for selecting sampling pixels according to a predetermined rule, but this is not limited to this example. For example, the preprocessing unit 210 may set pixel positions for selecting sampling pixels in response to an instruction from outside the recognition processing unit 20b or from outside the information processing device 1b including the recognition processing unit 20b.
[0097] (2-1-3. Subsampling Process According to the First Embodiment) Next, the subsampling process in the recognition process according to the first embodiment will be described in more detail. FIG. 13A and FIG. 13B are schematic diagrams for explaining the subsampling process in the recognition process according to the first embodiment. Here, for the purpose of explanation, as shown in section (b) of FIG. 13A, the divided area 35 is a 2 pixel by 2 pixel area. In each divided area 35, the upper left pixel position is set to the coordinates [0,0] of the origin, and the upper right, lower left and lower right pixel positions are set to the coordinates [1,0], [0,1] and [1,1], respectively. Moreover, the sampling of the pixel 300 is performed in each divided area 35 in the order of coordinates [1,1], [1,0], [0,1] and [0,0], with the lower right pixel position [1,1] as the base point.
[0098] In section (a) of Fig. 13A, the passage of time is shown from bottom to top. The example of Fig. 13A corresponds to Fig. 12A to Fig. 12E described above, and image data 32a is image [T] at the most recent time T, followed by image data 32b, image data 32c, and image data 32d, in the order of time T-1, T-2, and T-3, which are image [T-1], image [T-2], and image [T-3] based on image data 32 that are one frame older each.
[0099] At time T-3, the preprocessing unit 210 selects the pixel 300 at the coordinate [1,1] of each divided region 35 as a sampling pixel for the image data 32a (step S10a), and the recognition unit 220 extracts a feature amount of the sampling image 36φ1 by the selected sampling pixel (step S11). The recognition unit 220 integrates the feature amount 50a extracted from the sampling image 36φ1 with, for example, a feature amount extracted during a predetermined period of time before that (step S12), and performs recognition processing based on the integrated feature amount (step S13).
[0100] Here, for example, by the subsampling process (step S10a) in each divided region 35 of the image data 32a described above, a sampled image 36φ1 can be obtained by uniformly thinning out the image data 32a. Using the feature amount 50a extracted in step S11 from this sampled image 36φ1, recognition processing can be performed on the entire image data 32a. By performing recognition processing on the sampled image using the sampling pixels selected by subsampling from the image data 32, it is possible to complete the recognition processing on the image data 32.
[0101] This series of processes of generating a sampling image from image data 32, extracting features from the generated sampling image, and performing recognition processing based on the extracted features is called one unit of processing. In the example of Fig. 13A, for example, one unit of processing includes the subsampling process in step S10a, the feature extraction process in step S11 for the sampling image 36φ1 generated by the subsampling process, the feature integration process in step S12, and the recognition process in step S13. The recognition unit 220 can perform recognition processing on the thinned image data 32 for each unit of processing (step S13).
[0102] Thereafter, in the same manner, the recognition processing unit 20b executes the above-mentioned unit processing for each of the image data 32b, 32c, and 32d that are updated sequentially in a frame cycle, and executes the recognition processing. At this time, the feature integration processing in step S12 and the recognition processing in step S13 can be common to the processing of each unit.
[0103] As described above, one unit of processing is performed on each of the image data 32a to 32d, thereby completing a cycle of selecting sampling pixels for each pixel position included in each divided region 35. Fig. 13B shows the next unit of processing after a cycle of selecting sampling pixels for each pixel position included in each divided region 35. That is, when a cycle of one unit of processing is performed on each of the image data 32a, 32b, 32c, and 32d, one unit of processing is performed on image data 32a' of the next frame input to the recognition processing unit 20b.
[0104] In this example, the feature amount 50d extracted based on the oldest image data 32d is discarded, and the feature amount 50a' is extracted from the new image data 32a'. That is, the preprocessing unit 210 selects each pixel 300 at the coordinate [1,1] of each divided area 35 of the image data 32a' as a sampling pixel, and generates a sampling image 36φ1. The recognition unit 220 extracts the feature amount 50a' from the sampling image 36φ1 selected from this image data 32a'. The recognition unit 220 integrates this feature amount 50a' with the feature amounts 50a, 50b, and 50c extracted up to that point, and performs recognition processing based on the integrated feature amount. In this case, the recognition unit 220 needs to perform feature amount extraction processing only for the newly acquired image data 32a'.
[0105] In this way, the recognition processing according to the first embodiment is performed by executing one unit of processing in the same processing system in the recognition processing unit 20b. More specifically, the recognition processing unit 20b repeats a processing system of subsampling processing and feature extraction processing for the image data 32 for each frame as one unit of processing, and integrates the features extracted by this repetition to perform the recognition processing.
[0106] The recognition processing unit 20b also performs sub-sampling processing including the pixel positions of all pixels 300 included in the image data 32 while periodically shifting the pixel positions at which the sampling pixels are selected. Furthermore, the recognition processing unit 20b performs recognition processing by integrating feature amounts as intermediate data extracted from the sampling images based on the sampling pixels selected from the image data 32 of each frame in step S11.
[0107] The recognition process according to the first embodiment configured as described above is a processing system that can be completed in one unit of processing, so that the recognition result can be obtained more quickly. Also, since sampling pixels are selected from the entire image data 32 in one unit, a wide range of recognition results can be confirmed in one unit of processing. Furthermore, since intermediate data (feature amounts) based on multiple image data 32 are integrated, it is possible to obtain more detailed recognition results that are obtained across multiple units.
[0108] In other words, by using the information processing device 1b according to the first embodiment, it is possible to improve the simultaneity of the recognition results while also obtaining recognition results that utilize the resolution of the captured image, thereby improving the characteristics of the recognition processing using the captured image.
[0109] (2-1-4. More specific configuration example according to the first embodiment) Next, a more specific example of the configuration for realizing the recognition process according to the first embodiment will be described. Fig. 14 is a time chart of an example showing the control of the readout and recognition process according to the first embodiment. In Fig. 14, the imaging period is a frame period, and is, for example, 50 [ms] (20 [fps (frames per second)]). Also, here, readout from the pixel circuits 1000 arranged in a matrix array in the pixel array section 1001 is performed line-sequentially by a rolling shutter method. Here, in Fig. 14, the passage of time is represented to the right, and the line position is represented from top to bottom.
[0110] For example, in the imaging process of frame #1, exposure is performed for a predetermined time on each line, and after the exposure is completed, pixel signals are transferred from each pixel circuit 1000 to the AD conversion unit 1003 via the vertical signal line VSL, and in the AD conversion unit 1003, each AD converter 1007 converts the transferred analog pixel signals into pixel data, which is a digital signal. When the pixel signals are converted into pixel data for all lines, image data 32a based on the pixel data of frame #1 is input to the pre-processing unit 210. The pre-processing unit 210 performs the above-mentioned sub-sampling process (shown as "SS" in the figure) on the input image data 32a to acquire pixels 300 from the pixel positions of the sampling pixels selected for each divided region 35, and generates a sampling image 36φ1 (step S10a).
[0111] The pre-processing unit 210 passes the sampling image 36φ1 to the recognition unit 220. At this time, the sampling image 36φ1 passed from the pre-processing unit 210 to the recognition unit 220 is an image in which the number of pixels has been reduced from the image data 32a by thinning out through sub-sampling processing. The recognition unit 220 executes recognition processing on this sampling image 36φ1. Here, the recognition processing is shown to include a feature extraction processing (step S11), a feature integration processing (step S12), and a recognition processing (step S13). The processing of steps S11 to S13 is performed within a period of one frame, for example. The recognition result φ1 based on the sampling image 36φ1 is output to the outside of the recognition processing unit 20b.
[0112] In parallel with the processing for frame #1 described above, processing for the next frame #2 is performed. Image data 32b consisting of pixel data of frame #2 is input to the pre-processing unit 210. The pre-processing unit 210 performs sub-sampling processing on the input image data 32b with a phase different from that of the image data 32a to generate a sampled image 36φ2.
[0113] The pre-processing unit 210 passes a sampled image 36φ2, the number of pixels of which has been reduced from that of the image data 32b by sub-sampling, to the recognition unit 220. The recognition unit 220 executes recognition processing on this sampled image 36φ2 within the period of one frame.
[0114] At this time, the recognition unit 220 integrates the feature amount 50b extracted from the sampling image 36φ2 and the feature amount 50a extracted by the feature amount extraction process for the image data 32a through a feature amount integration process in step S12. The recognition unit 220 executes the recognition process using the integrated feature amount. The recognition result φ2 by this recognition process is output to the outside of the recognition processing unit 20b.
[0115] Thereafter, in the same manner, the pre-processing unit 210 performs sub-sampling processing on the next frame #3 in parallel with processing on the image data 32b of the immediately preceding frame #2, and the recognition unit 220 extracts a feature amount 50c from the sampled image 36φ3 generated by the sub-sampling processing. The recognition unit 220 further integrates a feature amount obtained by integrating the feature amounts 50a and 50b extracted from the image data 32a and 32b, respectively, with the extracted feature amount 50c, and performs recognition processing based on the integrated feature amount. The recognition unit 220 outputs the recognition result φ3 obtained by this recognition processing to the outside.
[0116] Similarly, for the next frame #4, the recognition processing unit 20b performs subsampling and feature extraction processing in parallel with the processing for the image data 32c of the immediately preceding frame #3, to obtain a feature 50d. The recognition processing unit 20b further integrates the extracted feature 50d with the feature obtained by integrating the feature amounts 50a-50c extracted from the image data 32a-32c by the recognition unit 220, and executes recognition processing based on the integrated feature amount. The recognition unit 220 outputs the recognition result φ4 obtained by this recognition processing to the outside.
[0117] Here, in FIG. 14, the thickness of the vertical arrows indicating the image data 32a to 32d, the recognition processes from steps S10a to S10d, and the outputs of the recognition results φ1 to φ4 from each recognition process roughly indicates the amount of information.
[0118] More specifically, in the example of FIG. 14, the amount of data of the sampled images 36φ1 to φ4 that are subsampled by the pre-processing unit 210 through the processing of steps S10a to S10d and passed to the recognition unit 220 is smaller than the amount of data of each image data 32a to 32d input to the pre-processing unit 210 for the processing of steps S10a to S10d.
[0119] On the other hand, the amount of information of each recognition result φ1 to φ4 by the recognition process based on each image data 32a to 32d increases with each recognition process, indicating that the obtained recognition result becomes more detailed with each recognition process. This is because, for each recognition process, a feature amount obtained by integrating the feature amount obtained by shifting the phase of the sampling image up to that point and the feature amount newly obtained by further shifting the phase of the immediately preceding sampling image is used.
[0120] Next, more detailed functions of the preprocessing unit 210 and the recognition unit 220 according to the first embodiment will be described. Fig. 15 is a functional block diagram of an example for explaining more detailed functions of the preprocessing unit 210 and the recognition unit 220 according to the first embodiment. In Fig. 15, the preprocessing unit 210 includes a reading unit 211 and a utilization area acquisition unit 212. Furthermore, the recognition unit 220 includes a feature amount calculation unit 221, a feature amount accumulation control unit 222, a feature amount accumulation unit 223, and a utilization area determination unit 224.
[0121] The reading unit 211, utilization area acquisition unit 212, feature amount calculation unit 221, feature amount accumulation control unit 222, feature amount accumulation unit 223, and utilization area determination unit 224 are realized, for example, by an information processing program that runs on the CPU 1205. This information processing program can be stored in advance in the ROM 1206. Alternatively, the information processing program can be supplied from the outside via the interface 1204 and written to the ROM 1206.
[0122] Furthermore, the reading unit 211, utilization area acquisition unit 212, feature amount calculation unit 221, feature amount accumulation control unit 222, feature amount accumulation unit 223, and utilization area determination unit 224 may be realized by the CPU 1205 and DSP 1203 operating in accordance with an information processing program. Furthermore, some or all of the reading unit 211, utilization area acquisition unit 212, feature amount calculation unit 221, feature amount accumulation control unit 222, feature amount accumulation unit 223, and utilization area determination unit 224 may be configured by hardware circuits operating in cooperation with each other.
[0123] In the pre-processing unit 210, the readout unit 211 reads out image data 32 from the sensor unit 10b. The readout unit 211 passes the image data 32 read out from the sensor unit 10b to the utilization area acquisition unit 212. The utilization area acquisition unit 212 performs sub-sampling processing on the image data 32 passed out from the readout unit 211 in accordance with information indicating the utilization area passed out from a utilization area determination unit 224 (described later) to extract sampling pixels. The utilization area acquisition unit 212 generates a sampling image 36φx with a phase φx using the extracted sampling pixels.
[0124] The utilization area acquisition unit 212 passes the generated sampling image 36φx to the recognition unit 220. The recognition unit 220 passes the passed sampling image 36φx to the feature amount calculation unit 221.
[0125] In the recognition unit 220, the feature amount calculation unit 221 calculates the feature amount based on the delivered sampling image 36φx. That is, the feature amount calculation unit 221 functions as a calculation unit that calculates the feature amount of the sampling image 36φx composed of sampling pixels. Without being limited to this, the feature amount calculation unit 221 may obtain information for setting exposure and analog gain from the reading unit 211, for example, and calculate the feature amount by further using the obtained information. The feature amount calculation unit 221 passes the calculated feature amount to the feature amount accumulation control unit 222.
[0126] The feature amount accumulation control unit 222 accumulates the feature amount passed from the feature amount calculation unit 221 in the feature amount accumulation unit 223. At this time, the feature amount accumulation control unit 222 can generate an integrated feature amount by integrating past feature amounts already accumulated in the feature amount accumulation unit 223 with the feature amount passed from the feature amount calculation unit 221. Furthermore, when the feature amount accumulation unit 223 is initialized, for example, and no feature amount exists, the feature amount accumulation control unit 222 accumulates the feature amount passed from the feature amount calculation unit 221 in the feature amount accumulation unit 223 as the initial feature amount.
[0127] Furthermore, the feature amount accumulation control unit 222 can delete a feature amount that satisfies a predetermined condition from among the feature amounts accumulated in the feature amount accumulation unit 223. The feature amount accumulation control unit 222 can apply time information, an external instruction, an exposure condition, or the like as a condition for deleting a feature amount.
[0128] For example, when applying time information as a condition for deleting a feature, the feature accumulation control unit 222 can delete a feature based on a sampling image obtained by the first sub-sampling in the immediately preceding sub-sampling period from among the features accumulated in the feature accumulation unit 223.
[0129] Furthermore, in a case where the imaging section 1200 has an automatic exposure setting function, when detecting a change in exposure of a predetermined level or more, the feature amount accumulation control section 222 can determine that the scene of the captured image has changed, and delete the feature amounts that have been accumulated up to that point in time in the feature amount accumulation section 223. Furthermore, the feature amount accumulation control section 222 can also delete all the feature amounts accumulated in the feature amount accumulation section 223 and initialize the feature amount accumulation section 223 in response to, for example, an external instruction.
[0130] The conditions under which the feature accumulation control unit 222 deletes the feature accumulated in the feature accumulation unit 223 are not limited to the above-mentioned conditions.
[0131] The feature amount accumulation control unit 222 passes the feature amount accumulated in the feature amount accumulation unit 223, or the feature amount obtained by integrating the feature amount accumulated in the feature amount accumulation unit 223 and the feature amount passed from the feature amount calculation unit 221, to the utilization area determination unit 224 and the recognition process execution unit 225. The recognition process execution unit 225 executes recognition processing based on the feature amount passed from the feature amount accumulation control unit 222. The recognition process execution unit 225 performs object detection, person detection, face detection, and the like by the recognition process. The recognition process execution unit 225 outputs the recognition result obtained by the recognition process to the outside of the recognition processing unit 20b.
[0132] Here, when a predetermined condition is satisfied, the feature amount accumulation control unit 222 can pass the feature amount accumulated in the feature amount accumulation unit 223 to the utilization area determination unit 224 and the recognition process execution unit 225. For example, the feature amount accumulation control unit 222 can apply time information, an external instruction, an exposure condition, etc. as conditions for passing the feature amount to the utilization area determination unit 224 and the recognition process execution unit 225.
[0133] For example, when applying time information as a condition for accumulating features, the feature accumulation control unit 222 can integrate the new feature passed to the feature accumulation control unit 222 with the feature already accumulated in the feature accumulation unit 223, and pass the integrated feature to the utilization area determination unit 224 and the recognition processing execution unit 225.
[0134] Furthermore, in the case where the imaging unit 1200 has an automatic exposure setting function, when a change in exposure of a predetermined level or more is detected, the feature amount accumulation control unit 222 can determine that the scene of the captured image has changed, and pass only the feature amount newly passed to the feature amount accumulation control unit 222 to the utilization area determination unit 224 and the recognition process execution unit 225. At this time, as described above, the feature amount accumulation control unit 222 can delete the feature amount already accumulated in the feature amount accumulation unit 223. Furthermore, in response to an instruction from the outside, for example, the feature amount accumulation control unit 222 can select the feature amount according to the instruction from among the newly passed feature amount and one or more feature amounts already accumulated in the feature amount accumulation unit 223, and pass it to the utilization area determination unit 224 and the recognition process execution unit 225.
[0135] The conditions under which the feature accumulation control unit 222 passes the feature to the utilization area determination unit 224 and the recognition process execution unit 225 are not limited to the above-mentioned conditions.
[0136] The utilization region determination unit 224 determines pixel positions for reading out pixel data as sampling pixels from the image data 32 read by the readout unit 211. The utilization region determination unit 224 determines the pixel positions, for example, according to a predetermined pattern and timing. Without being limited to this, the utilization region determination unit 224 can also determine the pixel positions based on the feature transferred from the feature accumulation control unit 222. The utilization region determination unit 224 transfers information indicating the determined pixel positions to the preprocessing unit 210 as utilization region information, and the information is input to the utilization region acquisition unit 212.
[0137] As described above, the utilization area acquisition unit 212 performs subsampling on the image data 32 passed from the reading unit 211 in accordance with the information indicating the utilization area passed from the utilization area determination unit 224. That is, the utilization area acquisition unit 212 or the pre-processing unit 210 and the utilization area determination unit 224 described above function as a setting unit that sets sampling pixels for the divided areas 35 into which the image data 32, i.e., imaging information composed of pixels, is divided.
[0138] (2-2. Effects of the First Embodiment) Next, the effects of the recognition process according to the first embodiment will be described in comparison with existing techniques.
[0139] (2-2-1. Overview of Existing Technology) Here, consider the case where an information processing device that performs recognition processing based on a captured image is used for in-vehicle applications. In this case, there is a demand for recognizing distant objects, and in consideration of the camera's angle of view, installation position, recognition object, and vehicle movement speed, etc., a resolution of 1920 pixels x 1080 pixels or more (for example, 4096 pixels x 2160 pixels for 4K resolution) may be required. On the other hand, a recognizer that performs recognition processing has a limit to its processing speed, and it is difficult to process high-resolution images as they are. If the resolution is, for example, several hundred pixels x several hundred pixels, it is considered that recognition processing following the captured image is possible, but in this case, as described with reference to FIG. 8A and FIG. 8B, it may be difficult to recognize distant objects.
[0140] For this reason, methods for reducing the amount of calculation required for recognition processing have been proposed. An example of a method for reducing the amount of calculation required for recognition processing according to a conventional technique will be described with reference to Figs. 16 and 17.
[0141] FIG. 16 is a schematic diagram for explaining a first example of a method for reducing the amount of calculations using existing technology. This first example is an example in which the row and column sizes of an original image 320, which is a captured image, are each reduced to 1 / N (N is an integer equal to or greater than 2) to generate a reduced image 321. The original image 320 can be reduced by, for example, thinning out every predetermined number of pixels in both the row and column directions. For example, when every other pixel is thinned out, N=2, and the number of pixels in reduced image 321 is 1 / N×1 / N=1 / N of the original image 320. 2 In simple comparison, the amount of calculation required for the recognition process of the reduced image 321 is 1 / 4 of the amount of calculation required for the recognition process of the original image 320. 2 On the other hand, in this first example, the resolution of the reduced image 321 is 1 / N of the resolution of the original image 320. 2Hereinafter, this first example will be referred to as "1 / N × 1 / N reduction."
[0142] Fig. 17 is a schematic diagram for explaining a second example of a method for reducing the amount of calculations by the existing technology. This second example is an example in which the original image 320 is divided into N divided regions and then cropped for each divided region. In the example of Fig. 17, N=2, and the original image 320 is divided into four divided regions and then cropped for each divided region. Each of the cropped images 322#1, 322#2, 322#3, and 322#4 has a size that is 1 / 9 of the original image 320. 2 Each of the cropped images 303#1 to 303#4 has the same resolution as the original image 320, and the amount of calculation required for the recognition process is 1 / 4 that of the original image 320 in a simple comparison. 2 Hereinafter, this second example will be called the "1 / N x 1 / N crop."
[0143] (2-2-2. Comparison of the technology disclosed herein with existing technologies) (2-2-2-1. Comparison of recognition results) Next, the technology according to the present disclosure will be compared with existing technologies, and the effects of the technology according to the present disclosure will be described. FIG. 18 is a schematic diagram showing an example of evaluation of the above-mentioned "1 / N×1 / N reduction", "1 / N×1 / N crop", and the recognition process according to the first embodiment. In FIG. 18, each column represents, from the left, "1 / N×1 / N reduction", "1 / N×1 / N crop", and the recognition process according to the first embodiment (1 / N×1 / N phase-shifted subsampling + information integration). In FIG. 18, "○ (circle mark)" indicates an evaluation result that is considered to be favorable, and "× (cross mark)" indicates an evaluation result that is considered to be unfavorable.
[0144] In addition, in FIG. 18, each row represents, from the top, "distance," "angle of view," "cut-off," "latency," "frame rate," and "bus width." Of these, "distance" indicates the recognizable distance. "Angle of view" indicates the angle of view for the image to be recognized. "Cut-off" indicates whether an object included in the image to be recognized is cut off at the edge of the image. "Latency" indicates the delay in the recognition result for the captured image (original image 320). "Frame rate" indicates the update period of the recognition result. Furthermore, "bus width" indicates the bus width for the imaging unit 1200 to output the captured image. Here, with regard to "bus width," if the bus width does not decrease for the recognition process using the original image 320, it is indicated with an "X (cross mark)."
[0145] Now, let us explain "1 / N x 1 / N reduction." "Distance" is marked with "X." This is because in "1 / N x 1 / N reduction," the number of pixels in both the row and column directions is reduced to 1 / N. Meanwhile, "angle of view" and "cut-off" are marked with "O" because the entire original image 320 is thinned out and used. Also, "latency" and "frame rate" are marked with "O" because the number of pixels is reduced compared to the original image 320, enabling high-speed processing.
[0146] The method according to the first embodiment, "1 / N×1 / N phase shift subsampling + information integration", will be described. When recognition processing for one phase period is performed, "distance" is equivalent to the original image 320, and is marked as "○". "Angle of view" and "cut-off" are marked as "○" because the entire original image 320 is used. Also, "latency" and "frame rate" are marked as "○" because recognition results can be output at each phase.
[0147] The "1 / N×1 / N crop" will be described. Regarding "distance," the cropped images #1 to #4 have the same resolution as the original image 320, so they are marked "○." On the other hand, regarding "angle of view," in order to see the entire angle of view of the original image 320, 2As it takes N frames to see the entire angle of view of the original image 320, the number of frames per frame is 1 / N, which is why it is marked as "X." As for "cut-off," as each of the cropped images #1 to #4 is a division of the original image 320, there is a risk of cut-off at the division position, so it is marked as "X." As for "latency," just like with "angle of view," it takes N frames to see the entire angle of view of the original image 320, so 2 In addition, the frame rate is set to N in order to view the entire field of view of the original image 320. 2 Frame required, 1 / N 2 It doubles and is marked as "x".
[0148] In addition, in any case, the "bus width" is marked "x" since the captured image is output from the imaging unit 1200 at the resolution of the original image 320.
[0149] In the evaluation example shown in FIG. 18, a comparison is made between “1 / N×1 / N reduction”, a method that has produced favorable evaluation results in the existing technology, and “1 / N×1 / N phase-shifted subsampling + information integration”, a method according to the first embodiment.
[0150] FIG. 19A is a diagram that shows a schematic diagram of a recognition process by "1 / N×1 / N reduction" of the existing technology. FIG. 19B is a diagram that shows a schematic diagram of a recognition process by "1 / N×1 / N phase shift subsampling+information integration" according to the first embodiment. Note that in FIG. 19A and FIG. 19B described later, the passage of time is represented from left to right in the diagram. Also, in FIG. 19A and FIG. 19B, the captured image (original image 320) that is the subject of the recognition process is the same.
[0151] Here, in the "1 / N×1 / N reduction" shown in Fig. 19A, one pixel each is thinned out in the row and column directions of an original image 320 (not shown), to generate 1 / 2×1 / 2=1 / 4 reduced images 321a, 321b, 321c, and 321d. Similarly, in the "1 / N×1 / N phase shift subsampling + information integration" shown in Fig. 19B, subsampling is performed on each divided area 35 of the original image 320, thinning out one pixel each in the row and column directions.
[0152] Therefore, the resolution in one frame is the same in the case of "1 / N x 1 / N reduction" shown in Fig. 19A and the case of "1 / N x 1 / N phase shift sub-sampling + information integration" shown in Fig. 19B. In the example of Fig. 19B, the pixel position to be sampled (pixel phase) is further shifted by one pixel for each frame.
[0153] First, the case of "1 / N x 1 / N reduction" will be described. In Fig. 19A, a reduced image 321a is generated from a captured image at a certain frame timing, and feature extraction processing is performed on this reduced image 321a using, for example, CNN. Then, recognition processing is performed based on the extracted feature. In the example of Fig. 19A, recognition results 62 and 63 in which a person is recognized are obtained by the recognition processing on the reduced image 321a.
[0154] At the next frame timing, reduced image 321b is generated from the captured image. Similar to the case of reduced image 321a, recognition results 62 and 63 are obtained by a recognition process based on the feature amount extracted from reduced image 321b. Similarly, recognition results 62 and 63 are obtained for reduced image 321c at the next frame timing and reduced image 321d at the frame timing after that by a recognition process based on the extracted feature amount. Thus, in the case of "1 / N x 1 / N reduction," only objects up to a distance according to the resolution of reduced images 321a, 321b, ... can be recognized.
[0155] Next, the case of "1 / N×1 / N phase shift subsampling + information integration" will be described. In FIG. 19B, at a certain frame timing, subsampling is performed on the captured image at the first phase to generate a sampled image 36φ1 of the first phase. The size of this sampled image 36φ1 is the same as the size of the reduced image 321a described above. A feature extraction process is performed on this sampled image 36φ1 using, for example, CNN in the same manner as in the example of FIG. 19A, and the feature is extracted. Then, a recognition process is performed based on the extracted feature. In the example of FIG. 19B, the recognition process on the sampled image 36φ1 produces recognition results 62 and 63 in which people are recognized, as in the case of FIG. 19A.
[0156] At the next frame timing, a sampled image 36φ2 having a second phase, which is shifted by one pixel from the phase of the sampled image 36φ1, is generated from the captured image. Recognition processing is performed based on the feature amount extracted from this sampled image 36φ2.
[0157] At this time, the recognition process is performed based on the feature amount obtained by integrating the feature amount extracted from the sampling image 36φ2 and the feature amount extracted from the sampling image 36φ1 used in the immediately preceding recognition process. As a result, in addition to the recognition results 62 and 63 based on the sampling image 36φ1, a recognition result 64 is obtained in which a person who is farther away than the people recognized as the recognition results 62 and 63 is recognized.
[0158] For the sampling image 36φ3 of the next frame timing, the feature amount based on the sampling image 36φ3 is integrated with the feature amount used in the immediately preceding recognition process to obtain a recognition result 65 in which a person further away than the person recognized as the above-mentioned recognition result 64 is recognized. Similarly, for the sampling image 36φ4 of the next frame timing, the feature amount based on the sampling image 36φ4 is integrated with the feature amount used in the immediately preceding recognition process to obtain a recognition result 66 in which a person further away than the person recognized as the above-mentioned recognition result 65 is recognized.
[0159] In this way, in the first embodiment, the feature amounts based on the subsampled sampled images 36φ1 to 36φ4 of a plurality of frames are sequentially integrated for each frame, and recognition processing is performed based on the integrated feature amounts. Therefore, for example, if the images of the frames from which the sampled images 36φ1 to 36φ4 are generated have temporal continuity, the feature amount obtained by integrating the feature amounts extracted from the sampled images 36φ1 to 36φ4 can be considered to be equivalent to the feature amount extracted from one captured image without subsampling.
[0160] Therefore, according to the recognition process of the first embodiment, it is possible to fully utilize the resolution of the camera (imaging unit 1200) to recognize distant objects. Also, since the recognition process is performed for each frame and the recognition result can be obtained, it is possible to recognize large objects on the image in a short time. For example, a small object on the image can be recognized in a short time. 2 Although the recognition is delayed by one frame, since such an object is expected to be located far away, a certain delay in the recognition result is tolerable.
[0161] (2-2-2-2. Comparison of recognition processing control) Next, the control of the recognition process by the existing technology will be described and compared with the control of the readout and recognition process according to the first embodiment described with reference to Fig. 14. Fig. 20 is a time chart showing an example of the control of the readout and recognition process when the image data 32 of the captured image is used without being reduced according to the existing technology. The meanings of the parts in Fig. 20 are the same as those in Fig. 14 described above, and therefore the description here will be omitted.
[0162] In the example of Fig. 20, the recognition process (step S20) for image data 32a based on the captured image of frame #1 requires a time equivalent to four frames. Since the recognition process is performed using image data 32a that is not reduced, it is possible to recognize distant objects, for example. However, due to restrictions on the amount of processing in the CPU 1205 and DSP 1203 (see Fig. 5) that perform the recognition process, a delay occurs in the recognition result based on the image data 32a with respect to the image data 32a.
[0163] In addition, the recognition process for the unreduced image data 32a takes a long time, so the update interval of the recognition result becomes longer, and the frame rate for the recognition result decreases. Furthermore, when the recognition process is performed based on the unreduced image data 32a, for example, a large memory capacity needs to be reserved for the image data 32a. Furthermore, in FIG. 20, while the recognition process for the image data 32a is being performed, the recognition process for the captured images of frames #2 to #4 following frame #1 cannot be performed, and the capture process for frames #2 to #4 is wasted.
[0164] Fig. 21 is a time chart showing an example of control of the readout and recognition process in the case of "1 / N x 1 / N reduction" by the existing technology. The meaning of each part in Fig. 21 is the same as that in Fig. 14 described above, so the explanation here is omitted.
[0165] In the example of FIG. 21, for example, image data 32a based on a captured image of frame #1 is reduced by thinning or the like (step S30), and the image data of the reduced image is subjected to recognition processing to obtain recognition result #a (step S31). Since the recognition processing is performed on the image data of the reduced image, the recognition result can be updated in a shorter time than when the recognition processing is performed on the above-mentioned non-reduced image data 32a. For example, when the reduced image is thinned out by every other pixel in the row and column directions of the image data 32a, the recognition processing can be performed in the same time as the recognition processing according to the first embodiment described with reference to FIG. 14, that is, the recognition processing by "1 / N×1 / N phase shift subsampling + information integration".
[0166] On the other hand, because the recognition process is performed based on a reduced image obtained by thinning out and reducing the image data 32a, the resolution of the image data 32a cannot be utilized, making it difficult to recognize distant objects. The same is true for the recognition results #b, #c, and #d of the image data 32b, 32c, and 32d of frames #2, #3, and #4, respectively.
[0167] In contrast, in the recognition process according to the first embodiment described with reference to Fig. 14, the timing of the recognition process and the like is the same as in the case of "1 / N x 1 / N reduction" shown in Fig. 21. However, in the example of Fig. 14, the feature amount, which is intermediate data in the recognition process, is integrated with the feature amount extracted from the image data of the next frame, and the recognition process is performed on that image data. Furthermore, subsampling of the image data is performed with a phase shift for each frame.
[0168] Therefore, it is possible to recognize distant objects by fully utilizing the resolution of the camera (imaging unit 1200). Also, since the recognition process can be performed for each frame and the recognition result can be obtained, it is possible to recognize large objects on the image in a short time. For example, a small object on the image that can be recognized by fully utilizing the resolution of the captured image can be recognized by, for example, N 2 Although the recognition is delayed by one frame, since such an object is expected to be located far away, a certain delay in the recognition result is tolerable.
[0169] (2-2-2-3. Comparison of recognition processing delays) Next, a description will be given of a delay in the recognition process due to the existing technology and a delay in the recognition process according to the first embodiment. Fig. 22 is a schematic diagram for describing a delay in the recognition result according to the first embodiment and a delay in the recognition result due to the existing technology.
[0170] In FIG. 22, time passes from left to right. The upper section (a) typically shows captured images 3101 to 3109 captured by the imaging unit 1200. The middle section (b) typically shows an example of a recognition result obtained by phase-shifted subsampling in the recognition process according to the first embodiment. The lower section (c) typically shows an example of a recognition result obtained by using the resolution of the captured images as is according to existing technology. The images 3111, 3112, and 3113 in section (b) and the images 3121, 3122, and 3123 in section (c) typically show images that mimic the captured images at the respective times.
[0171] In this example, as the existing technology in section (c), a recognition process is performed using the resolution of the captured image as is, without thinning out the captured image, in consideration of the recognition of distant objects. The frame rate of the captured image is assumed to be, for example, 20 fps or higher.
[0172] Each of the captured images 3101, 3102, ..., 3109, ... shown in section (a) of Fig. 22 is taken by a camera (imaging unit 1200) mounted on a vehicle. At this time, the vehicle on which the camera is mounted is moving forward. For the sake of explanation, in section (a) of Fig. 22, each of the captured images 3101 to 3109 is taken at times t1, t2, ..., t9, with one second divided into four equal parts (0.25 [s (second)]).
[0173] In section (a) of Fig. 22, each of the captured images 3101, 3102, ..., 3109, ... includes a human object 43 in the distance. Also, the captured images 3105 to 3107 further include a human object 44 in addition to the object 43. The object 44 suddenly appears from behind the vehicle 45 in the captured image 3105, and moves toward the vehicle (assumed to be the own vehicle) on which the camera is mounted in the captured images 3106, 3107, .... The own vehicle is required to quickly recognize this object 44.
[0174] In the example of Fig. 22, the recognition process by phase shift subsampling shown in section (b) is updated at a frame rate of 20 [fps], i.e., every 0.05 [s]. In contrast, in the existing technology shown in section (c), the recognition process is updated at a frame rate of 2 [fps], i.e., every 0.5 [s].
[0175] In the recognition process by phase-shifted subsampling shown in section (b), the object 43 included in the captured image 3101 is captured at a time t1 delayed by 0.05 [s] from the capture timing of the captured image 3101, as shown in an image 3111. 11 In the recognition process using subsampling, the recognition result 70 is updated at 0.05 [s] intervals.
[0176] On the other hand, in the recognition process using the resolution of the captured image shown in section (c), the object 43 is recognized at time t 20 In the recognition process using the resolution of the captured image, the recognition result 70 is obtained at time t 20 Time t 0.5[s] has passed since 21 It becomes.
[0177] Next, consider a captured image 3105 in which a human object 44 appears from behind a vehicle 45. In this case, in the recognition process using phase-shifted subsampling shown in section (b), as shown in image 3112, the human object 44 appears at time t 0.05 [s] after time t 5 when the captured image 3105 is acquired. 12 In the recognition process using phase-shifted subsampling, the recognition result 71 is updated every 0.05 [s], and the recognition result 71 is updated every t 12 Time t 0.5[s] after 13 In this example, as shown in image 3113, an object 44 moving toward the vehicle is obtained as recognition result 72.
[0178] In contrast, in the recognition process using the resolution of the captured image shown in section (c), the object 44 is captured at time t 0.5 [s] after time t 5 when the captured image 3105 is acquired. 22 and a recognition result 71 corresponding to the object 44 is obtained. That is, in the recognition process using the resolution of the captured image, the object 44 is recognized with a delay of 0.45 [s] compared to the recognition process using phase-shifted subsampling.
[0179] In addition, in the recognition process using the resolution of the captured image shown in section (c), the time t 20 From time t 22 During this time, the recognition result is updated at time t 21 Therefore, it is extremely difficult to check the state of the object 44 with high concurrency.
[0180] The relationship between the recognition process and vehicle braking will be explained briefly with reference to Fig. 23A and Fig. 23B. In Fig. 23A and Fig. 23B, the left side diagrams show a captured image (or a view from the front of the vehicle), and the right side diagrams show an image including a recognition result by the recognition process. In Fig. 23A, the upper part corresponds to the state at time t5 and the lower part corresponds to the state at time t7 with reference to Fig. 22.
[0181] Fig. 23A is a diagram showing an example of recognition processing using the resolution of a captured image. Referring to Fig. 22, at time t5, an object 44, which is a person, appears from behind a vehicle 45 in a captured image 3105. Here, the distance from the host vehicle (imaging unit 1200) to the object 44 at time t5 is set to distance A. Here, distance A is set to a distance that allows the host vehicle to avoid contact or collision with the object 44 by braking the host vehicle, such as by decelerating or braking.
[0182] In the recognition process using the resolution of the captured image, this time t5 (= time t 2223A ), a recognition result based on captured image 3103 at time t3, 0.5 [s] earlier, is obtained due to a delay. Captured image 3103 includes object 43 but does not include object 44, so that only recognition result 70 for object 43 is obtained, as shown in image 3122 in the upper part of FIG. 23A .
[0183] At time t7, the object 44 approaches the vehicle to a very close distance B. Distance B is a distance at which the vehicle is highly likely to come into contact with or collide with the object 44 even if the vehicle is slowed down or braked. For example, if the vehicle speed is 30 km / h, the vehicle moves about 4.2 m in 0.5 s. If the moving speed of the object 44 is ignored, distance B is about 4.2 m closer to distance A.
[0184] In the recognition process using the resolution of the captured image, a recognition result is obtained at time t7 using the captured image 3105 captured at time t5, which is 0.5 [s] earlier than the delay. That is, at time t7, as shown in the image 3123 on the right side of the lower part of Fig. 23A, a recognition result 71 of the object 44 at the distance A is obtained, and there is a high possibility that braking, such as deceleration, will not be possible in time.
[0185] FIG. 23B is a diagram showing an example of recognition processing by phase-shifted subsampling. In this case, referring to FIG. 22, at time t 22 In this case, a recognition result based on the captured image 3105 at time t5, 0.05 [s] earlier due to a delay, is obtained. In this case, the recognition process produces a recognition result 71 corresponding to the object 44 included in the captured image 3105, as shown in the image 3112 on the right side of FIG. 23B. 22 In the example, the object 44 approaches the vehicle at a distance A' which is slightly closer than the distance A.
[0186] As described above, if the speed of the vehicle is 30 km / h, the distance traveled in 0.05 seconds is approximately 40 cm. Therefore, if the speed of the object 44 is ignored, the distance A' is approximately 40 cm closer to the distance A. In this case, the vehicle can be prevented from coming into contact with or colliding with the object 44 by applying the brakes such as slowing down or braking.
[0187] In this way, by using the recognition process according to the first embodiment (recognition process using phase-shifted subsampling), it is possible to recognize an object more quickly, and it is possible to more reliably perform avoidance operations, for example by braking the vehicle.
[0188] (2-2-2-4. Examples of improvements achieved by the recognition process according to the first embodiment) Next, an example of an improvement over the recognition processing of the existing technology by the recognition processing according to the first embodiment will be described with reference to Fig. 24. In Fig. 24, each column represents, from the left, "1 / 4 reduction" which is the existing technology, "no reduction" which is the existing technology, and "phase shift subsampling" which is the recognition processing according to the first embodiment. Also, each numerical value shown in Fig. 24 is an example numerical value for the purpose of explanation, and is not limited to these values.
[0189] Note that "1 / 4 reduction" corresponds to "1 / N x 1 / N reduction" explained with reference to Fig. 16. "No reduction" corresponds to the above-mentioned "recognition process using the resolution of the captured image." Furthermore, "phase shift subsampling" assumes that the size of the divided region 35 is 8 pixels x 8 pixels, and subsampling is performed by thinning out every third pixel in both the row and column directions, thereby selecting four pixels from the divided region 35 as sampling pixels.
[0190] In addition, in FIG. 24, each row represents, from the top, "camera resolution," "bus width," "CNN resolution," "longest recognition distance," "latency," "frame rate," and "idle distance." Of these, "camera resolution" is the resolution of the image captured by the imaging unit 1200, and in the example of FIG. 24, it is 1920 pixels x 1080 pixels in all of "1 / 4 reduction," "no reduction," and "phase shift subsampling." In addition, "bus width" indicates the bus width for the imaging unit 1200 to output the captured image, and in the example of FIG. 24, it is 120 [MBps (Mega byte per second)] in all of "1 / 4 reduction," "no reduction," and "phase shift subsampling."
[0191] "CNN resolution" is the resolution when extracting features using CNN in the recognition unit 220. In the example of Fig. 24, "no reduction" has a resolution of 1920 pixels x 1080 pixels, which is equivalent to the camera resolution, and "1 / 4 reduction" has a resolution of 480 pixels x 270 pixels, which is 1 / 4 of that in each matrix. In other words, "no reduction" has a CNN resolution that is 16 times (=4 x 4) that of "1 / 4 reduction".
[0192] On the other hand, in "phase shift subsampling", every third pixel is thinned out in both the row and column directions, so that the "CNN resolution" is set to 480 pixels x 270 pixels, which is 1 / 4 of the "camera resolution". Here, in the first embodiment, all pixel positions in the divided region 35 having a size of 8 pixels x 8 pixels are selected as pixel positions of the sampled pixels in one cycle. Therefore, the CNN resolution when one cycle of subsampling is completed is equivalent to 1920 pixels x 1080 pixels, which is 480 pixels x 270 pixels x 4 x 4.
[0193] The "maximum recognition distance" is the longest distance from the imaging unit 1200 to an object at which the object can be recognized. In the example of Fig. 24, it is 20 [m] for "1 / 4 reduction" and 80 [m] for "no reduction" and "phase-shifted subsampling". In the case of "phase-shifted subsampling", when one cycle of subsampling is completed, it becomes 80 [m], which is the same as "no reduction".
[0194] "Latency" indicates the delay in the recognition result for the captured image, and is 50 [ms] for "1 / 4 reduction" and 800 [ms] for "no reduction", which is 16 times (=4 x 4) that of "1 / 4 reduction". On the other hand, in "phase-shifted subsampling", it is 50 [ms] for one subsampling, and when one cycle of subsampling is completed, it is 800 [ms], which is the same as "no reduction". In "phase-shifted subsampling", the symbol "@ (at sign)" indicates the maximum recognition distance. According to this, in one subsampling, the latency is small at 50 [ms], while the maximum recognition distance is short at 20 [m]. Also, when one cycle of subsampling is completed, the maximum recognition distance is extended to 80 [m].
[0195] "Frame rate" indicates the update cycle of the recognition result, and in the example of Fig. 24, it is set to 20[fps] for "1 / 4 reduction" and "phase shift subsampling", and 1.25[fps] for "no reduction". Note that the frame rate of 20[fps] for "phase shift subsampling" is the frame rate for one subsampling.
[0196] The "free running distance" is the distance traveled from the appearance of an object that should stop the vehicle, for example, until braking or other braking is actually performed to stop the vehicle. More specifically, the "free running distance" is the distance traveled by the vehicle from the time an object appears until the object is recognized by a recognition process and braking of the vehicle is started according to the recognition result. Here, the free running distance is calculated assuming the vehicle's running speed is 50 [km / h]. In order to calculate the "free running distance", it is necessary to take into account the value of the "latency" described above.
[0197] The "free-running distance" is 0.7 [m] for "1 / 4 reduction" and 11 [m] for "no reduction". As mentioned above, the "latency" is 16 times that of "1 / 4 reduction". Therefore, the "free-running distance" is also 11 [m], which is approximately 16 times 0.7 [m]. In addition, while the free-running distance is short in "1 / 4 reduction", the maximum recognition distance is 20 [m], and it is difficult to recognize objects at a distance of more than 20 [m]. In addition, while it is possible to recognize objects at a distance of up to 80 [m] in "no reduction", the free-running distance is long at 11 [m]. In the case of "no reduction", for example, even for objects at a short distance of up to 20 [m], the free-running distance is 11 [m], and it may be difficult to avoid contact or collision with the object.
[0198] In contrast, with "phase-shifted subsampling," objects up to 20 m away can be recognized with a latency of 50 ms during the first subsampling of one cycle. Therefore, for objects up to 20 m away, the "free-running distance" is 0.7 m, the same as with "1 / 4 reduction." For objects between 20 m and 80 m away, the "free-running distance" is 11 m, the same as with "no reduction."
[0199] "Maximum recognition distance (pedestrian)" indicates the longest recognition distance when the vehicle needs to brake when a pedestrian is recognized. This "Maximum recognition distance (pedestrian)" is the value obtained by subtracting the "free running distance" from the "Maximum recognition distance" described above, and is approximately 19 [m] for "1 / 4 reduction" and approximately 69 [m] for "no reduction" and "phase shift subsampling".
[0200] Thus, the "phase shift subsampling" recognition processing method according to the first embodiment is a dramatic improvement over the recognition processing methods of the existing technology in that it can recognize objects at long distances while suppressing the free-running distance at short distances. Therefore, the recognition processing according to the first embodiment is capable of recognizing objects at long distances, and the free-running distance at short distances is short at 0.7 [m], making it easy to avoid contact or collision with objects at short distances.
[0201] (2-2-2-5. Effective application scenes of the recognition process according to the first embodiment) Next, an effective application scene of the recognition process according to the first embodiment will be described with reference to Fig. 25A and Fig. 25B. Note that states 400 to 402 in Fig. 25A and states 403 to 405 in Fig. 25B each show a state in chronological order of a vehicle 410 (hereinafter, referred to as the own vehicle 410) traveling on a road. The own vehicle 410 is equipped with an information processing device 1b capable of executing the recognition process according to the first embodiment, and is capable of controlling a braking control system and the like based on the recognition result by the information processing device 1b.
[0202] Fig. 25A is a schematic diagram showing a first example of an effective application scene of the recognition processing according to the first embodiment. Referring to a state 400 in Fig. 25A, another vehicle 411a is stopped ahead of a host vehicle 410 traveling forward, and a person 412a is present in a portion of the vehicle 411a that is in the shadow of the host vehicle 410. In addition, another vehicle 411b is traveling in the oncoming lane of the traveling lane in which the host vehicle 410 is traveling.
[0203] State 401 shows a state where a vehicle 411b strays from the lane 420 and enters the lane of the host vehicle 410 after a predetermined time has elapsed since state 400. In this state 402, the information processing device 1b mounted on the host vehicle 410 can recognize the distant vehicle 411b as an object. Therefore, the information processing device 1b can recognize the vehicle 411b straying from the lane 420, and control the host vehicle 410 according to the recognition result to cause the host vehicle 410 to take an avoidance action to avoid the vehicle 411b.
[0204] State 402 shows a schematic diagram of a case where a person 412a suddenly appears from behind a vehicle 411a after a predetermined time has elapsed since state 400. In this state 402, the information processing device 1b mounted on the vehicle 410 can recognize the person 412a at a short distance as an object with a short latency. Therefore, the information processing device 1b can recognize the person 412a and start an operation to control the vehicle 410 according to the recognition result in a short free running distance, thereby making it possible to avoid contact or collision with the person 412a.
[0205] Fig. 25B is a schematic diagram showing a second example of an effective application scene of the recognition processing according to the first embodiment. Referring to state 403 in Fig. 25B, an obstacle 413 such as a wall is present in an area outside the road ahead of the host vehicle 410 traveling forward, and a person 412b is present in a part of the obstacle 413 that is in the shadow of the host vehicle 410. Also, a person 412c is present at a long distance ahead in the lane in which the host vehicle 410 is traveling.
[0206] A state 404 is a schematic diagram showing a state in which a person 412c is recognized by the information processing device 1b of the vehicle 410. The information processing device 1b can recognize objects at a long distance. Therefore, when a person 412c at a long distance in the traveling direction of the vehicle 410 is recognized, the information processing device 1b can control the vehicle 410 to gradually decelerate.
[0207] State 405 shows a schematic diagram of a person 412b suddenly appearing from behind a shield 413 after a predetermined time has elapsed since state 403. In this state 405, the information processing device 1b mounted on the vehicle 410 can recognize the person 412b at a short distance as an object with short latency. Therefore, the information processing device 1b can recognize the person 412b and start an operation to control the vehicle 410 according to the recognition result within a short free running distance, thereby making it possible to avoid contact or collision with the person 412b.
[0208] (2-3. Modifications of the First Embodiment) Next, a modification of the first embodiment will be described.
[0209] (2-3-1. First modified example) First, a first modified example of the first embodiment will be described. The first modified example of the first embodiment is an example in which the spatial interval of sampling by sub-sampling is changed according to the object of recognition, the situation in which the recognition process is performed, etc. Figures 26A and 26B are schematic diagrams showing examples of divided areas 35 and 35' with different sampling intervals.
[0210] 26A is a schematic diagram showing an example in which pixels 300 included in the divided region 35 are thinned out every other pixel in the row and column directions, and four pixels 300sa1, 300sa2, 300sa3, and 300sa4 are selected as sampled pixels. In this case, the sampling interval is two pixels. In the example of FIG. 26A, the size of the divided region 35 is 4 pixels by 4 pixels.
[0211] Fig. 26B is a schematic diagram showing an example in which the pixels 300 included in the divided region 35 are thinned out to every third pixel in the row and column directions, and four pixels 300sa1, 300sa2, 300sa3, and 300sa4 are selected as sampled pixels. In this case, the sampling interval is four pixels. In the example of Fig. 26B, the size of the divided region 35 is 8 pixels x 8 pixels.
[0212] As in the example of Fig. 26B, when the sampling interval is widened from that of Fig. 26A, a smaller number of sampled pixels are selected in one subsampling, and the latency is reduced. In this case, for example, the recognizable distance in the recognition process by the first subsampling is shortened, and the resolution of the sampled image is low, making it suitable for recognizing large objects on the image.
[0213] On the other hand, if the sampling interval is narrowed, as in Fig. 26A, more sampled pixels are selected in one subsampling, and the latency increases. In this case, for example, the recognizable distance is longer in the recognition process by the first subsampling, and the resolution of the sampled image is higher than in the example of Fig. 26B, making it suitable for recognizing small objects on the image.
[0214] For example, when the information processing device 1b according to the first modified example of the first embodiment is used for vehicle-mounted applications, it is important to recognize objects at a greater distance when the vehicle equipped with the information processing device 1b is traveling at high speed. Therefore, the preprocessing unit 210 narrows the sampling interval, for example, as shown in FIG. 26A, and the recognition unit 220 performs the recognition process using a sampling image with a higher resolution. On the other hand, when the vehicle is traveling, for example, in an urban area, latency is important. Therefore, the preprocessing unit 210 widens the sampling interval, for example, as shown in FIG. 26B, and the recognition unit 220 performs the recognition process using a sampling image with a lower resolution.
[0215] For example, the information processing device 1b can acquire location information indicating a current location, and determine whether or not the current location is in an urban area based on the acquired location information and map information corresponding to the current location. The information processing device 1b can be configured to estimate the current location using, for example, SLAM (Simultaneous Localization And Mapping) or GNSS (Global Navigation Satellite System). In addition, the information processing device 1b acquires information indicating a traveling speed from a vehicle via the interface 1204. In the information processing device 1b, for example, the pre-processing unit 210 can set a sampling interval based on the current location information and the traveling speed information. The sampling interval can be dynamically set based on this information.
[0216] However, it is also possible to adjust the speed of the recognition process and the recognizable distance by controlling the clock frequency supplied to the preprocessing unit 210 and the recognition unit 220, or by narrowing the recognition area in the captured image that is the target of the recognition process. For example, when driving at high speed in an urban area, recognition results for both long and short distances are required. In such a case, control is performed by increasing the clock frequency, narrowing the recognition area, etc.
[0217] In this way, in the first modification of the first embodiment, the recognition result can be obtained more stably by appropriately switching between the recognition process by subsampling and the normal recognition process. Also, since the recognition process by subsampling is executed in addition to the normal recognition process, the load on the recognition unit 220 is reduced, and power consumption can be reduced.
[0218] (2-3-2. Second modified example) Next, a second modified example of the first embodiment will be described. The second modified example of the first embodiment is an example of controlling an external device according to the latency of the recognition process. As an example, when the information processing device 1b according to the second modified example of the first embodiment is used for vehicle-mounted use, the speed (vehicle speed) of the vehicle in which the information processing device 1b is mounted is controlled according to the latency of the recognition process. For example, when the sampling pixels are selected as shown in FIG. 26A described above, it is suitable for recognizing small-sized objects on the image, but the latency is large.
[0219] For example, in a school zone, there are many children as small objects on the image. Therefore, when the current location is determined to be a school zone based on the location information, for example, the information processing device 1b according to the second modification of the first embodiment selects sub-sampling using the sampling pixels of Fig. 26A, which has a higher resolution. Since sub-sampling using the sampling pixels of Fig. 26A has a large latency, the information processing device 1b limits the vehicle speed of the vehicle on which the information processing device 1b is installed.
[0220] (2-3-3. Third modified example) Next, a third modified example of the first embodiment will be described. The third modified example of the first embodiment is an example in which the recognition process by subsampling according to the above-mentioned first embodiment (hereinafter referred to as the recognition process by subsampling) and the normal recognition process without subsampling are switched and executed. Here, the normal recognition process refers to the recognition process using all the pixels of the captured image without performing subsampling and reduction on the captured image.
[0221] For example, normal recognition processing can be performed at regular time intervals in addition to the subsampling recognition processing. This makes it possible to verify the subsampling recognition processing, for example. Also, in an emergency, it is possible to switch from the subsampling recognition processing to the normal recognition processing, for example. This makes it possible to improve the stability of the recognition processing.
[0222] Here, when switching from the subsampling recognition process to the normal recognition process, for example, the latency increases, and the speed of the recognition result decreases. Therefore, when switching to the normal recognition process, it is preferable to increase the frequency of the clock supplied to the preprocessing unit 210 and the recognition unit 220.
[0223] Furthermore, the recognition unit 220 can switch between the recognition process using subsampling and the normal recognition process depending on the reliability of the recognition result obtained by the recognition process. For example, when the recognition unit 220 is performing the recognition process using subsampling, it acquires the reliability of the recognition result obtained by the recognition process. When the reliability is less than a predetermined value, for example, the recognition unit 220 instructs the pre-processing unit 210 to switch from the recognition process using subsampling to the normal recognition process. In response to this instruction, the pre-processing unit 210 stops the subsampling of the captured image and passes all pixels of the captured image to the recognition unit 220. The recognition unit 220 performs the normal recognition process based on all pixels of the captured image passed from the pre-processing unit 210.
[0224] In the normal recognition processing to which the recognition processing by subsampling has been switched in this way, the recognition unit 220 acquires the reliability of the recognition result by the recognition processing. When the reliability is, for example, equal to or greater than a predetermined value, the recognition unit 220 instructs the pre-processing unit 210 to switch from the normal recognition processing to the recognition processing by subsampling. In response to this instruction, the pre-processing unit 210 performs subsampling of the captured image and passes the selected sampled pixels to the recognition unit 220. The recognition unit 220 executes the recognition processing by subsampling based on the sampled pixels passed from the pre-processing unit 210.
[0225] In this way, in the third modified example of the first embodiment, the recognition result can be obtained more stably by switching between the subsampling recognition process and the normal recognition process depending on the reliability of the recognition result by the recognition process. Also, when the reliability of the recognition result is high, the subsampling recognition process is executed, so that the load on the recognition unit 220 is reduced and power consumption can be reduced.
[0226] (2-3-4. Fourth Modified Example) Next, a fourth modified example of the first embodiment will be described. In the fourth modified example of the first embodiment, a pixel 300 at a pixel position arbitrarily selected in a captured image is used as a sampling pixel by sub-sampling.
[0227] 27 is a schematic diagram for explaining subsampling according to the fourth modification of the first embodiment. As shown in FIG. 27, in the fourth modification of the first embodiment, the pre-processing unit 210 processes original images 320Rn1, 320Rn2, ..., 320Rn which are captured images of each frame. n For each frame, n frames are set as one cycle, and patterns R#m_1, R#m_2, ..., R#m_n are set based on a number of arbitrarily selected pixel positions. As an example, when the total number of pixels included in a frame is s and the number of frames in one cycle is n, the preprocessing unit 210 sets patterns R#m_1, R#m_2, ..., R#m_n each including (s / n) pixel positions.
[0228] Here, the arbitrarily selected pixel positions include, for example, discrete and non-periodic pixel positions. For example, the preprocessing unit 210 can select the pixel positions using pseudo-random numbers. In addition, it is preferable that the selected pixel positions are different for each frame, but some pixel positions may overlap between frames.
[0229] The recognition unit 220 recognizes the original images 320Rn1, 320Rn2, ..., 320Rn n From each of the patterns R#m_1, R#m_2, . . . , R#m_n, pixels 300 at pixel positions included in the patterns R#m_1, R#m_2, . . . , R#m_n are selected as sampled pixels, and recognition processing is performed based on the selected sampled pixels.
[0230] In the fourth modified example of the first embodiment, the original images 320Rn1, 320Rn2, . . . , 320Rn nThe recognition process is performed based on sampled pixels selected according to a pattern R#m_1, R#m_2, ..., R#m_n of multiple pixel positions that are arbitrarily selected from each of the above and that differ for each frame. Therefore, it is possible to reduce sampling artifacts compared to using pixels 300 at pixel positions that are periodically selected, for example, every other pixel or every third pixel in the row and column directions, as sampled pixels.
[0231] For example, the recognition process according to the fourth modification of the first embodiment can suppress the occurrence of erroneous recognition or failure to recognize a time-based periodic pattern such as a flicker. Also, the recognition process can suppress the occurrence of erroneous recognition or failure to recognize a spatial periodic pattern (such as a fence or a mesh-like structure).
[0232] In the above description, for example, the original images 320Rn1, 320Rn2, . . . , 320Rn n In each of the above, the sampling pixels are selected according to pixel positions that are arbitrarily set for the entire image, but this is not limited to the above example. For example, the sampling pixels may be selected according to pixel positions that are arbitrarily set within the divided region 35 that is obtained by dividing the original image 320.
[0233] (2-3-5. Fifth Modification) Next, a fifth modified example of the first embodiment will be described. The fifth modified example of the first embodiment is an example in which the arrangement of pixel positions of sampling pixels for performing recognition processing is changed according to the recognition result.
[0234] Fig. 28 is a schematic diagram for explaining the subsampling according to the fifth modified example of the first embodiment. In Fig. 28, original images 320φ1, 320φ2, 320φ3, and 320φ4 are schematic diagrams showing images for one period of subsampling in which pixel positions are selected every predetermined number of pixels in the row and column directions. The recognition unit 220 executes recognition processing based on the sampled pixels selected from these original images 320φ1, 320φ2, 320φ3, and 320φ4, respectively.
[0235] The recognition unit 220 sets an attention area for the captured image based on the recognition results for the original images 320φ1, 320φ2, 320φ3, and 320φ4. As an example, when the recognition unit 220 obtains a recognition result in which an object is recognized with low reliability, the recognition unit 220 sets an area of a predetermined range including the object in the captured image as an attention area. The pre-processing unit 210 sets the pixel positions of the sampling pixels for this attention area. In the example of FIG. 28, pixel positions Ptφ1, Ptφ2, Ptφ3, and Ptφ4 of the sampling pixels are set for each of the images 320Pt1, 320Pt2, 320Pt3, and 320Pt4 in which attention areas are set for the original images 320φ1, 320φ2, 320φ3, and 320φ4.
[0236] At this time, the pre-processing unit 210 can set all pixel positions in the region of interest as pixel positions of the sampling pixels without thinning out the pixels. Without being limited to this, the pre-processing unit 210 may set pixel positions of the sampling pixels for the region of interest at a sampling interval narrower than the sampling interval of the sampling pixels set for the original images 320φ1 to 320φ4. Furthermore, the pre-processing unit 210 may set pixel positions of the sampling pixels for the region of interest at a sampling interval equal to the sampling interval of the sampling pixels set for the original images 320φ1 to 320φ4.
[0237] In this way, in the fifth modified example of the first embodiment, pixel positions of the sampling pixels are set for the attention area set for the captured image based on the recognition results for the original images 320φ1 to 320φ4. This reduces the load on the recognition unit 220, making it possible to reduce power consumption. In addition, by setting pixel positions of the sampling pixels at narrower sampling intervals for the attention area, it becomes possible to obtain more accurate recognition results at a higher speed.
[0238] (2-3-6. Sixth Modification) Next, a sixth modified example of the first embodiment will be described. The sixth modified example of the first embodiment is an example in which exposure by the image capturing section 1200 is controlled in each phase of sub-sampling in one cycle.
[0239] Fig. 29 is a schematic diagram showing an example of exposure control according to the sixth modified example of the first embodiment. In this example, the divided regions 35 (not shown) are set to a size of 4 pixels x 4 pixels, and subsampling is performed in each divided region 35 by thinning out every other pixel in both the row and column directions. Sections (a), (b), (c), and (d) of Fig. 29 respectively show an original image 320Expφ1 in which the first phase is subsampled, an original image 320Expφ2 in which the second phase is subsampled, an original image 320Expφ4 in which the third phase is subsampled, and an original image 320Expφ4 in which the fourth phase is subsampled.
[0240] Here, the preprocessing unit 210 sequentially sets shorter exposure times for each of the original images 320Expφ1, 320Expφ2, 320Expφ3, and 320Expφ4 than the previous original image. In this way, by setting different exposure times for each of the original images 320Expφ1, 320Expφ2, 320Expφ3, and 320Expφ4 in one cycle of subsampling, the dynamic range for luminance can be expanded.
[0241] (2-3-7. Seventh Modification) Next, a seventh modified example of the first embodiment will be described. The seventh modified example of the first embodiment is an example in which an analog gain for a pixel signal by the imaging unit 1200 is controlled in each phase of subsampling in one period. For example, when reading out pixel signals of each original image 320 in which subsampling with different phases is performed from the pixel array unit 1001, the pre-processing unit 210 sets a different analog gain for each subsampling phase in the imaging unit 1200.
[0242] In the imaging unit 1200, the control unit 1100 passes information indicating the set analog gain to the AD conversion unit 1003. The AD conversion unit 1003 controls the gain of a pixel signal input to each AD converter 1007 included in the AD conversion unit 1003 via a vertical signal line VSL, in accordance with the information indicating the analog gain.
[0243] In this way, by setting different analog gains for the original images 320 having different subsampling phases in one period of subsampling, the dynamic range for luminance can be expanded.
[0244] The sixth modified example of the first embodiment and the seventh modified example of the first embodiment differ in whether the brightness of the original image 320 is controlled by the exposure time or the analog gain. When the brightness is controlled by the exposure time, a longer exposure time makes it possible to obtain the original image 320 with high brightness and suppress noise in the obtained original image 320. On the other hand, a longer exposure time increases blur in the original image 320.
[0245] In contrast, when brightness is controlled by analog gain, even when the analog gain is increased to obtain a bright original image 320, the blur in the original image 320 does not change. On the other hand, when the analog gain is increased, a high-brightness original image 320 can be obtained, but noise increases.
[0246] Therefore, it is preferable to use the sixth modified example of the first embodiment and the seventh modified example of the first embodiment depending on the purpose. For example, when performing recognition processing on a dynamic scene, the analog gain is increased to suppress the occurrence of blur in order to obtain a high-luminance original image 320. Also, when performing recognition processing on a static scene, the exposure time is lengthened to suppress the occurrence of noise in order to obtain a high-luminance original image 320.
[0247] [3. Second embodiment] Next, a second embodiment of the present disclosure will be described. The second embodiment of the present disclosure is an example in which a sensor unit 10b including a pixel array unit 1001, a recognition unit 220, and a configuration corresponding to a pre-processing unit 210 are integrally incorporated into a CIS having a layered structure.
[0248] (3-1. Configuration Example According to Second Embodiment) Fig. 30 is a block diagram showing an example of a configuration of an information processing device according to the second embodiment. In Fig. 30, the information processing device 1c includes a sensor unit 10c and a recognition unit 220. The sensor unit 10c also includes a pixel array unit 1001 and a readout control unit 230. The readout control unit 230 includes, for example, a function corresponding to the pre-processing unit 210 described in the first embodiment and a function of the control unit 1100 in the imaging unit 1200.
[0249] In addition, in FIG. 30, of the components described with reference to FIG. 5, the vertical scanning unit 1002, the AD conversion unit 1003, and the signal processing unit 1101 will be described as being included in the pixel array unit 1001.
[0250] The read control unit 230 supplies the pixel array unit 1001 with a control signal that specifies a pixel circuit 1000 from which a pixel signal is to be read out. For example, the read control unit 230 can specify a line from which a pixel signal is to be read out to the pixel array unit 1001. Without being limited to this, the read control unit 230 can also specify a pixel circuit 1000 from which a pixel signal is to be read out to the pixel array unit 1001 on a pixel circuit 1000 basis. At this time, the read control unit 230 can specify, to the pixel array unit 1001, a pixel circuit 1000 that corresponds to the pixel position of a sampling pixel by phase-shifted sub-sampling described in the first embodiment.
[0251] The pixel array unit 1001 converts a pixel signal read from a specified pixel circuit 1000 into digital pixel data, and passes this pixel data to the read control unit 230. The read control unit 230 passes one frame of pixel data passed from the pixel array unit 1001 to the recognition unit 220 as image data. This image data is a sampled image obtained by phase-shifted sub-sampling. The recognition unit 220 performs recognition processing on the passed image data.
[0252] In the second embodiment, the information processing device 1c can be configured by a two-layer stacked CIS in which semiconductor chips are stacked in two layers, as described with reference to FIG. 6A. Referring to FIG. 6A, a pixel section 2020a is formed in a first-layer semiconductor chip, and a memory+logic section 2020b is formed in a second-layer semiconductor chip. The pixel section 2020a includes at least the sensor section 10c in the information processing device 1c. The memory+logic section 2020b includes, for example, a drive circuit for driving the pixel array section 1001, and also includes a read control section 230 and a recognition section 220. The memory+logic section 2020b can further include a frame memory.
[0253] As another example, the information processing device 1c can be configured with a three-layered stacked CIS in which semiconductor chips are stacked in three layers, as described with reference to FIG. 6B. In this case, the pixel section 2020a described above is formed in the semiconductor chip of the first layer, the memory section 2020c including, for example, a frame memory is formed in the semiconductor chip of the second layer, and the logic section 2020d corresponding to the memory + logic section 2020b described above is formed in the semiconductor chip of the third layer. In this case, the logic section 2020d includes, for example, a drive circuit for driving the pixel array section, a read control section 230, and a recognition section 220. In addition, the memory section 2020c can include a frame memory and a memory 1202.
[0254] Next, a more specific configuration example for realizing the recognition process according to the second embodiment will be described. Fig. 31 is a time chart of an example showing the control of the readout and recognition process according to the second embodiment. The meanings of the parts in Fig. 31 are the same as those in Fig. 14 described above, so the description here will be omitted.
[0255] In the following, as described with reference to section (b) of Fig. 11, the divided region 35, which is the unit of subsampling, is assumed to be a region consisting of 4 pixels x 4 pixels, and the pixels to be sampled are selected by thinning out every other pixel in both the row and column directions. Furthermore, the captured image of frame #1 is subject to subsampling in a first phase, the captured image of frame #2 is subject to subsampling in a second phase, the captured image of frame #3 is subject to subsampling in a third phase, and the captured image of frame #4 is subject to subsampling in a fourth phase.
[0256] In the second embodiment, the read control unit 230 selectively reads out lines including sampling pixels in the imaging process of each of frames #1 to #4. For example, in frame #1, the sampling pixels are selected based on the top left pixel of the divided area 35, and in frame #2, the sampling pixels are selected based on the pixel adjacent to the top left pixel of the divided area 35. In other words, if the line at the top edge of the frame is the first line, odd-numbered lines are selectively read out in frames #1 and #2, and even-numbered lines are selectively read out in frames #3 and #4.
[0257] For example, in frame #1, the read control unit 230 selects pixel data of a sampling pixel from the pixel data of each read line, and generates a sampling image 36φ1 from the selected pixel data (step S10a). The read control unit 230 passes the generated sampling image 36φ1 to the recognition unit 220. The recognition unit 220 executes recognition processing based on the sampling image 36φ1 passed from the read control unit 230 (steps S11, S12, and S13), and outputs the recognition result φ1.
[0258] Thereafter, in the same manner for frames #2, #3, and #4, the read control unit 230 generates sampled images 36φ2, 36φ3, and 36φ4 from sampled pixels selected from the pixel data of each read line (steps S10b, S10c, and S10d). The recognition unit 220 executes recognition processing based on the sampled images 36φ2, 36φ3, and 36φ4 generated by the read control unit 230 (steps S11, S12, and S13), and outputs the recognition results φ2, φ3, and φ4.
[0259] Further, the recognition unit 220 sequentially integrates the feature amounts extracted from the sampling images 36φ1 to 36φ4 by the recognition process, similarly to the first embodiment. The recognition unit 220 uses the integrated feature amounts to output the recognition results φ1 to φ4 based on the sampling images 36φ1 to 36φ4, respectively. That is, the amount of information of the recognition results φ1 to φ4 by the recognition process based on the frames #1 to #4 increases with each recognition process, and the obtained recognition results become more detailed with each recognition process.
[0260] In this way, in the second embodiment, the sub-sampling process is performed in the sensor unit 10c. Therefore, it is not necessary to read from all pixel circuits 1000 included in the pixel array unit 1001. Therefore, it is possible to further reduce the delay in the recognition process compared to the first embodiment described above. In addition, since the pixel circuits 1000 of the line including the sampling pixel are selectively read from all pixel circuits 1000, it is possible to reduce the amount of pixel signals read from the pixel array unit 1001 and reduce the bus width.
[0261] In the second embodiment, each pixel circuit 1000 is read out by line thinning in the pixel array unit 1001. Therefore, it is possible to reduce distortion of an image captured by a rolling shutter. It is also possible to reduce power consumption during image capture in the pixel array unit 1001. Furthermore, in the lines thinned out by subsampling, it is also possible to capture an image by changing the imaging conditions such as exposure for the lines read out by subsampling.
[0262] (3-2. Application Example of the Second Embodiment) Next, an application example of the recognition process of the second embodiment will be described.
[0263] (3-2-1. First application example) The first application example is an application to recognition processing of a high-resolution captured image such as a 4K resolution image. Fig. 32 is a schematic diagram for explaining the first application example of the second embodiment. The 4K resolution image is illustrated, on the left side of Fig. 32, as an image of 1920 pixels x 1080 pixels (called an HD image), and on the right side, as a 4K resolution image of 4096 pixels x 2160 pixels. The 4K resolution image has twice the number of pixels in both the row and column directions of the HD image, and the number of pixels is four times that of the HD image.
[0264] In the second embodiment, at least a part of the subsampling process is performed inside the sensor unit 10c when pixel signals are read out from the pixel array unit 1001, so that the amount of data handled in one frame can be reduced. Also, the recognition unit 220 executes the recognition process for each frame based on each sampled image obtained by the subsampling process. Therefore, it is possible to obtain recognition results with high simultaneity for captured images. Furthermore, the recognition unit 220 sequentially integrates feature amounts between frames, so that it is possible to obtain more accurate recognition results that effectively utilize 4K resolution images.
[0265] (3-2-2. Second application example) The second application example is an application to a recognition process for a sampling image whose resolution has been reduced by subsampling. Here, as the second application example, a UI (User Interface) and an improvement in UX (User Experience) by the UI will be described. FIG. 33 is a schematic diagram for explaining a second application example of the second embodiment. The left side of FIG. 33 typically shows a UI according to an example in which a captured image is used without being reduced among existing techniques, and the right side typically shows a UI according to the second embodiment. As described above, in the recognition process according to the second embodiment, by sequentially integrating feature amounts for each frame, it is possible to obtain a recognition result with the same resolution as when a recognition process is performed using a captured image without being reduced.
[0266] According to the existing technology, as described with reference to FIG. 20, it takes a time for multiple frames to obtain a recognition result for a certain frame. Therefore, the recognition result is updated every multiple frames. This means that operation between frames is difficult, and is considered to be undesirable as a UX. In contrast, in the second embodiment, it is possible to update the recognition result for each frame, although at a low resolution, based on the subsampled sampling image. Therefore, operation between frames is easy, and is considered to be preferable as a UX. This makes it possible to improve the UX of the UI related to the recognition processing according to the second embodiment compared to the UI of the existing technology that uses the captured image without reducing it.
[0267] It should be noted that this second application example is applicable not only to the second embodiment but also to the above-mentioned first embodiment and each of its modified examples.
[0268] (3-2-3. Third application example) The third application example is an application example for reducing data transfer and power consumption. FIG. 34 is a schematic diagram for explaining the third application example of the second embodiment. Section (a) of FIG. 34 typically shows a configuration example of an information processing device 1c' that performs recognition processing using a captured image without reducing it, among existing technologies. Section (b) typically shows a configuration example of an information processing device 1c according to the second embodiment. Note that these information processing devices 1c and 1c' are assumed to perform recognition processing on captured images of the same size.
[0269] In the information processing device 1c according to the second embodiment, as described with reference to section (b) of Fig. 11, the divided region 35, which is the unit of subsampling, is an area consisting of 4 pixels x 4 pixels, and the pixels to be sampled are selected by thinning out every other pixel in both the row and column directions. Furthermore, each captured image of frames #1 to #4 is subject to subsampling in the first phase, the second phase, the third phase, and the fourth phase, respectively.
[0270] In the example of an information processing device 1c' that uses captured images without reducing them, shown in section (a) of Figure 34, in the recognition process for one frame of a captured image, all pixel data contained in one frame is transferred from the sensor unit 10c' to the recognition unit 220.
[0271] On the other hand, in the information processing device 1c according to the second embodiment shown in section (b) of Fig. 34, in the recognition process for a captured image of one frame, only pixel data of pixel positions selected by thinning for one frame is transferred. For example, when a divided area 35 shown in section (b) of Fig. 11 has a size of 4 pixels x 4 pixels, and subsampling is performed on this divided area 35 by thinning every other pixel in both the row and column directions, the pixel data transferred per frame is 1 / 4 of that in section (a).
[0272] Thus, in the information processing device 1c according to the second embodiment, the amount of pixel data transferred from the sensor unit 10c to the recognition unit 220 is smaller than in the information processing device 1c' in which the captured image is used for recognition processing without being reduced in size, and the bus width can be reduced, while the amount of processing per frame in the recognition unit 220 is reduced, enabling lower power consumption.
[0273] On the other hand, if it is possible to supply the information processing device 1c of the second embodiment with power equivalent to that of an information processing device 1c' that uses an image without reducing the captured image, as shown in section (a) of Figure 34, it is possible to operate multiple information processing devices 1c in parallel, as shown in section (c) of Figure 34.
[0274] [4. Third embodiment] Next, a third embodiment of the present disclosure will be described. The third embodiment is an example in which the sensor unit 10c and the recognition unit 220 in the information processing device 1c according to the above-described second embodiment are separated.
[0275] Fig. 35 is a block diagram showing an example of a configuration of an information processing device according to the third embodiment. In Fig. 35, an information processing device 1d includes a sensor unit 10d and a recognition processing unit 20d. The sensor unit 10d includes a pixel array unit 1001 and a read control unit 230. The recognition processing unit 20d includes a recognition unit 220.
[0276] Here, the sensor unit 10d is formed, for example, by a two-layer stacked CIS in which semiconductor chips are stacked in two layers, as described with reference to FIG. 6A. Referring to FIG. 6A, a pixel unit 2020a is formed in a first-layer semiconductor chip, and a memory+logic unit 2020b is formed in a second-layer semiconductor chip. The pixel unit 2020a includes at least the pixel array unit 1001 in the sensor unit 10d. The memory+logic unit 2020b includes, for example, a drive circuit for driving the pixel array unit 1001 and a readout control unit 230. The memory+logic unit 2020b may further include a frame memory.
[0277] The sensor unit 10d outputs image data of the sampling image from the read control unit 230 and supplies it to a recognition processing unit 20d included in hardware different from the sensor unit 10d. The recognition processing unit 20d inputs the image data supplied from the sensor unit 10d to the recognition unit 220. The recognition unit 220 executes recognition processing based on the input image data and outputs the recognition result to the outside.
[0278] As another example, the sensor unit 10d can be formed by a three-layered stacked CIS in which semiconductor chips are stacked in three layers, as described with reference to FIG. 6B. In this case, the pixel unit 2020a described above is formed in the semiconductor chip of the first layer, the memory unit 2020c including, for example, a frame memory is formed in the semiconductor chip of the second layer, and the logic unit 2020b corresponding to the memory+logic unit 2020b described above is formed in the semiconductor chip of the third layer. In this case, the logic unit 2020b includes, for example, a drive circuit for driving the pixel array unit 1001 and a readout control unit 230. The memory unit 2020c can include a frame memory and a memory 1202.
[0279] In this way, by configuring the recognition processing unit 20d (recognition unit 220) with hardware separate from the sensor unit 10d, it is possible to easily change the configuration of the recognition unit 220, for example, the recognition model.
[0280] In addition, since the sensor unit 10d performs the recognition process based on the subsampled sampled image, the load of the recognition process can be reduced compared to the case where the recognition process is performed using the image data 32 of the captured image as is. Therefore, for example, a CPU, DSP, or GPU with low processing power can be used in the recognition processing unit 20d, and the cost of the information processing device 1d can be reduced.
[0281] [5. Fourth embodiment] Next, a fourth embodiment of the present disclosure will be described. In the above-mentioned first to third embodiments, it has been described that one sub-sampling is performed for one image data 32, but this is not limited to this example. The fourth embodiment is an example in which one image data 32 is sub-sampling multiple times.
[0282] In the third embodiment, the information processing device 1b according to the first embodiment and each of its modified examples, the information processing device 1c according to the second embodiment, and the information processing device 1d according to the third embodiment can all be applied. In the following, the fourth embodiment will be described assuming that the information processing device 1b shown in FIG. 9 is applied.
[0283] Fig. 36 is a schematic diagram for explaining the subsampling process in the recognition process according to the fourth embodiment. For the sake of explanation, it is assumed that subsampling is performed in the same manner as in section (b) of Fig. 13A described above, as shown in section (b) of Fig. 36. That is, in each divided region 35 having a size of 2 pixels x 2 pixels, the upper left pixel position is set to the coordinates [0,0] of the origin, and the upper right, lower left and lower right pixel positions are set to the coordinates [1,0], [0,1] and [1,1], respectively.
[0284] Moreover, sampling of pixel 300 is performed in the order of coordinates [1,1], [1,0], [0,1], and [0,0], with the bottom right pixel position [1,1] as the base point in each divided region 35. Moreover, sampled images of sampled pixels acquired by sub-sampling with the coordinates [1,1], [1,0], [0,1], and [0,0] as the base point are respectively referred to as a sampled image of phase [1,1], a sampled image of phase [1,0], a sampled image of phase [0,1], a sampled image of phase [0,0], and so on.
[0285] In section (a) of Fig. 36, image data 32a is the latest image [T] at time T, and image data 32b is the image [T-1] at time T-1, which is the frame immediately before the image data 32a. In the following description, images at each time will be represented by image [T] at time T (image data 32a).
[0286] The pre-processing unit 210 performs sub-sampling on one image data 32a while periodically shifting the position (steps S10a to S10d). For example, during one frame period from when the image data 32a is acquired until the next image data is acquired, the pre-processing unit 210 sequentially performs sub-sampling with the coordinates [1,1], [1,0], [0,1], and [0,0] as the base point, and acquires a sampled image with a phase [1,1], a sampled image with a phase [1,0], a sampled image with a phase [0,1], and a sampled image with a phase [0,0], respectively.
[0287] The recognition unit 220 extracts features for each of the sampled images of phase [1,1], phase [1,0], phase [0,1], and phase [0,0] (step S11), and extracts features 50a, 50b, 50c, and 50d.
[0288] The recognition unit 220 performs an integration process of the feature amounts 50a, 50b, 50c, and 50d extracted from the image data 32a (step S12), and performs a recognition process based on the integrated feature amount of the feature amounts 50a, 50b, 50c, and 50d (step S13).
[0289] In this way, by performing feature extraction processing on sampled images of each phase obtained by sub-sampling one image data 32a with a shifted phase, recognition processing based on the integrated features can be performed more quickly.
[0290] In the above description, in the divided region 35 including 2 pixels x 2 pixels, subsampling is performed for all phases [1,1], [1,0], [0,1], and [0,0] to perform recognition processing for one image data 32a as a whole. This is not limited to this example, and subsampling may be selectively performed for a specific phase among the phases [1,1], [1,0], [0,1], and [0,0].
[0291] For example, among the phases [1,1], [1,0], [0,1], and [0,0], sub-sampling is performed only on the diagonally positioned phases [1,1] and [0,0], and the feature amounts of each sampled image of the phases [1,1] and [0,0] obtained are extracted and the recognition process is performed. This reduces the amount of processing for feature extraction and human initial processing, and suppresses power consumption in the recognition processing unit 20b.
[0292] Also, for example, the result of performing recognition processing based on the feature obtained by subsampling one of the phases [1,1], [1,0], [0,1], and [0,0] (for example, phase [1,1]) can be output as a preliminary result. In this case, after the preliminary result is output, subsampling is performed for the other phases (for example, [1,0], [0,1], and [0,0]), and recognition processing is performed based on the feature obtained by integrating the feature amounts for each of [1,1], [1,0], [0,1], and [0,0], and the recognition result is output.
[0293] In this case, if sufficient recognition results are obtained from the preliminary results, it is possible to omit subsequent processing (subsampling by other phases, feature extraction, etc.). In this case, processing of the next image data can be started immediately after the preliminary results are output, making it possible to further increase the frame rate.
[0294] [6. Fifth embodiment] (6-1. Application Examples of the Technology of the Present Disclosure) Next, as a fifth embodiment, an application example of the information processing devices 1b, 1c, and 1d according to the first embodiment and each of its modifications, the second embodiment, the third embodiment, and the fourth embodiment of the present disclosure will be described. FIG. 37 is a diagram showing a usage example of the information processing devices 1b, 1c, and 1d according to the first embodiment and each of its modifications, the second embodiment, the third embodiment, and the fourth embodiment. In the following, when there is no particular need to distinguish between the information processing devices 1b, 1c, and 1d, the information processing device 1b will be used as a representative.
[0295] The above-described information processing device 1a can be used in various cases, for example, as described below, for sensing light such as visible light, infrared light, ultraviolet light, and X-rays, and performing recognition processing based on the sensing results.
[0296] -Digital cameras, mobile devices with camera functions, and other devices that take images for viewing. Devices used for traffic purposes, such as on-board sensors that photograph the front, rear, surroundings, and interior of a vehicle for safe driving such as automatic stopping, and for recognition of the driver's condition, surveillance cameras that monitor moving vehicles and roads, and distance measuring sensors that measure distances between vehicles. A device used in home appliances such as TVs, refrigerators, and air conditioners to capture user gestures and operate the appliance in accordance with those gestures. - Equipment used for medical or healthcare purposes, such as endoscopes and devices that take blood vessel images by receiving infrared light. - Equipment used for security purposes, such as surveillance cameras for crime prevention and cameras for person authentication. - Equipment used for beauty purposes, such as skin measuring devices that take pictures of the skin and microscopes that take pictures of the scalp. Equipment used for sports, such as action cameras and wearable cameras for sports purposes. Agricultural equipment, such as cameras for monitoring fields and crop conditions.
[0297] (6-2. Application examples to moving objects) The technology according to the present disclosure (the present technology) can be applied to various products. For example, the technology according to the present disclosure may be realized as a device mounted on any type of moving body such as an automobile, an electric vehicle, a hybrid electric vehicle, a motorcycle, a bicycle, a personal mobility device, an airplane, a drone, a ship, a robot, etc.
[0298] FIG. 38 is a block diagram showing a schematic configuration example of a vehicle control system, which is an example of a moving object control system to which the technology according to the present disclosure can be applied.
[0299] The vehicle control system 12000 includes a plurality of electronic control units connected via a communication network 12001. In the example shown in Fig. 38, the vehicle control system 12000 includes a drive system control unit 12010, a body system control unit 12020, an outside-vehicle information detection unit 12030, an inside-vehicle information detection unit 12040, and an integrated control unit 12050. Also, as functional configurations of the integrated control unit 12050, a microcomputer 12051, an audio / video output unit 12052, and an in-vehicle network I / F (interface) 12053 are illustrated.
[0300] The drivetrain control unit 12010 controls the operation of devices related to the drivetrain of the vehicle in accordance with various programs. For example, the drivetrain control unit 12010 functions as a control device for a drive force generating device for generating a drive force of the vehicle, such as an internal combustion engine or a drive motor, a drive force transmission mechanism for transmitting the drive force to the wheels, a steering mechanism for adjusting the steering angle of the vehicle, and a braking device for generating a braking force of the vehicle.
[0301] The body system control unit 12020 controls the operation of various devices mounted on the vehicle body according to various programs. For example, the body system control unit 12020 functions as a control device for a keyless entry system, a smart key system, a power window device, or various lamps such as head lamps, back lamps, brake lamps, turn signals, and fog lamps. In this case, radio waves or signals of various switches transmitted from a portable device that replaces a key may be input to the body system control unit 12020. The body system control unit 12020 receives the input of these radio waves or signals and controls the door lock device, power window device, lamps, and the like of the vehicle.
[0302] The outside-vehicle information detection unit 12030 detects information outside the vehicle equipped with the vehicle control system 12000. For example, an imaging unit 12031 is connected to the outside-vehicle information detection unit 12030. The outside-vehicle information detection unit 12030 causes the imaging unit 12031 to capture an image outside the vehicle and receives the captured image. The outside-vehicle information detection unit 12030 may perform object detection processing or distance detection processing for a person, a car, an obstacle, a sign, or characters on a road surface, based on the received image.
[0303] The imaging unit 12031 is an optical sensor that receives light and outputs an electrical signal according to the amount of light received. The imaging unit 12031 can output the electrical signal as an image, or as distance measurement information. The light received by the imaging unit 12031 may be visible light or invisible light such as infrared light.
[0304] The in-vehicle information detection unit 12040 detects information inside the vehicle. For example, a driver state detection unit 12041 that detects the state of the driver is connected to the in-vehicle information detection unit 12040. The driver state detection unit 12041 includes, for example, a camera that captures an image of the driver, and the in-vehicle information detection unit 12040 may calculate the degree of fatigue or concentration of the driver based on the detection information input from the driver state detection unit 12041, or may determine whether the driver is dozing.
[0305] The microcomputer 12051 can calculate control target values for the driving force generating device, the steering mechanism, or the braking device based on the information inside and outside the vehicle acquired by the outside-of-vehicle information detection unit 12030 or the inside-vehicle information detection unit 12040, and output a control command to the drive system control unit 12010. For example, the microcomputer 12051 can perform cooperative control aimed at realizing the functions of an ADAS (Advanced Driver Assistance System), including vehicle collision avoidance or impact mitigation, following driving based on the inter-vehicle distance, vehicle speed maintenance driving, vehicle collision warning, vehicle lane departure warning, etc.
[0306] In addition, the microcomputer 12051 can perform cooperative control for the purpose of autonomous driving, which allows the vehicle to travel autonomously without relying on the driver's operation, by controlling the driving force generating device, steering mechanism, braking device, etc. based on information about the surroundings of the vehicle acquired by the outside vehicle information detection unit 12030 or the inside vehicle information detection unit 12040.
[0307] Furthermore, the microcomputer 12051 can output a control command to the body system control unit 12020 based on the information outside the vehicle acquired by the outside-vehicle information detection unit 12030. For example, the microcomputer 12051 can control the headlamps according to the position of a preceding vehicle or an oncoming vehicle detected by the outside-vehicle information detection unit 12030, and perform cooperative control for the purpose of preventing glare, such as switching from high beams to low beams.
[0308] The audio / video output unit 12052 transmits at least one output signal of audio and video to an output device capable of visually or audibly notifying information to passengers in the vehicle or the outside of the vehicle. In the example of Fig. 38, an audio speaker 12061, a display unit 12062, and an instrument panel 12063 are exemplified as the output device. The display unit 12062 may include at least one of an on-board display and a head-up display, for example.
[0309] FIG. 39 is a diagram showing an example of the installation position of the imaging unit 12031.
[0310] In FIG. 39, a vehicle 12100 has imaging units 12101, 12102, 12103, 12104, and 12105 as an imaging unit 12031.
[0311] The imaging units 12101, 12102, 12103, 12104, and 12105 are provided at positions such as the front nose, side mirrors, rear bumper, back door, and the upper part of the windshield inside the vehicle cabin of the vehicle 12100. The imaging unit 12101 provided at the front nose and the imaging unit 12105 provided at the upper part of the windshield inside the vehicle cabin mainly acquire images of the front of the vehicle 12100. The imaging units 12102 and 12103 provided at the side mirrors mainly acquire images of the sides of the vehicle 12100. The imaging unit 12104 provided at the rear bumper or back door mainly acquires images of the rear of the vehicle 12100. The images of the front acquired by the imaging units 12101 and 12105 are mainly used to detect a preceding vehicle, a pedestrian, an obstacle, a traffic light, a traffic sign, a lane, or the like.
[0312] 39 shows an example of the imaging ranges of the imaging units 12101 to 12104. Imaging range 12111 indicates the imaging range of the imaging unit 12101 provided on the front nose, imaging ranges 12112 and 12113 indicate the imaging ranges of the imaging units 12102 and 12103 provided on the side mirrors, respectively, and imaging range 12114 indicates the imaging range of the imaging unit 12104 provided on the rear bumper or back door. For example, an overhead image of the vehicle 12100 viewed from above is obtained by superimposing the image data captured by the imaging units 12101 to 12104.
[0313] At least one of the imaging units 12101 to 12104 may have a function of acquiring distance information. For example, at least one of the imaging units 12101 to 12104 may be a stereo camera made up of a plurality of imaging elements, or may be an imaging element having pixels for detecting a phase difference.
[0314] For example, the microcomputer 12051 can extract, as a preceding vehicle, a three-dimensional object that is the closest three-dimensional object on the travel path of the vehicle 12100 and travels at a predetermined speed (for example, 0 km / h or more) in approximately the same direction as the vehicle 12100, by calculating the distance to each three-dimensional object in the imaging ranges 12111 to 12114 and the change over time of this distance (relative speed with respect to the vehicle 12100) based on the distance information obtained from the imaging units 12101 to 12104. Furthermore, the microcomputer 12051 can set a vehicle distance to be secured in advance in front of the preceding vehicle, and perform automatic brake control (including follow-up stop control) and automatic acceleration control (including follow-up start control). In this way, cooperative control can be performed for the purpose of automatic driving that travels autonomously without relying on the driver's operation.
[0315] For example, the microcomputer 12051 classifies and extracts three-dimensional object data on three-dimensional objects into two-wheeled vehicles, ordinary vehicles, large vehicles, pedestrians, utility poles, and other three-dimensional objects based on distance information obtained from the imaging units 12101 to 12104, and can use the data for automatic obstacle avoidance. For example, the microcomputer 12051 distinguishes obstacles around the vehicle 12100 into obstacles that are visible to the driver of the vehicle 12100 and obstacles that are difficult to see. Then, the microcomputer 12051 determines a collision risk indicating the risk of collision with each obstacle, and when the collision risk is equal to or greater than a set value and a collision is possible, the microcomputer 12051 can provide driving assistance for collision avoidance by outputting an alarm to the driver via the audio speaker 12061 or the display unit 12062, or by performing forced deceleration or avoidance steering via the drive system control unit 12010.
[0316] At least one of the imaging units 12101 to 12104 may be an infrared camera that detects infrared rays. For example, the microcomputer 12051 can recognize a pedestrian by determining whether or not a pedestrian is present in the captured images of the imaging units 12101 to 12104. The recognition of such a pedestrian is performed, for example, by a procedure of extracting feature points in the captured images of the imaging units 12101 to 12104 as infrared cameras, and a procedure of performing pattern matching processing on a series of feature points that indicate the contour of an object to determine whether or not the object is a pedestrian. When the microcomputer 12051 determines that a pedestrian is present in the captured images of the imaging units 12101 to 12104 and recognizes the pedestrian, the audio / image output unit 12052 controls the display unit 12062 to superimpose a rectangular contour line for emphasis on the recognized pedestrian. The audio / image output unit 12052 may also control the display unit 12062 to display an icon or the like indicating a pedestrian at a desired position.
[0317] An example of a vehicle control system to which the technology according to the present disclosure can be applied has been described above. The technology according to the present disclosure can be applied to the imaging unit 12031 and the vehicle outside information detection unit 12030 among the above-described configurations. Specifically, for example, the sensor unit 10b of the information processing device 1b is applied to the imaging unit 12031, and the recognition processing unit 20b is applied to the vehicle outside information detection unit 12030. The recognition result output from the recognition processing unit 20b is passed to the integrated control unit 12050 via, for example, the communication network 12001.
[0318] In this way, by applying the technology disclosed herein to the imaging unit 12031 and the outside vehicle information detection unit 12030, it is possible to recognize both close-up objects and long-up objects, and it is also possible to recognize close-up objects with a high degree of synchronicity, thereby enabling more reliable driving assistance.
[0319] It should be noted that the effects described in this specification are merely examples and are not limiting, and other effects may also be obtained.
[0320] The present technology can also be configured as follows. (1) a setting unit that sets pixel positions for acquiring sampling pixels for each divided area into which imaging information composed of pixels is divided; a calculation unit for calculating a feature amount of a sampled image constituted by the sampled pixels; a recognition unit that performs a recognition process based on the feature amount of the sampled image and outputs a recognition process result; Equipped with The setting unit is setting different pixel positions between a first piece of imaging information and a second piece of imaging information acquired next to the first piece of imaging information in time series; Information processing device. (2) The recognition unit is executing a machine learning process by a recurrent neural network (RNN) using the sampling pixels set in the first imaging information and the sampling pixels set in the second imaging information, and performing the recognition process based on a result of the machine learning process; The information processing device according to (1). (3) The setting unit is the pixel positions are set so as to rotate in a constant cycle within the divided regions in response to acquisition of the imaging information; The information processing device according to (1) or (2). (4) The setting unit is the pixel positions are arbitrarily set within the divided regions in response to acquisition of the imaging information. The information processing device according to (1) or (2). (5) The setting unit is setting the pixel position within the divided region in response to acquisition of the imaging information based on an external instruction; The information processing device according to (1) or (2). (6) The setting unit is all pixel positions included in the divided region are set as the pixel positions across the plurality of pieces of imaging information that are consecutive in time series; 13. The information processing device according to claim 12, wherein the information processing device is a (7) The setting unit is all pixel positions included in the imaging information are set as the pixel positions across a plurality of pieces of imaging information that are consecutive in time series; 13. The information processing device according to claim 12, wherein the information processing device is a (8) a storage unit that stores the feature amount calculated by the calculation unit, The recognition unit is performing the recognition processing based on at least a part of the feature amounts stored in the storage unit, and outputting the recognition processing result; 8. An information processing device according to any one of (1) to (7). (9) The recognition unit is performing the recognition process based on a feature amount obtained by integrating the plurality of feature amounts stored in the storage unit; The information processing device according to (8). (10) The recognition unit is integrating the feature amount calculated by the calculation unit in response to acquisition of the imaging information with at least a part of the feature amount stored in the storage unit immediately before the acquisition, and performing the recognition process based on the integrated feature amount. The information processing device according to (8). (11) The recognition unit is performing the recognition process based on a feature quantity selected in accordance with a predetermined condition from the feature quantities stored in the storage unit; The information processing device according to any one of (8) to (10). (12) The recognition unit is performing the recognition process based on a new feature quantity in time series among the feature quantities stored in the storage unit; The information processing device according to (11). (13) The recognition unit is discarding a feature quantity corresponding to a predetermined condition from among the feature quantities stored in the storage unit; The information processing device according to any one of (8) to (12). (14) The recognition unit is performing the recognition process on the feature amount of the sampling image based on the teacher data for each pixel corresponding to the pixel position of each of the divided regions; 14. The information processing device according to any one of (1) to (13). (15) The setting unit is In response to the recognition process by the recognition unit, the pixel positions for calculating the feature amount are set in a second pattern different from a first pattern in which the pixel positions constituting the sampling image on which the recognition process has been performed are set. 15. The information processing device according to any one of (1) to (14). (16) The setting unit is An exposure condition when the first imaging information is acquired is made different from an exposure condition when the second imaging information is acquired. 16. The information processing device according to any one of (1) to (15). (17) Executed by a processor, a setting step of setting pixel positions for acquiring sampling pixels for each divided area into which imaging information composed of pixels is divided; a calculation step of calculating a feature amount of a sampling image constituted by the sampling pixels; a recognition step of performing a recognition process based on the feature amount of the sampled image and outputting a recognition process result; having The setting step includes: setting different pixel positions between a first piece of imaging information and a second piece of imaging information acquired next to the first piece of imaging information in time series; Information processing methods. (18) a setting step of setting pixel positions for acquiring sampling pixels for each divided area into which imaging information composed of pixels is divided; a calculation step of calculating a feature amount of a sampling image constituted by the sampling pixels; a recognition step of performing a recognition process based on the feature amount of the sampled image and outputting a recognition process result; on the computer, The setting step includes: setting different pixel positions between a first piece of imaging information and a second piece of imaging information acquired next to the first piece of imaging information in time series; Information processing program for. [Explanation of symbols]
[0321] 1a, 1b, 1c, 1d Information processing device 10a, 10b, 10c, 10d Sensor section 20a, 20b, 20d Recognition processing unit 30a, 30b Captured images 32, 32a, 32a', 32b, 32c, 32d Image data 35,35' split area 36, 36φ1, 36φ1', 36φ2, 36φ3, 36φ4, 36φx Sampling images 40,60,61,62,63,64,65,66,70,71 Recognition result 41, 42, 43, 44 Object 50a, 50a', 50b, 50c, 50d Features 210 Pretreatment section 211 Reading section 212 Usage area acquisition unit 220 Recognition part 221 Feature Calculation Unit 222 Feature accumulation control unit 223 Feature Storage Unit 224 Usage area determination section 225 Recognition Processing Execution Unit 230 Readout control section 300 pixels 3101, 3102, 3103, 3104, 3105, 3106, 3107, 3108, 3109 Images 320,320Expφ1,320Expφ2,320Expφ3,320Expφ4,320Rn1,320Rn2,320Rn n ,320φ1,320φ2,320φ3,320φ4 Original image 321,321a,321b,321c,321d Reduced image 1000 pixel circuit 1001 Pixel array section 1200 Imaging unit 1202 Memory 1203 DSP 1205 CPU
Claims
1. a setting unit that sets pixel positions for acquiring sampling pixels for each divided area into which imaging information composed of pixels is divided; a calculation unit for calculating a feature amount of a sampled image constituted by the sampled pixels; a recognition unit that performs a recognition process based on the feature amount of the sampled image and outputs a recognition process result; Equipped with The setting unit is setting different pixel positions between a first piece of imaging information and a second piece of imaging information acquired next to the first piece of imaging information in time series; Information processing device.
2. The recognition unit is executing a machine learning process by a recurrent neural network (RNN) using the sampling pixels set in the first imaging information and the sampling pixels set in the second imaging information, and performing the recognition process based on a result of the machine learning process; The information processing device according to claim 1 .
3. The setting unit is the pixel positions are set so as to rotate in a constant cycle within the divided regions in response to acquisition of the imaging information; The information processing device according to claim 1 .
4. The setting unit is the pixel positions are arbitrarily set within the divided regions in response to acquisition of the imaging information. The information processing device according to claim 1 .
5. The setting unit is setting the pixel position within the divided region in response to acquisition of the imaging information based on an external instruction; The information processing device according to claim 1 .
6. The setting unit is all pixel positions included in the divided region are set as the pixel positions across the plurality of pieces of imaging information that are consecutive in time series; The information processing device according to claim 1 .
7. The setting unit is all pixel positions included in the imaging information are set as the pixel positions across a plurality of pieces of imaging information that are consecutive in time series; The information processing device according to claim 1 .
8. a storage unit that stores the feature amount calculated by the calculation unit, The recognition unit is performing the recognition processing based on at least a part of the feature amounts stored in the storage unit, and outputting the recognition processing result; The information processing device according to claim 1 .
9. The recognition unit is performing the recognition process based on a feature amount obtained by integrating the plurality of feature amounts stored in the storage unit; The information processing device according to claim 8.
10. The recognition unit is integrating the feature amount calculated by the calculation unit in response to acquisition of the imaging information with at least a part of the feature amount stored in the storage unit immediately before the acquisition, and performing the recognition process based on the integrated feature amount. The information processing device according to claim 8.
11. The recognition unit is performing the recognition process based on a feature quantity selected in accordance with a predetermined condition from the feature quantities stored in the storage unit; The information processing device according to claim 8.
12. The recognition unit is performing the recognition process based on a new feature amount in time series among the feature amounts stored in the storage unit; The information processing device according to claim 11.
13. The recognition unit is discarding a feature quantity corresponding to a predetermined condition from among the feature quantities stored in the storage unit; The information processing device according to claim 8.
14. The recognition unit is performing the recognition process on the feature amount of the sampling image based on the teacher data for each pixel corresponding to the pixel position of each of the divided regions; The information processing device according to claim 1 .
15. The setting unit is In response to the recognition process by the recognition unit, the pixel positions for calculating the feature amount are set in a second pattern different from a first pattern in which the pixel positions constituting the sampling image on which the recognition process has been performed are set. The information processing device according to claim 1 .
16. The setting unit is An exposure condition when the first imaging information is acquired is made different from an exposure condition when the second imaging information is acquired. The information processing device according to claim 1 .
17. Executed by a processor, a setting step of setting pixel positions for acquiring sampling pixels for each divided area into which imaging information composed of pixels is divided; a calculation step of calculating a feature amount of a sampling image constituted by the sampling pixels; a recognition step of performing a recognition process based on the feature amount of the sampled image and outputting a recognition process result; having The setting step includes: setting different pixel positions between a first piece of imaging information and a second piece of imaging information acquired next to the first piece of imaging information in time series; Information processing methods.
18. a setting step of setting pixel positions for acquiring sampling pixels for each divided area into which imaging information composed of pixels is divided; a calculation step of calculating a feature amount of a sampling image constituted by the sampling pixels; a recognition step of performing a recognition process based on the feature amount of the sampled image and outputting a recognition process result; Run the following on your computer: The setting step includes: setting different pixel positions between a first piece of imaging information and a second piece of imaging information acquired next to the first piece of imaging information in time series; Information processing program for.
Citation Information
Patent Citations
Imaging apparatus and method
JP2017112409A
Video compression and transmission system
WO2013191193A1
Motion video analysis device, motion video analysis system, motion video analysis method, and program
WO2019135270A1
