Information processing device, information processing method, and program

Abstract

An information processing device (2) comprises a detection unit (323) which, using a pre-trained model (330) into which an image (40) and a difference image (50) showing a change to an object contained in the image (40) are input, detects the object in the image (40). The pre-trained model (330) contains an extraction layer (331) which extracts features of the image (40), and a detection layer (333) which detects the object in the image (40) on the basis of the extraction result of the extraction layer (331) and the difference image (50).

Description

[Technical field]

[0001] The present disclosure relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] In recent years, technology for detecting objects reflected in a field of view by image recognition has been rapidly developing. For example, Patent Document 1 discloses a technology for detecting objects reflected in an image by inputting an image combined with difference information indicating pixels that have changed relative to a background image into a recognizer using a neural network. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2017-191501 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, in the above conventional technology, only one bit of information indicating whether a pixel has changed relative to the background image is combined with the image, and other information such as color or shape is not used, so the accuracy of object detection cannot be sufficiently improved.

[0005] An object of the present disclosure is to provide an information processing device, an information processing method, and a program capable of further improving the accuracy of object detection. [Means for solving the problem]

[0006] An information processing device according to one aspect of the present disclosure includes a detection unit that detects objects in an image using a trained model that receives as input an image and a difference image that shows changes in objects contained in the image, and the trained model includes an extraction layer that extracts features of the image, and a detection layer that detects objects in the image based on the extraction results of the extraction layer and the difference image.

[0007] An information processing method according to one aspect of the present disclosure includes detecting an object in an image using a trained model that receives as input an image and a difference image showing changes in an object contained in the image, the trained model including an extraction layer that extracts features of the image, and a detection layer that detects an object in the image based on the extraction results of the extraction layer and the difference image.

[0008] A program relating to one aspect of the present disclosure is a program for causing a computer to function, and causes the computer to execute the steps of: detecting an object in an image using a trained model that has as input an image and a difference image that shows changes in an object contained in the image; the trained model includes an extraction layer that extracts features of the image, and a detection layer that detects an object in the image based on the extraction results of the extraction layer and the difference image. [Brief description of the drawings]

[0009] [Figure 1] 1 is a diagram illustrating an example of a schematic configuration of an imaging device according to an embodiment. [Diagram 2] 1 is a perspective view showing an outline of an example of the external configuration of an imaging device; [Diagram 3] FIG. 2 is a diagram illustrating an example of a functional block of a DSP. [Figure 4] FIG. 11 illustrates an example of an object detection process. [Diagram 5] 1 is a flowchart illustrating an example of generating a trained model. [Figure 6] 13 is a flowchart illustrating an example of generating a template image. [Figure 7]13 is a flowchart illustrating an example of an object detection process. [Figure 8] FIG. 13 is a diagram illustrating an example of a schematic configuration of an imaging device according to a modified example. [Figure 9] FIG. 13 is a diagram illustrating an example of a schematic configuration of an imaging device according to a modified example. [Figure 10] FIG. 13 is a diagram illustrating an example of a schematic configuration of an imaging device according to a modified example. [Figure 11] FIG. 13 is a diagram illustrating an example of a schematic configuration of an imaging device according to a modified example. [Figure 12] 1 is a block diagram showing an example of a schematic configuration of a vehicle control system; [Figure 13] 4 is an explanatory diagram showing an example of the installation positions of an outside-vehicle information detection unit and an imaging unit; FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In the following embodiments, the same components are designated by the same reference numerals, and duplicated descriptions will be omitted.

[0011] The present disclosure will be described in the following order. 1. Introduction 2. Embodiment 2.1 Example of imaging device configuration 2.2 Overview of the external configuration of the imaging device 2.3 Example of DSP function blocks 2.4 Object Detection Processing Example 2.5 Examples of trained model techniques 2.6 Example of template image generation 2.7 Discovery Process Example 3. Modifications 4. Effects 5. Mobile Applications

[0012] 1. Introduction As described above, in conventional techniques, only one bit of information indicating whether a pixel has changed relative to the background image is combined with the image, and other information such as color or shape is not used for object detection. This means that the detection results may be strongly influenced by how the difference region is generated, and it has not been possible to sufficiently improve the accuracy of object detection.

[0013] Furthermore, to generate a neural network, learning data with images and difference information as input is required, which requires a lot of work to prepare. That is, since the input is a data format that combines the input image and difference information (for example, a combination of an RGB image and 1-bit information expressing the difference), data including difference information is also essential when training the neural network. In order to improve the generalization performance of the neural network, it is necessary to train using a wide variety of data, but difference information is required for all of the wide variety of data, that is, consecutive frames are required, so there are strong restrictions on the learning data, and it is difficult to prepare the learning data.

[0014] According to one aspect of the present disclosure, there are provided an information processing device, an information processing method, and a program capable of improving detection accuracy. That is, one aspect of the present disclosure aims to improve accuracy by using information on a difference image in the time direction as additional information in a recognition process for detecting objects such as people and objects using images captured with as low a resolution and wide an angle as possible in order to reduce power consumption and calculation costs, and in a recognizer for detecting objects from a conventional still image in order to perform lightweight and accurate recognition processing on a sensor module.

[0015] For example, in low-resolution, wide-angle scenes, the target (such as a person) to be detected may appear very small in the image or may be distorted due to wide-angle shooting, making it difficult to uniformly recognize them. Also, in scenes with low contrast between the target and the background, scenes where the target is difficult to detect using machine learning-based algorithms alone due to their unusual posture, and cases where the recognizer is simplified to enable execution on a sensor module, conventional still image recognition processing alone may not provide sufficient accuracy.

[0016] To address these issues, a feature map calculated based on the difference image is overlaid on the layer after feature extraction of the recognizer using arithmetic operations, and the result is input into the object recognition layer, making it possible to take into account information on the movement of moving objects, which cannot be obtained by processing still images alone.

[0017] With this method, the number of input dimensions of the recognizer does not change whether or not there is a difference, and it is possible to train using normal 1-channel images (Gray) or 3-channel images (RGB or YUV, etc.). This means that it is possible to use conventional recognizers and there is no need to prepare special training data.

[0018] Furthermore, rather than simply inputting the presence or absence of a difference into the neural network, the difference region is evaluated based on a certain model to see whether it has a shape, color, etc. that is reminiscent of the target object, and then input, thereby improving the detection accuracy for the difference region.

[0019] Furthermore, since difference information is not necessarily required during learning, no special preparation is required for preparing a learning data set.

[0020] 2. Embodiment 2.1 Example of imaging device configuration An imaging device according to an embodiment of the present disclosure will be described below. In the following embodiment, the imaging device is an electronic device such as a camera, and is also an information processing device that performs signal processing on acquired image data. However, the imaging device is not limited to such an electronic device.

[0021] 1 is a block diagram showing an example of the configuration of an imaging device. The imaging device 2 includes an imaging block 20 and a signal processing block 30. The imaging block 20 and the signal processing block 30 are connected by connection lines CL1 to CL3.

[0022] The imaging block 20 generates image data by performing an imaging operation. The imaging block 20 includes an imaging unit 21, an imaging processing unit 22, an output control unit 23, an output I / F 24, and an imaging control unit 25.

[0023] The imaging unit 21 includes a plurality of pixels arranged two-dimensionally. When light from an optical system (not shown) is incident on the imaging unit 21, photoelectric conversion is performed in each pixel, and an analog pixel signal corresponding to the incident light is output.

[0024] The imaging processing unit 22 drives the imaging unit 21. The imaging processing unit 22 also converts analog pixel signals from the imaging unit 21 into digital pixel signals, and outputs one frame of digitally converted pixel signals as a captured image 40. The captured image 40 is sent to the output control unit 23 and also to the signal processing block 30 via a connection line CL2. The captured image 40 may be one frame of a moving image.

[0025] The output control unit 23 outputs the captured image 40 from the imaging processing unit 22 and / or the signal processing result 60 (described later) from the signal processing block 30 to the outside via the output I / F 24.

[0026] The output I / F 24 is an I / F that outputs the captured image 40 and the signal processing result 60 to the outside. As the output I / F 24, for example, a relatively high-speed I / F such as a Mobile Industry Processor Interface (MIPI) may be adopted.

[0027] The imaging control unit 25 includes a communication I / F 26 and a register group 27. The communication I / F 26 exchanges necessary information, such as information to be read from and written to the register group 27, with the outside of the imaging device 2. As the communication I / F 26, for example, a first communication I / F such as a serial communication I / F such as I2C (Inter-Integrated Circuit) may be adopted. The register group 27 stores information related to imaging by the imaging unit 21 and various other information.

[0028] The imaging control unit 25 controls the imaging processing unit 22 according to the imaging information stored in the register group 27, thereby controlling the imaging of an image by the imaging unit 21. The imaging control unit 25 is connected to the CPU 31 of the signal processing block 30 via a connection line CL1. Reading and writing of information from and to the register group 27 may be performed by the CPU 31.

[0029] The signal processing block 30 performs predetermined signal processing using the captured image 40 obtained by the imaging block 20, etc. The signal processing block 30 includes a CPU (Central Processing Unit) 31, a DSP (Digital Signal Processor), a memory 33, a communication I / F 34, an image compression unit 35, an input I / F 36, and a difference generation unit 37. These components of the signal processing block 30 are connected to each other via a bus, and exchange information as necessary.

[0030] By executing the program, the CPU 31 functions as an imaging information calculation unit that calculates imaging information using a signal processing result 60 obtained by signal processing in the DSP 32. The CPU 31 feeds back the calculated imaging information to the register group 27 of the imaging control unit 25 via a connection line CL1 for storage.

[0031] The DSP 32 executes a program stored in the memory 33 to perform signal processing using the captured image 40 supplied to the signal processing block 30 from the imaging processing unit 22 via the connection line CL2 and information received from the outside by the input I / F 36.

[0032] The memory 33 is composed of a static random access memory (SRAM), a dynamic RAM (DRAM), etc., and stores programs and the like necessary for processing of the signal processing block 30. Programs necessary for the operation of the imaging device 2, a trained model 330 and an information processing program 335 described later are also stored in the memory 33.

[0033] The communication I / F 34 is, for example, a second communication I / F such as a serial communication I / F such as SPI (Serial Peripheral Interface), and exchanges necessary information such as programs executed by the CPU 31 and DSP 32 with the outside.

[0034] The captured image 40 is supplied to the image compression unit 35 from the imaging processing unit 22 via a connection line CL2. The image compression unit 35 performs a compression process to compress the captured image 40, and generates a compressed image having a smaller amount of data than the captured image 40. The generated compressed image is supplied to the bus. Note that an uncompressed image that is not compressed by the image compression unit 35 may also be supplied to the bus. Hereinafter, unless otherwise specified, both the compressed image and the uncompressed image will be referred to as the captured image 40.

[0035] The input I / F 36 is an I / F that receives information from the outside. For example, the input I / F 36 receives an output of an external sensor (external sensor output) from the external sensor, and supplies the output to the memory 33 via a bus for storage.

[0036] The difference generating unit 37 is a generating unit that generates the difference image 50. The difference image 50 is an image that shows a change in an object included in the captured image 40. The method of generating the difference image 50 is not particularly limited. The difference image 50 may be generated from the captured image 40, and in this case, a background difference method, a simple difference method, or the like may be used. As described above, when the captured image 40 is one frame, it is possible to obtain a background image from another image, or to obtain a simple difference image with another image (such as a simple difference image between frames).

[0037] The difference image 50 may be generated based on information other than the captured image 40, which will be described later with reference to FIG.

[0038] 1 is omitted, and the output (captured image 40) of the imaging processing unit 22 is input to the CPU 31 or the DSP 32 via a bus.

[0039] 2.2 Overview of the external configuration of the imaging device FIG. 2 is a perspective view showing an outline of an example of the external configuration of the imaging device 2 of FIG.

[0040] The imaging device 2 can be configured as a one-chip semiconductor device having a stacked structure in which a plurality of dies are stacked, for example, as shown in FIG.

[0041] In FIG. 2, the imaging device 2 is configured by stacking two dies, dies 51 and 52.

[0042] In FIG. 2, the imaging unit 21 is mounted on the upper die 51, and the imaging processing unit 22 through the imaging control unit 25 and the CPU 31 through the difference generating unit 37 are mounted on the lower die 52.

[0043] The upper die 51 and the lower die 52 are electrically connected, for example, by forming a through hole that penetrates the die 51 and reaches the die 52, or by performing Cu-Cu bonding that directly connects the Cu wiring exposed on the underside of the die 51 with the Cu wiring exposed on the upper side of the die 52.

[0044] Here, in the imaging processing section 22, as a method for performing AD conversion on the image signal output from the imaging section 21, for example, a column parallel AD method or an area AD method can be adopted.

[0045] In the column-parallel AD method, for example, an ADC (AD Converter) is provided for each column of pixels constituting the imaging unit 21, and the ADC of each column is responsible for AD conversion of the pixel signals of the pixels of that column, thereby performing AD conversion of the image signals of the pixels of each column in one row in parallel. When the column-parallel AD method is adopted, a part of the imaging processing unit 22 that performs AD conversion of the column-parallel AD method may be mounted on the upper die 51.

[0046] In the area AD method, the pixels constituting the imaging unit 21 are divided into a plurality of blocks, and an ADC is provided for each block. Then, the ADC of each block is responsible for AD conversion of the pixel signals of the pixels of that block, and thus AD conversion of the image signals of the pixels of the plurality of blocks is performed in parallel. In the area AD method, the block is the smallest unit, and AD conversion (reading and AD conversion) of the image signals can be performed only on necessary pixels among the pixels constituting the imaging unit 21.

[0047] If it is acceptable for the area of ​​the imaging device 2 to be large, the imaging device 2 can be formed from a single die.

[0048] 2, two dies 51 and 52 are stacked to form the one-chip imaging device 2, but the one-chip imaging device 2 can be formed by stacking three or more dies. For example, when three dies are stacked to form the one-chip imaging device 2, the memory 33 in FIG. 2 can be mounted on a separate die.

[0049] Here, in an imaging device in which the sensor chip, memory chip, and DSP chip are connected in parallel with multiple bumps (hereinafter also referred to as a bump-connected imaging device), the thickness increases significantly and the device becomes larger compared to a one-chip imaging device 2 configured in a stacked structure.

[0050] Furthermore, in a bump-connected imaging device, it may become difficult to ensure a sufficient rate for outputting captured images from the imaging processing unit 22 to the output control unit 23 due to signal degradation at the bump connection parts, etc.

[0051] According to the imaging device 2 having a stacked structure, it is possible to prevent the device from becoming larger as described above, and to prevent a sufficient rate from being secured between the imaging processing unit 22 and the output control unit 23.

[0052] Therefore, according to the imaging device 2 having a layered structure, it is possible to realize a compact imaging device that outputs information required by a user.

[0053] If the information required by the user is a captured image, the imaging device 2 can output the captured image.

[0054] In addition, if the information required by the user can be obtained by signal processing using the captured image, the imaging device 2 can perform the signal processing in the DSP 32 to obtain and output the signal processing results as the information required by the user.

[0055] The signal processing performed by the imaging device 2, that is, the signal processing by the DSP 32, may be, for example, recognition processing for recognizing a predetermined recognition target from a captured image.

[0056] Furthermore, for example, the imaging device 2 can receive, via the input I / F 36, the output of a distance sensor such as a ToF (Time of Flight) sensor arranged to have a predetermined positional relationship with the imaging device 2. In this case, the signal processing of the DSP 32 can employ, for example, a fusion process that integrates the output of the distance sensor with the captured image to determine a highly accurate distance, such as a process that uses the captured image to remove noise in a distance image obtained from the output of the distance sensor received via the input I / F 36.

[0057] Furthermore, for example, the imaging device 2 can receive, via the input I / F 36, an image output by an image sensor arranged to have a predetermined positional relationship with the imaging device 2. In this case, as the signal processing of the DSP 32, for example, a self-position estimation process (SLAM (Simultaneously Localization and Mapping)) using the image received via the input I / F 36 and the captured image as a stereo image can be adopted.

[0058] In the imaging device 2 having the above configuration, the captured image 40 acquired by the imaging block 20 is processed by the signal processing block 30, and the signal processing result 60 resulting from the processing can be output to an element (including the AP shown in the figure, which will be described later) external to the imaging device 2. The processing of the signal processing block 30 in this embodiment includes object detection processing in the captured image 40. In one embodiment, the object detection processing is executed by the DSP 32 and the difference generation unit 37. An example of such a form will be described below.

[0059] 2.3 Example of DSP function blocks 3 is a diagram showing an example of functional blocks of the DSP 32. The DSP 32 includes an input unit 321, an evaluation unit 322, a detection unit 323, and an output unit 324.

[0060] The captured image 40 and the difference image 50 are input to the input unit 321. The input unit 321 acquires the captured image 40 from the imaging block 20 via a bus (FIG. 1). The input unit 321 also acquires the difference image 50 from the difference generation unit 37 via the bus.

[0061] The evaluation unit 322 evaluates the difference image 50. The evaluation of the difference image 50 is performed using, for example, template matching. In template matching, matching is performed between each template image generated in advance and the difference image 50. The template image may be an image showing the shape of various assumed moving parts of objects. Matching is established when the difference image 50 and the template image are determined to be identical or similar. The similarity may be determined based on the similarity calculated using a cross-correlation coefficient of both images. Matching may be performed based on the shape or color. The difference image 50 for which matching is established is evaluated as a difference feature amount and sent to the detection unit 323.

[0062] Prior to evaluation of the differential image 50, the evaluation unit 322 may perform a binarization process (differential binarize) on the differential image 50. In this case, the template image is also a binarized image, and matching is performed on the silhouette shape. In addition to the binarization process, morphology processing, inversion processing, probability distribution processing, noise reduction processing, etc. may be performed.

[0063] The detection unit 323 detects an object in the captured image 40. The detection unit 323 detects an object in the captured image 40 by using a trained model 330 ( FIG. 1 ) that receives the captured image 40 and the difference image 50 input to the input unit 321. The trained model 330 is a trained model that is generated in advance by using training data so as to detect an object in the captured image 40 based on the captured image 40 and the difference image 50. Further details of the trained model 330 will be described later with reference to FIG. 4.

[0064] The output unit 324 outputs the detection result of the detection unit 323. The detection result includes, for example, information on the captured image 40 and an object detected in the captured image 40. The detection result is output as a signal processing result 60 (or a part thereof).

[0065] 2.4 Object Detection Processing Example 4 is a diagram showing an example of object detection processing by the difference generation unit 37 and the DSP 32. In this example, the trained model 330 is a CNN (Convolutional Neural Network) and includes an extraction layer (Feature-extractor) 331 and a detection layer (Object-detection) 333. In the figure, a connection portion between the extraction layer 331 and the detection layer 333 is conceptually shown as a node 332. FIG. 4 shows, as three consecutive time-series images, a captured image 401, a captured image 402, and a captured image 403 captured in sequence.

[0066] The extraction layer 331 of the trained model 330 extracts features of the captured image 403 from the captured image 403. The features extracted by the extraction layer 331 are sent to a node 332.

[0067] On the other hand, the difference generation unit 37 generates a difference image 50 using the captured image 401, the captured image 402, and the captured image 403. The difference image 50 is an image indicating the difference between the captured image 403 and other images (the captured image 401 and the captured image 402). The evaluation unit 322 evaluates the difference image 50 generated by the difference generation unit 37. The evaluation result (difference feature amount) of the evaluation unit 322 is sent to the node 332.

[0068] The detection layer 333 detects an object in the captured image 403 from the extraction result (feature amount of the captured image 403) of the extraction layer 331 sent to the node 332 and the evaluation result (difference feature amount of the captured image 403) of the evaluation unit 322. Here, the difference feature amount of the captured image 403 may be concatenated with the feature amount of the captured image 403 as a new channel and input to the detection layer 333. Alternatively, the difference feature amount of the captured image 403 may be incorporated into the feature amount of the captured image 403 by arithmetic processing (four arithmetic operations, etc.) on the feature amount of the captured image 403 and then input to the detection layer 333. In either case, the size of the difference feature amount map of the captured image 403 may be adjusted to fit the size of the feature amount map of the captured image 403. Note that, although the CNN in which the extraction layer and the detection layer are clearly separated has been described as an example here, it is not necessary to use a CNN in which the extraction layer and the detection layer are separated. For example, in a CNN realized by one functional block, the above-mentioned difference feature may be combined with an intermediate layer or incorporated by any arithmetic processing (such as four arithmetic operations).

[0069] 2.5 Example of generating a trained model 5 is a flowchart showing an example of generating the trained model 330. The process of this flowchart may be executed by a person skilled in the art using an apparatus such as a computer as necessary.

[0070] In step S1, training data is prepared. An example of the training data is teacher data in which object information that can be a detection target (desired to be recognized) is associated with a label. The object information includes an image of the object and a difference image. The label is information for identifying an object in an image.

[0071] In step S2, the parameters of the trained model are trained. Specifically, the parameters of the trained model 330 are trained using the training data prepared in the previous step S1. If the trained model 330 is a CNN for image recognition, the parameters are the coefficients of the CNN. The training data may be a plurality of training data consisting of different combinations of teacher data. In this case, a plurality of parameters each optimized for the corresponding training data is obtained.

[0072] In step S3, the performance of the trained model 330 is confirmed. For example, the object detection accuracy of the trained model 330 is confirmed using data for validation or evaluation. If multiple parameters are obtained in the previous step S2, the detection performance of the trained model 330 configured with each parameter is confirmed.

[0073] In step S4, the parameter with the best performance is stored. That is, in the previous step S3, the detection performance of each of the trained models 330 configured with each parameter is compared, and the parameter with the best performance is stored as a parameter of the trained model 330.

[0074] After the process of step S4 is completed, the process of the flowchart ends.

[0075] 2.6 Example of template image generation 6 is a flowchart showing an example of generating a template image. The process of this flowchart can be executed by a person skilled in the art using a device such as a computer as necessary.

[0076] In step S11, a difference image is obtained. For example, time-series images of various possible objects are prepared, and a difference image is obtained from these time-series images.

[0077] In step S12, binarization processing is performed on the difference image acquired in the previous step S11. In addition, morphological processing, inversion processing, probability distribution processing, noise reduction processing, etc. may be performed.

[0078] In step S13, a template image is created using the image that has been binarized in the previous step S12. If there are multiple images of the same part of the same object, the template image may be created using their statistics (such as the average value).

[0079] After the process of step S13 is completed, the process of the flowchart ends.

[0080] 2.7 Discovery Process Example 7 is a flowchart showing an example of the detection process. The detection process is performed by the DSP 32 executing the information processing program 335 (FIG. 1) stored in the memory 33.

[0081] In step S21, an input image is acquired. That is, the captured image 40 is input to the input unit 321.

[0082] After the process of step S21 is completed, the processes of steps S22 to S25 and the process of step S26 are executed in parallel.

[0083] In step S22, a difference image is acquired. That is, a difference image 50 is input to the input unit 321.

[0084] In step S23, a binarization process is executed. That is, the evaluation unit 322 binarizes the difference image 50 input in the previous step S22. In addition to the binarization process, morphological processing, inversion processing, probability distribution processing, noise reduction processing, etc. may be executed.

[0085] In step S24, matching with a template is performed. That is, the evaluation unit 322 performs matching between the difference image 50 that has been subjected to the binarization process in the previous step S23 and the template image created as described above with reference to Fig. 6. The difference image 50 for which matching is established is obtained as the evaluation result (difference feature amount).

[0086] In step S25, the size of the difference feature map is adjusted. Specifically, the detection unit 323 adjusts the map size of the difference image 50 obtained as the evaluation result (difference feature) in the previous step S24 so as to fit the map size of the captured image 40. After completing the process of step S25, the process proceeds to step S27.

[0087] In step S26, feature amounts are extracted. Specifically, the extraction layer 331 of the trained model 330 extracts feature amounts of the captured image 40 acquired in the previous step S31. After completing the process of step S26, the process proceeds to step S27.

[0088] In step S27, the feature amount maps are merged. Specifically, the detection unit 323 combines the difference feature amount map adjusted in the previous step S25 with the feature amount map acquired in the previous step S26.

[0089] In step S28, an object is detected. Specifically, the detection layer 333 detects an object in the captured image 40 by using the captured image 40 acquired in the previous step S21 and the feature map (i.e., the feature and difference feature of the captured image 40) merged in the previous step S27.

[0090] After the process of step S28 is completed, the process of the flowchart ends.

[0091] Although the embodiments of the present disclosure have been described above, the embodiments of the present disclosure are not limited to the above examples.

[0092] 3. Modifications Fig. 8 is a block diagram showing a configuration example of an imaging device according to a modified example. The imaging device 2A shown in Fig. 8 differs from the imaging device 2 (Fig. 1) in that it includes a difference generation unit 37A1 and a difference generation unit 37A2 instead of the difference generation unit 37. However, the difference generation unit 37A2 may be omitted.

[0093] The difference generation unit 37A1 is, for example, a DVS (Dynamic Vision Sensor) and detects a change (for example, brightness difference information) in the imaging target of the imaging unit 21, i.e., the captured image 40. Such a detection result by the difference generation unit 37A1 is also one aspect of the difference image 50. The difference image 50 is supplied to the signal processing block 30 via the input I / F 36. The difference image 50 may be supplied to the bus directly from the input I / F 36, or, if a difference generation unit 37A2 is present, may be supplied to the bus via the difference generation unit 37A2.

[0094] The difference generation section 37A2 removes, for example, noise caused by flicker components, dark current, and the like in the difference image 50 output from the difference generation section 37A1.

[0095] In this way, the difference image 50 can also be generated by using the difference generation unit 37A1. Since the difference generation unit 37A1, which is a DVS, can immediately obtain the difference image 50, it is possible to omit the process of generating the difference image 50 from the captured images 40 as in the difference generation unit 37 (FIG. 1). In addition, there is no need to hold time-series images (imaging information of a plurality of frames) for generating the difference image 50.

[0096] Fig. 9 is a block diagram showing a configuration example of an imaging device according to another modified example. The imaging device 2B shown in Fig. 9 differs from the imaging device 2 (Fig. 1) in that it includes an imaging block 20B and a signal processing block 30B instead of the imaging block 20 and the signal processing block 30. Specifically, the imaging device 2B includes a difference generation unit 37 in the imaging control unit 25B of the imaging block 20B, not in the signal processing block 30B. The difference generation unit 37 generates a difference image 50 using the captured image 40 sent to the imaging control unit 25B. The generated difference image 50 is sent to the CPU 31 and then to the DSP 32 via the connection line CL1.

[0097] In the above embodiment, an example has been described in which the object detection process in the captured image 40 is executed within the imaging device. In this case, all the processes are performed in the imaging device, and only the results are sent to the subsequent stage, i.e., to the outside of the imaging device (including an application processor (AP) or the like), which has the advantage of reducing the processing load in the subsequent stage. Since the amount of information sent is small, a low-speed IF is sufficient, and there is also the advantage that there is no need to start an ISP on the AP side, and the information can be directly input to an NPU (Neural network Processing Unit) or the like. However, part of the object detection process may be executed outside the imaging device 2.

[0098] For example, a part of the trained model may be provided outside (at a later stage) the imaging device. FIG. 10 is a diagram showing an example of a schematic configuration of an imaging device according to such a modified example. In the imaging device 2C shown in FIG. 10, the previously described extraction by the extraction layer 331 and the evaluation by the evaluation unit 322 are executed in the imaging device 2C by the trained model 330C and the information processing program 335C stored in the memory 33C. The extraction result by the extraction layer 331 and the evaluation result by the evaluation unit 322 are sent to the application processor 70C via the output I / F 24 as the signal processing result 60. The imaging device 2C includes a configuration equivalent to the detection layer 333 in the application processor 70C. The application processor 70C is configured using, for example, a CPU (Central Processing Unit) and executes an operating system, various application software, and the like. The application processor 70C may be equipped with functions such as a GPU (Graphics Processing Unit) and a baseband processor. In addition to detecting objects in captured images, the application processor 70C performs various processing as necessary on image data and machine learning results, displays them to the user, and transmits them to an external cloud server 90 via a specified network 80.

[0099] Note that various networks can be applied to the network 80, such as the Internet, a wired LAN (Local Area Network), a wireless LAN, a mobile communication network, Bluetooth (registered trademark), etc. In addition, the destination of the image data and the machine learning results is not limited to the cloud server 90, and may be various information processing devices (systems) having communication functions, such as a single operating server, a file server that stores various data, or a communication terminal such as a mobile phone.

[0100] According to the configuration of the imaging device 2C, the application processor 70C can use the feature amount to perform recognition processing suited to various use cases, thereby increasing versatility.

[0101] Also, for example, all object recognition processes may be executed in a later stage of the imaging device 2. FIG. 11 is a diagram showing a schematic configuration of an imaging device according to such a modified example. The imaging device 2D shown in FIG. 11 is different from the imaging device 2 (FIG. 1) in that it includes a signal processing block 30D instead of the signal processing block 30. The signal processing block 30D does not include a difference generation unit 37, and does not include the trained model 330 and the information processing program 335 (FIG. 1) in the memory 33D. The imaging device 2D includes configurations corresponding to the difference generation unit 37, the trained model 330, and the information processing program 335 in the application processor 70D. That is, in the imaging device 2D, all object recognition processes are executed by the application processor 70D. In this case, a dedicated imaging device (or DSP) configured to execute object detection processes is not required, and object detection processes can be realized by combining a general-purpose imaging device and an application processor.

[0102] 4. Effects The imaging device (one aspect of an information processing device) described above is specified, for example, as follows. That is, as illustrated in Figs. 1 to 4, the imaging device 2 includes a detection unit 323. The detection unit 323 detects an object in the captured image 40 by using a trained model 330 that receives as input the captured image 40 and a difference image 50 indicating a change in an object included in the captured image 40. The trained model 330 includes an extraction layer 331 and a detection layer 333. The extraction layer 331 extracts a feature amount of the captured image 40. The detection layer 333 detects an object in the captured image 40 based on the extraction result of the input unit 321 and the difference image 50.

[0103] According to the imaging device 2, an object in the captured image 40 is detected based not only on the feature amount of the captured image 40 but also on the difference image 50. Therefore, it is possible to improve the accuracy of object detection.

[0104] 4, the imaging device 2 may further include an evaluation unit 322. The evaluation unit 322 evaluates the difference image 50. The detection layer 333 may detect an object in the captured image 40 based on the extraction result of the extraction layer 331 and the evaluation result of the evaluation unit 322. By using the evaluation result of the difference image 50 in this manner, it is possible to improve the accuracy of object detection.

[0105] The evaluation section 322 may evaluate the shape of the difference image 50. By using the evaluation result of the shape of the difference image 50, it is possible to improve the accuracy of object detection.

[0106] The evaluation section 322 may evaluate the color of the difference image 50. By using the evaluation result of the color of the difference image 50, it is possible to improve the accuracy of object detection.

[0107] The evaluation unit 322 may perform template matching to evaluate the difference image 50. For example, by evaluating the difference image 50 in this manner, it is possible to improve the accuracy of object detection.

[0108] The imaging device 2 may further include a difference generation section 37. The difference generation section 37 is a generation section that generates a difference image 50. This allows the imaging device 2 to generate the difference image 50.

[0109] The difference generation unit 37 may use a background subtraction method to generate the difference image 50. In this manner, for example, the difference image 50 can be generated.

[0110] The difference generating section 37 may use a simple difference method to generate the difference image 50. For example, the difference image 50 can be generated in this manner.

[0111] The difference generating unit 37 may be a DVS (Dynamic Vision Sensor). By immediately obtaining the difference image 50 by the DVS, it is possible to omit the process of generating the difference image 50 from the captured image 40, for example. In addition, it is possible to eliminate the need to hold time-series images (image information of multiple frames) for generating the difference image 50.

[0112] The imaging device 2 may further include an imaging block 20. The imaging block 20 is an imaging section that acquires a captured image 40. This allows the imaging device 2 to acquire the captured image 40.

[0113] For example, the information processing method shown in Fig. 7 is also one aspect of the present disclosure. That is, the information processing method includes detecting an object in a captured image 40 by using a trained model 330 that receives as input a captured image 40 and a difference image 50 showing a change in the object included in the captured image 40 (step S28). The trained model 330 includes an extraction layer 331 that extracts a feature amount of the captured image 40, and a detection layer 333 that detects an object in the captured image 40 based on the extraction result of the extraction layer 331 and the difference image 50. This information processing method also makes it possible to improve the detection accuracy of an object, as with the information processing device described above.

[0114] For example, the information processing program 335 shown in FIG. 1 is also an aspect of the present disclosure. That is, the information processing program 335 is a program for causing a computer to function. As illustrated in FIG. 7, the program causes a computer to execute a step (step S28) of detecting an object in a captured image 40 by using a trained model 330 to which a captured image 40 and a difference image 50 showing a change in an object included in the captured image 40 are input. The trained model 330 includes an extraction layer 331 that extracts a feature amount of the captured image 40, and a detection layer 333 that detects an object in the captured image 40 based on the extraction result of the extraction layer 331 and the difference image 50. With such a program, it is possible to improve the detection accuracy of an object, as with the information processing device described above.

[0115] 5. Mobile Applications The technology according to the present disclosure (the present technology) can be applied to various products. For example, the technology according to the present disclosure may be realized as a device mounted on any type of moving body such as an automobile, an electric vehicle, a hybrid electric vehicle, a motorcycle, a bicycle, a personal mobility device, an airplane, a drone, a ship, a robot, etc.

[0116] FIG. 12 is a block diagram showing a schematic configuration example of a vehicle control system, which is an example of a moving object control system to which the technology according to the present disclosure can be applied.

[0117] The vehicle control system 12000 includes a plurality of electronic control units connected via a communication network 12001. In the example shown in Fig. 12, the vehicle control system 12000 includes a drive system control unit 12010, a body system control unit 12020, an outside-vehicle information detection unit 12030, an inside-vehicle information detection unit 12040, and an integrated control unit 12050. Also, as functional configurations of the integrated control unit 12050, a microcomputer 12051, an audio / video output unit 12052, and an in-vehicle network I / F (Interface) 12053 are illustrated.

[0118] The drivetrain control unit 12010 controls the operation of devices related to the drivetrain of the vehicle in accordance with various programs. For example, the drivetrain control unit 12010 functions as a control device for a drive force generating device for generating a drive force of the vehicle, such as an internal combustion engine or a drive motor, a drive force transmission mechanism for transmitting the drive force to the wheels, a steering mechanism for adjusting the steering angle of the vehicle, and a braking device for generating a braking force of the vehicle.

[0119] The body system control unit 12020 controls the operation of various devices mounted on the vehicle body according to various programs. For example, the body system control unit 12020 functions as a control device for a keyless entry system, a smart key system, a power window device, or various lamps such as head lamps, back lamps, brake lamps, turn signals, and fog lamps. In this case, radio waves or signals of various switches transmitted from a portable device that replaces a key may be input to the body system control unit 12020. The body system control unit 12020 receives the input of these radio waves or signals and controls the door lock device, power window device, lamps, and the like of the vehicle.

[0120] The outside-vehicle information detection unit 12030 detects information outside the vehicle equipped with the vehicle control system 12000. For example, an imaging unit 12031 is connected to the outside-vehicle information detection unit 12030. The outside-vehicle information detection unit 12030 causes the imaging unit 12031 to capture an image outside the vehicle and receives the captured image. The outside-vehicle information detection unit 12030 may perform object detection processing or distance detection processing for a person, a car, an obstacle, a sign, or characters on a road surface, based on the received image.

[0121] The imaging unit 12031 is an optical sensor that receives light and outputs an electrical signal according to the amount of light received. The imaging unit 12031 can output the electrical signal as an image, or as distance measurement information. The light received by the imaging unit 12031 may be visible light or invisible light such as infrared light.

[0122] The in-vehicle information detection unit 12040 detects information inside the vehicle. For example, a driver state detection unit 12041 that detects the state of the driver is connected to the in-vehicle information detection unit 12040. The driver state detection unit 12041 includes, for example, a camera that captures an image of the driver, and the in-vehicle information detection unit 12040 may calculate the degree of fatigue or concentration of the driver based on the detection information input from the driver state detection unit 12041, or may determine whether the driver is dozing.

[0123] The microcomputer 12051 can calculate control target values ​​for the driving force generating device, the steering mechanism, or the braking device based on the information inside and outside the vehicle acquired by the outside-of-vehicle information detection unit 12030 or the inside-vehicle information detection unit 12040, and output a control command to the drive system control unit 12010. For example, the microcomputer 12051 can perform cooperative control aimed at realizing the functions of an ADAS (Advanced Driver Assistance System), including vehicle collision avoidance or impact mitigation, following driving based on the inter-vehicle distance, vehicle speed maintenance driving, vehicle collision warning, vehicle lane departure warning, etc.

[0124] In addition, the microcomputer 12051 can perform cooperative control for the purpose of autonomous driving, which allows the vehicle to travel autonomously without relying on the driver's operation, by controlling the driving force generating device, steering mechanism, braking device, etc. based on information about the surroundings of the vehicle acquired by the outside vehicle information detection unit 12030 or the inside vehicle information detection unit 12040.

[0125] Furthermore, the microcomputer 12051 can output a control command to the body system control unit 12030 based on the information outside the vehicle acquired by the outside-vehicle information detection unit 12030. For example, the microcomputer 12051 can control the headlamps according to the position of a preceding vehicle or an oncoming vehicle detected by the outside-vehicle information detection unit 12030, and perform cooperative control for the purpose of preventing glare, such as switching from high beams to low beams.

[0126] The audio / video output unit 12052 transmits at least one output signal of audio and image to an output device capable of visually or audibly notifying information to passengers in the vehicle or the outside of the vehicle. In the example of Fig. 12, an audio speaker 12061, a display unit 12062, and an instrument panel 12063 are exemplified as the output device. The display unit 12062 may include at least one of an on-board display and a head-up display, for example.

[0127] FIG. 13 is a diagram showing an example of the installation position of the imaging unit 12031.

[0128] In FIG. 13, the imaging unit 12031 includes imaging units 12101, 12102, 12103, 12104, and 12105.

[0129] The imaging units 12101, 12102, 12103, 12104, and 12105 are provided at positions such as the front nose, side mirrors, rear bumper, back door, and the upper part of the windshield in the vehicle interior of the vehicle 12100. The imaging unit 12101 provided at the front nose and the imaging unit 12105 provided at the upper part of the windshield in the vehicle interior mainly acquire images of the front of the vehicle 12100. The imaging units 12102 and 12103 provided at the side mirrors mainly acquire images of the sides of the vehicle 12100. The imaging unit 12104 provided at the rear bumper or back door mainly acquires images of the rear of the vehicle 12100. The imaging unit 12105 provided at the upper part of the windshield in the vehicle interior is mainly used to detect a preceding vehicle, a pedestrian, an obstacle, a traffic light, a traffic sign, a lane, or the like.

[0130] 13 shows an example of the imaging ranges of the imaging units 12101 to 12104. An imaging range 12111 indicates the imaging range of the imaging unit 12101 provided on the front nose, imaging ranges 12112 and 12113 indicate the imaging ranges of the imaging units 12102 and 12103 provided on the side mirrors, respectively, and an imaging range 12114 indicates the imaging range of the imaging unit 12104 provided on the rear bumper or back door. For example, an overhead image of the vehicle 12100 viewed from above is obtained by superimposing the image data captured by the imaging units 12101 to 12104.

[0131] At least one of the imaging units 12101 to 12104 may have a function of acquiring distance information. For example, at least one of the imaging units 12101 to 12104 may be a stereo camera made up of a plurality of imaging elements, or may be an imaging element having pixels for detecting a phase difference.

[0132] For example, the microcomputer 12051 can extract, as a preceding vehicle, a three-dimensional object that is the closest three-dimensional object on the travel path of the vehicle 12100 and travels at a predetermined speed (for example, 0 km / h or more) in approximately the same direction as the vehicle 12100, by calculating the distance to each three-dimensional object in the imaging ranges 12111 to 12114 and the change over time of this distance (relative speed with respect to the vehicle 12100) based on the distance information obtained from the imaging units 12101 to 12104. Furthermore, the microcomputer 12051 can set a vehicle distance to be secured in advance in front of the preceding vehicle, and perform automatic brake control (including follow-up stop control) and automatic acceleration control (including follow-up start control). In this way, cooperative control can be performed for the purpose of automatic driving that travels autonomously without relying on the driver's operation.

[0133] For example, the microcomputer 12051 classifies and extracts three-dimensional object data on three-dimensional objects into two-wheeled vehicles, ordinary vehicles, large vehicles, pedestrians, utility poles, and other three-dimensional objects based on distance information obtained from the imaging units 12101 to 12104, and can use the data for automatic obstacle avoidance. For example, the microcomputer 12051 distinguishes obstacles around the vehicle 12100 into obstacles that are visible to the driver of the vehicle 12100 and obstacles that are difficult to see. Then, the microcomputer 12051 determines a collision risk indicating the risk of collision with each obstacle, and when the collision risk is equal to or greater than a set value and a collision is possible, the microcomputer 12051 can provide driving assistance for collision avoidance by outputting an alarm to the driver via the audio speaker 12061 or the display unit 12062, or by performing forced deceleration or avoidance steering via the drive system control unit 12010.

[0134] At least one of the imaging units 12101 to 12104 may be an infrared camera that detects infrared rays. For example, the microcomputer 12051 can recognize a pedestrian by determining whether or not a pedestrian is present in the captured images of the imaging units 12101 to 12104. The recognition of such a pedestrian is performed, for example, by a procedure of extracting feature points in the captured images of the imaging units 12101 to 12104 as infrared cameras, and a procedure of performing pattern matching processing on a series of feature points that indicate the contour of an object to determine whether or not the object is a pedestrian. When the microcomputer 12051 determines that a pedestrian is present in the captured images of the imaging units 12101 to 12104 and recognizes the pedestrian, the audio / image output unit 12052 controls the display unit 12062 to superimpose a rectangular contour line for emphasis on the recognized pedestrian. The audio / image output unit 12052 may also control the display unit 12062 to display an icon or the like indicating a pedestrian at a desired position.

[0135] An example of a vehicle control system to which the technology according to the present disclosure can be applied has been described above. The technology according to the present disclosure can be applied to the imaging unit 12031 among the above-described configurations. By applying the technology according to the present disclosure to the imaging unit 12031, it is possible to obtain a captured image that is easier to see, thereby reducing the fatigue of the driver.

[0136] Note that the effects described in this disclosure are merely examples and are not limited to the disclosed contents. Other effects may also be obtained.

[0137] Although the embodiments of the present disclosure have been described above, the technical scope of the present disclosure is not limited to the above-described embodiments, and various modifications are possible without departing from the gist of the present disclosure. In addition, components of different embodiments and modifications may be appropriately combined.

[0138] Furthermore, the effects of each embodiment described in this specification are merely examples and are not limiting, and other effects may also be provided.

[0139] The present technology can also be configured as follows. (1) A detection unit that detects an object in an image by using a trained model that receives an image and a difference image showing a change in the object included in the image as input, The trained model is An extraction layer that extracts features of the image; a detection layer that detects the object in the image based on the extraction result of the extraction layer and the difference image; Including, Information processing device. (2) An evaluation unit for evaluating the difference image is further provided, The detection layer detects an object in the image based on the extraction result of the extraction layer and the evaluation result of the evaluation unit. An information processing device as described in (1). (3) The evaluation unit evaluates a shape of the difference image. An information processing device as described in (2). (4) The evaluation unit evaluates a color of the difference image. An information processing device according to (2) or (3). (5) The evaluation unit evaluates the difference image by performing template matching. An information processing device according to any one of (2) to (4). (6) A generating unit that generates the difference image is further provided. An information processing device according to any one of (1) to (5). (7) The generating unit generates the difference image using a background subtraction method. (6) An information processing device. (8) The generation unit generates the difference image using a simple difference method. (6) An information processing device. (9) The generation unit is a DVS (Dynamic Vision Sensor), (6) An information processing device. (10) Further comprising an imaging unit for acquiring the image. An information processing device according to any one of (1) to (9). (11) Detecting the object in the image using a trained model that receives as input an image and a difference image that indicates a change in the object included in the image; The trained model is An extraction layer that extracts features of the image; a detection layer that detects the object in the image based on an extraction result of the extraction layer and a difference image; Including, Information processing methods. (12) A program for causing a computer to function, Detecting an object in an image using a trained model that receives as input an image and a difference image showing a change in the object included in the image; causing the computer to execute The trained model is An extraction layer that extracts image features; a detection layer that detects the object in the image based on an extraction result of the extraction layer and a difference image; Including, program. [Explanation of symbols]

[0140] 2. Imaging device 20 Imaging block 21 Imaging unit 22 Imaging processing section 23 Output control section 24 Output I / F 25 Imaging control section 26 Communication I / F 27 Registers 30 Signal Processing Blocks 31 CPU 32 DSP 33 Memory 34 Communication I / F 35 Image Compression Section 36 Input I / F 37 Difference generation part 40 Captured images 50 Difference Images 51 Die 52 Die 60 Signal Processing Results 70 Application Processors 80 Network 90 Cloud Server 330 trained models 331 Extraction Layer 332 nodes 333 Detection Layer 335 Information Processing Program

Claims

1. A detection unit that detects an object in an image by using a trained model that receives an image and a difference image that indicates a change in an object included in the image as input, The trained model is An extraction layer that extracts features of the image; a detection layer that detects the object in the image based on the extraction result of the extraction layer and the difference image; Including, Information processing device.

2. An evaluation unit for evaluating the difference image is further provided, The detection layer detects an object in the image based on the extraction result of the extraction layer and the evaluation result of the evaluation unit. The information processing device according to claim 1 .

3. The evaluation unit evaluates a shape of the difference image. The information processing device according to claim 2 .

4. The evaluation unit evaluates a color of the difference image. The information processing device according to claim 2 .

5. The evaluation unit evaluates the difference image by performing template matching. The information processing device according to claim 2 .

6. A generating unit that generates the difference image is further provided. The information processing device according to claim 1 .

7. The generating unit generates the difference image using a background subtraction method. The information processing device according to claim 6.

8. The generation unit generates the difference image using a simple difference method. The information processing device according to claim 6.

9. The generation unit is a Dynamic Vision Sensor (DVS), The information processing device according to claim 6.

10. Further comprising an imaging unit for acquiring the image. The information processing device according to claim 1 .

11. Detecting the object in the image using a trained model that receives as input an image and a difference image that indicates a change in the object included in the image; The trained model is An extraction layer that extracts features of the image; a detection layer that detects the object in the image based on an extraction result of the extraction layer and a difference image; Including, Information processing methods.

12. A program for causing a computer to function, Detecting an object in an image using a trained model that receives as input an image and a difference image showing a change in the object included in the image; causing the computer to execute The trained model is An extraction layer that extracts image features; a detection layer that detects the object in the image based on an extraction result of the extraction layer and a difference image; Including, program.

Images