Image recognition device and apparatus

Abstract

To provide an image recognition device that, when performing recognition processing by using event data, performs the recognition processing with high accuracy.SOLUTION: An image recognition device according to the present disclosure has: an event data acquisition unit that acquires event data indicating a change in the quantity of light from pixels; a meta data acquisition unit that acquires meta data related to the position of an object to be recognized; and a processing unit that performs recognition processing on the object to be recognized by using the event data. The processing unit controls the reference time of the event data in the recognition processing according to the acquired meta data.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The technology of the present disclosure relates to an image recognition device and device. [Background technology]

[0002] With the recent spread of IoT, AI, autonomous driving, and the like, there is a demand for image sensors with lower power consumption and higher speed. Non-Patent Document 1 shows an event-based sensor that monitors changes in the amount of light with each pixel arranged in a two-dimensional array and outputs a signal when a change is detected. As shown in Non-Patent Document 1, the event-based sensor outputs a signal only when a change in the amount of light occurs, enabling low power consumption and high-speed operation. In addition, Non-Patent Document 2 discloses a technology in which the output signal of the event-based sensor is input to a neural network and image recognition processing is performed. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Finateu et al., "A 1280x720 Back-Illuminated Stacked Temporal Contrast Event-Based Vision Sensor with 4.86μm Pixels, 1.066GEPS Readout, Programmable Event-Rate Controller and Compressive Data-Formatting Pipeline," 2020 IEEE International Solid-State Circuits Conference, (USA), February 2020, pp.112-114. [Non-Patent Document 2] Gehrig, and 3 others, "End-to-End Learning of Representations for Asynchronous Event-Based Data", ICCV 2019 IEEE International Solid-State Circuits Conference Summary of the Invention [Problem to be solved by the invention]

[0004] In the technology disclosed in the non-patent document, data obtained by an event-based sensor (hereinafter, event data) is subjected to a predetermined processing, and then sampled at a predetermined time interval to form input data (tensor) to a neural network. In this case, the time range of the event data that affects the tensor is uniform in any area of ​​the image. However, since the amount of generated event data differs depending on, for example, the distance to the subject to be recognized, sufficient information for recognition cannot be obtained, particularly when the subject is far away, which may lead to a decrease in recognition accuracy.

[0005] The technology disclosed herein has been made in consideration of the above-mentioned problems, and has an object to provide a technology for performing recognition processing with high accuracy when performing recognition processing using event data. [Means for solving the problem]

[0006] In order to achieve the above-mentioned object, the image recognition device according to the present disclosure includes an image recognition device having an event data acquisition unit that acquires event data indicating a change in the amount of light of a pixel, a metadata acquisition unit that acquires metadata regarding the position of an object to be recognized, and a processing unit that performs a recognition process of the object to be recognized using the event data, wherein the processing unit controls a reference time of the event data in the recognition process in accordance with the acquired metadata.

[0007] In addition, in order to achieve the above-mentioned objective, the equipment disclosed herein includes equipment equipped with the above-mentioned image recognition device, characterized in that it further includes at least one of an optical device corresponding to the image recognition device, a control device that controls the image recognition device, a processing device that processes signals output from the image recognition device, a display device that displays information obtained by the image recognition device, a storage device that stores information obtained by the image recognition device, and a mechanical device that operates based on the information obtained by the image recognition device. Effect of the Invention

[0008] According to the technology of the present disclosure, when the recognition process is performed using event data, the recognition process can be performed with high accuracy. [Brief description of the drawings]

[0009] [Figure 1] 1 is a diagram illustrating an example of the configuration of a recognition system according to a first embodiment. [Diagram 2] 1 is a diagram illustrating an example of the configuration of an image recognition device according to a first embodiment. [Diagram 3] 4 is a diagram illustrating an example of the configuration of a recognition processing unit according to the first embodiment. FIG. [Figure 4] FIG. 11 is a diagram illustrating an example of the configuration of a recognition system according to a second embodiment. [Diagram 5] FIG. 13 is a diagram illustrating an example of the configuration of a recognition system according to a third embodiment. [Figure 6] FIG. 13 is a diagram illustrating an example of the configuration of a recognition system according to a fourth embodiment. [Figure 7] FIG. 13 is a diagram illustrating an example of the configuration of a recognition system according to a fifth embodiment. [Figure 8] FIG. 13 is a diagram illustrating a configuration example of an apparatus including a semiconductor device according to a sixth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, the embodiments of the present invention will be described in detail with reference to the drawings. In the following description, terms indicating specific directions or positions (for example, "up", "down", "right", "left" and other terms including these terms) will be used as necessary. The use of these terms is for the purpose of facilitating understanding of the embodiments with reference to the drawings, and the meaning of these terms does not limit the technical scope of the present invention. Furthermore, the use of these terms is for the purpose of embodying the technical ideas of the present invention, and does not limit the present invention. Components of each embodiment can be added to another embodiment or replaced with components of another embodiment. Furthermore, the size and positional relationship of the members shown in each drawing may be exaggerated to clarify the explanation.

[0011] First Embodiment A configuration of a recognition system having an image recognition device according to a first embodiment will be described with reference to FIG. 1. As shown in FIG. 1, recognition system 1 has event-based sensor 110, distance sensor 120, and image recognition device 130. Image recognition device 130 acquires event data indicating changes in light intensity of pixels constituting event-based sensor 110 from event-based sensor 110. Image recognition device 130 also acquires distance data as metadata related to the position of a recognition target object from distance sensor 120. Then, image recognition device 130 controls the reference time of the event data in the recognition process according to the acquired metadata, and performs recognition process of the recognition target object (for example, detects the position of a pedestrian) using the event data.

[0012] [Explanation of image recognition device] Next, the configuration of image recognition device 130 according to the first embodiment will be described with reference to FIG. 2. As shown in FIG. 2, image recognition device 130 has event data acquisition unit 131, distance information acquisition unit 132, reference range determination unit 133, and recognition processing unit 134. Event data acquisition unit 131 is connected to event-based sensor 110, and acquires event data from event-based sensor 110. The event data is composed of four pieces of data represented by (x, y, t, p). Here, of the four pieces of data that make up the event data, x and y indicate the coordinates of a pixel of event-based sensor 110, and t indicates the time when the event was detected. , p indicates the polarity of the event (increase or decrease in the amount of light). The event data acquisition unit 210 has an internal storage unit (not shown) and can store the acquired event data in the storage unit.

[0013] The distance information acquisition unit 132 is connected to a distance sensor 120 capable of distance measurement, such as Lidar or Radar, and acquires distance data (depth) indicating the distance measurement result of the recognition target object from the distance sensor 120. In general, the distance data and other additional information are also called metadata. The distance information acquisition unit 132 converts the acquired distance data into a two-dimensional image-like depth map according to the coordinate system indicated by the event data. Note that the distance information acquisition unit 132 may output the distance data acquired from the distance sensor 120 to the subsequent reference range determination unit 133 without converting it into a depth map. The distance information acquisition unit 132 is an example of a metadata acquisition unit that acquires metadata regarding the position of the recognition target object.

[0014] The reference range determination unit 133 determines the time-direction range (reference time) of the event data for each pixel (or each region) according to the depth map output from the distance information acquisition unit 132. Here, based on the data output from the distance information acquisition unit 132, the reference range determination unit 133 widens the reference range in the time direction as the distance from the distance sensor 120 becomes longer, and narrows the reference range in the time direction as the distance from the distance sensor 120 becomes shorter.

[0015] Here, the reason why reference range determination unit 133 controls the reference range in this manner will be described. The farther the subject's position is from distance sensor 120, the smaller the apparent speed (speed in a two-dimensional plane on the image) of the subject, which is the object to be recognized, and the smaller the change in the amount of light at each pixel of event-based sensor 110. That is, the frequency of occurrence of events indicating changes in the amount of light at event-based sensor 110 decreases, and therefore the number of events per unit time and / or unit area at event-based sensor 110 (hereinafter referred to as event density) decreases. On the other hand, the closer the subject's position is to distance sensor 120, the larger the apparent speed of the subject becomes, and the larger the change in the amount of light at each pixel of event-based sensor 110 becomes. That is, the higher the event density at event-based sensor 110 becomes. Therefore, there is a negative correlation between the distance to the subject and the event density.

[0016] In general, when a recognition process of a recognition target object is performed using event data, if the number of events used in the recognition process is small, a sufficient amount of information cannot be obtained for the recognition process, which may result in a decrease in recognition accuracy. Therefore, when the event density is low, the reference range determination unit 133 can increase the number of events used in the recognition process by expanding the reference range in the time direction. The reference range (window width) determined by the reference range determination unit 133 is set to W(x,y). In this case, if the reference time is T, the reference range determination unit 133 controls the recognition processing unit 134 so that the occurrence time t of an event at pixel coordinates (x,y) refers to an event that satisfies the following formula (1).

number

[0017] The recognition processing unit 134 acquires event data of events occurring within the reference range from the event data acquisition unit 131 according to the reference range information (window width information) determined by the reference range determination unit 133, and performs recognition processing using the acquired event data. Note that the reference range determination unit 133 and the recognition processing unit 134 are an example of a processing unit that controls the reference time of the event data in the recognition processing according to the acquired metadata, and performs recognition processing of the recognition target object using the event data.

[0018] The recognition process executed by the recognition processing unit 134 will be described in detail with reference to FIG. As shown in FIG. 3, the recognition processing unit 134 includes an integration unit 135 and a neural network unit 136. The neural network unit 136 uses a CNN (Convolutional Neural Network) as a learning model that accepts event data. The neural network unit 136 detects a recognition target object using an object detection algorithm based on the CNN, and outputs a detection result. However, since input data to the CNN is generally a tensor (multidimensional data) of a fixed size, event data in which the length in the time direction varies for each pixel cannot be input to the CNN. Therefore, as a pre-processing before inputting the event data to the neural network unit 136, the integration unit 135 performs integration processing in the time direction on the event data using the reference range information. Examples of the integration processing include calculation of the number of events in the reference range determined by Equation (1), calculation of the occurrence frequency of events in the reference range, and calculation of the logical sum in the time direction in the reference range (i.e., generation of information indicating whether an event has occurred even once). As a result, the integration unit 135 generates fixed-size data by adjusting the size in the time direction of the event data for input to the algorithm according to the acquired metadata.

[0019] Therefore, according to the image recognition device 130 of this embodiment, by controlling the reference time of the event data according to the distance from the device (distance sensor) to the recognition object, it is possible to maintain high accuracy of the recognition process even if the distance to the recognition object is far. Furthermore, according to the image recognition device 130, when the distance to the recognition object is close, the reference time of the event data is short, so that low-latency recognition process can be realized. In particular, it is possible to realize obstacle detection when the image recognition device 130 is applied to an autonomous moving body such as a drone, and collision avoidance in automatic driving when the image recognition device 130 is applied to a vehicle. Thus, according to the image recognition device 130 of this embodiment, it is possible to realize low-latency recognition process, which is important when the distance to the recognition object is close.

[0020] In the above description, the distance data, which is metadata acquired from the distance sensor 120, is used to calculate the reference range of the event data. However, the distance data can also be input to the neural network unit 320 to further improve the accuracy of the recognition process. The neural network can input multiple channels. For example, when an RGB image is input, three channels corresponding to R, G, and B can be input. Therefore, when distance data is input to the neural network, for example, the event data is input to the first channel, and the distance data is input to the second channel. In this way, the neural network unit 320 can perform more accurate recognition processing by using the distance data in addition to the event data to recognize the recognition target.

[0021] <Second embodiment> Next, an image recognition device according to a second embodiment will be described. In the following description, the same components as those in the first embodiment are denoted by the same reference numerals, and detailed description thereof will be omitted.

[0022] Fig. 4 shows the configuration of recognition system 2 according to the second embodiment. As shown in Fig. 4, recognition system 2 has an event-based sensor 410 and an image recognition device 420. Furthermore, image recognition device 420 has a distance information calculation unit 421 and an image recognition unit 422. In the first embodiment, image recognition device 130 acquires event data and metadata (distance data) from separate sensors, but in this embodiment, image recognition device 420 acquires event data and distance data from event-based sensor 410.

[0023] The pixels constituting the event-based sensor 410 are, for example, a so-called dual pixel configuration having a unit consisting of one microlens and two photodiodes (imaging elements), and each photodiode detects an event. In one pixel, the event data obtained from one photodiode is called image A, and the event data obtained from the other photodiode is called image B. The event data is assumed to be image B. At this time, distance information calculation unit 421 of image recognition device 420 calculates the amount of deviation between image A and image B obtained from each pixel of a predetermined region of event-based sensor 410 as a difference in information of the recognized object. In this way, distance information calculation unit 421 can estimate the distance from event-based sensor 410 to the recognized object. Distance information calculation unit 421 outputs the estimated distance to the recognized object to image recognition unit 422 as distance data of the estimated result. In addition, image recognition unit 422 of image recognition device 420 can acquire data obtained by adding image A and image B obtained from one pixel of event-based sensor 410 as event data. Distance information calculation unit 421 is an example of an estimation unit that estimates the distance to the recognized object based on the difference in information of the recognized object obtained from each of a plurality of imaging elements.

[0024] Image recognition unit 422 generates event data using data acquired from event-based sensor 410, and executes recognition processing of the recognition target object in the same manner as in the first embodiment, using the generated event data and distance data acquired from distance information calculation unit 421. However, unlike image recognition device 130 of the first embodiment, image recognition device 420 of the present embodiment acquires distance data and event data from the same pixel of event-based sensor 410, that is, from pixels having the same optical axis of a microlens. Therefore, in this embodiment, conversion processing of the coordinate system for distance data and event data as performed in the first embodiment is not required.

[0025] As a modification of this embodiment, all or some of event-based sensor 410, distance information calculation unit 421, and image recognition unit 422 may be implemented in one sensor. When event-based sensor 410, distance information calculation unit 421, and image recognition unit 422 are implemented in one sensor, a series of processes from acquisition of event data and metadata to recognition of the recognition target can be completed within the sensor. This makes it possible to realize a sensor that outputs only the recognition result of the recognition target.

[0026] Furthermore, as another modification of this embodiment, event-based sensor 410 may have a so-called hybrid configuration capable of capturing both frame data (images captured by an image sensor) and event data. In this case, distance information calculation unit 421 can calculate distance data using images A and B of frame data acquired from event-based sensor 410. In addition, a technique for estimating three-dimensional information of the periphery of the device from images, such as Visual SLAM (Simultaneous Localization and Mapping), may be used as a process for calculating distance data. This allows distance information calculation unit 421 to calculate from event data or frame data, or calculate distance from event density.

[0027] <Third embodiment> Next, an image recognition device according to a third embodiment will be described. In the following description, the same components as those in the above embodiment are denoted by the same reference numerals, and detailed description thereof will be omitted.

[0028] Fig. 5 shows an example of the configuration of a recognition system 3 according to this embodiment. As shown in Fig. 5, the recognition system 3 has an event-based sensor 110, a distance sensor 120, and an image recognition device 510. The image recognition device 510 also has a decoder 511, a distance information acquisition unit 512, a parameter determination unit 513, and a spiking neural network unit 514. In the above embodiment, the image recognition device executes the recognition process using a CNN, but the image recognition device 510 of this embodiment executes the recognition process of an object to be recognized using an SNN (Spiking Neural Network).

[0029] The decoder 511 decodes the event data input in the (x, y, t, p) format as in the above embodiment, and converts it into a two-dimensional time series pulse train for each pixel. The data obtained by the above process is equivalent to the raw data before a timestamp is added by event-based sensor 110 when an event is detected by event-based sensor 110. As a modification of this embodiment, when image recognition device 510 acquires raw data from event-based sensor 110, image recognition device 510 may be configured without decoder 511.

[0030] Similar to the first embodiment, the distance information acquisition unit 512 acquires distance data as metadata from the distance sensor 120. In addition, the parameter determination unit 513 determines the time constant of the spiking neural network used by the spiking neural network unit 514 based on the distance data acquired by the distance information acquisition unit 512.

[0031] In the LIF (Leaky Integrate and Fire) neuron used in the spiking neural network of this embodiment, the membrane potential V(t) is calculated by the following equation (2).

number

[0032] where τ is the time constant and V R is a predetermined reset potential, and F(t) is a spike train that is an input to the LIF neuron. That is, the membrane potential V(t) behaves as if it is integrating the input F(t) and decaying over time. The rate of decay is controlled by the time constant τ. Specifically, when the time constant τ is increased, the decay becomes gentler, and the time during which one spike has an effect becomes longer. Conversely, when the time constant τ is decreased, the decay becomes steeper, and the time during which one spike has an effect becomes shorter. That is, in this embodiment, by controlling the time constant τ, the reference range of the event data in the spiking neural network unit 514 in the time direction can be controlled. In this embodiment, the time constant τ is controlled so that the decay rate becomes slower as the distance from the distance sensor 120 to the recognition object becomes longer, and the decay rate becomes faster as the distance from the distance sensor 120 to the recognition object becomes shorter.

[0033] The spiking neural network unit 514 executes recognition processing using the two-dimensional time series pulse train data output from the decoder 511 as an input to the spiking neural network. The spiking neural network unit 514 also uses the time constant determined by the parameter determination unit 513 as the time constant of the LIF neuron.

[0034] As a result, in the recognition processing of the recognition target by the spiking neural network unit 514, when the distance to the recognition target is far, it is possible to suppress a decrease in recognition accuracy even if the event density is low. Also, in the recognition processing of the recognition target by the spiking neural network unit 514, when the distance to the recognition target is close, it is possible to realize low-delay recognition processing, which is one of the advantages of SNN.

[0035] <Fourth embodiment> Next, an image recognition device according to a fourth embodiment will be described. In the following description, the same components as those in the above-described embodiment are denoted by the same reference numerals, and detailed description thereof will be omitted.

[0036] In the above embodiment, an example was shown in which the reference range in the time direction of the event data is controlled according to the distance to the recognition target object. In this embodiment, however, the reference range in the time direction is controlled according to the number of events generated in the event-based sensor 110. FIG. 6 shows an example of the configuration of the recognition system 4 in this embodiment. As shown in FIG. 6, the recognition system 4 is an event-based sensor. The image recognition device 610 includes an event data acquisition unit 611, a waiting unit 612, and a neural network unit 613.

[0037] The event data acquisition unit 611 acquires event data from the event-based sensor 110 in the same manner as in the above embodiment.

[0038] The waiting unit 612 counts the number of events for each predetermined pixel region of the event-based sensor 110 based on the event data output from the event data acquisition unit 611, and holds the event data in the waiting unit 612 until the number of events exceeds a predetermined number. When the counted number of events exceeds a predetermined number, the waiting unit 612 integrates the held event data and outputs the integrated event data to the neural network unit 613. When the waiting unit 612 outputs the event data to the neural network unit 613, the waiting unit 612 resets the counted number of events to 0, and repeats counting the number of events using the event data output from the event data acquisition unit 611 after the reset. Holding the event data using the waiting unit 612 in this way corresponds to controlling the reference range of events in the time direction by the event density. In other words, the event density can also be considered as a kind of metadata.

[0039] The neural network unit 613 uses the event data output from the waiting unit 612 to execute the recognition process of the recognition target object in the same manner as described above.

[0040] In this embodiment, the farther the distance to the recognition object, the lower the event density, and the longer the event data retention time in the waiting unit 612, so the wider the reference range of the event data in the time direction. On the other hand, the closer the distance to the recognition object, the higher the event density, and the shorter the event data retention time in the waiting unit 612, so the narrower the reference range of the event data in the time direction. Therefore, in the image recognition device 610 according to this embodiment, as in the first embodiment, the reference time of the event data can be controlled according to the distance to the recognition object. Furthermore, in this embodiment, even if the event density varies due to factors other than the distance (for example, the contrast difference between the recognition object and the background), the retention time of the event data by the waiting unit 612 changes, and the reference time of the event data can be controlled.

[0041] When counting the number of events, waiting unit 612 may decrease the number of counted events by a predetermined number as time passes. As a result, even if an event that occurs suddenly due to noise or the like is counted in the number of events in event-based sensor 110, the number of events decreases while waiting unit 612 holds the event data. As a result, in image recognition device 610, waiting unit 612 can accumulate event data without actually counting the suddenly occurring event as the number of events.

[0042] <Fifth embodiment> Next, an image recognition device according to a fifth embodiment will be described. In the following description, the same components as those in the above-described embodiments are denoted by the same reference numerals, and detailed description thereof will be omitted.

[0043] In the above embodiment, an example was shown in which the reference range of the event data in the time direction was controlled depending on the distance to the recognition target object, but the image recognition device of this embodiment generates attribute information for each area within the imaging range of the event-based sensor. Here, the attribute information is information related to priority including recognition accuracy and recognition speed in recognition processing using event data. Then, the image recognition device of this embodiment controls the reference range of the event data in the time direction for each area within the imaging range based on the generated attribute information.

[0044] Fig. 7 shows an example of the configuration of recognition system 5 in this embodiment. As shown in Fig. 7, recognition system 5 has event-based sensor 110 and image recognition device 710. In addition, image recognition device 710 has event data acquisition unit 711, attribute information generation unit 712, reference range determination unit 713, and recognition processing unit 714.

[0045] As in the first embodiment, the event data acquisition unit 711 acquires event data from the event-based sensor 110. The attribute information generation unit 712 generates attribute information based on information acquired from outside the recognition system 5.

[0046] For example, when recognition system 5 is applied to an automobile, image recognition device 710 stores in advance in a storage unit (not shown) position information of an intersection having a blind spot that may cause a delay in the driver's visual recognition. Then, attribute information generation unit 712 acquires GPS information from an external source and identifies the position of the automobile from the acquired GPS information. From the position indicated by the position information stored in the storage unit and the identified position of the automobile, attribute information generation unit 712 identifies an area near the blind spot within the range currently captured by event-based sensor 110 as an area in which recognition speed is prioritized in recognition processing. Then, attribute information generation unit 712 generates attribute information indicating that recognition speed is prioritized for that area.

[0047] As another example of the recognition system 5 being applied to an automobile, for example, when the automobile tracks a specific vehicle, the image recognition device 710 stores vehicle information such as the license plate number, model, and body color of the tracked vehicle in the storage unit. Then, it is assumed that the image recognition device 710 uses the vehicle information to capture an image of the vehicle by the event-based sensor 110. At this time, the attribute information generating unit 712 specifies an area in the vicinity of the vehicle area indicated by the previous recognition result of the vehicle in the range currently captured by the event-based sensor 110 as an area for which recognition accuracy is to be improved in the recognition process. Then, the attribute information generating unit 712 generates attribute information indicating that the area prioritizes recognition accuracy. The attribute information for each area in the image capture range of the event-based sensor is also called ROI (Region Of Interest) information, and can be considered as a type of metadata.

[0048] Then, based on the attribute information output from attribute information generation unit 712, reference range determination unit 713 determines a reference range in the time direction of event data in the area indicated by the attribute information within the range currently captured by event-based sensor 110. Specifically, reference range determination unit 713 widens the reference range in the time direction when the attribute information indicates that recognition accuracy is prioritized, and narrows the reference range in the time direction when the attribute information indicates that recognition speed is prioritized. Recognition processing unit 714 executes recognition processing of the recognition target object using the event data output from event data acquisition unit 711 and information on the reference range of the event data output from reference range determination unit 713.

[0049] In this way, image recognition device 710 of this embodiment can set whether to give priority to recognition accuracy or recognition speed for a portion of the area within the range currently captured by event-based sensor 110. This makes it possible to realize more suitable image recognition processing depending on the application of image recognition device 710.

[0050] Sixth embodiment Any of the first to fifth embodiments can be applied to the sixth embodiment. FIG. 8 is a schematic diagram for explaining a device 9191 including a semiconductor device 930 of this embodiment. The semiconductor device 930 can be any of the image recognition devices explained in the first to fifth embodiments, or an image recognition device that combines these embodiments. The device 9191 including the semiconductor device 930 will be explained in detail. The semiconductor device 930 has a semiconductor layer. In addition to the semiconductor device 910, it may include a package 920 that houses the semiconductor device 910. The package 920 may include a base to which the semiconductor device 910 is fixed, and a lid such as glass that faces the semiconductor device 910. The package 920 may further include a bonding member such as a bonding wire or a bump that connects a terminal provided on the base and a terminal provided on the semiconductor device 910.

[0051] The device 9191 can include at least one of an optical device 940, a control device 950, a processing device 960, a display device 970, a storage device 980, and a mechanical device 990. The optical device 940 corresponds to the semiconductor device 930. The optical device 940 is, for example, a lens, a shutter, or a mirror. The control device 950 controls the semiconductor device 930. The control device 950 is, for example, a semiconductor device such as an ASIC.

[0052] The processing device 960 processes the signal output from the semiconductor device 930. The processing device 960 is a semiconductor device such as a CPU or ASIC for configuring an AFE (analog front end) or a DFE (digital front end). The display device 970 is an EL display device or a liquid crystal display device that displays information (images) obtained by the semiconductor device 930. The storage device 980 is a magnetic device or a semiconductor device that stores information (images) obtained by the semiconductor device 930. The storage device 980 is a volatile memory such as an SRAM or a DRAM, or a non-volatile memory such as a flash memory or a hard disk drive.

[0053] The mechanical device 990 has a moving part or a propulsion part such as a motor or an engine. In the device 9191, the signal output from the semiconductor device 930 is displayed on the display device 970, or transmitted to the outside by a communication device (not shown) included in the device 9191. For this purpose, the device 9191 preferably further includes a memory device 980 and a processing device 960 in addition to the memory circuit and arithmetic circuit included in the semiconductor device 930. The mechanical device 990 may be controlled based on the signal output from the semiconductor device 930.

[0054] The device 9191 is also suitable for electronic devices such as information terminals (e.g., smartphones and wearable devices) with a photographing function and cameras (e.g., interchangeable lens cameras, compact cameras, video cameras, and surveillance cameras). The mechanical device 990 in the camera can drive components of the optical device 940 for zooming, focusing, and shutter operation. Alternatively, the mechanical device 990 in the camera can move the semiconductor device 930 for vibration isolation operation.

[0055] The device 9191 may be transportation equipment such as a vehicle, a ship, or an aircraft. The mechanical device 990 in the transportation equipment may be used as a moving device. The device 9191 as a transportation equipment is suitable for transporting the semiconductor device 930 or for assisting and / or automating driving (piloting) by using a photographing function. The processing device 960 for assisting and / or automating driving (piloting) can perform processing for operating the mechanical device 990 as a moving device based on information obtained by the semiconductor device 930. Alternatively, the device 9191 may be a medical device such as an endoscope, a measuring device such as a distance measuring sensor, an analytical device such as an electron microscope, an office machine such as a copier, or an industrial device such as a robot.

[0056] According to the sixth embodiment described above, it is possible to obtain good pixel characteristics. Therefore, it is possible to increase the value of the semiconductor device 930. Increasing the value here means at least one of adding a function, improving performance, improving characteristics, improving reliability, improving manufacturing yield, reducing environmental load, reducing costs, making the device smaller, and reducing weight.

[0057] Therefore, if the semiconductor device 930 according to the sixth embodiment is used in the equipment 9191, the value of the equipment can be improved. For example, if the semiconductor device 930 is mounted on a transport equipment, Excellent performance can be obtained when photographing the outside of the equipment and measuring the external environment. Therefore, when manufacturing and selling transportation equipment, it is advantageous to decide to mount the semiconductor device 930 according to the sixth embodiment on the transportation equipment in order to improve the performance of the transportation equipment itself. In particular, the semiconductor device 930 is suitable for transportation equipment that performs driving assistance and / or automatic driving of the transportation equipment using information obtained by the semiconductor device 930.

[0058] The above-described embodiments may be modified as appropriate without departing from the technical concept. The disclosure of this specification includes not only what is described in this specification, but also all matters that can be understood from this specification and the drawings attached hereto. The disclosure of this specification also includes the complement of the concepts described in this specification. In other words, if this specification contains a statement that "A is greater than B," even if the statement that "A is not greater than B" is omitted, this specification still discloses that "A is not greater than B." This is because when it contains a statement that "A is greater than B," it is assumed that the case in which "A is not greater than B" is taken into consideration.

[0059] The disclosure of this embodiment includes the following configuration. (Configuration 1) an event data acquisition unit that acquires event data indicating a change in the amount of light of a pixel; a metadata acquisition unit for acquiring metadata relating to a position of a recognition target object; a processing unit that performs a recognition process on the recognition target object using the event data; having The processing unit controls a reference time of the event data in the recognition process in accordance with the acquired metadata. 1. An image recognition device comprising: (Configuration 2) the recognition process is a process using an algorithm based on a learning model that accepts fixed-size data as input; The processing unit adjusts a size of the event data for input to the algorithm in response to the acquired metadata. 2. The image recognition device according to configuration 1. (Configuration 3) The processing unit generates the data of the fixed size by adjusting a size of the event data in a time direction for input to the algorithm. 3. The image recognition device according to configuration 2. (Configuration 4) The recognition process is a process using an algorithm based on a spiking neural network, The processing unit changes a time constant of the spiking neural network that processes the event data in response to the acquired metadata. 2. The image recognition device according to configuration 1. (Configuration 5) The metadata includes data indicating a distance measurement result of the recognition object, The processing unit extends the reference time of the event data as the distance to the recognition object increases based on the distance measurement result. 5. The image recognition device according to any one of configurations 1 to 4. (Configuration 6) The pixel has a unit including one microlens and a plurality of image pickup elements, The image recognition device further includes an estimation unit that estimates a distance to the recognition object based on a difference in information of the recognition object obtained from each of the plurality of image pickup elements, The metadata acquisition unit acquires, as the metadata, an estimation result of the distance to the recognition object by the estimation unit. 6. The image recognition device according to any one of configurations 1 to 5. (Configuration 7) 7. The image recognition device according to any one of configurations 1 to 6, wherein the metadata includes data indicating an occurrence frequency of a change in the amount of light of the pixel. (Configuration 8) 8. The image recognition device according to any one of configurations 1 to 7, wherein the metadata includes data on priority including at least a recognition accuracy and a recognition speed of the object to be recognized. (Configuration 9) An apparatus including the image recognition device according to any one of configurations 1 to 8, an optical device corresponding to the image recognition device; A control device for controlling the image recognition device; a processing device for processing a signal output from the image recognition device; a display device for displaying information obtained by the image recognition device; a storage device that stores information obtained by the image recognition device; and and a mechanical device that operates based on information obtained by the image recognition device. [Explanation of symbols]

[0060] 130 Image recognition device, 131 Event data acquisition unit, 132 Distance information acquisition unit, 133 Reference range determination unit, 134 Recognition processing unit

Claims

1. An event data acquisition unit that acquires event data indicating a change in the light intensity of a sensor pixel, A metadata acquisition unit that acquires metadata regarding the location of the object to be recognized, A processing unit that uses the event data to perform recognition processing of the object to be recognized using an algorithm based on a spiking neural network, It has, The processing unit controls the reference time of the event data in the recognition process according to the acquired metadata. An image recognition device characterized by the following features.

2. An event data acquisition unit that acquires event data indicating a change in the light intensity of a sensor pixel, A metadata acquisition unit that acquires metadata regarding the location of the object to be recognized, A processing unit that uses the event data to perform recognition processing of the object to be recognized using an algorithm based on a learning model that accepts fixed-size data as input, It has, The processing unit adjusts the size of the event data in the recognition process for input to the algorithm, according to the acquired metadata. An image recognition device characterized by the following features.

3. The processing unit generates the fixed-size data by adjusting the temporal size of the event data for input to the algorithm. The image recognition device according to feature 2.

4. The processing unit changes the time constant of the spiking neural network that processes the event data according to the acquired metadata. The image recognition device according to feature 1.

5. An event data acquisition unit that acquires event data indicating a change in the light intensity of a sensor pixel, A metadata acquisition unit that acquires metadata regarding the location of the object to be recognized, A processing unit that performs recognition processing of the object to be recognized using the event data, It has, The metadata includes data indicating the distance measurement result of the recognized object, The processing unit, based on the distance measurement result, increases the time spent referencing the event data in the recognition process as the distance to the object to be recognized increases. An image recognition device characterized by the following features.

6. The aforementioned pixel has a unit consisting of one microlens and multiple image sensors, The image recognition device further includes an estimation unit that estimates the distance to the object to be recognized based on the difference in the information of the object to be recognized obtained from each of the plurality of image sensors. The metadata acquisition unit acquires the distance estimation result to the recognized object by the estimation unit as metadata. The image recognition device according to feature 1.

7. The pixel has a unit consisting of one microlens and a plurality of image sensors, The image recognition device further includes an estimation unit that estimates the distance to the object to be recognized based on the difference in the information of the object to be recognized obtained from each of the plurality of image sensors. The metadata acquisition unit acquires the distance estimation result to the recognized object by the estimation unit as metadata. The image recognition device according to feature 2.

8. The pixel has a unit consisting of one microlens and a plurality of image sensors, The image recognition device further includes an estimation unit that estimates the distance to the object to be recognized based on the difference in the information of the object to be recognized obtained from each of the plurality of image sensors. The metadata acquisition unit acquires the distance estimation result to the recognized object by the estimation unit as metadata. The image recognition device according to feature 5.

9. The image recognition device according to claim 1, characterized in that the metadata includes data indicating the frequency of occurrence of changes in the light intensity of the pixels.

10. The image recognition device according to claim 2, characterized in that the metadata includes data indicating the frequency of occurrence of changes in the light intensity of the pixels.

11. The image recognition device according to claim 5, characterized in that the metadata includes data indicating the frequency of occurrence of changes in the light intensity of the pixels.

12. The image recognition apparatus according to claim 1, characterized in that the metadata includes at least priority data including the recognition accuracy and recognition speed of the object to be recognized.

13. The image recognition device according to claim 2, characterized in that the metadata includes at least priority data including the recognition accuracy and recognition speed of the object to be recognized.

14. The image recognition device according to claim 5, characterized in that the metadata includes at least priority data including the recognition accuracy and recognition speed of the object to be recognized.

15. A device comprising an image recognition device according to any one of claims 1 to 14, Optical device corresponding to the aforementioned image recognition device, Control device for controlling the image recognition device, A processing unit that processes the signal output from the image recognition device, A display device that displays the information obtained by the image recognition device. A storage device for storing information obtained by the image recognition device, and The apparatus is characterized by further comprising at least one of the following: a mechanical device that operates based on information obtained by the image recognition device.

Images