Event recognition system, robot system, and event recognition method

Abstract

This event recognition system comprises: an imaging unit; a filter unit that extracts temporal and spatial features from imaging data; a first processing unit; a second processing unit; and a control unit. In a first mode, the first processing unit processes data corresponding to the features extracted by the filter unit using a first neural network, and the control unit determines whether or not a predetermined event has occurred on the basis of an output from the first neural network, and activates a second mode when it is determined that the event has occurred. In the second mode, the second processing unit processes the imaging data using a second neural network, and the control unit recognizes details of the event on the basis of an output from the second neural network.

Description

Event recognition system, robot system, and event recognition method

[0001] One aspect of the present disclosure relates to an event recognition system, a robot system, and an event recognition method.

[0002] For example, Patent Literature 1 describes a system that uses a neural network to recognize the pose of an object in an image. This system uses deep learning for pose recognition.

[0003] US Patent Application Publication No. 2022 / 0301304

[0004] When the above-mentioned system is applied to the recognition of actions and gestures (events), it is believed that high-precision recognition is possible. However, when this system is actually operated in the field, deep learning is a large-scale neural network and requires a huge amount of calculation, so there is a risk that the system will consume a lot of power or the processing will stop due to a lack of computing resources.

[0005] An object of one aspect of the present disclosure is to provide an event recognition system, a robot system, and an event recognition method that enable highly accurate recognition while reducing the amount of calculation.

[0006] An event recognition system according to one aspect of the present disclosure is [1] "an event recognition system comprising: an imaging unit that acquires imaging data; a filter unit that performs filtering using a time filter and a spatial filter on the imaging data, which is time-series data acquired by the imaging unit, and extracts temporal and spatial features from the imaging data; a first processing unit that performs processing using a first neural network; a second processing unit that performs processing using a second neural network different from the first neural network; and a control unit, wherein the event recognition system includes a first mode and a second mode as operation modes, wherein in the first mode, the first processing unit processes data corresponding to the features extracted by the filter unit using the first neural network, and the control unit determines whether a predetermined event has occurred based on an output from the first neural network, and activates the second mode when it determines that the event has occurred, and in the second mode, the second processing unit processes the imaging data acquired by the imaging unit using the second neural network, and the control unit recognizes details of the event based on the output from the second neural network."

[0007] In this event recognition system, in a first mode, data corresponding to temporal and spatial features extracted from imaging data is processed using a first neural network, and whether a predetermined event has occurred is determined based on the output from the first neural network. If the determination results in the occurrence of an event, a second mode is activated. In the second mode, the imaging data is processed using a second neural network, and details of the event are recognized based on the output from the second neural network. As a result, in the first mode, smaller data (data corresponding to features extracted from the imaging data) is processed using the first neural network compared to the second mode. Therefore, the first neural network can be made smaller than the second neural network, thereby reducing the amount of computation. Furthermore, because the data input to the first neural network is small, slowdowns or interruptions in processing can be suppressed even when, for example, the imaging data is large. Furthermore, in the second mode, details of the event can be recognized using a second neural network that is larger than the first neural network, enabling highly accurate recognition. Therefore, this event recognition system enables highly accurate recognition while reducing the amount of computation.

[0008] An event recognition system according to an aspect of the present disclosure may be [2] "the event recognition system according to [1], in which the number of nodes in the input layer of the first neural network is smaller than the number of nodes in the input layer of the second neural network." In this case, the first neural network can be made smaller in size, and the amount of calculation can be reduced.

[0009] An event recognition system according to an aspect of the present disclosure may be [3] "the event recognition system according to [1] or [2], in which the amount of calculation in the first neural network to calculate an output from an input is less than the amount of calculation in the second neural network to calculate an output from an input." In this case, the amount of calculation can be reduced.

[0010] An event recognition system according to one aspect of the present disclosure may be [4] "the event recognition system according to any one of [1] to [3], wherein the first neural network is a reservoir computing model having an input layer, a reservoir layer connected to the input layer, and an output layer connected to the reservoir layer, and in which only the connection weights between the reservoir layer and the output layer are trained." In this case, the first neural network can be made smaller, and the amount of calculation can be reduced.

[0011] An event recognition system according to one aspect of the present disclosure may be [5] "the event recognition system according to any one of [1] to [4], wherein the second neural network is a deep learning model having an input layer, at least three or more intermediate layers connected to the input layer, and an output layer connected to the intermediate layers." In this case, the details of the event can be recognized using a large-scale second neural network, enabling highly accurate recognition.

[0012] An event recognition system according to one aspect of the present disclosure may be [6] "the event recognition system according to any one of [1] to [5], wherein the imaging unit has a two-dimensional sensor that acquires imaging data, and the filter unit is configured by a circuit unit formed on a substrate separate from the two-dimensional sensor." In this case, it is possible to easily realize a filter unit that extracts temporal and spatial features.

[0013] An event recognition system according to one aspect of the present disclosure may be [7] "the event recognition system according to any one of [1] to [5], wherein the imaging unit has a two-dimensional sensor that acquires imaging data, and the filter unit is incorporated within the two-dimensional sensor." In this case, the configuration can be simplified compared to, for example, a case where the filter unit is configured by a circuit unit formed on a substrate separate from the sensor.

[0014] An event recognition system according to one aspect of the present disclosure may be [8] "the event recognition system according to any one of [1] to [7], wherein the first neural network is a trained model based on training data corresponding to the event, and in the first mode, the control unit determines whether the event has occurred based on a comparison between an output from the first neural network and the training data." In this case, it is possible to easily determine whether an event has occurred.

[0015] An event recognition system according to an aspect of the present disclosure may be [9] "the event recognition system according to any one of [1] to [8], wherein in the first mode, one data set corresponding to the feature extracted by the filter unit is input to the first neural network at one processing timing." In this case, the first neural network can be made smaller in size than when, for example, multiple data sets are input to the first neural network at one processing timing.

[0016] An event recognition system according to one aspect of the present disclosure may be

[10] "the event recognition system according to any one of [1] to [9], wherein the imaging unit has a first two-dimensional sensor and a second two-dimensional sensor that acquire imaging data, the filter unit extracts the temporal and spatial features from the imaging data acquired by the first two-dimensional sensor, and in the first mode, the first processing unit processes data corresponding to the features extracted by the filter unit from the imaging data acquired by the first two-dimensional sensor using the first neural network, and in the second mode, the second processing unit processes the imaging data acquired by the second two-dimensional sensor using the second neural network." In this case, by separating the first two-dimensional sensor used in the first mode from the second two-dimensional sensor used in the second mode, it is possible to reduce circuit complexity.

[0017] An event recognition system according to one aspect of the present disclosure may be

[11] "the event recognition system according to any one of [1] to [9], wherein the imaging unit has a two-dimensional sensor that acquires imaging data, and in the first mode, the first processing unit processes data corresponding to the features extracted by the filter unit from the imaging data acquired by the two-dimensional sensor using the first neural network, and in the second mode, the second processing unit processes the imaging data acquired by the two-dimensional sensor using the second neural network." In this case, by using a common two-dimensional sensor for the first mode and the second mode, the imaging unit can be made smaller.

[0018] An event recognition system according to an aspect of the present disclosure may be

[12] "the event recognition system according to any one of [1] to

[10] , wherein, when it is determined that the event has occurred in the first mode, the operation mode is changed from the first mode to the second mode." In this case, the first mode is turned off when the second mode is activated, thereby reducing power consumption.

[0019] An event recognition system according to an aspect of the present disclosure may be

[13] "the event recognition system according to any one of [1] to

[10] , wherein, when it is determined that the event has occurred in the first mode, the second mode is activated while the first mode remains activated." In this case, the occurrence of an event can be detected in the first mode even when the second mode is activated.

[0020] An event recognition system according to one aspect of the present disclosure may be

[14] "the event recognition system according to any one of [1] to

[13] , wherein in the first mode, the filter unit applies the filtering to the imaging data acquired by the imaging unit, and in the second mode, the second processing unit uses the second neural network to process the imaging data having a data size smaller than the imaging data to which the filtering is applied in the first mode." In this case, by reducing the size of the imaging data to be processed using the second neural network in the second mode, it is possible to increase the speed and accuracy of processing in the second mode.

[0021] A robot system according to one aspect of the present disclosure is

[15] "a robot system including the event recognition system according to any one of [1] to

[14] and a robot, wherein the robot performs a predetermined action or process based on details of the event recognized in the second mode." For the reasons described above, this robot system can reduce the amount of calculation while enabling highly accurate recognition.

[0022] An event recognition method according to one aspect of the present disclosure is

[16] "an event recognition method including: a first step of filtering imaging data, which is time-series data acquired by an imaging unit, using a time filter and a spatial filter, extracting temporal and spatial features from the imaging data, processing data corresponding to the extracted features using a first neural network, determining whether a predetermined event has occurred based on an output from the first neural network, and executing a second step if it is determined that the event has occurred; and a second step of processing the imaging data acquired by the imaging unit using a second neural network different from the first neural network, and recognizing details of the event based on the output from the second neural network."

[0023] In this event recognition method, in a first step, data corresponding to features extracted from imaging data is processed using a first neural network, and whether a predetermined event has occurred is determined based on the output from the first neural network. If the determination results in the occurrence of an event, a second step is executed, in which the imaging data is processed using a second neural network, and details of the event are recognized based on the output from the second neural network. As a result, in the first step, smaller data (data corresponding to features extracted from the imaging data) than in the second step is processed using the first neural network, allowing the first neural network to be smaller in size than the second neural network and reducing the amount of calculation. Furthermore, because the data input to the first neural network is small, slowdowns or interruptions in processing can be suppressed even when, for example, the imaging data is large. Furthermore, in the second step, details of the event can be recognized using a second neural network larger than the first neural network, enabling highly accurate recognition. Therefore, this event recognition method allows for highly accurate recognition while reducing the amount of calculation.

[0024] According to one aspect of the present disclosure, it is possible to provide an event recognition system, a robot system, and an event recognition method that enable highly accurate recognition while reducing the amount of calculation.

[0025] 1 is a configuration diagram of a robot system including an event recognition system; FIG. 2 is a configuration diagram of a filter unit; FIG. 3 is a configuration diagram of a reservoir model; FIG. 4 is a configuration diagram of a deep learning model; FIG. 5 is a flowchart for explaining processing in a first mode; FIG. 6 is a diagram for explaining processing in a first mode; (a) is a diagram showing examples of output values ​​and target values ​​when an event has not occurred, and (b) is a diagram showing examples of output values ​​and target values ​​when an event has occurred; (a) is a diagram showing an example of a value obtained by squaring the difference between an output value and a target value when an event has not occurred, and (b) is a diagram showing an example of a value obtained by squaring the difference between an output value and a target value when an event has occurred; FIG. 7 is a flowchart for explaining processing in a second mode; (a) is a configuration diagram of a robot system of a first modified example; (b) is a diagram showing a configuration diagram of a robot system of a second modified example; and (c) is a flowchart for explaining processing in a second mode of the third modified example.

[0026] Hereinafter, an embodiment according to one aspect of the present disclosure will be described in detail with reference to the drawings. In the following description, the same or equivalent elements will be designated by the same reference numerals, and redundant description will be omitted.

[0027] 1 , the robot system 1 includes an event recognition system 2 and a robot 3 communicatively connected to the event recognition system 2. Generally speaking, the event recognition system 2 determines whether a predetermined event (phenomenon) has occurred based on imaging data acquired by an imaging unit 11 (described later). Furthermore, if the event recognition system 2 determines that an event has occurred, it recognizes details of the event based on the imaging data acquired by the imaging unit 11. The robot 3 executes a predetermined operation or process based on the details of the event recognized by the event recognition system 2.

[0028] The event to be recognized may be any event, and one or more (at least one) events may be set as the recognition target. For example, the event may include a predetermined posture or movement (e.g., gesture) of the object OJ (e.g., a person). That is, the event recognition system 2 may be a movement recognition system. The object OJ may be a machine, in which case the event may include an abnormal posture or abnormal movement of the machine. The object OJ may be an object, in which case the event may include the object falling, or the discovery of an obstacle.

[0029] "Recognizing details of an event" means, for example, performing more advanced (more information-rich) recognition of the event than determining that an event has occurred. For example, recognizing details of an event may include recognizing attributes of the object OJ (e.g., person, object, machine). Recognizing details of an event may include recognizing the posture or type of movement of the object OJ. As an example, if it is determined that "approach of an object OJ" has occurred as an event, it may be recognized as details of the event that "the approaching object OJ is a person or an object (e.g., a ball)."

[0030] The robot 3 is, for example, a mechanical device including an actuator device and configured to be movable and / or operable. The robot 3 may have any shape, and may be, for example, a humanoid robot or a non-humanoid robot. The robot 3 may be a service robot that performs a predetermined service, or may be, for example, an industrial robot used in a factory.

[0031] The robot 3 performs a predetermined action or process based on the details of the event recognized by the event recognition system 2. The action or process may be any action or process. As an example, the robot 3 may perform an avoidance action when an object OJ approaches. In this case, for example, when the event recognition system 2 determines that an event "approaching an object OJ" has occurred, it recognizes, as the event details, whether the approaching object OJ is a person or an object (e.g., a ball). When the event details are recognized as "the approaching object OJ is a person," the robot 3 performs an avoidance action against the person so as to maintain a relatively large distance between the robot 3 and the person during the avoidance action. On the other hand, when the event details are recognized as "the approaching object OJ is an object (e.g., a ball)," the robot 3 performs an avoidance action against the object so as to maintain a relatively small distance between the robot 3 and the object during the avoidance action. As another example, the robot 3 may stop operating when an abnormality occurs in the object OJ, which is a machine. For example, if it is determined that a "machine abnormality" has occurred as an event, and if an "abnormality that requires operation to be stopped" is recognized as the details of the event, the robot 3 may stop operating.

[0032] When the event recognition system 2 determines that "movement of the object OJ" has occurred as an event, it may recognize as details of the event, "whether the moving object OJ (person) is waving and calling." When the robot 3 recognizes that "the object OJ (person) is waving and calling" as details of the event, it may perform an action of approaching the person waving. Alternatively, when the event recognition system 2 determines that "movement of the object OJ" has occurred as an event, it may recognize as details of the event, "whether the moving object OJ is a person or an object." When the robot 3 recognizes that "the object OJ is a person" as details of the event, it may perform actions such as measuring the size (height) of the person or recording their face. This measurement or recording may be performed as a calculation process or recording process within the robot 3. In this way, the robot 3 may perform a physical action or a predetermined internal process based on the details of the recognized event.

[0033] 1, the event recognition system 2 includes an imaging unit 11, a filter unit 12, a preprocessing unit 13, a first processing unit 14, a second processing unit 15, and a control unit 16. The imaging unit 11 has a first image sensor 21 (first two-dimensional sensor, first imaging unit) and a second image sensor 22 (second two-dimensional sensor, second imaging unit).

[0034] Each of the first image sensor 21 and the second image sensor 22 is an image sensor (two-dimensional sensor) having a plurality of pixels arranged two-dimensionally, such as a camera. Each of the first image sensor 21 and the second image sensor 22 acquires image data consisting of the brightness values ​​of each pixel at a predetermined period (frame rate). In this manner, each of the first image sensor 21 and the second image sensor 22 acquires imaging data (data acquired by imaging, in this example, image data) that is time-series data. In this example, the first image sensor 21 captures an image of the periphery of the robot 3 via a light-receiving lens 23, and the second image sensor 22 captures an image of the periphery of the robot 3 via a light-receiving lens 24. When an object OJ is present within the imaging range, the first image sensor 21 and the second image sensor 22 acquire imaging data in which the object OJ is captured.

[0035] The filter unit 12 is configured by a circuit unit 18 formed on a substrate 17 (e.g., a circuit board) separated from the first image sensor 21 and the second image sensor 22. In other words, the substrate 17 is a substrate separate from the substrate on which the first image sensor 21 and the second image sensor 22 are formed. The circuit unit 18 is, for example, a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC).

[0036] 2 , the filter unit 12 includes a first spatial filter 31, a first temporal filter 32, a second spatial filter 33, a second temporal filter 34, a first rectifier 35, an inverter 36, and a second rectifier 37. The filter unit 12 performs filtering on the imaging data D1, which is time-series data acquired by the first image sensor 21, using the first temporal filter 32 and the second temporal filter 34, and the first spatial filter 31 and the second spatial filter 33, to extract temporal and spatial features from the imaging data D1. In other words, the filter unit 12 is a spatiotemporal filter that extracts spatiotemporal features from the imaging data D1.

[0037] The filter unit 12 is a neuromorphic filter that mimics the structure and function of the human brain or retina and extracts temporal and spatial features from the imaging data D1. The filter unit 12 includes a first time filter 32 and a second time filter 34, thereby extracting specific features related to temporal changes. For example, in this example, the time filter is configured with bandpass filters (e.g., a low-pass filter, a high-pass filter, and a band-pass filter), and only changes in a specific frequency band are extracted. The time filter is configured to include at least one of a low-pass filter, a high-pass filter, and a band-pass filter. As described below, in this example, the time filter is configured with a low-pass filter and a band-pass filter.

[0038] The temporal change relates to, for example, the moving speed of the object OJ (moving object) captured by the first image sensor 21. This speed can be expressed, for example, by the number of frames elapsed between when an object OJ is captured by a pixel of the first image sensor 21 and when it is captured by another pixel of the first image sensor 21. For example, a low-pass filter passes only information at speeds below a certain level, so providing a low-pass filter prevents fast moving objects from being detected. Furthermore, a high-pass filter passes only information at speeds above a certain level, so providing a high-pass filter prevents slow (or stationary) moving objects from being detected. As a result, for example, providing a low-pass filter and a high-pass filter makes it possible to detect only moving objects moving within a specific speed range. In other words, in this case, the time filter constitutes a band-pass filter and does not respond to DC (constant) components that do not change over time or to fast-moving moving objects. The time filter may also be configured to include a band-pass filter that passes only information within a specific speed range. In this case, moving objects moving within a specific speed range can still be detected.

[0039] Furthermore, the filter unit 12 extracts features related to spatial changes by including a first spatial filter 31 and a second spatial filter 33. For example, as described below, in this example, the spatial filter is configured by a smoothing filter (e.g., a Gaussian filter), and the contour (edge) portion of the captured object OJ is emphasized and extracted.

[0040] The first spatial filter 31 is a spatial filter that mimics horizontal cells in the human retina and is configured, for example, by a Gaussian filter with a relatively small half-width. The first spatial filter 31 receives time-series image data D1 (image data in this example) acquired by the first image sensor 21. By passing through the first spatial filter 31, the image is blurred, smoothing out changes in luminance values ​​between adjacent pixels. The first temporal filter 32 is a temporal filter that mimics horizontal cells in the human retina and is configured, for example, by a low-pass filter. The image blurred by the first spatial filter 31 is input to the first temporal filter 32 (low-pass filter), which removes fast-changing components of moving objects, thereby further reducing the amount of information.

[0041] The second spatial filter 33 is a spatial filter that mimics the photoreceptor cells of the human retina and is configured, for example, by a Gaussian filter with a smaller half-width than the first spatial filter 31. The image data D1 is input to the second spatial filter 33. By passing through the second spatial filter 33, the image is blurred, and the change in luminance value between adjacent pixels becomes smoother. The degree of blurring of the image that has passed through the second spatial filter 33 is smaller than the degree of blurring of the image that has passed through the first spatial filter 31.

[0042] The difference between the image that has passed through the first temporal filter 32 and the image that has passed through the second spatial filter 33 is input to the second temporal filter 34. By taking the difference between the images that have passed through the first spatial filter 31 and the second spatial filter 33, which have different half-widths (two images with different degrees of blurring), the contours of objects in the image are emphasized. The second temporal filter 34 is a temporal filter that mimics the bipolar cells of the human retina and is configured, for example, by a band-pass filter. By inputting the image to the second temporal filter 34, only the velocity components of the moving object within a specific range are extracted, thereby further reducing the amount of information.

[0043] The image that has passed through the second temporal filter 34 is branched and input to a first rectifier 35 and an inverter 36. The first rectifier 35 is a rectifier circuit that passes and outputs only pixels with positive luminance among the pixels in the image. The output data from the first rectifier 35 is output from the filter unit 12 to the subsequent pre-processing unit 13. The inverter 36 inverts the positive and negative luminance of each pixel in the input image. The second rectifier 37 is a rectifier circuit that passes and outputs only pixels with positive luminance among the pixels in the image. In other words, by passing through the inverter 36 and the second rectifier 37, only pixels with negative luminance among the pixels in the image input to the inverter 36 are passed and output. The output data from the second rectifier 37 is output from the filter unit 12 to the subsequent pre-processing unit 13. The first rectifier 35 and the second rectifier 37 emphasize the amount of change between positive and negative. In the filter unit 12, temporal and spatial features are extracted from the imaging data D1 acquired by the first image sensor 21 through the above filtering process.

[0044] The preprocessing unit 13 is configured by a circuit unit 18 on which the filter unit 12 is implemented. That is, the filter unit 12 and the preprocessing unit 13 are implemented on the same circuit unit 18. The preprocessing unit 13 performs preprocessing on the data output from the filter unit 12 and outputs the preprocessed data to the first processing unit 14.

[0045] The first processing unit 14 is configured by a circuit unit 18 on which the filter unit 12 and the preprocessing unit 13 are implemented. That is, the filter unit 12, the preprocessing unit 13, and the first processing unit 14 are implemented on the same circuit unit 18. The first processing unit 14 stores a reservoir model 40 ( FIG. 3 ) as a first neural network, and the first processing unit 14 processes the data output from the preprocessing unit 13 using the reservoir model 40 (reservoir computing model). The data output from the first processing unit 14 is input to the control unit 16.

[0046] As shown in FIG. 3 , the reservoir model 40 includes an input layer 41, a reservoir layer 42 connected to the input layer 41, and an output layer 43 connected to the reservoir layer 42. The input layer 41 includes multiple nodes 41a. The reservoir layer 42 includes multiple nodes 42a. Each node 41a in the input layer 41 is connected to at least one of the multiple nodes 42a in the reservoir layer 42. In the reservoir layer 42, each node 42a is connected to one or more other nodes 42a. The multiple nodes 42a may include one or more recurrent nodes 42a1 configured to input at least a portion of their own output. In this example, the reservoir layer 42 is configured as an echo state network, forming a state (echo state) in which past information in the input data, which is time-series data, reverberates and remains. The output layer 43 includes multiple nodes 43a. Each node 43 a of the output layer 43 is coupled to at least one of the plurality of nodes 42 a of the reservoir layer 42 .

[0047] In the reservoir model 40, each connection between the nodes 41a, 42a, and 43a has a weight. When data is input to the input layer 41, a product-sum operation is performed at each node 42a, 43a. Finally, the value of each node 43a in the output layer 43 is output as output data from the reservoir model 40. Each node 42a in the reservoir layer 42 may perform a nonlinear transformation on the weighted sum of the input values. An example of this nonlinear transformation is a calculation using a sigmoid function (activation function). Instead of encoding information using real activity values, information may be encoded using spikes. That is, instead of configuring each node 42a as an artificial neuron model, each node 42a may be configured as a spiking neuron model. The reservoir layer 42 is not limited to an echo state network and may have any configuration. For example, the reservoir layer 42 may be a physical reservoir utilizing physical phenomena such as light or fluids.

[0048] In the reservoir model 40, only the connection weights between the reservoir layer 42 and the output layer 43 are learned based on target values ​​(learning data), and the connection weights between the input layer 41 and the reservoir layer 42 and the connection weights between the nodes 42a in the reservoir layer 42 are not learned. This enables faster learning than, for example, learning all the connection weights in a neural network.

[0049] The second processing unit 15 is configured by a circuit separate from the circuit unit 18. The second processing unit 15 is configured by, for example, a graphics processing unit (GPU) or a central processing unit (CPU). The circuit that configures the second processing unit 15 is larger in scale than the circuit that configures the first processing unit 14 (e.g., has a larger amount of memory for calculations or a larger number of cores). In this example, the circuit that configures the second processing unit 15 is larger in scale than the filter unit 12, the preprocessing unit 13, and the circuit unit 18 that configures the first processing unit 14. The second processing unit 15 stores a deep learning model 50 ( FIG. 4 ) as a second neural network different from the first neural network, and the second processing unit 15 processes data output from the second image sensor 22 using the deep learning model 50. The data output from the second processing unit 15 is input to the control unit 16.

[0050] As shown in FIG. 4 , the deep learning model 50 has an input layer 51, a plurality of intermediate layers 52 connected to the input layer 51, and an output layer 53 connected to the intermediate layers 52. The input layer 51 has a plurality of nodes 51a. At least three intermediate layers 52 are provided. Each intermediate layer 52 has a plurality of nodes 52a. Each node 52a of the input layer 51 is connected to a respective node 52a of the intermediate layer 52A located most upstream. Each node 52a of one intermediate layer 52 is connected to a respective node 52a of the next intermediate layer 52. The output layer 53 has a plurality of nodes 53a. Each node 53a of the output layer 53 is connected to a respective node 52a of the intermediate layer 52B located most downstream.

[0051] In the deep learning model 50, each connection between nodes 51a, 52a, and 53a has a weight. When data is input to the input layer 51, a product-sum operation is performed at each node 52a and 53a, and finally, the value of each node 53a in the output layer 53 is output as output data from the deep learning model 50. Each node 52a in the intermediate layer 52 may perform a nonlinear transformation on a value obtained by weighting and adding the input values. An example of this nonlinear transformation is a calculation using a sigmoid function (activation function). In the deep learning model 50, the connection weights between the input layer 41 and the intermediate layer 52, the connection weights between the intermediate layers 52, and the connection weights between the intermediate layer 52 and the output layer 53 are learned based on target values ​​(learning data).

[0052] The reservoir model 40, which is the first neural network, is smaller in scale than the deep learning model 50, which is the second neural network. In this example, the number of nodes 41a arranged in the input layer 41 of the reservoir model 40 (the number of nodes) is smaller than the number of nodes 51a arranged in the input layer 51 of the deep learning model 50 (the number of nodes). In other words, the amount of data input to the reservoir model 40 (the number of data dimensions) is smaller than the amount of data input to the deep learning model 50 (the number of data dimensions). The amount of calculation required to calculate an output from an input in the reservoir model 40 is smaller than the amount of calculation required to calculate an output from an input in the deep learning model 50. Therefore, the power consumption per unit time in the reservoir model 40 is smaller than the power consumption per unit time in the deep learning model 50.

[0053] The control unit 16 is configured by, for example, a computer including a processor such as a CPU and storage media such as a RAM and a ROM. The control unit 16 processes data output from the first processing unit 14 and the second processing unit 15. The control unit 16 is communicatively connected to the robot 3 and controls the operation of the robot 3. The control unit 16 may be a control unit of the robot 3, and may also be considered to be a part of the robot 3.

[0054] The event recognition system 2 includes a first mode and a second mode as operation modes. The first mode is a mode for determining whether or not a recognition target event has occurred, and the second mode is a mode for recognizing details of an event determined to have occurred in the first mode. The first mode and the second mode are the first and second steps, respectively, of an event recognition method executed by the event recognition system 2.

[0055] FIG. 5 is a flowchart illustrating processing in the first mode. In the first mode, the control unit 16 activates the first image sensor 21, which acquires imaging data, which is time-series data (step S1). The first image sensor 21 may remain activated in the first mode. Next, the filter unit 12 performs filtering on the imaging data acquired by the first image sensor 21 to extract temporal and spatial features from the imaging data (step S2). Next, the preprocessing unit 13 performs preprocessing on the data output from the filter unit 12 and outputs the preprocessed data to the first processing unit 14 (step S3). Details of the processing in step S3 will be described later, along with details of the processing in steps S4, S5, and S6. Next, the first processing unit 14 processes the data output from the preprocessing unit 13 using the reservoir model 40 (step S4). Next, the control unit 16 compares the output from the reservoir model 40 with the learning data (step S5) and determines whether an event has occurred based on the comparison result (step S6). If it is determined in step S6 that an event has occurred (YES in step S6), the operation mode is changed to the second mode (the second mode is activated) (step S7). If it is determined in step S6 that an event has not occurred (NO in step S6), the process returns to step S1.

[0056] FIG. 6 is a diagram illustrating processing in the first mode. As shown in FIG. 6, the preprocessing unit 13 performs preprocessing on the data D2 output from the filter unit 12, and the preprocessed data is input to the reservoir model 40 (step S3). The data D2 is time-series data, representing the temporal and spatial features extracted by the filter unit 12. In the preprocessing, for example, the data D2 output from the filter unit 12 is compressed into data with a number of dimensions corresponding to the number of nodes in the input layer 41 of the reservoir model 40. In this example, the number of nodes in the input layer 41 is 12, and the data D2 output from the filter unit 12 is compressed into 12-dimensional vector data and input to the reservoir model 40. The data input to the reservoir model 40 is time-series data, and in this example, it includes a total of 29 frames of data, from frame 0 to frame 28. The data in each frame includes 12-dimensional vector data. The time interval between frames corresponds to the data acquisition period (frame rate) of the first image sensor 21 and the second image sensor 22. The data input to the reservoir model 40 is data corresponding to the temporal and spatial features extracted by the filter unit 12 (in this example, data obtained by compressing data D2 representing the temporal and spatial features). In this example, only one data set corresponding to the features extracted by the filter unit 12 is input to the reservoir model 40 in one processing timing (one frame).

[0057] In this example, the reservoir model 40 predicts and outputs data one frame later based on input data (step S4). That is, the reservoir model 40 is trained in advance using training data (teacher data, normal data) so as to predict a data set in the frame following an input data set. For example, in this example, for frame 0, time series data from frames 1 to 29 is predicted from time series data from frames 0 to 28. The training data is data corresponding to the event to be recognized, such as image data actually acquired by the first image sensor 21 when no event has occurred. When normal data is input to the reservoir model 40, the reservoir model 40 can correctly predict (reproduce) data one frame later. However, when abnormal data is input to the reservoir model 40, the reservoir model 40 cannot correctly predict (reproduce) data one frame later. Therefore, by comparing the output from the reservoir model 40 with the training data, it is possible to determine whether an event has occurred (steps S5 and S6). The reservoir model 40 is trained before the event recognition system 2 starts operating, and the reservoir model 40 is not trained while the event recognition system 2 is operating.

[0058] 7(a) is a diagram showing an example of an output value from the reservoir model 40 and a target value in the learning data when no event has occurred, and FIG. 7(b) is a diagram showing an example of an output value and a target value when an event has occurred. FIG. 8(a) is a diagram showing an example of a value obtained by squaring the difference between an output value and a target value when no event has occurred, and FIG. 8(b) is a diagram showing an example of a value obtained by squaring the difference between an output value and a target value when an event has occurred. FIG. 8(a) corresponds to FIG. 7(a), and FIG. 8(b) corresponds to FIG. 7(b). In FIGS. 7 and 8, one of the 12-dimensional data in each frame is extracted and shown.

[0059] 7A, when no event occurs, the output value from the reservoir model 40 is a value (waveform) close to the target value (normal value) in the training data. As shown in FIG. 7B, when an event occurs, the output value from the reservoir model 40 is a value (waveform) that is significantly different from the target value in the training data.

[0060] In this example, in step S6, the control unit 16 calculates the square of the difference between the output value from the reservoir model 40 in each frame and the target value in the training data, and determines that an event has occurred if any of the squares of the difference for each frame exceeds a predetermined threshold (e.g., 200). As shown in FIG. 8( a), if an event has not occurred, the square of the difference is relatively small. As shown in FIG. 8( b), if an event has occurred, the square of the difference is relatively large. Note that the control unit 16 may determine whether an event has occurred based on the difference between the output value from the reservoir model 40 and the target value in the training data, and the determination method is not limited to the above example. For example, the average value of the differences between the output values ​​and the target values ​​in all frames may be compared with a threshold. Through the above processing, it is determined whether an event to be recognized has occurred in the first mode.

[0061] 9 is a flowchart illustrating processing in the second mode. In the second mode, the control unit 16 activates the second image sensor 22, and the second image sensor 22 acquires imaging data, which is time-series data (step S11). Subsequently, the second processing unit 15 processes the imaging data acquired by the second image sensor 22 using the deep learning model 50 (step S12). Subsequently, the control unit 16 recognizes details of the event based on the output from the deep learning model 50 (step S13).

[0062] The deep learning model 50 is trained in advance using training data so as to be able to recognize details of an event from the imaging data output from the second image sensor 22. The training data is data corresponding to the details of the event to be recognized, such as imaging data actually acquired by the second image sensor 22 when the event occurs. For example, when it is determined that an event has occurred in which an "approach of an object OJ" has occurred and the details of the event are recognized as "whether the approaching object OJ is a person or an object," the training data is imaging data acquired by the second image sensor 22 when the person approaches and when the object approaches. The training of the deep learning model 50 is completed in advance before the event recognition system 2 starts operating, and the training of the deep learning model 50 is not performed while the event recognition system 2 is operating.

[0063] Through the above processing, in the second mode, details of the event determined to have occurred in the first mode are recognized. After the details of the event are recognized in the second mode, the control unit 16 causes the robot 3 to perform a predetermined action based on the details of the event recognized in the second mode. The action performed by the robot 3 is, for example, the above-mentioned avoidance behavior. [Operation and Effect]

[0064] In the event recognition system 2, in the first mode, data corresponding to temporal and spatial features extracted from the imaging data is processed using a reservoir model 40 (first neural network), and it is determined whether a predetermined event has occurred based on the output from the reservoir model 40. If it is determined that an event has occurred as a result of the determination, the operation mode is changed to the second mode (the second mode is activated). In the second mode, the imaging data is processed using a deep learning model 50 (second neural network), and details of the event are recognized based on the output from the deep learning model 50. As a result, in the first mode, smaller data (data corresponding to features extracted from the imaging data) is processed using the reservoir model 40 compared to the second mode. Therefore, the reservoir model 40 can be made smaller in size than the deep learning model 50, and the amount of calculation can be reduced. Furthermore, because the data input to the reservoir model 40 is small, slowdowns or stops in processing can be suppressed even when, for example, the imaging data is large. Furthermore, in the second mode, details of the event can be recognized using a deep learning model 50 that is larger than the reservoir model 40, enabling high-precision recognition. Therefore, the event recognition system 2 can reduce the amount of calculation while enabling highly accurate recognition.

[0065] The number of nodes in the input layer 41 of the reservoir model 40 is smaller than the number of nodes in the input layer 51 of the deep learning model 50. This allows the reservoir model 40 to be made smaller in size, and the amount of calculation can be reduced.

[0066] The amount of calculation required to calculate an output from an input in the reservoir model 40 is less than the amount of calculation required to calculate an output from an input in the deep learning model 50. This allows the amount of calculation to be reduced.

[0067] The reservoir model 40 has an input layer 41, a reservoir layer 42 connected to the input layer 41, and an output layer 43 connected to the reservoir layer 42, and only the connection weights between the reservoir layer 42 and the output layer 43 are trained. This allows the reservoir model 40 to be made smaller, and the amount of calculation required to be reduced.

[0068] The deep learning model 50 has an input layer 41, at least three or more intermediate layers 52 connected to the input layer 41, and an output layer 43 connected to the intermediate layers 52. This makes it possible to recognize details of an event using the large-scale deep learning model 50, enabling highly accurate recognition.

[0069] The imaging unit 11 has a first image sensor 21 and a second image sensor 22 (two-dimensional sensors) that acquire imaging data, and the filter unit 12 is configured by a circuit unit 18 formed on a substrate 17 that is separated from the first image sensor 21 and the second image sensor 22. This makes it possible to easily realize the filter unit 12 that extracts temporal and spatial features.

[0070] The reservoir model 40 is a trained model based on training data corresponding to an event, and in the first mode, the control unit 16 determines whether an event has occurred based on a comparison between the output from the reservoir model 40 and the training data. This makes it easy to determine whether an event has occurred.

[0071] In the first mode, at one processing timing (at each processing timing), one data set corresponding to the features extracted by the filter unit 12 is input to the reservoir model 40. This allows the reservoir model 40 to be smaller in size than when, for example, multiple data sets are input to the reservoir model 40 at one processing timing.

[0072] The imaging unit 11 has a first image sensor 21 (first two-dimensional sensor) and a second image sensor 22 (second two-dimensional sensor) that acquire imaging data. The filter unit 12 performs filtering on the imaging data acquired by the first image sensor 21 and extracts temporal and spatial features from the imaging data. In the first mode, the first processing unit 14 processes data corresponding to the features extracted by the filter unit 12 from the imaging data acquired by the first image sensor 21 using a reservoir model 40. In the second mode, the second processing unit 15 processes the imaging data acquired by the second image sensor 22 using a deep learning model 50. As a result, by separating the first image sensor 21 used in the first mode from the second image sensor 22 used in the second mode, it is possible to reduce circuit complexity.

[0073] When it is determined that an event has occurred in the first mode, the operation mode is changed from the first mode to the second mode (step S7). As a result, the first mode is turned off when the second mode is activated (the first mode process is not executed when the second mode process is executed), thereby reducing power consumption. [Modification]

[0074] As in a first modified example shown in Fig. 10 , the imaging unit 11 may have only one image sensor 21 (two-dimensional sensor). In this case, imaging data acquired by the image sensor 21 is input to the filter unit 12 and the second processing unit 15. That is, in the first mode, the first processing unit 14 processes data corresponding to features extracted by the filter unit 12 from the imaging data acquired by the image sensor 21 using a reservoir model 40. In the second mode, the second processing unit 15 processes the imaging data acquired by the image sensor 21 using a deep learning model 50. In this way, by using a common two-dimensional sensor for the first mode and the second mode, the imaging unit 11 can be made smaller.

[0075] As in a second modified example shown in FIG. 11 , the filter unit 12 may be incorporated into the first image sensor 21. That is, the first image sensor 21 and the filter unit 12 may be configured on a single substrate 19 (chip or device). In this case, the filter unit 12 may be formed as a circuit on the substrate constituting the image sensor, or may be realized as a physical configuration formed on the substrate constituting the image sensor. According to the second modified example, the configuration can be simplified compared to, for example, a case in which the filter unit 12 is configured by a circuit unit formed on a substrate separate from the image sensor. Furthermore, by incorporating the filter unit 12 into the first image sensor 21, the transfer speed of signals from the first image sensor 21 to the filter unit 12 can be increased, thereby enabling faster processing. Note that incorporating the filter unit 12 into the first image sensor 21 also includes a case in which a substrate on which the filter unit 12 is formed as a circuit is stacked and integrated on the substrate constituting the first image sensor 21.

[0076] 12 , the event recognition system 2 may further include an image extraction unit 60. The image extraction unit 60 is disposed between the second image sensor 22 and the second processing unit 15 (deep learning model 50), and cuts out (extracts, trims) a partial area from the imaging data acquired by the second image sensor 22. The partial area to be cut out is identified by the control unit 16 based on, for example, an area in the imaging data that is recognized as having a relatively large number of events occurring therein or an area that is determined to be effective for image recognition.

[0077] 13 , in the second mode of the third modified example, step S14 in which the image extraction unit 60 cuts out a portion of the imaging data is executed between step S11 in which imaging data is acquired by the second image sensor 22 and step S12 in which the second processing unit 15 processes the imaging data using the deep learning model 50. As a result, in the second mode of the third modified example, the image area is limited compared to the first mode, and the second processing unit 15 processes imaging data having a smaller data size than the imaging data to which filtering is applied in the first mode, using the deep learning model 50. As a result, the data size of the imaging data input to the deep learning model 50 can be reduced, thereby enabling faster processing and improved recognition accuracy.

[0078] The image extraction unit 60 may be formed integrally with the second image sensor 22. In this case, the imaging area of ​​the second image sensor 22 may be limited based on a signal from the control unit 16. Alternatively, the imaging area of ​​the second image sensor 22 may be limited by, for example, moving the light receiving lens 24. The image extraction unit 60 may be omitted. The image area of ​​the imaging data in the first mode and the image area of ​​the imaging data processed by the second neural network in the second mode may be the same.

[0079] The present disclosure is not limited to the above-described embodiment and modifications. The first neural network is not limited to the reservoir model 40 and may be any neural network. The second neural network is not limited to the deep learning model 50 and may be any neural network.

[0080] In the above embodiment, when it is determined that an event has occurred in the first mode, the operation mode is changed from the first mode to the second mode. However, the second mode may be activated while the first mode remains activated. That is, the first mode may be activated in parallel with the second mode. In this case, the occurrence of an event can be detected in the first mode even when the second mode is activated.

[0081] The number of nodes in the input layer 41 of the reservoir model 40 (first neural network) may be equal to or greater than the number of nodes in the input layer 51 of the deep learning model 50 (second neural network). The amount of calculation required to calculate an output from an input in the reservoir model 40 may be equal to or greater than the amount of calculation required to calculate an output from an input in the deep learning model 50.

[0082] In the above embodiment, the filter unit 12, pre-processing unit 13, and first processing unit 14 are configured by a single circuit, and the second processing unit 15 and control unit 16 are each configured by a separate circuit or processing device, but at least one of the filter unit 12, pre-processing unit 13, first processing unit 14, second processing unit 15, and control unit 16 may be configured by a common circuit or processing device, for example, the first processing unit 14 and control unit 16 may be configured by a single circuit or processing device, or the first processing unit 14, second processing unit 15, and control unit 16 may be configured by a single circuit or processing device.

[0083] In the above embodiment, in the second mode, imaging data, which is time-series data, is acquired by the second image sensor 22, and the imaging data is processed by the deep learning model 50 to recognize details of the event, but in the second mode, one frame of imaging data acquired by the second image sensor 22 may be processed by the deep learning model 50 to recognize details of the event. In this way, the imaging data input to the deep learning model 50 in the second mode does not have to be time-series data, and the second image sensor 22 does not necessarily have to acquire time-series data.

[0084] The two-dimensional sensor included in the imaging unit 11 is not limited to an image sensor and may be any sensor capable of acquiring imaging data. For example, the two-dimensional sensor may be an event camera (event sensor, event-driven camera). Similar to an image sensor, an event camera has multiple pixels arranged two-dimensionally. However, in an event camera, the output of each pixel is compared by a comparator, and information (address, on / off signal) of pixels whose output exceeds a threshold is output as data. The data output from the event camera can also be considered imaging data acquired by imaging. However, in the above embodiment, temporal and spatial features are extracted using the first image sensor 21 and the filter unit 12. This reduces power consumption and the amount of data output to the reservoir model 40 compared to, for example, an event camera. The application of the event recognition system 2 is not limited to the robot system 1 and may be applied to other objects.

[0085] Although not included in the invention according to claim 1 of the scope of claims at the time of filing, the event recognition system may be configured as in the following examples 1 and 2. <Example 1> In the event recognition system 2, the two-dimensional sensor provided in the imaging unit 11 is configured by an event camera, and the filter unit 12 is not provided. That is, the event recognition system may include "an event camera having a plurality of pixels and acquiring image data, the event camera extracting temporal features by comparing outputs from the plurality of pixels using a comparator; a two-dimensional sensor; a first processing unit performing processing using a first neural network; a second processing unit performing processing using a second neural network different from the first neural network; and a control unit, the event recognition system including a first mode and a second mode as operation modes, wherein in the first mode, the first processing unit processes data corresponding to the features extracted by the event camera using the first neural network, and the control unit determines whether a predetermined event has occurred based on an output from the first neural network, and activates the second mode when it determines that the event has occurred; and in the second mode, the second processing unit processes image data acquired by the event camera or the two-dimensional sensor using the second neural network, and the control unit recognizes details of the event based on the output from the second neural network." The two-dimensional sensor may be the event camera (or may be shared with the event camera), or may be a sensor separate from the event camera (e.g., an RGB camera). The event camera may compare the outputs of multiple pixels using a comparator and extract only the pixel outputs whose difference in change is greater than a reference value (e.g., voltage value) as a positive event (bright area) or a negative event (dark area). The reference value may be reset each time an event occurs. In this case, temporal features can be extracted using the event camera, and the amount of data can be reduced compared to, for example, using a normal camera.In an event camera, by using a comparator, it is possible to extract only fast-moving objects, similar to when a high-pass filter (temporal filter) is used, and temporal features can be extracted. However, because an event camera does not have the function of extracting spatial features, the configuration of the above embodiment, which includes a filter unit that extracts temporal and spatial features after the imaging unit, has a greater effect on data reduction. <Example 2> In an event recognition system 2, the two-dimensional sensor included in the imaging unit 11 is configured by an event camera, and data output from the event camera is filtered by a filter unit 12 that does not have a temporal filter but only a spatial filter. That is, the event recognition system may be an event recognition system including: an event camera having a plurality of pixels and acquiring imaging data, the event camera extracting temporal features by comparing outputs of the plurality of pixels using a comparator; a filter unit performing filtering processing on the imaging data acquired by the event camera using a spatial filter to extract spatial features from the imaging data; a two-dimensional sensor; a first processing unit performing processing using a first neural network; a second processing unit performing processing using a second neural network different from the first neural network; and a control unit; the event recognition system including a first mode and a second mode as operation modes, wherein in the first mode, the first processing unit processes data corresponding to the features extracted by the event camera and the filter unit using the first neural network, the control unit determines whether or not a predetermined event has occurred based on an output from the first neural network, and activates the second mode when it determines that the event has occurred; and in the second mode, the second processing unit processes imaging data acquired by the event camera or the two-dimensional sensor using the second neural network, and the control unit recognizes details of the event based on the output from the second neural network. The two-dimensional sensor may be the event camera (or may be shared with the event camera), or may be a sensor (for example, an RGB camera) separate from the event camera.The event camera may compare the outputs of multiple pixels using a comparator and extract only pixel outputs with a difference in change greater than a reference value (e.g., voltage value) as positive events (bright areas) or negative events (dark areas). The reference value may be reset each time an event occurs. In this case, the amount of data input to the first processing unit can be reduced by extracting temporal features using the event camera and spatial features using a spatial filter. However, although the event camera can extract temporal features using a comparator, it outputs pixel information that has changed regardless of the bandwidth, resulting in a large amount of data. The configuration of the above embodiment, which includes a filter unit that extracts temporal and spatial features downstream of the event camera, can more reliably reduce the amount of data.

[0086] 1...Robot system, 2...Event recognition system, 3...Robot, 11...Imaging unit, 12...Filter unit, 14...First processing unit, 15...Second processing unit, 16...Control unit, 17...Substrate, 18...Circuit unit, 21...First image sensor (first two-dimensional sensor), 22...Second image sensor (second two-dimensional sensor), 40...Reservoir model (first neural network), 41...Input layer, 42...Reservoir layer, 43...Output layer, 50...Deep learning model (second neural network), 51...Input layer, 52...Intermediate layer, 53...Output layer.

Claims

1. An event recognition system comprising: an imaging unit that acquires imaging data; a filter unit that performs filtering using a time filter and a spatial filter on the imaging data, which is time-series data acquired by the imaging unit, and extracts temporal and spatial features from the imaging data; a first processing unit that performs processing using a first neural network; a second processing unit that performs processing using a second neural network different from the first neural network; and a control unit, the event recognition system having first and second operating modes, wherein in the first mode, the first processing unit processes data corresponding to the features extracted by the filter unit using the first neural network, and the control unit determines whether a predetermined event has occurred based on an output from the first neural network, and activates the second mode if it determines that the event has occurred; and in the second mode, the second processing unit processes the imaging data acquired by the imaging unit using the second neural network, and the control unit recognizes details of the event based on the output from the second neural network.

2. The event recognition system according to claim 1, wherein the number of nodes in the input layer of the first neural network is smaller than the number of nodes in the input layer of the second neural network.

3. The event recognition system according to claim 1 or 2, wherein the amount of calculation required to calculate an output from an input in the first neural network is less than the amount of calculation required to calculate an output from an input in the second neural network.

4. An event recognition system as described in any one of claims 1 to 3, wherein the first neural network is a reservoir computing model having an input layer, a reservoir layer connected to the input layer, and an output layer connected to the reservoir layer, and in which only the connection weights between the reservoir layer and the output layer are trained.

5. An event recognition system according to any one of claims 1 to 4, wherein the second neural network is a deep learning model having an input layer, at least three or more hidden layers connected to the input layer, and an output layer connected to the hidden layers.

6. An event recognition system according to any one of claims 1 to 5, wherein the imaging unit has a two-dimensional sensor that acquires imaging data, and the filter unit is configured by a circuit unit formed on a substrate separate from the two-dimensional sensor.

7. An event recognition system according to any one of claims 1 to 5, wherein the imaging unit has a two-dimensional sensor that acquires imaging data, and the filter unit is incorporated within the two-dimensional sensor.

8. An event recognition system according to any one of claims 1 to 7, wherein the first neural network is a trained model based on training data corresponding to the event, and in the first mode, the control unit determines whether the event has occurred based on a comparison of the output from the first neural network with the training data.

9. An event recognition system according to any one of claims 1 to 8, wherein in the first mode, one data set corresponding to the features extracted by the filter unit is input to the first neural network at one processing timing.

10. An event recognition system according to any one of claims 1 to 9, wherein the imaging unit has a first two-dimensional sensor and a second two-dimensional sensor that acquire imaging data, the filter unit extracts the temporal and spatial features from the imaging data acquired by the first two-dimensional sensor, in the first mode the first processing unit processes data corresponding to the features extracted by the filter unit from the imaging data acquired by the first two-dimensional sensor using the first neural network, and in the second mode the second processing unit processes the imaging data acquired by the second two-dimensional sensor using the second neural network.

11. An event recognition system according to any one of claims 1 to 9, wherein the imaging unit has a two-dimensional sensor that acquires imaging data, and in the first mode, the first processing unit processes data corresponding to the features extracted by the filter unit from the imaging data acquired by the two-dimensional sensor using the first neural network, and in the second mode, the second processing unit processes the imaging data acquired by the two-dimensional sensor using the second neural network.

12. An event recognition system according to any one of claims 1 to 11, wherein the operation mode is changed from the first mode to the second mode when it is determined that the event has occurred in the first mode.

13. An event recognition system according to any one of claims 1 to 11, wherein when it is determined that the event has occurred in the first mode, the second mode is activated while the first mode remains activated.

14. An event recognition system as described in any one of claims 1 to 13, wherein in the first mode, the filter unit applies the filtering process to the imaging data acquired by the imaging unit, and in the second mode, the second processing unit uses the second neural network to process the imaging data having a smaller data size than the imaging data to which the filtering process is applied in the first mode.

15. A robot system comprising: an event recognition system according to any one of claims 1 to 14; and a robot, wherein the robot executes a predetermined action or process based on details of the event recognized in the second mode.

16. An event recognition method comprising: a first step of filtering imaging data, which is time-series data acquired by an imaging unit, using a time filter and a spatial filter to extract temporal and spatial features from the imaging data, processing data corresponding to the extracted features using a first neural network, determining whether a predetermined event has occurred based on the output from the first neural network, and executing a second step if it is determined that the event has occurred; and a second step of processing the imaging data acquired by the imaging unit using a second neural network different from the first neural network, and recognizing details of the event based on the output from the second neural network.

Images