Event recognition system, robotic system, event recognition method

Abstract

This invention provides an event recognition system, a robot system, and an event recognition method that enable high-precision recognition while reducing computational load. [Solution] The event recognition system 2 comprises an imaging unit 11, a filter unit 12 that extracts temporal and spatial features from imaging data, a first processing unit 14, a second processing unit 15, and a control unit 16. In the first mode, the first processing unit 14 processes the data corresponding to the features extracted by the filter unit 12 using a first neural network, and the control unit 16 determines whether a predetermined event has occurred based on the output from the first neural network. If it determines that an event has occurred, it activates the second mode. In the second mode, the second processing unit 15 processes the imaging data using a second neural network, and the control unit 16 recognizes the details of the event based on the output from the second neural network.

Description

Technical Field

[0001] The present invention relates to an event recognition system, a robot system, and an event recognition method.

Background Art

[0002] For example, Patent Document 1 describes a system that uses a neural network to recognize the pose of an object in an image. In this system, deep learning is used for pose recognition.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] When applying a system as described above to, for example, the recognition of operations or gestures (events), it is considered that high-precision recognition becomes possible. However, when operating the system in an actual field, since deep learning is a large-scale neural network and the amount of calculation is enormous, there is a risk that the power consumption increases or the processing stops due to insufficient computing resources.

[0005] An object of the present invention is to provide an event recognition system, a robot system, and an event recognition method that can reduce the amount of calculation while enabling high-precision recognition.

Means for Solving the Problems

[0006] The event recognition system of the present invention is [1] an event recognition system comprising: an imaging unit that acquires imaging data; a filter unit that applies filtering processing by a time filter and a spatial filter to the imaging data, which is time-series data acquired by the imaging unit, to extract temporal and spatial features from the imaging data; a first processing unit that performs processing using a first neural network; a second processing unit that performs processing using a second neural network different from the first neural network; and a control unit, wherein the operating modes include a first mode and a second mode, in which the first processing unit processes data corresponding to the features extracted by the filter unit using the first neural network, the control unit determines whether a predetermined event has occurred based on the output from the first neural network, and if it determines that the event has occurred, activates the second mode, in which the second processing unit processes the imaging data acquired by the imaging unit using the second neural network, and the control unit recognizes the details of the event based on the output from the second neural network.

[0007] In this event recognition system, in the first mode, data corresponding to temporal and spatial features extracted from imaging data is processed using a first neural network, and it is determined whether a predetermined event has occurred based on the output of the first neural network. If it is determined that an event has occurred, the second mode is activated, and in the second mode, the imaging data is processed using a second neural network, and the details of the event are recognized based on the output of the second neural network. As a result, in the first mode, smaller data (data corresponding to features extracted from imaging data) is processed using the first neural network compared to the second mode, allowing the first neural network to be made smaller than the second neural network, thus reducing the computational load. Also, because the data input to the first neural network is small, even if the imaging data is large, processing slowdowns or crashes can be suppressed. Furthermore, in the second mode, the details of the event can be recognized using a second neural network that is larger than the first neural network, enabling high-precision recognition. Therefore, this event recognition system enables high-precision recognition while reducing the computational load.

[0008] The event recognition system of the present invention may also be [2] "the event recognition system according to [1] in which the number of nodes in the input layer of the first neural network is less than the number of nodes in the input layer of the second neural network." In this case, the first neural network can be made smaller and the amount of computation can be reduced.

[0009] The event recognition system of the present invention may also be [3] "the event recognition system according to [1] or [2] in which the computational amount required to calculate the output from the input in the first neural network is less than the computational amount required to calculate the output from the input in the second neural network." In this case, the computational amount can be reduced.

[0010] The event recognition system of the present invention may also be [4] "the event recognition system according to any one of [1] to [3], wherein the first neural network comprises an input layer, a reservoir layer coupled to the input layer, and an output layer coupled to the reservoir layer, and only the coupling weights between the reservoir layer and the output layer are learned, which is a reservoir computing model." In this case, the first neural network can be made smaller and the computational load can be reduced.

[0011] The event recognition system of the present invention may also be [5] "the event recognition system according to any one of [1] to [4], wherein the second neural network is a deep learning model having an input layer, at least three or more intermediate layers coupled to the input layer, and an output layer coupled to the intermediate layers." In this case, a large-scale second neural network can be used to recognize the details of an event, enabling highly accurate recognition.

[0012] The event recognition system of the present invention may also be [6] "the event recognition system according to any one of [1] to [5], wherein the imaging unit has a two-dimensional sensor that acquires imaging data, and the filter unit is composed of a circuit unit formed on a substrate separated from the two-dimensional sensor." In this case, a filter unit that extracts temporal and spatial features can be easily realized.

[0013] The event recognition system of the present invention may also be [7] "the event recognition system according to any one of [1] to [5], wherein the imaging unit has a two-dimensional sensor that acquires imaging data, and the filter unit is incorporated into the two-dimensional sensor." In this case, the configuration can be simplified compared to, for example, the case in which the filter unit is made up of a circuit unit formed on a substrate separated from the sensor.

[0014] The event recognition system of the present invention may also be [8] "the event recognition system according to any one of [1] to [7], wherein the first neural network is a trained model based on training data corresponding to the event, and in the first mode, the control unit determines whether or not the event has occurred based on a comparison of the output from the first neural network with the training data." In this case, it is possible to easily determine whether or not an event has occurred.

[0015] The event recognition system of the present invention may also be [9] "the event recognition system according to any one of [1] to [8], wherein in the first mode, at one processing timing, one dataset corresponding to the features extracted by the filter unit is input to the first neural network." In this case, for example, the first neural network can be made smaller compared to the case where multiple datasets are input to the first neural network at one processing timing.

[0016] The event recognition system of the present invention may also be

[10] "the event recognition system according to any one of [1] to [9], wherein the imaging unit has a first two-dimensional sensor and a second two-dimensional sensor for acquiring imaging data, the filter unit extracts the temporal and spatial features from the imaging data acquired by the first two-dimensional sensor, in the first mode, the first processing unit processes the data corresponding to the features extracted by the filter unit from the imaging data acquired by the first two-dimensional sensor using the first neural network, and in the second mode, the second processing unit processes the imaging data acquired by the second two-dimensional sensor using the second neural network." In this case, the complexity of the circuit can be suppressed by separating the first two-dimensional sensor used in the first mode and the second two-dimensional sensor used in the second mode.

[0017] The event recognition system of the present invention may also be

[11] "the event recognition system according to any one of [1] to [9], wherein the imaging unit has a two-dimensional sensor for acquiring imaging data, in the first mode, the first processing unit processes data corresponding to the features extracted from the imaging data acquired by the two-dimensional sensor by the filter unit using the first neural network, and in the second mode, the second processing unit processes the imaging data acquired by the two-dimensional sensor using the second neural network." In this case, the imaging unit can be miniaturized by using the same two-dimensional sensor for the first mode and the same two-dimensional sensor for the second mode.

[0018] The event recognition system of the present invention may also be

[12] "an event recognition system according to any one of [1] to

[10] wherein, when it is determined that the event has occurred in the first mode, the operating mode is changed from the first mode to the second mode." In this case, since the first mode is turned off when the second mode is started, power consumption can be reduced.

[0019] The event recognition system of the present invention may also be

[13] "an event recognition system according to any one of [1] to

[10] , wherein if it is determined that the event has occurred in the first mode, the second mode is activated while the first mode remains activated." In this case, even when the second mode is activated, the occurrence of an event can be detected by the first mode.

[0020] The event recognition system of the present invention may also be

[14] "an event recognition system according to any one of [1] to

[13] , wherein in the first mode, the filter unit performs the filtering process on the imaging data acquired by the imaging unit, and in the second mode, the second processing unit processes the imaging data having a smaller data size than the imaging data that is filtered in the first mode using the second neural network." In this case, by reducing the size of the imaging data processed using the second neural network in the second mode, the processing speed and accuracy in the second mode can be improved.

[0021] The robot system of the present invention is

[15] "a robot system comprising an event recognition system described in any of [1] to

[14] and a robot, wherein the robot performs a predetermined operation or processing based on the details of the event recognized in the second mode." According to this robot system, for the reasons described above, it is possible to reduce the amount of computation while enabling high-precision recognition.

[0022] The event recognition method of the present invention is

[16] "an event recognition method comprising: a first step of applying filtering processing by a time filter and a spatial filter to imaging data which is time-series data acquired by an imaging unit, extracting temporal and spatial features from the imaging data, processing the data corresponding to the extracted features using a first neural network, determining whether a predetermined event has occurred based on the output from the first neural network, and executing a second step if it is determined that the event has occurred; and a second step of processing the imaging data acquired by the imaging unit using a second neural network different from the first neural network, and recognizing the details of the event based on the output from the second neural network."

[0023] In this event recognition method, in the first step, data corresponding to features extracted from imaging data is processed using a first neural network, and it is determined whether a predetermined event has occurred based on the output from the first neural network. Then, if it is determined that an event has occurred as a result of the determination, the second step is executed. In the second step, the imaging data is processed using a second neural network, and the details of the event are recognized based on the output from the second neural network. Thus, in the first step, smaller data (data corresponding to features extracted from imaging data) compared to the second step is processed using the first neural network, so the first neural network can be made smaller in scale compared to the second neural network, and the amount of computation can be reduced. Also, since the data input to the first neural network is small, it is possible to suppress the processing from slowing down or stopping even when, for example, the imaging data is large. Further, in the second step, the details of the event can be recognized using a second neural network that is larger in scale than the first neural network, enabling high-precision recognition. Therefore, according to this event recognition method, it is possible to reduce the amount of computation while enabling high-precision recognition.

Advantages of the Invention

[0024] According to the present invention, it becomes possible to provide an event recognition system, a robot system, and an event recognition method that can reduce the amount of computation while enabling high-precision recognition.

Brief Description of the Drawings

[0025] [Figure 1] It is a configuration diagram of a robot system including an event recognition system. [Figure 2] It is a configuration diagram of a filter unit. [Figure 3] It is a configuration diagram of a reservoir model. [Figure 4] It is a configuration diagram of a deep learning model. [Figure 5]This is a flowchart to explain the processing in the first mode. [Figure 6] This is a diagram illustrating the processing in the first mode. [Figure 7] (a) is a diagram showing examples of output values ​​and target values ​​when no event occurs, and (b) is a diagram showing examples of output values ​​and target values ​​when an event occurs. [Figure 8] (a) is a diagram showing an example of the squared difference between the output value and the target value when no event occurs, and (b) is a diagram showing an example of the squared difference between the output value and the target value when an event occurs. [Figure 9] This is a flowchart to explain the processing in the second mode. [Figure 10] This is a diagram showing the configuration of the robot system in the first modified example. [Figure 11] This is a diagram illustrating the configuration of the robot system in the second modified example. [Figure 12] This is a diagram illustrating the configuration of the robot system in the third modified example. [Figure 13] This is a flowchart illustrating the processing in the second mode of the third modified example. [Modes for carrying out the invention]

[0026] Embodiments of the present invention will be described in detail below with reference to the drawings. In the following description, the same or equivalent elements will be denoted by the same reference numerals, and redundant explanations will be omitted.

[0027] As shown in Figure 1, the robot system 1 comprises an event recognition system 2 and a robot 3 that is communicatively connected to the event recognition system 2. In general terms, the event recognition system 2 determines whether a predetermined event (occurrence) has occurred based on imaging data acquired by the imaging unit 11, which will be described later. Furthermore, if the event recognition system 2 determines that an event has occurred, it recognizes the details of the event based on the imaging data acquired by the imaging unit 11. The robot 3 then performs a predetermined operation or process based on the details of the event recognized by the event recognition system 2.

[0028] The event to be recognized can be any event, and one or more (at least one) events may be set as the event to be recognized. For example, an event may include a predetermined posture or action (e.g., a gesture) of an object OJ (e.g., a person). In other words, the event recognition system 2 may be an action recognition system. The object OJ may be a machine, in which case the event may include an abnormal posture or abnormal operation of the machine. The object OJ may be an object, in which case the event may include the object falling, or the detection of an obstacle.

[0029] "Recognizing the details of an event" means performing a more advanced (more informative) recognition of the event than simply determining that an event has occurred. For example, recognizing the details of an event may include recognizing the attributes of the object OJ (e.g., person, object, machine). Recognizing the details of an event may also include recognizing the posture or type of motion of the object OJ. As an example, if it is determined that "an object OJ has approached" has occurred as an event, the details of the event may include recognizing whether "the approaching object OJ is a person or an object (e.g., a ball)."

[0030] Robot 3 is a mechanical device configured to be movable and / or operable, for example, including an actuator device. Robot 3 may have any shape and may be, for example, a humanoid robot or a non-humanoid robot. Robot 3 may be a service robot that performs a predetermined service or an industrial robot used, for example, in a factory.

[0031] Robot 3 performs a predetermined action or process based on the details of the event recognized by the event recognition system 2. The action or process performed may be any action or process. For example, Robot 3 may perform an evasive action when an object OJ approaches. In this case, for example, if the event recognition system 2 determines that "approach of object OJ" has occurred as an event, it recognizes as an event detail whether "the approaching object OJ is a person or an object (e.g., a ball)". If Robot 3 recognizes as an event detail that "the approaching object OJ is a person", it performs an evasive action toward the person such that a relatively large distance is maintained between Robot 3 and the person during the evasive action. On the other hand, if Robot 3 recognizes as an event detail that "the approaching object OJ is an object (e.g., a ball)", it performs an evasive action toward the object such that a relatively small distance is maintained between Robot 3 and the object during the evasive action. As another example, Robot 3 may stop its operation if an abnormality occurs in the object OJ, which is a machine. For example, if it is determined that a "machine malfunction" has occurred as an event, and the details of the event indicate that "the malfunction should cause the robot to stop," then robot 3 may stop operating.

[0032] If the event recognition system 2 determines that an event has occurred in which "object OJ movement" has occurred, it may recognize as an event detail whether "the moving object OJ (person) is waving its hand to call out." If the robot 3 recognizes as an event detail whether "the object OJ (person) is waving its hand to call out," it may perform an action to approach the person waving. Alternatively, if the event recognition system 2 determines that an event has occurred in which "object OJ movement" has occurred, it may recognize as an event detail whether "the moving object OJ is a person or an object." If the robot 3 recognizes as an event detail whether "the object OJ is a person," it may perform actions such as measuring the person's size (height) or recording their face. This measurement or recording may be performed as an internal calculation or recording process within the robot 3. In this way, the robot 3 may perform physical actions or perform predetermined internal processing based on the recognized event details.

[0033] As shown in Figure 1, the event recognition system 2 comprises an imaging unit 11, a filter unit 12, a preprocessing unit 13, a first processing unit 14, a second processing unit 15, and a control unit 16. The imaging unit 11 includes a first image sensor 21 (first two-dimensional sensor, first imaging unit) and a second image sensor 22 (second two-dimensional sensor, second imaging unit).

[0034] Each of the first image sensor 21 and the second image sensor 22 is an image sensor (two-dimensional sensor) having multiple pixels arranged in two dimensions, for example, a camera. Each of the first image sensor 21 and the second image sensor 22 acquires image data consisting of the brightness value of each pixel at a predetermined period (frame rate). In this way, each of the first image sensor 21 and the second image sensor 22 acquires imaging data (data acquired by imaging, in this example, image data), which is time-series data. In this example, the first image sensor 21 images the area around the robot 3 via the light-receiving lens 23, and the second image sensor 22 images the area around the robot 3 via the light-receiving lens 24. When an object OJ is within the imaging range, the first image sensor 21 and the second image sensor 22 acquire imaging data in which the object OJ has been imaged.

[0035] The filter unit 12 is composed of a circuit unit 18 formed on a substrate 17 (e.g., a circuit board) that is separated from the first image sensor 21 and the second image sensor 22. That is, the substrate 17 is a different substrate from the one on which the first image sensor 21 and the second image sensor 22 are formed. The circuit unit 18 is, for example, an FPGA (Field Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit).

[0036] As shown in Figure 2, the filter unit 12 includes a first spatial filter 31, a first temporal filter 32, a second spatial filter 33, a second temporal filter 34, a first rectifier unit 35, an inverter unit 36, and a second rectifier unit 37. The filter unit 12 applies filtering processing to the imaging data D1, which is time-series data acquired by the first image sensor 21, using the first temporal filter 32, the second temporal filter 34, the first spatial filter 31, and the second spatial filter 33, to extract temporal and spatial features from the imaging data D1. In other words, the filter unit 12 is a spatiotemporal filter that extracts spatiotemporal features from imaging data D1.

[0037] The filter unit 12 is a neuromorphic filter that mimics the structure and function of the human brain or retina, and extracts temporal and spatial features from the imaging data D1. The filter unit 12 includes a first temporal filter 32 and a second temporal filter 34, thereby extracting specific features related to temporal changes. For example, in this example, the temporal filter is composed of a bandpass filter (e.g., a low-pass filter, a high-pass filter, or a bandpass filter), and only changes in a specific frequency band are extracted. The temporal filter is composed of, for example, at least one of a low-pass filter, a high-pass filter, and a bandpass filter. As described below, in this example, the temporal filter is composed of a low-pass filter and a bandpass filter.

[0038] The temporal change relates, for example, to the speed of movement of an object OJ (moving object) captured by the first image sensor 21. This speed can be represented, for example, by the number of frames that have elapsed between the time an object OJ is captured by one pixel of the first image sensor 21 and the time it is captured by another pixel of the first image sensor 21. For example, since a low-pass filter is a filter that allows only information below a certain speed to pass through, fast-moving objects will not be detected by providing a low-pass filter. Similarly, since a high-pass filter is a filter that allows only information above a certain speed to pass through, slow-moving (or stationary) objects will not be detected by providing a high-pass filter. As a result, for example, by providing a low-pass filter and a high-pass filter, it is possible to detect only moving objects moving within a specific speed range. That is, in this case, the time filter constitutes a band-pass filter and does not respond to DC (constant) components that do not change over time or to fast-moving objects. The time filter may also be configured to include a band-pass filter that allows only information within a specific speed range to pass through. In this case as well, moving objects moving within a specific speed range can be detected.

[0039] Furthermore, the filter unit 12 includes a first spatial filter 31 and a second spatial filter 33, thereby extracting features related to spatial changes. For example, as described below, in this example the spatial filter is composed of a smoothing filter (e.g., a Gaussian filter), and the contour (edge) portion of the imaged object OJ is emphasized and extracted.

[0040] The first spatial filter 31 is a spatial filter that mimics the horizontal cells of the human retina and is composed of, for example, a Gaussian filter with a relatively small half-width. The first spatial filter 31 receives imaging data D1 (image data in this example), which is time-series data acquired by the first image sensor 21. Passing through the first spatial filter 31 blurs the image and smooths the change in brightness values ​​between adjacent pixels. The first temporal filter 32 is a temporal filter that mimics the horizontal cells of the human retina and is composed of, for example, a low-pass filter. When the image blurred by the first spatial filter 31 is input to the first temporal filter 32 (low-pass filter), the rapidly changing components of moving objects are removed, further reducing the amount of information.

[0041] The second spatial filter 33 is a spatial filter that mimics the photoreceptor cells of the human retina, and is composed of, for example, a Gaussian filter with a smaller half-width than the first spatial filter 31. The imaging data D1 is input to the second spatial filter 33. By passing through the second spatial filter 33, the image is blurred and the change in brightness values ​​between adjacent pixels becomes smoother. The degree of blurring in the image after passing through the second spatial filter 33 is less than the degree of blurring in the image after passing through the first spatial filter 31.

[0042] The difference between the image that has passed through the first time filter 32 and the image that has passed through the second spatial filter 33 is input to the second time filter 34. By taking the difference between the images that have passed through the first spatial filter 31 and the second spatial filter 33, which have different half-widths (two images with different degrees of blurring), the contours of objects in the image are enhanced. The second time filter 34 is a time filter that mimics the bipolar cells of the human retina and is composed of, for example, a bandpass filter. By inputting the image to the second time filter 34, only the velocity components of a specific range of the moving object are extracted, further reducing the amount of information.

[0043] The image that has passed through the second time filter 34 is branched and input to the first rectifier 35 and the inverter 36, respectively. The first rectifier 35 is a rectifier circuit that allows only pixels with positive brightness in the image to pass through and outputs the result. The output data from the first rectifier 35 is output to the subsequent preprocessing unit 13 as the output from the filter unit 12. The inverter 36 inverts the sign of the brightness of each pixel in the input image. The second rectifier 37 is a rectifier circuit that allows only pixels with positive brightness in the image to pass through and outputs the result. That is, by passing through the inverter 36 and the second rectifier 37, only pixels with negative brightness in the image input to the inverter 36 are allowed to pass through and output the result. The output data from the second rectifier 37 is output to the subsequent preprocessing unit 13 as the output from the filter unit 12. The amount of change in sign is amplified by the first rectifier 35 and the second rectifier 37. In the filter unit 12, temporal and spatial features are extracted from the imaging data D1 acquired by the first image sensor 21 through the filtering process described above.

[0044] The preprocessing unit 13 is composed of a circuit unit 18 on which the filter unit 12 is mounted. In other words, the filter unit 12 and the preprocessing unit 13 are mounted on the same circuit unit 18. The preprocessing unit 13 performs preprocessing on the data output from the filter unit 12 and outputs the preprocessed data to the first processing unit 14.

[0045] The first processing unit 14 is composed of a circuit unit 18 on which the filter unit 12 and the preprocessing unit 13 are implemented. That is, the filter unit 12, the preprocessing unit 13, and the first processing unit 14 are implemented in the same circuit unit 18. The first processing unit 14 stores a reservoir model 40 (Figure 3) as the first neural network, and the first processing unit 14 processes the data output from the preprocessing unit 13 using the reservoir model 40 (reservoir computing model). The data output from the first processing unit 14 is input to the control unit 16.

[0046] As shown in Figure 3, the reservoir model 40 includes an input layer 41, a reservoir layer 42 coupled to the input layer 41, and an output layer 43 coupled to the reservoir layer 42. The input layer 41 has a plurality of nodes 41a. The reservoir layer 42 has a plurality of nodes 42a. Each node 41a of the input layer 41 is coupled to at least one of the plurality of nodes 42a of the reservoir layer 42. In the reservoir layer 42, one node 42a is coupled to one or more other nodes 42a. Among the plurality of nodes 42a, there may be one or more recurrent nodes 42a1 configured to input at least a portion of its own output to itself. In this example, the reservoir layer 42 is configured as an echo-state network such that a state (echo state) is formed in which past information in the input data, which is time-series data, echoes and remains. The output layer 43 has a plurality of nodes 43a. Each node 43a of the output layer 43 is connected to at least one of several nodes 42a of the reservoir layer 42.

[0047] In the reservoir model 40, each connection between nodes 41a, 42a, and 43a has a weight. When data is input to the input layer 41, sum-of-products operations are performed at each node 42a and 43a, and finally the values ​​of each node 43a in the output layer 43 are output as output data from the reservoir model 40. At each node 42a of the reservoir layer 42, a nonlinear transformation may be applied to the weighted sum of the input values. An example of this nonlinear transformation is an operation using a sigmoid function (activation function). Instead of encoding information using such real-valued activation values, information may be encoded using spikes. That is, instead of configuring each node 42a as an artificial neuron model, each node 42a may be configured as a spiking neuron model. The reservoir layer 42 is not limited to an echo-state network and may have any configuration. For example, the reservoir layer 42 may be a physical reservoir that utilizes physical phenomena such as light or fluid.

[0048] In the reservoir model 40, only the connection weights between the reservoir layer 42 and the output layer 43 are learned based on the target values ​​(training data), while the connection weights between the input layer 41 and the reservoir layer 42, and the connection weights between nodes 42a in the reservoir layer 42, are not learned. Therefore, compared to learning all the connection weights in a neural network, for example, faster learning is possible.

[0049] The second processing unit 15 is composed of a circuit separate from the circuit unit 18. The second processing unit 15 is composed of, for example, a GPU (Graphics Processing Unit) or a CPU (Central Processing Unit). The circuit that constitutes the second processing unit 15 is larger in scale than the circuit that constitutes the first processing unit 14 (for example, it has a larger amount of memory or a larger number of cores for calculations). In this example, the circuit that constitutes the second processing unit 15 is larger in scale than the filter unit 12, the preprocessing unit 13, and the circuit unit 18 that constitutes the first processing unit 14. The second processing unit 15 stores a deep learning model 50 (Figure 4) as a second neural network different from the first neural network, and the second processing unit 15 processes the data output from the second image sensor 22 using the deep learning model 50. The data output from the second processing unit 15 is input to the control unit 16.

[0050] As shown in Figure 4, the deep learning model 50 has an input layer 51, a plurality of hidden layers 52 coupled to the input layer 51, and an output layer 53 coupled to the hidden layers 52. The input layer 51 has a plurality of nodes 51a. There are at least three hidden layers 52. Each hidden layer 52 has a plurality of nodes 52a. Each node 52a of the input layer 51 is coupled to each node 52a of the upstream hidden layer 52A. Each node 52a of one hidden layer 52 is coupled to each node 52a of the next hidden layer 52. The output layer 53 has a plurality of nodes 53a. Each node 53a of the output layer 53 is coupled to each node 52a of the downstream hidden layer 52B.

[0051] In the deep learning model 50, each connection between nodes 51a, 52a, and 53a has a weight. When data is input to the input layer 51, sum-of-products operations are performed at each node 52a and 53a, and finally the values ​​of each node 53a in the output layer 53 are output as output data from the deep learning model 50. In each node 52a of the hidden layer 52, a nonlinear transformation may be applied to the weighted sum of the input values. An example of this nonlinear transformation is an operation using a sigmoid function (activation function). In the deep learning model 50, the connection weights between the input layer 41 and the hidden layer 52, the connection weights between the hidden layers 52 themselves, and the connection weights between the hidden layer 52 and the output layer 53 are learned based on the target values ​​(training data).

[0052] The first neural network, the reservoir model 40, is smaller in scale than the second neural network, the deep learning model 50. In this example, the number of nodes 41a in the input layer 41 of the reservoir model 40 is less than the number of nodes 51a in the input layer 51 of the deep learning model 50. In other words, the amount of data (number of data dimensions) input to the reservoir model 40 is less than the amount of data (number of data dimensions) input to the deep learning model 50. The computational effort required to calculate the output from the input in the reservoir model 40 is less than the computational effort required to calculate the output from the input in the deep learning model 50. Therefore, the power consumption per unit time in the reservoir model 40 is less than the power consumption per unit time in the deep learning model 50.

[0053] The control unit 16 is composed of a computer, for example, a processor such as a CPU and a storage medium such as RAM or ROM. The control unit 16 processes the data output from the first processing unit 14 and the second processing unit 15. The control unit 16 is connected to the robot 3 in a communicative manner and controls the operation of the robot 3. The control unit 16 may also be the control unit of the robot 3, or it may be considered as a part of the robot 3.

[0054] The event recognition system 2 includes a first mode and a second mode as operating modes. The first mode is a mode for determining whether or not a target event has occurred, and the second mode is a mode for recognizing the details of the event that was determined to have occurred in the first mode. The first mode and the second mode are the first and second steps of the event recognition method executed by the event recognition system 2, respectively.

[0055] Figure 5 is a flowchart illustrating the processing in the first mode. In the first mode, the control unit 16 activates the first image sensor 21, and the first image sensor 21 acquires imaging data, which is time-series data (step S1). The first image sensor 21 may remain activated in the first mode. Next, the filter unit 12 applies a filter to the imaging data acquired by the first image sensor 21 to extract temporal and spatial features from the imaging data (step S2). Subsequently, the preprocessing unit 13 preprocesses the data output from the filter unit 12 and outputs the preprocessed data to the first processing unit 14 (step S3). Details of the processing in step S3 will be described later, along with details of the processing in steps S4, S5, and S6. Next, the first processing unit 14 processes the data output from the preprocessing unit 13 using the reservoir model 40 (step S4). Next, the control unit 16 compares the output from the reservoir model 40 with the training data (step S5) and determines whether or not an event has occurred based on the comparison result (step S6). If the determination in step S6 determines that an event has occurred (YES in step S6), the operating mode is changed to the second mode (the second mode is activated) (step S7). If the determination in step S6 determines that no event has occurred (NO in step S6), the process returns to step S1.

[0056] Figure 6 is a diagram illustrating the processing in the first mode. As shown in Figure 6, the preprocessing unit 13 preprocesses the data D2 output from the filter unit 12, and the preprocessed data is input to the reservoir model 40 (step S3). Data D2 is time-series data representing temporal and spatial features extracted by the filter unit 12. In the preprocessing, for example, the data D2 output from the filter unit 12 is compressed into data with a number of dimensions corresponding to the number of nodes in the input layer 41 of the reservoir model 40. In this example, the number of nodes in the input layer 41 is 12, and the data D2 output from the filter unit 12 is compressed into 12-dimensional vector data and input to the reservoir model 40. The data input to the reservoir model 40 is time-series data, and in this example, it includes data from a total of 29 frames, from frame 0 to frame 28. The data in each frame includes 12-dimensional vector data. The time interval between frames corresponds to the data acquisition period (frame rate) of the first image sensor 21 and the second image sensor 22. The data input to the reservoir model 40 is data corresponding to the temporal and spatial features extracted by the filter unit 12 (in this example, compressed data D2 representing the temporal and spatial features). In this example, at one processing timing (1 frame), only one dataset corresponding to the features extracted by the filter unit 12 is input to the reservoir model 40.

[0057] In this example, the reservoir model 40 predicts and outputs the data for the next frame based on the input data (step S4). That is, the reservoir model 40 is pre-trained using training data (training data, normal data) to predict the data set for the next frame of a given input dataset. For example, in this example, in frame 0, it predicts the time series data for frames 1 to 29 from the time series data for frames 0 to 28. The training data is data corresponding to the event to be recognized, for example, imaging data actually acquired by the first image sensor 21 when no event has occurred. When normal data is input to the reservoir model 40, the reservoir model 40 can correctly predict (reproduce) the data for the next frame. On the other hand, when abnormal data is input to the reservoir model 40, the reservoir model 40 cannot correctly predict (reproduce) the data for the next frame. Therefore, by comparing the output from the reservoir model 40 with the training data, it is possible to determine whether or not an event has occurred (steps S5, S6). The reservoir model 40 is trained in advance before the event recognition system 2 starts operating, and no training of the reservoir model 40 takes place while the event recognition system 2 is operating.

[0058] Figure 7(a) shows an example of the output value from the reservoir model 40 and the target value in the training data when no event occurs, and (b) shows an example of the output value and target value when an event occurs. Figure 8(a) shows an example of the squared difference between the output value and the target value when no event occurs, and Figure 8(b) shows an example of the squared difference between the output value and the target value when an event occurs. Figure 8(a) corresponds to Figure 7(a), and Figure 8(b) corresponds to Figure 7(b). In Figures 7 and 8, one of the 12-dimensional data points in each frame is extracted and shown.

[0059] As shown in Figure 7(a), when no event occurs, the output value from the reservoir model 40 is close to the target value (normal value) in the training data (waveform). As shown in Figure 7(b), when an event occurs, the output value from the reservoir model 40 is significantly different from the target value in the training data (waveform).

[0060] In this example, in step S6, the control unit 16 calculates the square of the difference between the output value from the reservoir model 40 and the target value in the training data for each frame, and determines that an event has occurred if any of these values ​​for each frame exceeds a predetermined threshold (e.g., 200). As shown in Figure 8(a), if no event has occurred, the value will be relatively small. As shown in Figure 8(b), if an event has occurred, the value will be relatively large. Note that the control unit 16 only needs to determine whether an event has occurred based on the difference between the output value from the reservoir model 40 and the target value in the training data, and the method of determination is not limited to the above example. For example, the average of the differences between the output value and the target value for all frames may be compared with the threshold. Through the above processing, it is determined whether or not an event to be recognized has occurred in the first mode.

[0061] Figure 9 is a flowchart illustrating the processing in the second mode. In the second mode, the control unit 16 activates the second image sensor 22, and the second image sensor 22 acquires imaging data, which is time-series data (step S11). Subsequently, the second processing unit 15 processes the imaging data acquired by the second image sensor 22 using the deep learning model 50 (step S12). Subsequently, the control unit 16 recognizes the details of the event based on the output from the deep learning model 50 (step S13).

[0062] The deep learning model 50 is pre-trained using training data so that it can recognize event details from the imaging data output from the second image sensor 22. The training data is data corresponding to the details of the event to be recognized, for example, imaging data actually acquired by the second image sensor 22 when the event occurs. For example, if it is determined that "approach of object OJ" has occurred as an event, and the details of the event are recognized as "whether the approaching object OJ is a person or an object", then the training data is imaging data acquired by the second image sensor 22 when a person approaches and when an object approaches. The training of the deep learning model 50 is completed in advance before the start of operation of the event recognition system 2, and the training of the deep learning model 50 is not performed while the event recognition system 2 is operating.

[0063] Through the above processing, the details of the event determined to have occurred in the first mode are recognized in the second mode. After the details of the event are recognized in the second mode, the control unit 16 causes the robot 3 to perform a predetermined action based on the details of the event recognized in the second mode. The action performed by the robot 3 is, for example, the avoidance action described above. [Mechanism of Action and Effects]

[0064] In the event recognition system 2, in the first mode, data corresponding to temporal and spatial features extracted from imaging data is processed using the reservoir model 40 (first neural network), and it is determined whether a predetermined event has occurred based on the output from the reservoir model 40. If it is determined that an event has occurred, the operating mode is changed to the second mode (the second mode is activated), and in the second mode, the imaging data is processed using the deep learning model 50 (second neural network), and the details of the event are recognized based on the output from the deep learning model 50. As a result, in the first mode, smaller data (data corresponding to features extracted from imaging data) is processed using the reservoir model 40 compared to the second mode, so the reservoir model 40 can be made smaller than the deep learning model 50, and the amount of computation can be reduced. In addition, because the data input to the reservoir model 40 is small, even if the imaging data is large, it is possible to suppress slowdowns or stoppages in processing. Furthermore, in the second mode, a larger deep learning model 50 than the reservoir model 40 can be used to recognize the details of events, enabling highly accurate recognition. Therefore, the event recognition system 2 can reduce the amount of computation while enabling highly accurate recognition.

[0065] The number of nodes in the input layer 41 of the reservoir model 40 is less than the number of nodes in the input layer 51 of the deep learning model 50. This allows the reservoir model 40 to be made smaller, thereby reducing the computational load.

[0066] The computational complexity required to calculate the output from the input in the reservoir model 40 is less than that required to calculate the output from the input in the deep learning model 50. This allows for a reduction in computational complexity.

[0067] The reservoir model 40 has an input layer 41, a reservoir layer 42 coupled to the input layer 41, and an output layer 43 coupled to the reservoir layer 42, and only the coupling weights between the reservoir layer 42 and the output layer 43 are learned. This allows the reservoir model 40 to be made smaller and the computational load to be reduced.

[0068] The deep learning model 50 includes an input layer 41, at least three hidden layers 52 coupled to the input layer 41, and an output layer 43 coupled to the hidden layers 52. This allows for the recognition of event details using a large-scale deep learning model 50, enabling highly accurate recognition.

[0069] The imaging unit 11 has a first image sensor 21 and a second image sensor 22 (two-dimensional sensor) for acquiring imaging data, and the filter unit 12 is composed of a circuit unit 18 formed on a substrate 17 separated from the first image sensor 21 and the second image sensor 22. This makes it easy to realize a filter unit 12 that extracts temporal and spatial features.

[0070] The reservoir model 40 is a trained model based on training data corresponding to events. In the first mode, the control unit 16 determines whether an event has occurred based on a comparison between the output from the reservoir model 40 and the training data. This makes it easy to determine whether an event has occurred.

[0071] In the first mode, one dataset corresponding to the features extracted by the filter unit 12 is input to the reservoir model 40 at a single processing timing. This allows the reservoir model 40 to be made smaller compared to, for example, when multiple datasets are input to the reservoir model 40 at a single processing timing.

[0072] The imaging unit 11 includes a first image sensor 21 (first two-dimensional sensor) and a second image sensor 22 (second two-dimensional sensor) for acquiring imaging data. The filter unit 12 applies filtering to the imaging data acquired by the first image sensor 21 to extract temporal and spatial features from the imaging data. In the first mode, the first processing unit 14 processes data corresponding to the features extracted by the filter unit 12 from the imaging data acquired by the first image sensor 21 using a reservoir model 40. In the second mode, the second processing unit 15 processes the imaging data acquired by the second image sensor 22 using a deep learning model 50. By separating the first image sensor 21 used in the first mode and the second image sensor 22 used in the second mode, the complexity of the circuit can be suppressed.

[0073] If an event is detected in the first mode, the operating mode is changed from the first mode to the second mode (step S7). As a result, the first mode is turned off when the second mode is started (the processing of the first mode is not executed when the processing of the second mode is executed), thus reducing power consumption. [Differentiation]

[0074] As shown in the first modified example in Figure 10, the imaging unit 11 may have only one image sensor 21 (two-dimensional sensor). In this case, the imaging data acquired by the image sensor 21 is input to the filter unit 12 and the second processing unit 15. That is, in the first mode, the first processing unit 14 processes data corresponding to features extracted by the filter unit 12 from the imaging data acquired by the image sensor 21 using the reservoir model 40. In the second mode, the second processing unit 15 processes the imaging data acquired by the image sensor 21 using the deep learning model 50. By making the two-dimensional sensor used in the first mode and the two-dimensional sensor used in the second mode common, the imaging unit 11 can be miniaturized.

[0075] As shown in the second modified example in Figure 11, the filter unit 12 may be incorporated into the first image sensor 21. That is, the first image sensor 21 and the filter unit 12 may be configured on a single substrate 19 (chip or device). In this case, the filter unit 12 may be formed as a circuit on the substrate constituting the image sensor, or it may be realized by a physical configuration formed on the substrate constituting the image sensor. According to the second modified example, the configuration can be simplified compared to the case in which the filter unit 12 is configured by a circuit formed on a substrate separated from the image sensor. Furthermore, by incorporating the filter unit 12 into the first image sensor 21, the signal transfer speed from the first image sensor 21 to the filter unit 12 can be increased, and processing can be accelerated. Note that the incorporation of the filter unit 12 into the first image sensor 21 includes the case in which the substrate on which the filter unit 12 is formed as a circuit is stacked and integrated on the substrate constituting the first image sensor 21.

[0076] As shown in the third modified example in Figure 12, the event recognition system 2 may further include an image extraction unit 60. The image extraction unit 60 is positioned between the second image sensor 22 and the second processing unit 15 (deep learning model 50), and extracts (trimmes) a portion of the image data acquired by the second image sensor 22. The portion of the image to be extracted is identified by the control unit 16 based, for example, on a region in the image data where a relatively large number of events are recognized, or a region that is judged to be effective for image recognition.

[0077] As shown in Figure 13, in the second mode of the third modified example, between step S11, in which imaging data is acquired by the second image sensor 22, and step S12, in which the second processing unit 15 processes the imaging data using the deep learning model 50, step S14 is performed in which the image extraction unit 60 extracts a portion of the imaging data. As a result, in the second mode of the third modified example, the image area is limited compared to the first mode, and the second processing unit 15 processes imaging data with the deep learning model 50 that has a smaller data size than the imaging data that is filtered in the first mode. This makes it possible to reduce the data size of the imaging data input to the deep learning model 50, thereby speeding up processing and improving recognition accuracy.

[0078] The image extraction unit 60 may be formed integrally with the second image sensor 22. In this case, the imaging area of ​​the second image sensor 22 may be limited based on a signal from the control unit 16. Alternatively, the imaging area of ​​the second image sensor 22 may be limited by moving, for example, the light-receiving lens 24. The image extraction unit 60 may be omitted. The image area of ​​the imaging data in the first mode and the image area of ​​the imaging data processed by the second neural network in the second mode may coincide.

[0079] The present invention is not limited to the embodiments and modifications described above. The first neural network is not limited to the reservoir model 40, but may be any neural network. The second neural network is not limited to the deep learning model 50, but may be any neural network.

[0080] In the above embodiment, the operating mode was changed from the first mode to the second mode when it was determined that an event had occurred in the first mode. However, the second mode may be started while the first mode remains active. That is, the first mode may be started in parallel while the second mode is active. In this case, even when the second mode is active, the first mode can still detect the occurrence of an event.

[0081] The number of nodes in the input layer 41 of the reservoir model 40 (first neural network) may be equal to or greater than the number of nodes in the input layer 51 of the deep learning model 50 (second neural network). The computational cost of calculating the output from the input in the reservoir model 40 may be equal to or greater than the computational cost of calculating the output from the input in the deep learning model 50.

[0082] In the above embodiment, the filter unit 12, the pre-processing unit 13, and the first processing unit 14 were configured by a single circuit, and the second processing unit 15 and the control unit 16 were configured by separate circuits or processing units. However, at least one of the filter unit 12, the pre-processing unit 13, the first processing unit 14, the second processing unit 15, and the control unit 16 may be configured by a common circuit or processing unit. For example, the first processing unit 14 and the control unit 16 may be configured by a single circuit or processing unit, or the first processing unit 14, the second processing unit 15, and the control unit 16 may be configured by a single circuit or processing unit.

[0083] In the above embodiment, in the second mode, imaging data, which is time-series data, is acquired by the second image sensor 22, and the imaging data is processed by the deep learning model 50 to recognize the details of the event. However, in the second mode, the imaging data of one frame acquired by the second image sensor 22 may be processed by the deep learning model 50 to recognize the details of the event. Thus, the imaging data input to the deep learning model 50 in the second mode does not have to be time-series data, and the second image sensor 22 does not necessarily have to acquire time-series data.

[0084] The two-dimensional sensor provided by the imaging unit 11 is not limited to an image sensor; any sensor capable of acquiring imaging data is acceptable. For example, the two-dimensional sensor may be an event camera (event sensor, event-driven camera). An event camera is similar to an image sensor in that it has multiple pixels arranged in two dimensions, but in an event camera, the output of each pixel is compared by a comparator, and information (address, on / off signal) of the pixel whose output exceeds a threshold is output as data. The data output from the event camera can also be considered as imaging data acquired by imaging. However, in the above embodiment, since the first image sensor 21 and the filter unit 12 are used to extract temporal and spatial features, power consumption can be reduced compared to, for example, using an event camera, and the amount of data output to the reservoir model 40 can be reduced. The application target of the event recognition system 2 is not limited to the robot system 1, but may be applied to other targets.

[0085] Although not included in the invention described in claim 1 of the patent claims at the time of filing, the event recognition system may be configured as shown in the following Examples 1 and 2. <Example 1> In the event recognition system 2, the two-dimensional sensor provided in the imaging unit 11 is configured by an event camera, and the filter unit 12 is not provided. That is, the event recognition system may be defined as "an event recognition system comprising: an event camera having a plurality of pixels and acquiring imaging data, the event camera extracting temporal features by comparing the output of each of the plurality of pixels with a comparator; a two-dimensional sensor; a first processing unit that processes using a first neural network; a second processing unit that processes using a second neural network different from the first neural network; and a control unit, with an operating mode including a first mode and a second mode, wherein in the first mode, the first processing unit processes data corresponding to the features extracted by the event camera using the first neural network, the control unit determines whether a predetermined event has occurred based on the output from the first neural network, and if it determines that the event has occurred, it activates the second mode, and in the second mode, the second processing unit processes imaging data acquired by the event camera or the two-dimensional sensor using the second neural network, and the control unit recognizes the details of the event based on the output from the second neural network." The two-dimensional sensor may be the event camera (or may be shared with the event camera), or it may be a different sensor from the event camera (for example, an RGB camera). The event camera may compare the output of each of multiple pixels using a comparator and extract only the output of pixels where the difference in change is larger than a reference value (for example, a voltage value) as a positive event (bright area) or a negative event (dark area). The reference value may be reset each time an event occurs. In this case, temporal features can be extracted using an event camera, reducing the amount of data compared to, for example, using a regular camera. By using a comparator with the event camera, it is possible to extract only fast-moving objects, similar to using a high-pass filter (temporal filter), thereby extracting temporal features. However, since the event camera does not have the function to extract spatial features, the above embodiment, which includes a filter unit that extracts both temporal and spatial features after the imaging unit, is more effective in reducing data. <Example 2> In the event recognition system 2, the two-dimensional sensor provided by the imaging unit 11 is configured as an event camera, and the data output from the event camera is filtered by a filter unit 12 that has only a spatial filter and no temporal filter. In other words, the event recognition system may be defined as "an event camera having multiple pixels and acquiring imaging data, the event camera extracting temporal features by comparing the output of each of the multiple pixels using a comparator, a filter unit that applies a spatial filter to the imaging data acquired by the event camera and extracts spatial features from the imaging data, a two-dimensional sensor, a first processing unit that processes using a first neural network, a second processing unit that processes using a second neural network different from the first neural network, and a control unit, and includes a first mode and a second mode as operating modes, wherein in the first mode, the first processing unit processes data corresponding to the features extracted by the event camera and the filter unit using the first neural network, the control unit determines whether a predetermined event has occurred based on the output from the first neural network, and if it determines that the event has occurred, it activates the second mode, and in the second mode, the second processing unit processes imaging data acquired by the event camera or the two-dimensional sensor using the second neural network, and the control unit recognizes the details of the event based on the output from the second neural network." The two-dimensional sensor may be the event camera (or may be shared with the event camera), or it may be a different sensor from the event camera (for example, an RGB camera). The event camera may compare the output of each of multiple pixels using a comparator and extract only the output of pixels where the difference in change is larger than a reference value (for example, a voltage value) as a positive event (bright area) or a negative event (dark area). The reference value may be reset each time an event occurs. In this case, the amount of data input to the first processing unit can be reduced by extracting temporal features using an event camera and spatial features using a spatial filter. However, although the event camera can extract temporal features using a comparator, it outputs pixel information that has changed regardless of bandwidth, resulting in a large amount of data. The configuration of the above embodiment, which includes a filter unit that extracts both temporal and spatial features after the event camera, can more reliably reduce the amount of data. [Explanation of Symbols]

[0086] 1...Robot system, 2...Event recognition system, 3...Robot, 11...Imaging unit, 12...Filter unit, 14...First processing unit, 15...Second processing unit, 16...Control unit, 17...Substrate, 18...Circuit unit, 21...First image sensor (first two-dimensional sensor), 22...Second image sensor (second two-dimensional sensor), 40...Reservoir model (first neural network), 41...Input layer, 42...Reservoir layer, 43...Output layer, 50...Deep learning model (second neural network), 51...Input layer, 52...Hidden layer, 53...Output layer.

Claims

1. The imaging unit acquires imaging data, A filter unit applies time-series data, which is imaging data acquired by the imaging unit, to the imaging data using a time filter and a spatial filter to extract temporal and spatial features from the imaging data. A first processing unit that performs processing using a first neural network, A second processing unit that performs processing using a second neural network different from the first neural network, It comprises a control unit and, The operating modes include a first mode and a second mode. In the first mode, the first processing unit processes the data corresponding to the features extracted by the filter unit using the first neural network, and the control unit determines whether a predetermined event has occurred based on the output from the first neural network. If it determines that the event has occurred, it activates the second mode. In the second mode, the second processing unit processes the imaging data acquired by the imaging unit using the second neural network, and the control unit recognizes the details of the event based on the output from the second neural network, in an event recognition system.

2. The event recognition system according to claim 1, wherein the number of nodes in the input layer of the first neural network is less than the number of nodes in the input layer of the second neural network.

3. The event recognition system according to claim 1, wherein the computational load for calculating an output from an input in the first neural network is less than the computational load for calculating an output from an input in the second neural network.

4. The event recognition system according to claim 1, wherein the first neural network is a reservoir computing model having an input layer, a reservoir layer coupled to the input layer, and an output layer coupled to the reservoir layer, wherein only the coupling weights between the reservoir layer and the output layer have been learned.

5. The event recognition system according to claim 1, wherein the second neural network is a deep learning model having an input layer, at least three or more intermediate layers coupled to the input layer, and an output layer coupled to the intermediate layers.

6. The imaging unit has a two-dimensional sensor that acquires imaging data. The event recognition system according to any one of claims 1 to 5, wherein the filter section is composed of a circuit section formed on a substrate separated from the two-dimensional sensor.

7. The imaging unit has a two-dimensional sensor that acquires imaging data. The event recognition system according to any one of claims 1 to 5, wherein the filter unit is incorporated within the two-dimensional sensor.

8. The first neural network is a trained model based on training data corresponding to the event, In the first mode, the control unit determines whether or not the event has occurred based on a comparison between the output from the first neural network and the training data, according to any one of claims 1 to 5.

9. The event recognition system according to any one of claims 1 to 5, wherein in the first mode, at one processing timing, one dataset corresponding to the features extracted by the filter unit is input to the first neural network.

10. The imaging unit has a first two-dimensional sensor and a second two-dimensional sensor for acquiring imaging data. The filter unit extracts the temporal and spatial features from the imaging data acquired by the first two-dimensional sensor. In the first mode, the first processing unit processes the data corresponding to the features extracted by the filter unit from the imaging data acquired by the first two-dimensional sensor using the first neural network. The event recognition system according to any one of claims 1 to 5, wherein in the second mode, the second processing unit processes the imaging data acquired by the second two-dimensional sensor using the second neural network.

11. The imaging unit has a two-dimensional sensor that acquires imaging data. In the first mode, the first processing unit processes the data corresponding to the features extracted by the filter unit from the imaging data acquired by the two-dimensional sensor using the first neural network. The event recognition system according to any one of claims 1 to 5, wherein in the second mode, the second processing unit processes the imaging data acquired by the two-dimensional sensor using the second neural network.

12. The event recognition system according to any one of claims 1 to 5, wherein if it is determined that the event has occurred in the first mode, the operating mode is changed from the first mode to the second mode.

13. The event recognition system according to any one of claims 1 to 5, wherein if it is determined that the event has occurred in the first mode, the second mode is activated while the first mode remains activated.

14. In the first mode, the filter unit performs the filtering process on the imaging data acquired by the imaging unit. The event recognition system according to any one of claims 1 to 5, wherein in the second mode, the second processing unit processes the imaging data having a smaller data size than the imaging data subjected to the filtering in the first mode using the second neural network.

15. An event recognition system according to any one of claims 1 to 5, Equipped with a robot, The robot is a robot system that performs predetermined actions or processes based on the details of the events recognized in the second mode.

16. The first step involves applying a time filter and a spatial filter to the imaging data, which is time-series data acquired by the imaging unit, to extract temporal and spatial features from the imaging data, processing the data corresponding to the extracted features using a first neural network, determining whether a predetermined event has occurred based on the output from the first neural network, and executing the second step if it is determined that the event has occurred. An event recognition method comprising the second step of processing imaging data acquired by the imaging unit using a second neural network different from the first neural network, and recognizing the details of the event based on the output from the second neural network.

Images