Information processing device, system and method

Abstract

To provide an information processing device for specifying an area where a change occurs in a real space to be sensed.SOLUTION: An information processing device comprises area specifying means for specifying an area in a real space corresponding to a detection signal on the basis of the detection signal detected by a sensor capable of detecting a change in luminance of incident light for every pixel as a signal. The area specifying means specifies the area in the real space corresponding to the detection signal on the basis of correspondence between a position of the pixel and the area in the real space. The information processing device includes event specifying means for specifying an event that has occurred in the real space according to areas in the real space specified by the area specifying means.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an information processing device, system, and method. [Background technology]

[0002] There are sensors that can detect changes in the brightness of light incident on each pixel as a signal, which are so-called event sensors. Various systems that use these event sensors have been studied. For example, a system has been disclosed that generates an event image by collecting event signals that detect droplets ejected from a dispenser in units of a predetermined integration time, and controls the ejection of droplets by the dispenser based on the generated event image (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2023-080101 Summary of the Invention [Problem to be solved by the invention]

[0004] The objects of the present invention can be exemplified as follows. A first object of the present invention is to identify an area where a change is occurring in the real space that is the target of sensing. A second object of the present invention is to make it possible to exclude each area made up of a plurality of pixels of an event sensor from being subject to predetermined processing. A third object of the present invention is to identify an event that occurs in the real space that is the target of sensing. A fourth object of the present invention is to determine whether the state of the real space that is the target of sensing is normal or not. A fifth object of the present invention is to determine whether an object exists or to identify the location of an object even if the real space being sensed is in a state where it is difficult for humans to see. A sixth object of the present invention is to identify the position where an object exists after the object has moved, or the area where the object exists, in the real space of the sensing target. [Means for solving the problem]

[0005] The object of the present invention is to [1] An information processing device comprising: an area specifying means for specifying an area in real space corresponding to a detection signal detected by a sensor capable of detecting a change in luminance of incident light for each pixel as a signal; [2] The information processing device according to [1], wherein the area specifying means specifies the area in real space corresponding to the detection signal based on the correspondence between the pixel position and the area in real space; [3] The information processing device according to [1] or [2], further comprising: an event identification means for identifying an event occurring in the real space according to the area of ​​the real space identified by the area identification means; [4] The information processing device according to any one of [1] to [3] above, further comprising: information output means for outputting change information indicating that a state change has occurred in a real-space area identified by the area identification means when a plurality of pixels correspond to one real-space area and the total amount of detection signals corresponding to one real-space area exceeds a threshold; or information storage means for storing the change information; [5] The information processing device according to [4], wherein the threshold value differs for each region in the real space; [6] The information processing device according to [4] or [5], wherein the threshold value varies depending on time in the real space; [7] A method executed in an information processing device, comprising: a region specifying step of specifying a region in real space corresponding to a detection signal detected by a sensor capable of detecting a change in luminance of incident light for each pixel as a signal; [8] An information processing device comprising: a sensor capable of detecting a change in luminance of incident light for each pixel as a signal; a determination means for determining whether the total amount of signals corresponding to one region exceeds a threshold when the sensor divides the pixels into a plurality of regions according to the positions of the pixels; and the threshold differs for each region; [9] The information processing device according to [8], further comprising a threshold value specifying means for specifying a threshold value for each region based on a detection signal detected by the sensor in a state where no change occurs in the real space of the object to be detected by the sensor, and the determination means makes a determination based on the threshold value specified by the threshold value specifying means;

[10] A method executed in an information processing device, the method comprising a step of determining whether a total amount of signals corresponding to one area exceeds a threshold when a plurality of pixels of a sensor capable of detecting a change in luminance of incident light for each pixel as a signal is divided into a plurality of areas according to the positions of the pixels, and the threshold is different for each area;

[11] An information processing device including an event identification means for identifying an event that has occurred in real space based on the detection signal detected by a sensor capable of detecting a change in luminance of incident light for each pixel as a signal, using a prediction model trained by machine learning with an event that has occurred in the real space of the object to be detected by the sensor as output data;

[12] A method executed in an information processing device, comprising an event identification step of identifying an event that occurred in real space based on the detection signal detected by a sensor capable of detecting a change in luminance of incident light for each pixel as a signal, using a prediction model trained by machine learning with an event that occurred in the real space of the object to be detected by the sensor as output data;

[13] An information processing device including a determination means for determining whether the state of the real space of the object to be detected by the sensor is normal or not based on the detection signal detected by the sensor as a signal, using a prediction model machine-learned as input data a detection signal detected by a sensor capable of detecting a change in luminance of incident light for each pixel as a signal, and output data information regarding whether the state of the real space of the object to be detected by the sensor is normal or not;

[14] The information processing device according to

[13] , further comprising: an output means for outputting information indicating that the state of the real space is abnormal when the determination means determines that the state of the real space is abnormal; or an output control means for controlling another device to output information indicating that the state of the real space is abnormal;

[15] A method comprising a determination step of determining whether the state of the real space of the object to be detected by the sensor is normal or not based on the detection signal detected by the sensor as a signal representing the change in luminance of incident light for each pixel, using a prediction model machine-learned as input data and information regarding whether the state of the real space of the object to be detected by the sensor is normal or not as output data;

[16] A system comprising: a detection means for detecting a detection signal using a sensor capable of detecting a change in luminance of incident light for each pixel as a signal; and a vibration generation means for generating vibrations in at least a part of an object existing in a real space to be detected by the sensor, or in the sensor;

[17] A system comprising: a detection means for detecting a detection signal using a sensor capable of detecting a change in luminance of incident light for each pixel as a signal; and a light irradiation means for irradiating light onto at least a part of a real space to be detected by the sensor;

[18] The system according to

[16] or

[17] , further comprising: a position specifying means for specifying a position of a pixel corresponding to an object existing in real space from a position of a pixel corresponding to a detection signal detected by the detection means due to a change in brightness caused by vibrations generated by the vibration generating means, or a detection signal detected by the detection means due to a change in brightness caused by light irradiation by the light irradiating means;

[19] The system according to

[18] , further comprising: a display unit that displays an object on a display screen in a manner that allows the presence of the object to be recognized at the pixel position corresponding to the object when the detection signal is no longer detected because the vibration has ceased, or when the detection signal is no longer detected because the irradiation of light has ended; or a display control unit that controls the display screen of another device to display an object on a display screen in a manner that allows the presence of the object to be recognized;

[20] The system according to

[18] or

[19] , further comprising a registration means for registering the presence of an object at a pixel position corresponding to the object when the detection signal is no longer detected due to the vibrations having ceased or when the light irradiation has ceased;

[21] The system according to any one of

[16] to

[20] , further comprising an object identification means for identifying attributes of an object present in the real space to be detected by the sensor, based on the detection signal detected by the detection means, using a prediction model trained by machine learning with input data of a detection signal detected by a sensor capable of detecting a change in luminance of the incident light for each pixel as a signal, and output data of the attributes of the object;

[22] A method comprising: a detection step of detecting a detection signal using a sensor capable of detecting a change in luminance of incident light for each pixel as a signal; and a vibration generation step of generating vibrations in at least a part of an object existing in a real space to be detected by the sensor, or in the sensor;

[23] A method comprising: a detection step of detecting a detection signal using a sensor capable of detecting a change in luminance of incident light for each pixel as a signal; and a light irradiation step of irradiating light onto at least a part of a real space to be detected by the sensor;

[24] An information processing device comprising: a position specifying means for specifying the position of a pixel corresponding to a detection signal detected immediately before the detection signal becomes unavailable in a sensor capable of detecting a change in luminance of incident light for each pixel as a signal;

[25] The information processing device according to

[24] , further comprising: a display means for displaying on a display screen in a manner that enables the user to recognize the presence of an object at the pixel position identified by the position identification means; or a display control means for controlling the display screen of another device to display on a display screen in a manner that enables the user to recognize the presence of an object at the pixel position identified by the position identification means;

[26] The information processing device according to

[24] or

[25] , further comprising a registration means for registering the presence of an object at the position of the identified pixel;

[27] The information processing device according to

[26] , wherein, when a detection signal is detected again at a pixel position corresponding to an object registered by the registration means and then the detection signal is no longer detected again, the information processing device registers that the object registered by the registration means has moved to a pixel position corresponding to a detection signal detected just before the detection of the detection signal again became no longer detected;

[28] A method comprising a position specifying step of specifying the position of a pixel corresponding to a detection signal detected immediately before the detection signal becomes unavailable in a sensor capable of detecting a change in luminance of incident light for each pixel as a signal; This can be solved by: [Effects of the Invention]

[0006] The effects of the present invention can be exemplified as follows. A first effect of the present invention is that it is possible to identify an area where a change is occurring in the real space that is the target of sensing. The second effect of the present invention is that each area made up of a plurality of pixels of the event sensor can be excluded from the target of predetermined processing. The third effect of the present invention is that it is possible to identify an event that has occurred in the real space that is the target of sensing. A fourth effect of the present invention is that it is possible to determine whether the state of the real space that is the target of sensing is normal or not. A fifth object of the present invention is to determine whether an object exists or to identify the location of the object even if the real space being sensed is in a state where it is difficult for a person to see. A sixth object of the present invention is to be able to identify the position where an object exists after the object has moved, or the area where the object exists, in the real space of the sensing target. [Brief explanation of the drawings]

[0007] [Figure 1] 1 is a diagram illustrating a configuration of a system according to an embodiment of the present invention. [Figure 2] 1 is a diagram showing a configuration of an information processing device according to an embodiment of the present invention; [Figure 3] FIG. 10 is a flowchart showing an event identification process according to an embodiment of the present invention. [Figure 4] FIG. 2 is a diagram illustrating a relationship between regions and pixels according to an embodiment of the present invention. [Figure 5] FIG. 10 is a diagram showing a table showing the correspondence between areas and events according to an embodiment of the present invention. [Figure 6] FIG. 10 is a flowchart showing a determination process according to an embodiment of the present invention. [Figure 7] FIG. 10 is a flowchart showing an event identification process according to an embodiment of the present invention. [Figure 8] FIG. 10 is a flowchart showing a determination process according to an embodiment of the present invention. [Figure 9] 1 is a diagram illustrating a configuration of a system according to an embodiment of the present invention. [Figure 10] FIG. 10 is a flowchart showing an object detection process according to an embodiment of the present invention. [Figure 11] 1 is a diagram illustrating a configuration of a system according to an embodiment of the present invention. [Figure 12] FIG. 10 is a flowchart showing an object detection process according to an embodiment of the present invention. [Figure 13] FIG. 10 is a flowchart showing a movement detection process according to the embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0008] Hereinafter, embodiments of the present invention will be described with reference to the drawings, but the present invention is not limited to the following embodiments unless it is contrary to the spirit of the present invention. Furthermore, the description of the effects is one aspect of the effects of the embodiments of the present invention, and is not limited to those described here.

[0009] (First embodiment) The first embodiment relates to an information processing device that identifies an area in real space corresponding to a signal detected by a sensor (i.e., an event sensor) that can detect changes in the brightness of incident light for each pixel as a signal, or identifies an event that has occurred in real space.

[0010] Fig. 1 is a diagram showing the configuration of a system according to an embodiment of the present invention. The system 1 in Fig. 1 includes an event sensor 2 and an information processing device 3. An arbitrary object 5 may exist in a real space 4 that is the target of sensing by the event sensor 2.

[0011] The event sensor 2 is also called an EVS (event-based vision sensor). The event sensor 2 converts incident light into an electrical signal using a light-receiving unit of the sensor and outputs the converted electrical signal as an event signal. The event sensor 2 includes a camera, and each pixel of the camera includes a light-receiving unit that receives incident light and a detection unit that detects changes in luminance. When a change in the luminance of the incident light for each pixel exceeds a set threshold, the event signal is detected and the position coordinate of the pixel where the event signal occurred, the time when the event signal occurred, and the polarity (whether it is a light-up signal or a dark-down signal) can be output. The position coordinate of the pixel where the event signal occurred can be expressed, for example, in an xy coordinate system aligned with the up, down, left, and right directions of the event sensor 2. Furthermore, the detection of the event signal can be displayed on the display screen of the display unit 15 of the information processing device 3 (described later) in a manner that allows the user to see which pixel of the event sensor 2 detected the event signal, corresponding to the position coordinate.

[0012] Normally, the event sensor 2 is installed without changing its position or orientation. The camera of the event sensor 2 stays in the same position and facing the same direction even over time, and the real space 4 that is the target of sensing is also fixed.

[0013] When an arbitrary object 5 moves in the real space 4, the movement of the object 5 causes a change in the brightness of the light incident on the event sensor 2, and an event signal is detected by the event sensor 2. When the object 5 is stationary, no event signal is detected. The event sensor 2 can detect the event signal even in conditions where it is difficult for people to see the presence or movement of an object due to darkness, such as at night, or backlighting, and can detect changes in the situation, such as changes in the object 5 in the real space 4.

[0014] The information processing device 3 executes various information processes based on the event data output from the event sensor 2. Figure 2 is a diagram showing the configuration of an information processing device according to an embodiment of the present invention. The information processing device 3 includes a control unit 11, a RAM 12, a storage unit 13, an input unit 14, a display unit 15, and a communication interface 16, all of which are connected via a bus.

[0015] The control unit 11 is composed of a CPU and a ROM. The control unit 11 executes programs stored in the storage unit 13 and controls the information processing device 3. The RAM 12 is a work area for the control unit 11. The storage unit 13 is a storage medium for saving programs and data. The control unit 11 performs arithmetic processing based on the event signal received from the event sensor 2, the programs and data read from the RAM 12, and / or data input via the input unit 14.

[0016] The display unit 15 has a display screen. The control unit 11 outputs a video signal for displaying an image on the display screen according to the result of the arithmetic processing. Here, the display screen of the display unit 15 may be a touch panel equipped with a touch sensor. In this case, the touch panel functions as the input unit 14. Note that the output of the result of the arithmetic processing from the information processing device 3 is not limited to display on the display screen of the display unit 15, but may also be audio output by an audio output unit (not shown) or transmission of information to another device via the communication interface 16.

[0017] The communication interface 16 can be connected to a communication network wirelessly or via a wire, and the information processing device 3 can send and receive data to and from other devices via the communication network. The other devices are not particularly limited as long as they are capable of receiving information and outputting the received information, and may, for example, be equipped with a control unit, RAM, a storage unit, an input unit, an audio output unit, a communication interface, etc., similar to the information processing device 3.

[0018] The real space 4 may be either indoors or outdoors, and is not particularly limited. The real space 4 can be appropriately selected depending on the application.

[0019] The object 5 may be any of a person, an animal, and a non-living tangible object, and is not particularly limited. The non-living tangible object may be a vehicle such as an automobile that has a function of moving, or may be a piece of furniture that does not have a function of moving, and is not particularly limited.

[0020] Next, the event identification process will be described. The event identification process is executed by the information processing device 3. Fig. 3 is a diagram showing a flowchart of the event identification process according to the embodiment of the present invention.

[0021] The information processing device 3 receives an event signal from the event sensor 2 (step S11). Based on the position of the pixel corresponding to the received event signal, an area in real space is identified (step S12). The correspondence between the pixel position and the area in real space is determined in advance, and multiple pixels correspond to one area in real space. The area in real space is identified according to this correspondence. Detection of an event signal in any of the multiple pixels corresponding to one area means that some kind of change is occurring in that area. Continuous detection of an event signal in a pixel corresponding to one area means that changes are continuing to occur in that area.

[0022] Next, an event that has occurred in the real space 4 is identified according to the identified real space area (step S13). Here, identifying an event is a concept that includes estimating that an event is occurring. For each event, a condition is set in advance as to which area, if identified in step S12, will be identified as the occurrence of a specific event, and step S13 is executed according to this condition. FIG. 5 is a diagram showing a table representing the correspondence between areas and events according to an embodiment of the present invention. In step S13, an event can be identified by referring to the table shown in FIG. 5. Note that the condition as to which area, if identified, will be identified as the occurrence of a specific event varies depending on the layout of the object 5 in the real space 4 and the position and orientation of the event sensor 2, and therefore needs to be set appropriately according to the layout of the object 5 and the position and orientation of the event sensor 2.

[0023] In step S13, for example, if a change occurs in area A (i.e., area A is identified), it can be determined that event α has occurred, and if a change occurs in area B, it can be determined that event β has occurred. Furthermore, if a change occurs in area B within a predetermined time after a change occurs in area A, it can be determined that event γ has occurred. That is, in addition to identifying an event based on the area in which a change occurred identified in step S12, if changes occur in multiple areas over time, it can also be identified based on the order in which the changes occurred, or the time from when a change occurs or ends in one area until a change occurs in another area, or the time until a change ends in another area. It can also be identified based on the length of time from when a change occurs in one area until it ends. In this way, an event can be identified based on the identified area and the time element related to the change.

[0024] For example, assume that the real space being sensed is one in which a patient is hospitalized and sleeping in bed at night. Unlike conventional camera monitoring, event sensors can detect changes in the room even when the lights are off at night and in near-dark conditions. Furthermore, because the patient's condition is not captured in detail, the patient's privacy is protected. For example, if there is a change in the "bed" area and no subsequent changes in other areas, it can be determined that the patient has turned over in their sleep. If there is a change in the "bed" area, followed by a change in the "floor" area, and then no subsequent changes in other areas, it can be determined that the patient has fallen to the floor. If there is a change in the "bed" area, followed by a change in the "floor" area, and then a change in the "door" area, it can be determined that the patient has opened the door and moved out of the room.

[0025] Note that the area in real space associated with multiple pixels does not have to coincide with an object that exists when a straight line is extended from each pixel on the visual axis in real space. For example, an area in which a change in brightness may occur when a patient moves on the bed can be set as the "bed" area. In this way, areas can be set based on the expected range of movement or movement of people or objects.

[0026] Next, an output indicating that the event identified in step S13 has occurred is output (step S14). The output in step S13 may be a display on the display unit 15 of the information processing device 3, an audio output, or transmission to another device. When the identified event is transmitted to another device, the other device can output the identified event by a method such as displaying the event on a display unit or outputting audio. In the other embodiments below, the "output" in the information processing device 3 and the other device is the same.

[0027] In the real space 4, the change information is stored as a history indicating that an event has occurred, in association with the time at which the event signal was detected (step S15).

[0028] The processes of steps S16 and S17 can be executed in parallel with the processes of steps S13 to S15, or only one of them can be executed.

[0029] If the total amount of event signals per time corresponding to one real space region exceeds a threshold, change information indicating that a state change has occurred in the real space region identified in step S12 is output (step S16). The change information is stored in association with the time at which the event signal was detected as a history indicating that a change has occurred in the real space 4 (step S17). The event identification process ends with steps S11 to S15, or S11, S12, S16 and S17.

[0030] Also, different thresholds can be used for each region. For example, a high threshold can be set for the region "bed" because sleeping patients often move their bodies at night, while a low threshold can be set for the region "floor" because changes occur less frequently than in the region "bed."

[0031] Furthermore, even for the same region, different thresholds can be used depending on the time in real space. "Using different thresholds depending on the time" is a concept that includes not only using different thresholds depending on the time of day, but also using different thresholds depending on the time of day, day, week, month, season, etc. For example, since the amount of movement of a sleeping patient differs between late at night and early in the morning, the threshold can be set higher during late at night than during early in the morning.

[0032] It is also possible to use different thresholds for each region, and to use different thresholds for the same region depending on the time.

[0033] The information processing device 3 of the first embodiment can be used in medical institutions, nursing homes, and homes for purposes such as monitoring and protecting hospitalized patients and the elderly, and watching over people who require care. Even if the real space 4 being sensed is in a situation where visibility is difficult, such as at night, movements and actions can be identified.

[0034] (Second embodiment) The second embodiment relates to an information processing device that determines whether the total amount of signals corresponding to one area exceeds a threshold when multiple pixels of an event sensor are divided into multiple areas according to the positions of the pixels. The system configuration in Fig. 1 and the information processing device configuration in Fig. 2 described above are also applied to the second embodiment.

[0035] The determination process will be described below. The determination process is executed by the control unit 11 of the information processing device 3. Fig. 6 is a diagram showing a flowchart of the determination process according to the embodiment of the present invention.

[0036] In the second embodiment, a plurality of regions (a plurality of groups) are set according to the positions of a plurality of pixels that receive light by the event sensor. Note that the "region" in the second embodiment is a different concept from the "region in real space" in the first embodiment, but may or may not have a corresponding relationship with the region in real space in the first embodiment.

[0037] The information processing device 3 first sets a threshold value (step S21). The threshold value differs for each region. For example, since the event signals detected by the event sensor 2 may be noisy in some regions and quieter in others, it is preferable to set a threshold value for each region. A threshold value can be set for each region based on the detected event signals when no changes occur in the real space. For example, the average amount of event signals per frame for each region can be calculated from the total amount of event signals detected for each region per predetermined time period. Based on this average amount of event signals, a threshold value for each frame for each region can be set. The threshold value is preferably greater than the average amount of event signals, and can be, for example, a value obtained by multiplying the average amount by a coefficient greater than 1. The event signals for setting the threshold value are preferably detected immediately after the event sensor 2 starts operating in the real space 4 being sensed.

[0038] Once the threshold is set, the event sensor 2 starts sensing the real space 4. For example, when a change occurs, such as the movement of an object 5 present in the real space 4, an event signal is detected (step S22). Based on the detected event signal, it is determined whether the total amount of signals corresponding to each region exceeds the threshold (step S23). For regions where the threshold is exceeded, a predetermined process is executed (step S24). The predetermined process is not particularly limited, but for example, a predetermined calculation or output process is executed using the event signal corresponding to the region or information indicating that the region has exceeded the threshold. An example of the predetermined process is a process of displaying on the display screen of the display unit 15 in a manner that indicates that a state change has occurred in the region where the threshold is exceeded. On the other hand, regions where the threshold is not exceeded are not subject to the execution of the predetermined process. By executing steps S21 to S24, the determination process ends.

[0039] The determination process explained in the second embodiment can also be executed in other embodiments (for example, the first embodiment, and the third to seventh embodiments).

[0040] As described above, by setting a threshold, it is possible to exclude noise from the event signal and execute predetermined processing, but instead of excluding noise, it is also possible to exclude areas of changes that have occurred in real space 4 that are larger than a threshold and execute predetermined processing, or to exclude areas that are smaller than the threshold and execute predetermined processing. For example, in a case where it is desired to exclude detection of event signals based on the breathing of a person sleeping on the headset and detect only event signals based on the person turning over in their sleep in an area related to a bed, a threshold can be set that allows detection of only event signals based on the breathing of the person sleeping on the headset, and detection of event signals that do not meet the threshold can be excluded from the target of predetermined processing.

[0041] (Third embodiment) The third embodiment relates to an information processing device that uses an event signal detected by an event sensor as input data and an event occurring in the real space of the detection target as output data, and identifies an event occurring in the real space based on the event signal detected by the event sensor in the real space that is the sensing target. The system configuration in Fig. 1 and the information processing device configuration in Fig. 2 described above are also applied to the third embodiment.

[0042] The event identification process will be described below. The event identification process is executed by the control unit 11 of the information processing device 3. Fig. 7 is a diagram showing a flowchart of the event identification process according to the embodiment of the present invention.

[0043] First, an event that has occurred is identified based on an event signal detected at a predetermined time (step S31). The length of the predetermined time can be designed appropriately depending on the real space 4 that is the sensing target and the event to be identified.

[0044] The occurrence of an event in step S31 is identified using a prediction model trained by machine learning based on the following input data and output data. Specifically, when an event occurs in real space, information about the event can be used as output data, and while the event is occurring, an event signal detected by sensing the real space where the event is occurring with an event sensor can be used as input data.

[0045] It is preferable that the prediction model used in step S31 uses as input data an event signal detected by an event sensor installed in substantially the same position and orientation as the event identification process in a real space that is substantially the same as the real space 4 that is the target of the event identification process (i.e., including not only the same real space but also a real space that is in a different location but has the same configuration with the same area and the same object arrangement).

[0046] In step S31, this prediction model can be used to identify an event that has occurred in the real space that is the sensing target, based on the event signal detected by the event sensor 2.

[0047] The machine learning algorithm is not particularly limited and any known algorithm can be used, but it is preferable to use deep learning using a multilayer neural network. The multilayer neural network has an input layer, an output layer, and multiple intermediate layers. Weights are set for the edges connecting nodes in each layer. Weights corresponding to each input to the node are set for the edges, and the input to the node is multiplied by the weight corresponding to each input, and the value obtained by multiplying these weights and the bias are added. The value obtained by the addition is subjected to nonlinear transformation using an activation function to calculate an activation value. The calculated activation value becomes the input value passed to the node in the next layer. The number of intermediate layers can be designed as appropriate.

[0048] In step S31, the events to be identified are not particularly limited, and may be any state change in the real space 4, such as the movement of a person or animal, or the movement of another object, that can cause a change in the brightness of light incident on the event sensor 2. If the real space 4 that is the target of sensing is a hospital room in a medical institution, events such as "the patient turned over in bed," "the patient fell off the bed onto the floor," or "the patient got off the bed and moved to another room" can be identified.

[0049] When an event is identified in step S31, the information processing device 3 outputs the identified event in a form that can be understood by a person (step S32).

[0050] Next, the information processing device 3 associates the identified event with the time when the event occurred (for example, the time when detection of the event signal corresponding to the event started) or the duration of the event (for example, the time when detection of the event signal corresponding to the event started and the time when detection of the event signal ended) and stores the associated event in the storage unit 13 as an event occurrence history (step S33).

[0051] The event identification process ends with steps S31 to S33. The process of steps S31 to S33 is repeatedly executed at predetermined time intervals.

[0052] As with the first embodiment, the information processing device 3 of the third embodiment can be used in medical institutions, nursing homes, and homes for purposes such as monitoring and protecting hospitalized patients and the elderly, and watching over people who require care. Even if the real space 4 being sensed is in a situation where visibility is difficult, such as at night, movements and actions can be identified.

[0053] (Fourth embodiment) The fourth embodiment relates to an information processing device that uses an event signal detected by an event sensor as input data and information regarding whether the state of the real space being detected by the event sensor is normal as output data, and uses a machine-learned prediction model to determine whether the state of the real space being sensed is normal based on the event signal detected by the event sensor in the real space being sensed.

[0054] Here, information regarding whether something is normal or not is a concept that includes not only information that means "normal" or "not normal," but also information that means "abnormal" or "not abnormal."

[0055] The above-described system configuration in FIG. 1 and the configuration of the information processing device in FIG. 2 are also applied to the fourth embodiment.

[0056] The determination process will be described below. The determination process is executed by the control unit 11 of the information processing device 3. Fig. 8 is a diagram showing a flowchart of the event identification process according to the embodiment of the present invention.

[0057] First, based on the event signal detected at a predetermined time, it is determined whether the state of the real space is normal (step S41). The length of the predetermined time can be designed appropriately depending on the real space 4 that is the sensing target and the event to be identified.

[0058] In step S41, the determination of whether the state of the real space is normal is made using a prediction model machine-learned based on the following input data and output data. Specifically, when the real space is in a normal state, an event signal detected by sensing the real space with an event sensor can be used as input data, and information indicating normality can be used as output data. Furthermore, when the real space is in an abnormal state, an event signal detected by sensing the real space with an event sensor can be used as input data, and information indicating abnormality (or abnormality) can be used as output data. Furthermore, a prediction model machine-learned can be used based on input data and output data corresponding to the case where the real space is in a normal state and the case where the real space is in an abnormal state.

[0059] It is preferable that the predictive model used in step S41 uses as input data an event signal detected by an event sensor installed in substantially the same position and orientation as in the judgment process in a real space that is substantially the same as the real space 4 that is the subject of the judgment process (i.e., including not only the same real space, but also a real space that is in a different location but has the same configuration with the same area and the same object arrangement).

[0060] The method for determining whether the real space is "normal" or "abnormal" in the output data of the prediction model is not particularly limited, and may be determined based on, for example, the result of a human visually determining whether the real space is normal. The method for determining whether the real space is "abnormal" or "not abnormal" in the output data of the prediction model is not particularly limited, and may be determined based on, for example, the result of a human visually determining whether the real space is abnormal. In this way, by determining whether the output data is "normal" or "abnormal" (or whether the output data is "abnormal" or "not abnormal") based on the result of a human visual determination, the determination in step S41 can also be made in line with the human visual determination. It is preferable that the determination of whether the state of the real space, which is the output data of the prediction model, is normal is made in accordance with predetermined conditions, predetermined standards, and / or predetermined rules.

[0061] For example, if the flow of people moving within a given facility (e.g., inside a train station) is the target of sensing by an event sensor, a state in which most people walking within the station are moving in the same direction as the flow of people moving without going against the flow of people moving can be determined to be a normal state, and a state in which some people are moving against the flow of people moving can be determined to be abnormal.By using a prediction model that specifies output data based on such criteria, it becomes possible to determine that a state in the real space that is the target of sensing, in which some people are moving against the flow of people moving, is abnormal.

[0062] In addition, when the flow of vehicles moving on a road is the target of sensing by an event sensor, a state in which vehicles traveling on the road are moving in a direction of travel determined for each road at a predetermined speed or less can be considered a normal state, and a state in which some vehicles are moving in the opposite direction to the direction of travel or moving at a speed exceeding the predetermined speed can be considered an abnormal state. The criteria for whether something is normal or abnormal are not particularly limited and can be designed as appropriate depending on the real space that is the target of sensing and the abnormality that is sought to be detected.

[0063] In step S41, using such a prediction model, it is possible to determine whether or not the real space that is the sensing target is normal based on the event signal detected by the event sensor 2 in the real space.

[0064] The machine learning algorithm is not particularly limited and any known algorithm can be used, but it is preferable to use deep learning using a multilayer neural network, which can have the same configuration as that of the third embodiment.

[0065] If it is determined in step S41 that the state of the real space 4 is normal (YES in step S42), the determination process ends. On the other hand, if it is determined in step S41 that the state of the real space 4 is abnormal (NO in step S42), information indicating that the state of the real space 4 is not normal or abnormal is output (step S43).

[0066] Next, the information processing device 3 associates the time when the abnormality occurred (for example, the time when detection of an event signal corresponding to the occurrence of the abnormality started) or the duration of the abnormality (for example, the time when detection of an event signal corresponding to the occurrence of the abnormality started and the time when detection of the event signal ended) and stores the associated history of the abnormality in the storage unit 13 (step S44).

[0067] The determination process ends with steps S41 to S44. The determination process of steps S41 to S44 is repeatedly executed at predetermined time intervals.

[0068] The information processing device 3 of the fourth embodiment can be used, for example, to determine whether the flow of moving objects such as people and vehicles is normal on a road where many people and vehicles are moving, or in a facility where many people are moving.

[0069] (Fifth embodiment) The fifth embodiment relates to a system that detects an event signal using an event sensor and detects at least a part of an object existing in a real space that is the detection target of the event sensor, or an event signal that can be detected by generating a vibration in the event sensor. The configuration of the information processing device shown in Fig. 2 described above is also applicable to the fifth embodiment. In the fifth embodiment, the system configuration shown in Fig. 9 can be applied instead of the system configuration shown in Fig. 1.

[0070] Fig. 9 is a diagram showing the configuration of a system according to an embodiment of the present invention. The system 1 in Fig. 9 includes an event sensor 2 and an information processing device 3. The system 1 also includes a vibration generator 6 for generating vibrations in the event sensor 2 and / or an object 5. Any object 5 may exist in a real space 4 that is the target of sensing by the event sensor 2.

[0071] Typically, when an object 5 moves in the real space 4, the movement of the object 5 causes a change in the brightness of light incident on the event sensor 2, and the event sensor 2 detects an event signal. If the object 5 is stationary, the event signal is not detected. Therefore, in a situation where it is difficult for a person to see the presence or movement of an object due to darkness, such as at night, or backlight, if the object 5 is stationary, the presence of the object 5 cannot be detected. However, by temporarily (for a short period of time, such as 0.1 seconds) applying vibration to the event sensor 2 and / or the object 5 by the vibration generator 6 to an extent that does not significantly change the state of the real space 4 (for example, an extent that does not change the position or orientation of the event sensor 2 and the object 5), it becomes possible to detect an event signal even if the object 5 is stationary. The duration of the vibration generation and the magnitude of the vibration are not particularly limited and can be set appropriately depending on the object 5 assumed to exist in the real space 4.

[0072] The real space 4 may be either indoors or outdoors, and is not particularly limited. The real space 4 can be appropriately selected depending on the purpose. The object 5 may be any of a person, an animal, or a tangible object other than a living thing, and is not particularly limited. The tangible object other than a living thing may be an object that has a function of moving, such as a vehicle such as an automobile, or an object that does not have a function of moving, such as furniture, and is not particularly limited.

[0073] The object detection process will be described below. The object detection process is mainly executed by the control unit 11 and the vibration generator 6 of the information processing device 3. Fig. 10 is a diagram showing a flowchart of the object detection process according to the embodiment of the present invention.

[0074] First, the information processing device 3 transmits a vibration generation request to the vibration generator 6 to generate vibration (step S51). The vibration generation request is transmitted at predetermined time intervals. When the vibration generator 6 receives the vibration generation request (step S52), the vibration generator 6 generates a slight vibration for a short period of time (step S53). The generated vibration is also transmitted to the object 5 or the event sensor 2 present in the real space 4. As the vibration is transmitted to the object 5 or the event sensor 2 in this way, the light reflected by or transmitted through the object 5 is received by the event sensor 2, and an event signal due to a change in the brightness of the incident light can be detected.

[0075] When a change occurs in the brightness of light incident on the event sensor 2 due to vibrations generated by the vibration generator 6, an event signal is detected (step S54). Next, the attribute (type) of the object 5 corresponding to the detected event signal is identified (step S55). The identification of the attribute of the object 5 in step S55 can utilize a machine-learned prediction model. The prediction model can use a machine-learned prediction model that uses an event signal, which is detected when light reflected and / or transmitted from an object is incident on a sensor that can detect changes in brightness of the incident light for each pixel as an event signal, as input data, and the attribute of the object as output data. Here, the attribute of the object is not particularly limited as long as it can identify, for example, the name, use, material, properties, etc. of the object.

[0076] The machine learning algorithm is not particularly limited and any known algorithm can be used, but it is preferable to use deep learning using a multilayer neural network, which can have the same configuration as that of the third embodiment.

[0077] Next, the position of the pixel corresponding to the detected event signal is identified, and from the identified positions of the multiple pixels, the position of the pixel (pixel position coordinates) corresponding to the object 5 is identified (step S56). The method for identifying the position of the pixel corresponding to the object 5 in step S56 is not particularly limited, but since event signals are detected in multiple pixels in multiple frames, the position of the pixel corresponding to the object 5 can be identified by averaging the position coordinates of these pixels.

[0078] Next, the presence of the object 5 at the identified position is registered in the RAM 12 or the storage unit 13 (step S57). At this time, along with the presence of the object 5, the attributes of the object 5 identified in step S55 may also be registered.

[0079] Next, the position of the pixel of the identified object 5 is output (step S58). When the position of the pixel of the object 5 is displayed on the display unit 15, it can also be displayed in a manner that makes it clear that it is a position corresponding to the object 5 on a display screen that displays a color different from the background at the pixel where the event signal is detected. After steps S51 to S58, the object detection process ends.

[0080] (Sixth embodiment) The sixth embodiment relates to a system that irradiates light onto at least a part of a real space that is a sensing target and that detects an event signal with an event sensor. The configuration of the information processing device shown in FIG. 2 described above is also applied to the sixth embodiment. In the sixth embodiment, the system configuration shown in FIG. 11 can be applied instead of the system configuration shown in FIG. 1.

[0081] Fig. 11 is a diagram showing the configuration of a system according to an embodiment of the present invention. The system 1 in Fig. 11 includes an event sensor 2 and an information processing device 3. The system 1 also includes an illumination device 7 for illuminating at least a portion of a real space 4 with light. An arbitrary object 5 may be present in the real space 4 that is the target of sensing by the event sensor 2.

[0082] Typically, when an object 5 moves in the real space 4, the movement of the object 5 causes a change in the brightness of the light incident on the event sensor 2, and the event sensor 2 detects an event signal. If the object 5 is stationary, the event signal is not detected. Therefore, in a situation where it is difficult for a person to see the presence or movement of an object due to darkness, such as at night, or backlight, if the object 5 is stationary, the presence of the object 5 cannot be detected. However, by temporarily (for a short period of time, such as 0.1 seconds) illuminating at least a portion of the real space 4 with the illumination device 7, the event sensor 2 can detect an event signal even if the object 5 is stationary. The length of illumination time and the illuminance of the light emitted by the illumination device 7 are not particularly limited. Furthermore, the illumination by the illumination device 7 may be direct illumination of the object 5, or may be indirect illumination of the object 5 by illuminating a portion of the real space.

[0083] The real space 4 may be either indoors or outdoors, and is not particularly limited. The real space 4 can be appropriately selected depending on the purpose. The object 5 may be any of a person, an animal, or a tangible object other than a living thing, and is not particularly limited. The tangible object other than a living thing may be an object that has a function of moving, such as a vehicle such as an automobile, or an object that does not have a function of moving, such as furniture, and is not particularly limited.

[0084] The object detection process will be described below. The object detection process is mainly executed by the control unit 11 and the illumination device 7 of the information processing device 3. Fig. 12 is a diagram showing a flowchart of the object detection process according to the embodiment of the present invention.

[0085] First, the information processing device 3 transmits an illumination request to the illumination device 7 to illuminate at least a part of the real space 4 (step S61). The illumination request is transmitted at predetermined time intervals. When the illumination device 7 receives the illumination request (step S62), the illumination device 7 temporarily illuminates the real space 4 with light (step S63). As a result, the event sensor 2 receives the light reflected by or transmitted through the object 5, and an event signal due to a change in the luminance of the incident light can be detected.

[0086] When a change occurs in the luminance of light incident on the event sensor 2, an event signal is detected (step S64). Next, the attribute (type) of the object 5 corresponding to the detected event signal is identified (step S65). The identification of the attribute of the object 5 in step S65 can utilize a machine-learned prediction model. The prediction model can use a machine-learned prediction model that uses an event signal, which is detected when light reflected and / or transmitted from an object is incident on a sensor that can detect a change in luminance of the incident light for each pixel as an event signal, as input data, and the attribute of the object as output data. Here, the attribute of the object is not particularly limited as long as it can identify, for example, the name, use, material, properties, etc. of the object.

[0087] The machine learning algorithm is not particularly limited and any known algorithm can be used, but it is preferable to use deep learning using a multilayer neural network. The details of the multilayer neural network can be applied to the third embodiment.

[0088] Next, the position of the pixel corresponding to the detected event signal is identified, and from the identified positions of the multiple pixels, the position of the pixel (pixel position coordinates) corresponding to the object 5 is identified (step S66). The method for identifying the position of the pixel corresponding to the object 5 in step S66 is not particularly limited, but since event signals are detected in multiple pixels in multiple frames, the position of the pixel corresponding to the object 5 can be identified by averaging the position coordinates of these pixels.

[0089] Next, the presence of the object 5 at the identified position is registered in the RAM 12 or the storage unit 13 (step S67). At this time, along with the presence of the object 5, the attributes of the object 5 identified in step S65 may also be registered.

[0090] Next, the position of the pixel of the identified object 5 is output (step S68). When the position of the pixel of the object 5 is displayed on the display unit 15, it can also be displayed in a manner that makes it clear that it is a position corresponding to the object 5 on a display screen that displays a color different from the background at the pixel where the event signal is detected. After steps S61 to S68, the object detection process ends.

[0091] (Seventh embodiment) The seventh embodiment relates to an information processing device that, when an event signal is detected by a sensor capable of detecting a change in luminance of each pixel of incident light as an event signal and then the event signal is no longer detected, identifies the position of a pixel corresponding to the event signal detected just before the event signal is no longer detected. The system configuration of Fig. 1 and the information processing device configuration of Fig. 2 described above are also applied to the third embodiment.

[0092] The movement detection process will be described below. The movement detection process is mainly executed by the control unit 11 of the information processing device 3. Fig. 13 is a diagram showing a flowchart of the movement detection process according to the embodiment of the present invention.

[0093] When an object 5 existing in the real space 4 moves or moves for some reason, the event sensor 2 starts detecting an event signal (step S71). Then, when the object 5 moves or moves, the detection of the event signal ends (step S72).

[0094] Next, the position of the pixel corresponding to the event signal detected immediately before the event signal became undetectable (the event signal detected in one frame or several frames immediately before the event signal became undetectable) is identified, and the position of the pixel corresponding to the object 5 is identified from the identified positions of the multiple pixels (step S73). There are no particular limitations on the method for identifying the position of the pixel corresponding to the object 5 (position coordinates on the screen) in step S73, but since event signals are detected in multiple pixels in multiple frames, the position of the pixel corresponding to the object 5 can be identified by averaging the position coordinates of these pixels.

[0095] Next, the presence of the object 5 at the identified position is registered in the RAM 12 or the storage unit 13 (step S74). Next, the identified position of the object 5 is output (step S75). When the position of the object 5 is displayed on the display unit 15, it can also be displayed in a manner that makes it clear that it is a position corresponding to the object 5 on a display screen that displays a color different from the background in pixels that detect an event signal. Note that when the position etc. of the object 5 are transmitted to another device, the position etc. of the object 5 are also output in the other device.

[0096] When the movement of the object 5 resumes, in step S75, detection of an event signal is started again at the pixel position corresponding to the registered object (step S76). In this way, if the pixel position of the event signal detected immediately before the end of detection of the first event signal in step S72 and the pixel position of the event signal immediately after detection of the second event signal is started in step S76 (the event signal detected in one frame immediately after detection is started or in the immediately preceding few frames), match or are within a predetermined range, the first event signal and the second event signal can be treated as originating from the same object 5.

[0097] Then, when the movement of the object 5 is completed, the detection of the second event signal ends (step S77). Next, the position of the pixel corresponding to the event signal detected just before the second event signal was no longer detected, that is, the position of the pixel corresponding to the object 5, is identified (step S78).

[0098] Next, the fact that the object 5 has moved from the position registered in step S74 to the position of the identified pixel, and the position coordinates of the pixel corresponding to the object 5 after the movement are registered in the RAM 12 or the storage unit 13 (step S79). Next, the position of the identified object 5 is output (step S80). Steps S71 to S80 complete the movement detection process.

[0099] In the seventh embodiment, when a moving object 5 stops moving and then starts moving again, if the position determined based on the event signal detected just before the detection of the first event signal ends and the position determined based on the event signal when the detection of the second event signal starts match or are within a predetermined range, the first event signal and the second event signal can be treated as being caused by the same object 5.

[0100] According to the present invention, based on a signal detected by an event sensor, it is possible to identify the area in real space corresponding to the signal, thereby making it possible to identify the area in which a change is occurring in the real space that is the subject of sensing. Furthermore, based on the area of ​​real space identified as having a change, it is possible to identify an event that has occurred in the real space. A plurality of pixels correspond to one real-space region, and when the total amount of signals corresponding to one real-space region exceeds a threshold, change information indicating that a state change has occurred in the identified real-space region can be output or stored. In this way, it is possible to prevent outputting or storing information indicating that a state change has occurred even in cases where noise has occurred. In this case, by setting different threshold values ​​for different areas of the real space, it is possible to set a higher threshold value in areas where a large amount of noise occurs and a lower threshold value in areas where a small amount of noise occurs. Furthermore, in this case, by varying the threshold value according to the time in the real space, since the environment such as the brightness of the space varies depending on the time, the threshold value can be made higher during times when there is a lot of noise and lower during times when there is little noise.

[0101] According to the present invention, when a plurality of pixels of an event sensor are divided into a plurality of regions according to the positions of the pixels, by determining whether the total amount of signals corresponding to one region exceeds a threshold, it is possible to exclude signals corresponding to at least some of the regions from being subjected to predetermined processing according to the signals detected for each region. By specifying a threshold for each region based on signals detected when no changes occur in the real space that is the target of sensing, it is possible to appropriately remove noise for each region.

[0102] According to the present invention, a prediction model trained by machine learning is used, which uses a signal detected by an event sensor as input data and an event that occurs in the real space that is the target of sensing by the event sensor as output data, and it is possible to identify an event that has occurred in the real space that is the target of sensing based on a signal that detects the change in brightness of each pixel of light that is incident on the event sensor.

[0103] According to the present invention, a prediction model developed by machine learning is used, which uses a signal detected by an event sensor as input data and information on whether the state of the real space being detected by the event sensor is normal as output data, and it is possible to determine whether the state of the real space being sensed is normal based on a signal detected as a change in the brightness of each pixel of light incident on the event sensor in the real space being sensed. Furthermore, if it is determined that the state of the real space being sensed is not normal, information indicating that the state of the real space is not normal can be output, or another device can be controlled to output information indicating that the state of the real space is not normal.

[0104] According to the present invention, by generating vibrations in at least a portion of an object present in the real space to be detected by the event sensor, or in the event sensor and detecting a signal with the event sensor, it is possible to determine whether an object exists or to identify the location of the object, even if the real space being sensed is in a state where it is difficult for a person to see the object. Furthermore, by irradiating light onto at least a portion of the real space to be detected by the event sensor and detecting a signal with the event sensor, it is possible to determine whether an object exists or to identify the location of the object even if the real space being sensed is in a state where it is difficult for a person to see the object. In this case, when the vibration subsides and the signal is no longer detected, or when the light irradiation ends and the signal is no longer detected, the signal can be displayed on the display screen in a manner that makes it possible to recognize the presence of an object at the pixel position where the signal was previously detected, or the signal can be displayed on the display screen of another device in a manner that makes it possible to recognize the presence of an object. Furthermore, when the vibration subsides and a signal is no longer detected, or when the light irradiation ends and a signal is no longer detected, it is possible to register the presence of an object at the pixel position where the signal had been detected up until then. Furthermore, when light reflected and / or transmitted from an object is incident on an event sensor, the detected signal is used as input data, and the attributes of the object are used as output data, using a machine-learned predictive model to identify the attributes of the object in the real space that is the target of sensing.

[0105] According to the present invention, when an event sensor detects a signal and then the signal is no longer detected, by identifying the position of the pixel corresponding to the signal detected just before the signal was no longer detected, it is possible to identify the position or area in which the object is located in the real space of the sensing target after the object has stopped moving. In this case, the signal can be displayed on a display screen in a manner that makes it possible to recognize the presence of an object at the position of the pixel where the signal was detected, or on a display screen of another device in a manner that makes it possible to recognize the presence of an object. Furthermore, it is possible to register the presence of an object at the position of the pixel where the signal is detected. Furthermore, if a signal is detected again at the position of a pixel corresponding to a registered object or at a pixel surrounding that pixel, and then the signal is no longer detected again, it is possible to detect that the registered object has moved to the position of the pixel corresponding to the signal detected just before the signal was no longer detected again by the registration means. [Explanation of symbols]

[0106] 1 System 2 Event Sensor 3. Information processing equipment 4 Real Space 5 objects 6. Vibration generator 7 Irradiation device 11 Control section 12 RAM 13 Storage Section 14 Input section 15 Display section 16 Communication Interface

Claims

1. a detecting means for detecting a detection signal using a sensor capable of detecting a change in luminance of incident light for each pixel as a signal; a light irradiating means for irradiating light onto at least a part of the real space to be detected by the sensor; A system comprising:

2. a position specifying means for specifying a position of a pixel corresponding to an object existing in real space from a position of a pixel corresponding to a detection signal detected by the detecting means due to a change in luminance caused by irradiation of light by the light irradiating means; The system of claim 1 , comprising:

3. When the detection signal is no longer detected due to the end of irradiation of light, a display means displays on the display screen at the position of the pixel corresponding to the object in a manner that makes it possible to recognize the presence of the object, or a display control means controls to display on the display screen of another device in a manner that makes it possible to recognize the presence of the object. The system of claim 2 , comprising:

4. A registration means for registering the presence of an object at the pixel position corresponding to the object when the detection signal is no longer detected due to the end of light irradiation. The system of claim 2 , comprising:

5. An object identification means for identifying the attributes of an object present in the real space to be detected by the sensor based on the detection signal detected by the detection means, using a prediction model trained by machine learning with input data of a detection signal detected by a sensor capable of detecting a change in brightness of each pixel of the incident light as a signal and output data of the attributes of the object. The system according to claim 1 or 2, comprising:

6. a detecting step of detecting a detection signal by a sensor capable of detecting a change in luminance of incident light for each pixel as a signal; a light irradiation step of irradiating light onto at least a part of a real space to be detected by the sensor; A method comprising:

Images