Information processing apparatus, system, and method

Abstract

In the real space to be sensed, identify the area where a change is occurring. 【Solution means】 An information processing apparatus including an area specifying means for specifying an area in the real space corresponding to the detection signal based on the detection signal detected by a sensor capable of detecting a change in luminance for each pixel of the incident light as a signal. The area specifying means specifies the area in the real space corresponding to the detection signal based on the correspondence between the position of the pixel and the area in the real space. An event specifying means for specifying an event occurring in the real space is provided according to the area in the real space specified by the area specifying means.

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, a system, and a method.

Background Art

[0002] There is a sensor, so-called an event sensor, capable of detecting a change in the luminance of incident light for each pixel as a signal. Various systems using this event sensor have been studied. For example, a system is disclosed that generates an event image by collecting event signals detected for droplets ejected from a dispenser in units of a predetermined integration time, and controls the ejection of droplets by the dispenser based on the generated event image (see, for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Examples of the object of the present invention can be exemplified as follows. A first object of the present invention is to identify a region where a change is occurring in the real space being sensed. A second object of the present invention is to be able to exclude from the target of a predetermined process for each region composed of a plurality of pixels of an event sensor. A third object of the present invention is to identify an event that has occurred in the real space being sensed. A fourth object of the present invention is to determine whether the state of the real space being sensed is normal or not. The fifth object of the present invention is to determine whether an object exists or to specify the position where the object exists, even if the real space to be sensed is in a situation where it is difficult for a person to visually recognize it. The sixth object of the present invention is to specify the position where an object exists or the area where the object exists after the object has moved in the real space to be sensed.

Means for Solving the Problems

[0005] The problems of the present invention are [1] An information processing apparatus including region specifying means for specifying a region of the real space corresponding to the detection signal based on the detection signal detected by a sensor capable of detecting a change in luminance for each pixel of the incident light as a signal; [2] The information processing apparatus according to [1] above, wherein the region specifying means specifies a region of the real space corresponding to the detection signal based on the correspondence relationship between the position of the pixel and the region of the real space; [3] The information processing apparatus according to [1] or [2] above, including event specifying means for specifying an event that has occurred in the real space according to the region of the real space specified by the region specifying means; [4] When a plurality of pixels correspond to one region of the real space and the total amount of the detection signals corresponding to one region of the real space exceeds a threshold value, change information indicating that there has been a change in state in the region of the real space specified by the region specifying means is output, or information storage means for storing the change information. The information processing apparatus according to any one of [1] to [3] above; [5] The information processing apparatus according to [4] above, wherein the threshold value is different for each region of the real space; [6] The information processing apparatus according to [4] or [5] above, wherein the threshold value is different according to the time of the real space; [7] A method executed in an information processing apparatus, the method having a region specifying step of specifying a region of the real space corresponding to the detection signal based on the detection signal detected by a sensor capable of detecting a change in luminance for each pixel of the incident light as a signal; [8] When a plurality of pixels of a sensor capable of detecting changes in the luminance of incident light for each pixel as signals are divided into a plurality of regions according to the positions of the pixels, determination means for determining whether or not the total amount of signals corresponding to one region exceeds a threshold value, the threshold value being different for each region, an information processing apparatus; [9] Threshold value specifying means for specifying a threshold value for each region based on the detection signal detected by the sensor in a state where no change has occurred in the real space of the detection target by the sensor, and the determination means determines based on the threshold value specified by the threshold value specifying means, the information processing apparatus according to [8] above;

[10] A method executed in an information processing apparatus, having a determination step of determining whether or not the total amount of signals corresponding to one region exceeds a threshold value when a plurality of pixels of a sensor capable of detecting changes in the luminance of incident light for each pixel as signals are divided into a plurality of regions according to the positions of the pixels, the threshold value being different for each region, a method;

[11] An information processing apparatus including event specifying means for specifying an event that has occurred in the real space based on a detection signal detected by a sensor that can detect changes in the luminance of incident light for each pixel as signals, using a prediction model that has been machine-learned with the detection signal detected by the sensor as input data and the event that has occurred in the real space of the detection target by the sensor as output data;

[12] A method to be executed in an information processing apparatus, having an event specifying step of specifying an event that has occurred in the real space based on a detection signal detected by a sensor that can detect changes in the luminance of incident light for each pixel as signals, using a prediction model that has been machine-learned with the detection signal detected by the sensor as input data and the event that has occurred in the real space of the detection target by the sensor as output data;

[13] Using the detection signal detected by a sensor capable of detecting the change in luminance for each pixel of the incident light as input data, and using a prediction model that has been machine-learned with information regarding whether the state of the real space of the detection target by the sensor is normal as output data, based on the detection signal obtained by detecting the change in luminance for each pixel of the light incident on the sensor as a signal, there is provided an information processing apparatus comprising determination means for determining whether the state of the real space of the detection target by the sensor is normal;

[14] When it is determined by the determination means that the state of the real space is not normal, there is provided the information processing apparatus according to

[13] above, comprising output means for outputting information indicating that the state of the real space is not normal, or output control means for controlling, in another apparatus, to output information indicating that the state of the real space is not normal;

[15] Using the detection signal detected by a sensor capable of detecting the change in luminance for each pixel of the incident light as input data, and using a prediction model that has been machine-learned with information regarding whether the state of the real space of the detection target by the sensor is normal as output data, based on the detection signal obtained by detecting the change in luminance for each pixel of the light incident on the sensor as a signal, there is provided a method having a determination step of determining whether the state of the real space of the detection target by the sensor is normal;

[16] There is provided a system comprising detection means for detecting a detection signal by a sensor capable of detecting the change in luminance for each pixel of the incident light, and at least a part of an object existing in the real space of the detection target by the sensor, or vibration generating means for generating vibration in the sensor;

[17] There is provided a system comprising detection means for detecting a detection signal by a sensor capable of detecting the change in luminance for each pixel of the incident light, and light irradiation means for irradiating light to at least a part of the real space of the detection target by the sensor;

[18] Based on the position of a pixel corresponding to the detection signal detected by the detection means due to a change in luminance caused by the vibration generated by the vibration generating means, or based on the position of a pixel corresponding to the detection signal detected by the detection means due to a change in luminance caused by the irradiation of light by the light irradiation means, there is provided the system according to

[16] or

[17] above, comprising position specifying means for specifying the position of a pixel corresponding to an object existing in the real space;

[19] When the detection signal is no longer detected due to the vibration subsiding, or when the detection signal is no longer detected due to the end of light irradiation, a display means for displaying on the display screen in a manner that enables the presence of the object to be grasped at the position of the pixel corresponding to the object, or a display control means for controlling to display on the display screen of another device in a manner that enables the presence of the object to be grasped, the system according to

[18] above;

[20] When the detection signal is no longer detected due to the vibration subsiding, or when the detection signal is no longer detected due to the end of light irradiation, a registration means for registering the presence of the object at the position of the pixel corresponding to the object, the system according to

[18] or

[19] above;

[21] Using, as input data, a detection signal detected by having light reflected and / or transmitted from an object incident on a sensor capable of detecting a change in luminance for each pixel of the incident light as a signal, and using a prediction model machine-learned with the attribute of the object as output data, an object identification means for identifying the attribute of the object existing in the real space of the detection target by the sensor based on the detection signal detected by the detection means, the system according to any one of

[16] to

[20] above;

[22] A method having: a detection step of detecting a detection signal with a sensor capable of detecting a change in luminance for each pixel of the incident light as a signal; and a vibration generation step of generating vibration in at least a part of the object in the real space to be detected by the sensor, or in the sensor;

[23] A method having: a detection step of detecting a detection signal with a sensor capable of detecting a change in luminance for each pixel of the incident light as a signal; and a light irradiation step of irradiating light on at least a part of the real space of the detection target by the sensor;

[24] An information processing apparatus comprising a position identification means for identifying the position of the pixel corresponding to the detection signal detected immediately before the detection signal ceases to be detected when the detection signal that has been detected by a sensor capable of detecting a change in luminance for each pixel of the incident light as a signal ceases to be detected;

[25] Display means for displaying on a display screen in a manner that enables the detection of the presence of an object at the position of the pixel specified by the position specifying means, or display control means for controlling to display on the display screen of another device in a manner that enables the detection of the presence of an object at the position of the pixel specified by the position specifying means, the information processing apparatus according to the above

[24] ;

[26] The information processing apparatus according to the above

[24] or

[25] , comprising registration means for registering the presence of an object at the position of the specified pixel;

[27] After the detection signal is detected again at the position of the pixel corresponding to the object registered by the registration means, when the detection signal is no longer detected again, the object registered by the registration means is moved to the position of the pixel corresponding to the detection signal detected immediately before the detection signal is no longer detected. The information processing apparatus according to the above

[26] ;

[28] A method having a position specifying step of specifying the position of a pixel corresponding to a detection signal detected immediately before the detection signal that has been detected is no longer detected, when the detection signal that has been detected is no longer detected by a sensor capable of detecting a change in luminance of incident light for each pixel as a signal; It can be solved by.

Advantages of the Invention

[0006] As the effects of the present invention, for example, the following can be exemplified. The first effect of the present invention is that it is possible to specify a region where a change is occurring in the real space that is the object of sensing. The second effect of the present invention is that it is possible to exclude from the object of a predetermined process for each region composed of a plurality of pixels of the event sensor. The third effect of the present invention is that it is possible to specify an event that has occurred in the real space that is the object of sensing. The fourth effect of the present invention is that it is possible to determine whether the state of the real space that is the object of sensing is normal. A fifth object of the present invention is to determine whether an object exists or to identify the position where the object exists, even when the real space to be sensed is in a situation where it is difficult for a person to visually recognize it. A sixth object of the present invention is to be able to identify the position where an object exists or the area where the object exists after the object has moved in the real space to be sensed.

Brief Description of the Drawings

[0007]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Best Mode for Carrying Out the Invention

[0008] Hereinafter, embodiments of the present invention will be described with reference to the drawings. However, the present invention is not limited to the following embodiments as long as it does not conflict with the gist of the present invention. Also, the description of the effects is one aspect of the effects of the embodiments of the present invention and is not limited to what is described here.

[0009] (First Embodiment) The first embodiment relates to an information processing apparatus that identifies a region in the real space corresponding to a signal or identifies an event that has occurred in the real space, based on a signal detected by a sensor (i.e., an event sensor) capable of detecting a change in the luminance of incident light for each pixel as a signal.

[0010] FIG. 1 is a diagram showing the configuration of a system according to an embodiment of the present invention. The system 1 in FIG. 1 includes an event sensor 2 and an information processing apparatus 3. An arbitrary object 5 may exist in the real space 4 that is the target of sensing by the event sensor 2.

[0011] The event sensor 2 is also called an EVS (event-based vision sensor). The event sensor 2 converts incident light into an electrical signal at the light-receiving part of the sensor and outputs the converted electrical signal as an event signal. The event sensor 2 is equipped with a camera, and each pixel of the camera includes a light-receiving part that receives incident light and a detection part that detects changes in luminance. When the change in the luminance of the light incident on each pixel exceeds a set threshold value, it is detected as an event signal, and the position coordinates of the pixel where the event signal occurred, the time when the event signal occurred, and the polarity (whether it is a light-on signal or a light-off signal) can be output. The position coordinates of the pixel where the event signal occurred can be represented, for example, in an xy coordinate system aligned with the up-down, left-right directions of the event sensor 2. Also, on the display screen of the display unit 15 of the information processing device 3 described later, it is possible to display that an event signal has been detected corresponding to the position coordinates in a manner that shows which pixel of the event sensor 2 the event signal was detected at.

[0012] Normally, the event sensor 2 is installed without changing its position or orientation. The camera of the event sensor 2 faces the same position and the same direction even as time passes, and the real space 4 to be sensed is also fixed.

[0013] When an arbitrary object 5 moves in the real space 4, due to the movement of the object 5, the luminance of the light incident on the event sensor 2 changes, and an event signal is detected in the event sensor 2. When the object 5 is in a stationary state, no event signal is detected. The event sensor 2 can detect an event signal even in a state where it is difficult for a person to visually recognize the presence or movement of an object due to darkness such as at night or backlight, and can detect changes in the situation such as changes in the object 5 in the real space 4.

[0014] The information processing apparatus 3 executes various information processes based on the event data output from the event sensor 2. FIG. 2 is a diagram showing the configuration of the information processing apparatus according to an embodiment of the present invention. The information processing apparatus 3 includes a control unit 11, a RAM 12, a storage unit 13, an input unit 14, a display unit 15, and a communication interface 16, which are connected by a bus respectively.

[0015] The control unit 11 is composed of a CPU and a ROM. The control unit 11 executes the program stored in the storage unit 13 to control the information processing apparatus 3. The RAM 12 is a work area of the control unit 11. The storage unit 13 is a storage medium for storing programs and data. The control unit 11 performs arithmetic processing based on the event signal received from the event sensor 2, the programs and data read from the RAM 12, and / or the data input at the input unit 14.

[0016] The display unit 15 has a display screen. The control unit 11 outputs a video signal for displaying an image on the display screen according to the result of the arithmetic processing. Here, the display screen of the display unit 15 may be a touch panel provided with a touch sensor. In this case, the touch panel functions as the input unit 14. Note that the output of the result of the arithmetic processing from the information processing apparatus 3 is not limited to the display on the display screen of the display unit 15, and may be the output of sound by a sound output unit (not shown) or the transmission of information to other devices via the communication interface 16.

[0017] The communication interface 16 can be connected to a communication network wirelessly or by wire, and the information processing apparatus 3 can transmit and receive data to and from other devices via the communication network. The other devices are not particularly limited as long as they can receive information and output the received information. For example, similar to the information processing apparatus 3, they may include a control unit, a RAM, a storage unit, an input unit, a sound output unit, a communication interface, etc.

[0018] The real space 4 may be either indoor or outdoor and is not particularly limited. The real space 4 can be appropriately selected according to its use.

[0019] The object 5 may be any tangible object other than a human, an animal, or a living thing, and is not particularly limited. The tangible object other than a living thing may have a function of moving like a vehicle such as an automobile, or may not have a function of moving like furniture, etc., and is not particularly limited.

[0020] Next, the event identification process will be described. The event identification process is executed by the information processing device 3. FIG. 3 is a diagram showing a flowchart of the event identification process according to an embodiment of the present invention.

[0021] The information processing device 3 receives an event signal from the event sensor 2 (step S11). Based on the position of the pixel corresponding to the received event signal, the area in the real space is identified (step S12). The correspondence relationship between the position of the pixel and the area in the real space is determined in advance, and a plurality of pixels correspond to one area in the real space. According to this correspondence relationship, the area in the real space is identified. Detecting an event signal in any one of the plurality of pixels corresponding to one area means that some change has occurred in that area. When the detection of the event signal continues in the pixels corresponding to one area, it means that the change is continuing to occur in that area.

[0022] Next, according to the identified region of the real space, identify the event that occurred in the real space 4 (step S13). Here, the identification of the event is a concept that includes estimating that the event would occur. For each event, in step S12, conditions are predetermined as to which event is to be identified as having occurred when a certain region is identified, and step S13 is executed according to these conditions. FIG. 5 is a diagram showing a table representing the correspondence between regions and events according to an embodiment of the present invention. In step S13, the event can be identified by referring to the table shown in FIG. 5. Note that the conditions for identifying which event is to be identified as having occurred when a certain region is identified also vary depending on the layout of the object 5 in the real space 4, the position and orientation of the event sensor 2, and thus need to be set appropriately according to the layout of the object 5, the position and orientation of the event sensor 2.

[0023] In step S13, for example, when a change occurs in region A (that is, when region A is identified), it can be identified that event α has occurred, and when a change occurs in region B, it can be identified that event β has occurred. Further, after a change occurs in region A, if a change occurs in region B within a predetermined time, it may be identified that event γ has occurred. That is, not only identify the event according to the region where the change occurred, which was identified in step S12, but also when changes occur in a plurality of regions according to the passage of time, the event may be identified according to the order in which the changes occurred, the time from when a change occurred in one region until a change occurs in another region after the change in the one region has started or ended, or until the change in the other region ends, or the length of time from when a change occurs until it ends in one region. Thus, the event may be identified according to the identified region and the time element related to the change.

[0024] For example, assume that the real space to be sensed is a situation where a patient is admitted to the hospital and is sleeping in bed at night. When using an event sensor, unlike monitoring with a normal camera, even in a state where the lights are turned off at night and it is close to darkness, changes in the room can be detected, and since the patient's condition is not detailed in the video, it also leads to the protection of the patient's privacy. For example, if there is a change in the area "bed" and then no change in other areas, it can be determined that the patient sleeping in the bed has turned over or the like. Also, if there is a change in the area "bed", then a change in the area "floor", and then no change in other areas, it can be determined that the patient who was sleeping in the bed has fallen to the floor. Further, if there is a change in the area "bed", then a change in the area "floor", and then a change in the area "door", it can be determined that the patient who was sleeping in the bed has opened the door and moved outside the room.

[0025] Note that the area of the real space associated with a plurality of pixels does not have to match the object that exists when a straight line is extended on the optical axis from each pixel in the real space. For example, an area where a change in luminance may occur when the patient performs some action on the bed can be set as the area "bed". In this way, the area can be set based on the assumed movement or movement range of people and objects.

[0026] Next, it is output that the event specified in step S13 has occurred (step S14). The output in step S13 includes display on the display unit 15 of the information processing device 3, audio output, or transmission to another device. When the specified event is transmitted to another device, the other device can output it by methods such as displaying it on the display unit or outputting it as audio. Hereinafter, in other embodiments as well, the "output" in the information processing device 3 and other devices is the same.

[0027] In the real space 4, as a history indicating that an event has occurred, change information is stored in association with the time when the event signal was detected (step S15).

[0028] In parallel with the processing of steps S13 to S15, the processing of steps S16 to S17 can also be executed. Or, only one of them can be executed.

[0029] When the total amount of event signals corresponding to the area of 1 real space per unit time exceeds the threshold value, in the area of the real space specified in step S12, change information indicating that a state change has occurred is output (step S16). In the real space 4, as a history indicating that a change has occurred, change information is stored in association with the time when the event signal was detected (step S17). The event identification process ends by steps S11 to S15, or S11, S12, S16, and S17.

[0030] Also, different threshold values can be used for each area. For example, in the area "bed", since patients sleeping at night often move their bodies, a large threshold value is set. In the area "floor", since the frequency of changes is less than that in the area "bed", a small threshold value can be set.

[0031] Furthermore, even for the same area, different threshold values can be used according to the time in the real space. "Using different threshold values according to the time" is a concept that includes not only the case of using different threshold values according to the time but also the case of using different threshold values according to time zones, days, weeks, months, or seasons, etc. For example, between the late-night time zone and the early morning time zone, since the movement of patients during sleep is different, the threshold value for the late-night time zone can be set higher than that for the early morning.

[0032] Note that while using different threshold values for each area, different threshold values can also be used according to the time even for the same area.

[0033] The information processing apparatus 3 according to the first embodiment can be used for applications such as monitoring and protecting inpatients and the elderly, and watching over people in need of care in medical institutions, nursing facilities, and each household. Even if the real space 4 to be sensed is in a situation where visual recognition is difficult, such as at night, the actions and behaviors can be specified.

[0034] (Second Embodiment) The second embodiment relates to an information processing apparatus that determines whether the total amount of signals corresponding to one region exceeds a threshold value when a plurality of pixels of an event sensor are divided into a plurality of regions according to the positions of the pixels. The configurations of the system in FIG. 1 and the information processing apparatus in FIG. 2 described above are also applicable in the second embodiment.

[0035] Hereinafter, the determination process will be described. The determination process is executed by the control unit 11 of the information processing apparatus 3. FIG. 6 is a diagram showing a flowchart of the determination process according to the embodiment of the present invention.

[0036] In the second embodiment, a plurality of regions (a plurality of groups) are set according to the positions of a plurality of pixels that receive light with the event sensor. Note that the "region" in the second embodiment is a different concept from the "region of the real space" in the first embodiment, but it may have a correspondence relationship with the region of the real space in the first embodiment, or may not have a correspondence relationship with the region of the real space.

[0037] The information processing apparatus 3 first sets a threshold value (step S21). The threshold value is different for each area. For example, for the event signal detected by the event sensor 2, since there are areas with a lot of noise and areas with little noise, it is preferable to set the threshold value for each area. As a method for setting the threshold value, in a state where no change has occurred in the real space, the threshold value can be set for each area based on the detected event signal. For example, from the total amount of event signals detected for each area per predetermined time for each area, the average amount of event signals per frame for each area can be obtained, and based on this average amount of event signals, the threshold value per frame for each area can be set. The threshold value is preferably a value larger than this average amount of event signals. For example, it can be a value obtained by multiplying the average amount by a coefficient larger than 1. The detection of the event signal for setting the threshold value is preferably immediately after starting the operation of the event sensor 2 in the real space 4 that is the object of sensing.

[0038] When the threshold value is set, sensing of the real space 4 is started in the event sensor 2. For example, when a change such as the movement of the object 5 existing in the real space 4 occurs, an event signal is detected (step S22). Based on the detected event signal, it is determined whether or not the total amount of the signals corresponding to each area exceeds the threshold value (step S23). For the area that exceeds the threshold value, a predetermined process is executed (step S24). The predetermined process is not particularly limited. For example, using the event signal corresponding to this area or the information indicating that this area has exceeded the threshold value, a process such as a predetermined calculation or output is executed. As the predetermined process, for example, a process of displaying on the display screen of the display unit 15 or the like in such a manner that it can be understood that there has been a change in the state in the area that exceeds the threshold value can be exemplified. On the other hand, for the area that does not exceed the threshold value, it is not the target of executing the predetermined process. By executing steps S21 to S24, the determination process ends.

[0039] The determination process described in the second embodiment can also be executed in other embodiments (for example, the first embodiment, the third embodiment to the seventh embodiment).

[0040] Above, it was described that by setting a threshold value, noise in the event signal can be excluded and a predetermined process can be executed. However, instead of excluding noise, among the changes that occurred in the real space 4, an area larger than the threshold value can be excluded to execute a predetermined process, or an area smaller than the threshold value can be excluded to execute a predetermined process. For example, in the case of the area related to the bed, when it is desired to exclude the detection of an event signal based on the breathing of a person sleeping on the head and detect only the event signal based on the person turning over, a threshold value can be set such that only the event signal based on the breathing of the person sleeping on the head can be detected, and the detection of an event signal that does not reach the threshold value can be excluded from the target of the predetermined process.

[0041] (Third Embodiment) The third embodiment relates to an information processing apparatus that uses a prediction model learned by machine learning with the event signal detected by an event sensor as input data and the event that occurred in the real space to be detected as output data, and identifies the event that occurred in the real space based on the event signal detected by the event sensor in the real space to be sensed. The configurations of the system in FIG. 1 and the information processing apparatus in FIG. 2 described above are also applicable in the third embodiment.

[0042] Hereinafter, the event identification process will be described. The event identification process is executed by the control unit 11 of the information processing apparatus 3. FIG. 7 is a diagram showing a flowchart of the event identification process according to an embodiment of the present invention.

[0043] First, based on the event signal detected at a predetermined time, the occurred event is identified (step S31). The length of the predetermined time can be appropriately designed according to the real space 4 to be sensed and the event to be identified.

[0044] For identifying the event that occurred in step S31, a prediction model learned through machine learning is utilized based on the following input data and output data. Specifically, in the real space, when an event occurs, information regarding this occurred event is used as the output data, and during the occurrence of this event, an event signal detected by sensing the real space where this event is occurring with an event sensor can be used as the input data.

[0045] The prediction model used in step S31 is preferably such that in a real space that is substantially the same as the real space 4 which is the target of the event identification process (that is, not only the same real space, but also including real spaces with the same area and the same configuration of object arrangements but different locations), with the event sensor installed at a position and in an orientation that are substantially the same as those in the event identification process, the event signal detected by the event sensor is used as the input data.

[0046] In step S31, using this prediction model, in the real space that is the target of sensing, the event that occurred can be identified based on the event signal detected by the event sensor 2.

[0047] The algorithm of machine learning is not particularly limited, and known algorithms can be used. However, it is preferable to use deep learning using a multi-layer neural network. A multi-layer neural network has an input layer, an output layer, and a plurality of intermediate layers. Weights are set for the edges connecting the nodes of each layer. For each edge, weights corresponding to each input to the node are set, and the values obtained by multiplying these weights are added to the bias. The value obtained by the addition is non-linearly transformed using an activation function to calculate the activation value. The calculated activation value becomes the value of the input passed to the nodes of the next layer. The number of intermediate layers can be designed as appropriate.

[0048] In step S31, the event to be specified may be any change in the state in the real space 4 that can cause a change in the luminance of the light incident on the event sensor 2, such as the movement of a person or an animal or the movement of other objects, and is not particularly limited. When the real space 4 to be sensed is a hospital ward, events such as "the patient turned over in bed", "the patient fell from the bed to the floor", and "the patient got out of the bed and moved to another room" can be specified.

[0049] When an event is specified in step S31, in the information processing device 3, the specified event is output in a manner that can be grasped by a person (step S32).

[0050] Next, the information processing device 3 associates the specified event with the time when the event occurred (for example, the time when the detection of the event signal corresponding to the event started) or the time when the event is occurring (for example, the time when the detection of the event signal corresponding to the event started and the time when the detection of the event signal ended) as the event occurrence history and stores it in the storage unit 13 (step S33).

[0051] The event specification process ends by steps S31 to S33. The processes of steps S31 to S33 are repeatedly executed at predetermined time intervals.

[0052] Similar to the first embodiment, the information processing device 3 according to the third embodiment can be used in medical institutions, nursing facilities, and each household for applications such as monitoring, protecting inpatients and the elderly, and watching over people in need of care. Even when the real space 4 to be sensed is in a situation where visual recognition is difficult, such as at night, movements and behaviors can be specified.

[0053] (Fourth Embodiment) The fourth embodiment relates to an information processing apparatus that uses, as input data, an event signal detected by an event sensor, and uses a prediction model learned by machine learning with information regarding whether the state of the real space to be detected by the event sensor is normal as output data, and determines whether the state of the real space is normal based on the event signal detected by the event sensor in the real space to be sensed.

[0054] Here, the information regarding whether it is normal is a concept that includes not only information meaning "normal" and information meaning "not normal", but also information meaning "abnormal" and information meaning "not abnormal".

[0055] The configuration of the system in FIG. 1 and the configuration of the information processing apparatus in FIG. 2 described above are also applicable in the fourth embodiment.

[0056] Hereinafter, the determination process will be described. The determination process is executed by the control unit 11 of the information processing apparatus 3. FIG. 8 is a diagram showing a flowchart of the event identification process according to the embodiment of the present invention.

[0057] First, based on the event signal detected at a predetermined time, it is determined whether the state of the real space is normal (step S41). The length of the predetermined time can be appropriately designed according to the real space 4 to be sensed and the event to be specified.

[0058] In step S41, to determine whether the state of the real space is normal, a prediction model learned by machine learning is used based on the following input data and output data. Specifically, when the real space is in a normal state, the event signal detected by sensing the real space with an event sensor can be used as the input data, and the information indicating "normal" can be used as the output data. Also, when the real space is in an abnormal state, the event signal detected by sensing the real space with an event sensor can be used as the input data, and the information indicating "abnormal (or not normal)" can be used as the output data. Furthermore, a prediction model learned by machine learning can also be used based on the input data and output data corresponding to each of the case where the real space is in a normal state and the case where the real space is in an abnormal state.

[0059] The prediction model used in step S41 preferably uses, as the input data, the event signal detected by the event sensor in a state where the event sensor is installed at a position substantially the same as the position and in the same orientation as the position in the determination process in a real space substantially the same as the real space 4 (i.e., not only the same real space but also a real space having the same configuration with the same area and the same arrangement of objects but different in location) that is the target of the determination process.

[0060] In the output data of the prediction model, the method for determining whether the real space is "normal" or "not normal" is not particularly limited. For example, it may be determined based on the result of a person visually judging whether the real space is normal. In the output data of the prediction model, the method for determining whether the real space is "abnormal" or "not abnormal" is not particularly limited. For example, it may be determined based on the result of a person visually judging whether the real space is abnormal. In this way, based on the result of a person's visual judgment, by determining whether to make the output data "normal" or "not normal" (or whether to make it "abnormal" or "not abnormal"), the determination in step S41 can also be made in accordance with the visual judgment of a person. It is preferable that the determination of whether the state of the real space, which is the output data of the prediction model, is normal or not is made in accordance with predetermined conditions, predetermined criteria, and / or predetermined rules.

[0061] For example, when the flow of people moving within a predetermined facility (e.g., within the precincts of a station) is targeted for sensing by an event sensor, a state where most people walking within the precincts of the station are moving in the same direction without going against the flow of people's movement can be regarded as a normal state, and a state where some people are moving against the flow of people's movement can be regarded as a non-normal state. By using a prediction model that specifies the output data based on such criteria, it is also possible to determine that a state where some people are moving against the flow of people's movement in the real space that is the target of sensing is not normal.

[0062] In addition, when the flow of vehicles moving on a road is targeted for sensing by an event sensor, a state where vehicles running on the road are moving at a speed equal to or lower than a predetermined speed in the traveling direction determined for each road is regarded as a normal state, and a state where some vehicles are moving in the direction opposite to the traveling direction or a state where they are moving at a speed exceeding the predetermined speed can be regarded as an abnormal state. The criterion for whether it is normal or abnormal is not particularly limited, and it can be appropriately designed according to the real space to be sensed and the type of abnormality for which detection is required.

[0063] In step S41, using such a prediction model, it is possible to determine whether the real space is normal or not based on the event signal detected by the event sensor 2 in the real space that is the target of sensing.

[0064] The algorithm of machine learning is not particularly limited, and a known one can be used, but it is preferable to use deep learning using a multi-layer neural network. The multi-layer neural network can have the same configuration as in the third embodiment.

[0065] In step S41, when it is determined that the state of the real space 4 is normal (YES in step S42), the determination process ends. On the other hand, in step S41, when it is determined that the state of the real space 4 is abnormal (NO in step S42), information indicating that the state of the real space 4 is not normal or is abnormal is output (step S43).

[0066] Next, the information processing device 3 associates the time when the abnormality occurred (for example, the time when the detection of the event signal corresponding to the occurrence of the abnormality started) or the time during which the abnormality has occurred (for example, the time when the detection of the event signal corresponding to the occurrence of the abnormality started and the time when the detection of the event signal ended) as the history of the occurrence of the abnormality, and stores it in the storage unit 13 (step S44).

[0067] Steps S41 to S44 end the determination process. The determination process of steps S41 to S44 is repeatedly executed at predetermined time intervals.

[0068] The information processing apparatus 3 according to the fourth embodiment can be used, for example, to determine whether the flow of moving objects such as people and vehicles is normal on a road where many people and vehicles are moving or in a facility where many people are moving.

[0069] (Fifth Embodiment) The fifth embodiment relates to a system that detects an event signal with an event sensor and detects at least a part of an object existing in the real space of the detection target by the event sensor, or an event signal that can be detected by causing the event sensor to generate vibration. The configuration of the information processing apparatus in FIG. 2 described above is also applicable in the fifth embodiment. In the fifth embodiment, instead of FIG. 1, the configuration of the system shown in FIG. 9 can be applied.

[0070] FIG. 9 is a diagram showing the configuration of a system according to an embodiment of the present invention. The system 1 in FIG. 9 includes an event sensor 2 and an information processing apparatus 3. The system 1 also includes a vibration generator 6 for generating vibration in the event sensor 2 and / or the object 5. Any object 5 may exist in the real space 4 that is the sensing target of the event sensor 2.

[0071] Generally, when an arbitrary object 5 moves in the real space 4, due to the movement of the object 5, the luminance of the light incident on the event sensor 2 changes, and an event signal is detected in the event sensor 2. When the object 5 is in a stationary state, no event signal is detected. Therefore, in a state where it is difficult for a person to visually recognize the presence or movement of an object due to darkness such as at night or backlight, if the object 5 is stationary, it is impossible to detect the existence of the object 5. However, by the vibration generator 6, for a short time (for example, a short time such as 0.1 seconds), vibrations that do not significantly change the state of the real space 4 (for example, to the extent that the positions and orientations of the event sensor 2 and the object 5 do not change) are applied to the event sensor 2 and / or the object 5, so that even if the object 5 was stationary, it becomes possible to detect an event signal. Note that the length of time for generating vibrations and the magnitude of the vibrations are not particularly limited and can be appropriately set according to the object 5 assumed to exist in the real space 4.

[0072] The real space 4 may be either indoors or outdoors and is not particularly limited. The real space 4 can be appropriately selected according to its use. The object 5 may be any corporeal object other than a person, an animal, or a living thing and is not particularly limited. Corporeal objects other than living things may have a function of moving like a vehicle such as an automobile or may not have a function of moving like furniture and are not particularly limited.

[0073] Hereinafter, the object detection process will be described. The object detection process is mainly executed by the control unit 11 of the information processing device 3 and the vibration generator 6. FIG. 10 is a diagram showing a flowchart of the object detection process according to an embodiment of the present invention.

[0074] First, the information processing device 3 transmits a vibration generation request to the vibration generation device 6 to generate vibration (step S51). The vibration generation request is transmitted at regular intervals. When the vibration generation device 6 receives the vibration generation request (step S52), the vibration generation device 6 generates slight vibration for a short time (step S53). The generated vibration is also transmitted to the object 5 or the event sensor 2 existing in the real space 4. When the vibration is transmitted to the object 5 or the event sensor 2 in this way, the event sensor 2 can receive the light reflected or transmitted by the object 5 and detect an event signal based on the change in the luminance of the incident light.

[0075] When a change occurs in the luminance of the light incident on the event sensor 2 due to the vibration generated by the vibration generation device 6, an event signal is detected (step S54). Next, the attribute of the object 5 corresponding to the detected event signal (what kind of object it is) is specified (step S55). For specifying the attribute of the object 5 in step S55, a machine-learned prediction model can be used. The prediction model uses, as input data, an event signal detected when the light reflected and / or transmitted from an object is incident on a sensor capable of detecting a change in the luminance of the incident light for each pixel as an event signal, and can use a prediction model machine-learned with the attribute of the object as output data. Here, the attribute of the object is not particularly limited as long as it can specify, for example, the name, use, material, properties, etc. of the object.

[0076] The machine learning algorithm is not particularly limited, and a known algorithm can be used, but it is preferable to use deep learning using a multi-layer neural network. The multi-layer neural network can have the same configuration as in the third embodiment.

[0077] Next, identify the positions of the pixels corresponding to the detected event signals, and from the positions of the identified plurality of pixels, identify the positions (pixel position coordinates) of the pixels corresponding to the object 5 (step S56). The method for identifying the positions of the pixels corresponding to the object 5 in step S56 is not particularly limited. However, since event signals are detected in a plurality of pixels in a plurality of frames, the positions of the pixels corresponding to the object 5 can be identified by taking the average of the position coordinates of these pixels.

[0078] Next, register the presence of the object 5 at the identified position in the RAM 12 or the storage unit 13 (step S57). At this time, together with the presence of the object 5, the attributes of the object 5 identified in step S55 may be registered.

[0079] Next, the positions of the pixels of the identified object 5 are output (step S58). When displaying the positions of the pixels of the object 5 on the display unit 15, it can also be displayed in a manner that allows it to be recognized that the positions are those corresponding to the object 5 in a display screen that displays a color different from the background for the pixels where the event signals are detected. The object detection process ends in steps S51 to S58.

[0080] (Sixth Embodiment) The sixth embodiment relates to a system that irradiates at least a part of the real space that is the object of sensing with light and detects an event signal with an event sensor in at least a part of the real space that is the object of detecting the event signal. The configuration of the information processing apparatus in FIG. 2 described above is also applicable in the sixth embodiment. In the sixth embodiment, instead of FIG. 1, the configuration of the system shown in FIG. 11 can be applied.

[0081] FIG. 11 is a diagram showing the configuration of a system according to an embodiment of the present invention. The system 1 in FIG. 11 includes an event sensor 2 and an information processing apparatus 3. Further, the system 1 includes an irradiation device 7 for irradiating at least a part of the real space 4 with light. An arbitrary object 5 may exist in the real space 4 that is the object of sensing by the event sensor 2.

[0082] Generally, when an arbitrary object 5 moves in the real space 4, due to the movement of the object 5, the luminance of the light incident on the event sensor 2 changes, and an event signal is detected by the event sensor 2. When the object 5 is in a stationary state, no event signal is detected. Therefore, in a state where it is difficult to visually recognize the presence or movement of an object by a person due to darkness such as at night or backlighting, if the object 5 is stationary, it is impossible to detect the existence of the object 5. However, by irradiating at least a part of the real space 4 temporarily (for example, in a short time such as 0.1 seconds) with the irradiation device 7, even if the object 5 is stationary, it becomes possible to detect an event signal with the event sensor 2. Note that the length of the irradiation time by the irradiation device 7 and the illuminance of the irradiated light are not particularly limited. Also, the light irradiation by the irradiation device 7 may be direct irradiation on the object 5, or may be indirect irradiation on the object 5 by irradiating a part of the real space.

[0083] The real space 4 may be either indoors or outdoors and is not particularly limited. The real space 4 can be appropriately selected according to its use. The object 5 may be any corporeal object other than a person, an animal, or a living thing and is not particularly limited. The corporeal object other than a living thing may have a function of moving such as a vehicle like an automobile, or may not have a function of moving such as furniture and is not particularly limited.

[0084] Hereinafter, the object detection process will be described. The object detection process is mainly executed by the control unit 11 of the information processing device 3 and the irradiation device 7. FIG. 12 is a diagram showing a flowchart of the object detection process according to an embodiment of the present invention.

[0085] First, the information processing apparatus 3 transmits an irradiation request to the irradiation apparatus 7 to irradiate at least a part of the real space 4 (step S61). The irradiation request is transmitted at predetermined time intervals. When the irradiation apparatus 7 receives the irradiation request (step S62), the irradiation apparatus 7 temporarily irradiates the real space 4 with light (step S63). As a result, it becomes possible to receive, by the event sensor 2, the light reflected or transmitted by the object 5 and detect an event signal based on the change in the luminance of the incident light.

[0086] When a change occurs in the luminance of the light incident on the event sensor 2, an event signal is detected (step S64). Next, the attribute of the object 5 (what kind of object it is) corresponding to the detected event signal is specified (step S65). For specifying the attribute of the object 5 in step S65, a learned prediction model can be used. The prediction model uses, as input data, an event signal detected when light reflected and / or transmitted from an object is incident on a sensor capable of detecting a change in the luminance of the incident light for each pixel as an event signal, and uses, as output data, the attribute of the object, a prediction model learned by machine learning. Here, the attribute of the object is not particularly limited as long as it can specify, for example, the name, use, material, properties, etc. of the object.

[0087] The algorithm of machine learning is not particularly limited, and a known algorithm can be used, but it is preferable to use deep learning using a multi-layer neural network. Regarding the details of the multi-layer neural network, the content described in the third embodiment can be applied.

[0088] Next, the position of the pixel corresponding to the detected event signal is specified, and from the positions of the plurality of specified pixels, the position (pixel position coordinates) of the pixel corresponding to the object 5 is specified (step S66). The method for specifying the position of the pixel corresponding to the object 5 in step S66 is not particularly limited, but since event signals are detected for a plurality of pixels in a plurality of frames, the position of the pixel corresponding to the object 5 can be specified by taking the average of the position coordinates of these pixels.

[0089] Next, it is registered in the RAM 12 or the storage unit 13 that the object 5 exists at the specified position (step S67). At this time, together with the existence of the object 5, the attributes of the object 5 specified in step S65 may be registered.

[0090] Next, the position of the pixels of the specified object 5 is output (step S68). When displaying the position of the pixels of the object 5 on the display unit 15, it can also be displayed in a manner that allows it to be recognized that it is the position corresponding to the object 5 in the display screen that displays a color different from the background for the pixels where the event signal is detected. The object detection process ends in steps S61 to S68.

[0091] (Seventh Embodiment) In the seventh embodiment, after an event signal is detected by a sensor capable of detecting a change in luminance for each pixel of the incident light as the event signal, when the event signal stops being detected, the position of the pixel corresponding to the event signal detected immediately before the event signal stops being detected is specified. This relates to an information processing apparatus. The configuration of the system in FIG. 1 and the configuration of the information processing apparatus in FIG. 2 described above are also applicable in the third embodiment.

[0092] Hereinafter, the movement detection process will be described. The movement detection process is mainly executed by the control unit 11 of the information processing apparatus 3. FIG. 13 is a diagram showing a flowchart of the movement detection process according to an embodiment of the present invention.

[0093] If, for some reason, the object 5 existing in the real space 4 operates or moves, the detection of an event signal is started in the event sensor 2 (step S71). Then, when the object 5 operates or moves, the detection of the event signal ends (step S72).

[0094] Next, identify the positions of the pixels corresponding to the event signal detected immediately before the event signal ceases to be detected (the event signal detected in the one frame immediately before it ceases to be detected or the event signals detected in several frames immediately before). From the positions of the identified plurality of pixels, identify the positions of the pixels corresponding to the object 5 (step S73). The method for identifying the positions (position coordinates on the screen) of the pixels corresponding to the object 5 in step S73 is not particularly limited. However, since event signals are detected in a plurality of pixels in a plurality of frames, the positions of the object 5 can be identified by taking the average of the position coordinates of these pixels.

[0095] Next, register the presence of the object 5 at the identified position in the RAM 12 or the storage unit 13 (step S74). Next, the position of the identified object 5 is output (step S75). When the position of the object 5 is to be displayed on the display unit 15, it can also be displayed in a manner that allows it to be recognized as the position corresponding to the object 5 in the display screen where the pixels detecting the event signal are displayed in a color different from the background. In addition, when the position or the like of the object 5 is transmitted to another device, the position or the like of the object 5 is also output in the other device.

[0096] When the movement of the object 5 resumes, in step S75, detection of the event signal is restarted at the positions of the pixels corresponding to the registered object (step S76). In this way, when the position of the pixel of the event signal detected immediately before the first detection of the event signal ends in step S72 and the position of the pixel of the event signal immediately after the second detection of the event signal starts in step S76 (the event signal detected in the one frame immediately after the detection starts or the event signals detected in several frames immediately before) match or are within a predetermined range, the first event signal and the second event signal can be treated as originating from the same object 5.

[0097] Then, when the movement of the object 5 is completed, the detection of the second event signal ends (step S77). Next, the position of the pixel corresponding to the event signal detected immediately before the second event signal stops being detected, that is, the position of the pixel corresponding to the object 5, is specified (step S78).

[0098] Next, from the position registered in step S74, it is registered in the RAM 12 or the storage unit 13 that the object 5 has moved from the registered position to the position of the specified pixel, and the position coordinates of the pixel corresponding to the object 5 after the movement (step S79). Next, the position of the specified object 5 is output (step S80). By steps S71 to S80, the movement detection process ends.

[0099] In the seventh embodiment, when the movement of the moving object 5 ends and the object 5 starts moving again, based on the event signal detected immediately before the detection of the first event signal ends and the event signal detected when the detection of the second event signal starts, if the specified positions match or are within a predetermined range, the first event signal and the second event signal can be treated as originating from the same object 5.

[0100] According to the present invention, since the region of the real space corresponding to the signal can be specified based on the signal detected by the event sensor, the region where a change has occurred in the real space that is the sensing target can be specified. Also, based on the region of the real space specified as having changed, the event that has occurred in the real space can be specified. When a plurality of pixels correspond to one region of the real space and the total amount of the signals corresponding to one region of the real space exceeds the threshold value, change information indicating that a state change has occurred in the specified region of the real space can be output, or the change information can be stored. By doing so, it is possible to prevent outputting that a state change has occurred or storing that a state change has occurred even when noise has occurred. Also, in this case, by varying the threshold value for each region of the real space, it is possible to increase the threshold value in a region where the amount of noise generated is large and decrease the threshold value in a region where the amount of noise generated is small. Furthermore, in this case, by varying the threshold value according to the time in the real space, since the environment such as the brightness of the space varies with time, it is possible to increase the threshold value in a time band where the amount of noise generated is large and decrease the threshold value in a time band where the amount of noise generated is small.

[0101] According to the present invention, when a plurality of pixels of an event sensor are divided into a plurality of regions according to the positions of the pixels, by determining whether or not the total amount of signals corresponding to one region exceeds a threshold value, according to the signals detected for each region, signals corresponding to at least some of the regions can be excluded from the target of a predetermined process. By specifying a threshold value for each region based on the signals detected in a state where no change has occurred in the real space that is the sensing target, noise can be appropriately removed for each region.

[0102] According to the present invention, using a prediction model that is machine-learned with the signals detected by an event sensor as input data and the events that occurred in the real space that is the detection target of the event sensor as output data, in the real space that is the sensing target, based on the signals detected by taking the change in the luminance of each pixel of the light incident on the event sensor as a signal, the events that occurred in the real space can be specified.

[0103] According to the present invention, using a prediction model that is machine-learned with the signals detected by an event sensor as input data and information regarding whether or not the state of the real space that is the detection target of the event sensor is normal as output data, in the real space that is the sensing target, based on the signals detected by taking the change in the luminance of each pixel of the light incident on the event sensor as a signal, it is possible to determine whether or not the state of the real space is normal. In addition, when it is determined that the state of the real space to be sensed is not normal, information indicating that the state of the real space is not normal can be output, or in another device, it is possible to control so as to output information indicating that the state of the real space is not normal.

[0104] According to the present invention, at least a part of an object existing in the real space to be detected by the event sensor, or by causing the event sensor to generate vibration and detecting a signal with the event sensor, even if the real space to be sensed is a situation where it is difficult for a person to visually recognize an object, it is possible to determine whether an object exists or to specify the position where the object exists. In addition, by irradiating at least a part of the real space to be detected by the event sensor with light and detecting a signal with the event sensor, even if the real space to be sensed is a situation where it is difficult for a person to visually recognize an object, it is possible to determine whether an object exists or to specify the position where the object exists. In this case, when the vibration subsides and the signal is no longer detected, or when the light irradiation ends and the signal is no longer detected, it is possible to display on the display screen in a manner that enables the understanding that an object exists at the position of the pixel where the signal has been detected until then, or to display on the display screen of another device in a manner that enables the understanding that an object exists. In addition, when the vibration subsides and the signal is no longer detected, or when the light irradiation ends and the signal is no longer detected, it is possible to register that an object exists at the position of the pixel where the signal has been detected until then. Furthermore, by using a prediction model that is machine-learned with the signal detected when the light reflected and / or transmitted from the object is incident on the event sensor as input data and the attribute of the object as output data, it is possible to specify the attribute of the object existing in the real space to be sensed.

[0105] According to the present invention, after a signal is detected by an event sensor, when the signal stops being detected, by specifying the position of the pixel corresponding to the signal detected immediately before the signal stops being detected, in the real space of the sensing target, after the object has finished moving, the position where the object exists or the area where the object exists can be specified. In this case, it can be displayed on the display screen in a manner that enables the understanding that an object exists at the position of the pixel that detected the signal, or can be displayed on the display screen of another device in a manner that enables the understanding that an object exists. Also, it is possible to register that an object exists at the position of the pixel that detected the signal. Furthermore, in the position of the pixel corresponding to the registered object or the pixels around that pixel, after the signal is detected again, when the signal stops being detected again, it is possible to detect that the object registered by the registration means has moved to the position of the pixel corresponding to the signal detected immediately before the signal stops being detected again.

Explanation of Signs

[0106] 1 System 2 Event Sensor 3 Information Processing Device 4 Real Space 5 Object 6 Vibration Generation Device 7 Irradiation Device 11 Control Unit 12 RAM 13 Storage Unit 14 Input Unit 15 Display Unit 16 Communication Interface

Claims

1. Detection means for detecting a detection signal with a sensor capable of detecting a change in luminance for each pixel of the incident light as a signal; Light irradiation means for irradiating light onto at least a part of the real space of the detection target by the sensor; Position specifying means for specifying the position of a pixel corresponding to an object existing in the real space from the position of a pixel corresponding to the detection signal detected by the detection means due to a change in luminance caused by the irradiation of light by the light irradiation means; Display means for displaying on a display screen in a manner that enables the presence of an object to be grasped at the position of a pixel corresponding to the object when the detection signal is no longer detected due to the end of the light irradiation, or display control means for controlling to display on the display screen of another device in a manner that enables the presence of an object to be grasped; A system comprising the above.

2. Detection means for detecting a detection signal with a sensor capable of detecting a change in luminance for each pixel of the incident light as a signal; Light irradiation means for irradiating light onto at least a part of the real space of the detection target by the sensor; Position specifying means for specifying the position of a pixel corresponding to an object existing in the real space from the position of a pixel corresponding to the detection signal detected by the detection means due to a change in luminance caused by the irradiation of light by the light irradiation means; Registration means for registering the presence of an object at the position of a pixel corresponding to the object when the detection signal is no longer detected due to the end of the light irradiation; A system comprising the above.

3. Object specifying means for specifying the attribute of an object existing in the real space of the detection target by the sensor based on the detection signal detected by the detection means, using a prediction model that is machine-learned with the detection signal detected when light reflected and / or transmitted from the object is incident on a sensor capable of detecting a change in luminance for each pixel of the incident light as input data and the attribute of the object as output data; The system according to claim 1 or 2, comprising the above.

4. A detection step of detecting a detection signal with a sensor capable of detecting a change in luminance for each pixel of the incident light as a signal; A light irradiation step of irradiating light onto at least a part of the real space of the detection target by the sensor; A position specifying step of specifying the position of a pixel corresponding to an object existing in the real space from the position of a pixel corresponding to the detection signal detected in the detection step due to a change in luminance caused by the irradiation of light in the light irradiation step; When the detection signal is no longer detected due to the termination of light irradiation, a display step of displaying on the display screen in a manner that enables the presence of the object to be grasped at the position of the pixel corresponding to the object, or a display control step of controlling to display on the display screen of another device in a manner that enables the presence of the object to be grasped A method comprising the above steps

5. A detection step of detecting a detection signal with a sensor capable of detecting a change in luminance of incident light for each pixel as a signal, A light irradiation step of irradiating at least a part of the real space of the detection target by the sensor with light, A position specifying step of specifying the position of the pixel corresponding to the object existing in the real space from the position of the pixel corresponding to the detection signal detected in the detection step due to the change in luminance caused by the light irradiation in the light irradiation step, A registration step of registering the presence of the object at the position of the pixel corresponding to the object when the detection signal is no longer detected due to the termination of light irradiation A method comprising the above steps

Images