Observation system

Abstract

To provide an observation system capable of individually determining the situation of each object person taking a group action without using a dedicated imaging device.SOLUTION: An observation system 1 observing a target person comprises a video cutting-out unit 10, an object person identification unit 11, a motion detecting unit 12, and a situation determination unit 13. The video cutting-out unit 10 cuts out a video M during a time during which the action of a kindergarten child T appearing in the video meets observation conditions from the video M. The object person identification unit 11 references identification data 6 of the kindergarten child T, identifies the kindergarten child T appearing in the video M on the basis of the video M cut out by the video cutting-out unit 10, and detects the position of the kindergarten child T in the video M. The motion detecting unit 12 detects the motion of the kindergarten child T on the basis of the position in the video M of the kindergarten child T detected by the object person identification unit 11. The situation determination unit 13 determines the situation of the kindergarten child T on the basis of the motion of the kindergarten child T detected by the motion detecting unit 12.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a monitoring system. [Background technology]

[0002] A system has been disclosed that analyzes a learner's concentration level based on facial expressions (see Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 7106159 Summary of the Invention [Problem to be solved by the invention]

[0004] However, in order for this system to analyze the concentration level of a learner, it is necessary to capture the facial expressions of the learner in detail. For this reason, it is difficult to use videos captured by already installed surveillance cameras for analysis. In addition, this system is not suitable for individually judging the status of subjects engaged in group activities. This is because it is difficult to judge the status of multiple subjects engaged in group activities at once if one tries to judge the status of each subject.

[0005] The present invention has been made in light of the above-mentioned circumstances, and aims to provide an observation system that can individually determine the status of subjects engaging in group behavior without using dedicated imaging equipment. [Means for solving the problem]

[0006] In order to achieve the above object, the observation system according to the present invention comprises: 1. An observation system for observing a subject, comprising: a video cutout unit that cuts out a portion of the video during which the behavior of a subject shown in the video satisfies an observation condition; a subject identification unit that identifies a subject appearing in the video based on the video cut out by the video cutout unit by referring to identification information of the subject, and detects a position of the subject within the video; a motion detection unit that detects a motion of the subject based on a position of the subject in the moving image detected by the subject identification unit; a situation determination unit that determines a situation of a subject based on the motion of the subject detected by the motion detection unit; Equipped with.

[0007] The subject identification unit is When a plurality of subjects appear in the video cut out by the video cutout unit, each subject is individually identified; The movement detection unit detects the movement of each subject, The situation determination unit determines the situation of each subject. This may also be the case.

[0008] The motion detection unit detects a head movement speed of a subject appearing in a video; The situation determination unit determines a concentration level of the subject as the situation of the subject based on the rocking speed detected by the motion detection unit. This may also be the case.

[0009] The concentration is The ratio of the time during which the subject's head sway velocity is equal to or less than the threshold value to the total time during which the subject's head sway velocity is detected is expressed as This may also be the case.

[0010] The motion detection unit detects a trajectory of a position of a subject appearing in a video; The situation determination unit determines a utilization degree of a facility or equipment as a situation of the subject based on a trajectory of the subject's position detected by the movement detection unit and a position of the facility or equipment. This may also be the case.

[0011] The motion detection unit detects a trajectory of a position of a subject appearing in a video; The situation determination unit determines a motion state of the subject based on a trajectory of the subject's position detected by the motion detection unit. This may also be the case.

[0012] The motion detection unit detects a trajectory of a position of each subject appearing in a video; The situation determination unit determines a degree of interaction between the subjects as the situation of the subjects based on the trajectories of the positions of the subjects detected by the movement detection unit. This may also be the case.

[0013] The situation determination unit determines a degree of synchronization between the movements of the subjects as the situation of the subjects based on the movements of the respective subjects detected by the movement detection unit. This may also be the case.

[0014] An activity content acquisition unit that acquires activity content of the subject in a time period before or including a time period that satisfies the observation condition; an information integration unit that associates the subject's status determined by the status determination unit with the subject's activity content acquired by the activity content acquisition unit for each subject and stores the information in a storage device; Equipped with This may also be the case.

[0015] The activity content includes an amount of physical activity of the subject measured by an activity meter worn by the subject. This may also be the case. Effect of the Invention

[0016] According to the present invention, the state of each subject engaged in group behavior can be determined individually without using dedicated imaging equipment. [Brief description of the drawings]

[0017] [Figure 1] 1 is a block diagram showing a configuration of an observation system according to a first embodiment of the present invention. [Diagram 2] FIG. 1 is a diagram showing an example of a video captured by a surveillance camera. [Diagram 3] 1A is a schematic diagram showing the processing executed by the video extraction unit, and FIG. 1B is a schematic diagram showing the observation conditions. [Figure 4] FIG. 2 is a schematic diagram showing identification information of a subject. [Diagram 5] 1A is a schematic diagram showing keystone correction, and FIG. 1B is a schematic diagram showing distortion correction. [Figure 6] 1A is a schematic diagram showing an example of a subject's movement detected by a movement detection unit, FIG. 1B is a graph showing an example of a subject's movement detected by the movement detection unit, and FIG. 1C is a graph showing the time change of the METs value measured by an activity meter. [Figure 7] 13(A) and (B) are diagrams showing an activity meter worn by a subject. [Figure 8] 11 is a table illustrating an example of log data. [Figure 9] FIG. 2 is a block diagram showing a hardware configuration of the observation system. [Figure 10] 11 is a flowchart showing an observation process in the observation system. [Figure 11] FIG. 11 is a schematic diagram showing functions of an observation system according to a second embodiment of the present invention. [Figure 12] FIG. 11 is a schematic diagram showing functions of an observation system according to a third embodiment of the present invention. [Figure 13] FIG. 11 is a schematic diagram showing functions of an observation system according to embodiment 4 of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. In each drawing, the same or equivalent parts are denoted by the same reference numerals.

[0019] First embodiment First, a first embodiment of the present invention will be described.

[0020] [Overview of the observation system] As shown in Fig. 1, an observation system 1 according to this embodiment observes a subject. In this embodiment, the place to be observed is a kindergarten room S, and a child T attending the kindergarten is the subject.

[0021] For example, for the purpose of crime prevention, multiple, specifically, four, surveillance cameras 2 are installed in a room S. The four surveillance cameras 2 are fixed cameras that capture images of the same location, i.e., the inside of the room S, in different directions, for example, from the upper corners of the room S toward the lower center, as shown in Fig. 2.

[0022] 2 shows an example of videos M1 to M4 captured by four surveillance cameras 2. In the following, the four videos M1 to M4 captured by the surveillance cameras 2 are collectively referred to as video M (see FIG. 1) for convenience. The surveillance cameras 2 are constantly capturing images of the inside of room S, including the time when child T is in room S.

[0023] The video data captured by the surveillance camera 2 is sent to the storage device 3 and stored as video data 5. The storage device 3 may be constructed in a server computer on a cloud capable of transmitting and receiving data to the observation system 1 and the surveillance camera 2, or may be a storage device provided in the observation system 1.

[0024] The video data 5 includes time information about the time at which the video was captured. This allows you to specify a time and extract a frame image at that time from the video data 5, and by extracting a frame image from the video data 5, you can obtain the time at which the frame image was captured.

[0025] The observation system 1 observes the child T who appears in the video M that is played back based on this video data 5. As indicated by the arrow extending from the portion surrounded by the dotted line in Fig. 1 (here, four signal lines are combined into one), the observation system 1 may receive the video data 5 sent from the surveillance camera 2 in real time, and observe the behavior of the child T who appears in the video M that is played back based on the received video data 5.

[0026] [Observation system configuration] As shown in Figure 1, the observation system 1 includes a video clipping unit 10 that clips out a video M, a subject identification unit 11 that identifies a child T, a motion detection unit 12 that detects the motion of the child T, and a situation determination unit 13 that determines the state of the child T.

[0027] [Video Extraction Section] As shown in Fig. 3(A), the video clipping unit 10 clips video M represented by video data 5 for a period during which the behavior of child T shown in the video M satisfies an observation condition. Such observation conditions are determined according to the state of child T to be judged. In this embodiment, the state of child T to be judged is the concentration level of child T when reading aloud. In this case, the observation condition is that the content of video M can be judged to be an image when reading aloud.

[0028] In this case, for example, as shown in Fig. 3(B), the observation condition can be that a person who appears to be a teacher appears in the first area A1 and the number of people appearing in the second area A2 exceeds a predetermined number in the video M. The video cutout unit 10 cuts out the video M according to such an observation condition.

[0029] This determination is made based on whether or not a pattern of a person who looks like a teacher is shown in the first area A1, and further based on whether or not the observation conditions are met by counting the number of people who appear in the second area A2 by detecting a code figure (see FIG. 4) attached to the hat worn by the child T. A person who looks like a teacher can be detected by template matching using a reference image of a person sitting with a book or a panel, for example. As described later, since the child T is wearing a hat with a code figure attached, the number of people who appear in the second area A2 can be counted by detecting the code figure. The observation conditions can be preset in the video extraction unit 10 by operation input, as shown in FIG. 1. At this time, the ranges and the predetermined numbers of the first area A1 and the second area A2 can also be preset.

[0030] The method of cutting out the video M by the video cutout unit 10 is not limited to this. The video cutout unit 10 may cut out the video M for a period of time that satisfies the observation condition by deep learning. That is, the video cutout unit 10 may be a deep learning device. In this case, for example, the video M during reading aloud is captured, and the video data is used as teacher data to be learned by the deep learning device. The trained deep learning device may determine whether the video M satisfies the observation condition, and cut out the video M for a period of time that satisfies the observation condition.

[0031] [Subject Identification Unit] Returning to FIG. 1, the subject identification unit 11 identifies the child T appearing in the video M based on the video M cut out by the video cut-out unit 10. As described above, in order to identify the child T, each child T wears a hat to which a code figure showing a unique code is attached as shown in FIG. 4. The code figure is attached in three places in different orientations. In other words, the child T is wearing the code figure. The storage device 3 has identification data 6 that stores this code figure in association with unique information of the child T. As shown in FIG. 4, the stored unique information includes the attendance number and name of the child T. The subject identification unit 11 refers to the identification data 6 and identifies the child T based on the image of the code figure detected from the video M.

[0032] Depending on the orientation of the code graphic, the image of the code graphic may not be captured by any of the four surveillance cameras 2. However, as described above, the video clipping unit 10 clips the video M for a time period that satisfies the observation conditions from each video M captured by the multiple surveillance cameras 2 that capture the same location from different orientations. The subject identification unit 11 identifies the child T based on the video M that shows the code graphic from among the multiple videos M1 to M4 (see FIG. 2) clipped by the video clipping unit 10.

[0033] When identifying a child T, the position of that child T in the video M (the pixel position where child T appears in the video M) is detected. That is, the subject identification unit 11 detects the position of the identified child T in the video M. This position is a position in the video M, and is not an actual position in the room S. Also, as shown in FIG. 3(B), when multiple children T appear in the video M cut out by the video cut-out unit 10, the subject identification unit 11 identifies each child T individually and detects the position of each child T in the video M.

[0034] Furthermore, the subject identification unit 11 can specify the start and end of processing by operation input. As shown in Fig. 3(A), when start is specified by operation input, the subject identification unit 11 starts identifying the child T appearing in the video M based on the video M cut out by the video cut-out unit 10, and when end is specified by operation input, the subject identification unit 11 ends identifying the child T appearing in the video M. That is, the subject identification unit 11 identifies the child T appearing in the video M between when start is specified and when end is specified.

[0035] [Motion detection section] Returning to FIG. 1, the motion detection unit 12 detects the motion of the child T based on the position of the child T in the video M detected by the subject identification unit 11. First, the motion detection unit 12 converts the position of the child T in the video M detected by the subject identification unit 11 into the actual position of the child T in the room S. The surveillance camera 2 captures the image of the room S from the four corners of the ceiling. In this embodiment, as described later, in order to detect the movement of the head of the child T in the horizontal plane, the motion detection unit 12 corrects (coordinate conversion) the video MC captured from the four corners of the ceiling of the room S to a video MD as if it was captured from directly above the room S, as shown in FIG. 5(A). This is keystone correction. In addition, each of the surveillance cameras 2 has its own optical aberration, and the video ME is distorted with respect to the actual subject, as shown in FIG. 5(B). Distortion correction is used to correct this distortion. In this distortion correction, the video ME is corrected to a video MF. Through this keystone correction and distortion correction, the positions of the pixels in the video M correspond to the two-dimensional positions in the horizontal plane.

[0036] The motion detection unit 12 performs calibration for generating correction information used for keystone correction and distortion correction before correcting the moving image M. That is, the motion detection unit 12 functions as a calibration unit.

[0037] For example, as shown in Fig. 5(A), square patterns B with known shape and size are placed in the imaging field of view of the surveillance camera 2, i.e., in multiple locations within a room S. These square patterns B are placed on a desk so that their height from the floor is the same as the head height of a kindergartener T who is sitting while being read to. Here, it is assumed that square patterns B are placed at each lattice point of a square lattice.

[0038] In this state, the motion detection unit 12 calculates correction information for performing keystone correction based on the video M captured by the surveillance camera 2. Specifically, as shown in Fig. 5(A), based on the positions of the square patterns B in the video MC, correction information is generated for converting the coordinates of a grid pattern G1 formed by connecting multiple square patterns B into a square grid G2, that is, correction information for converting the position coordinates of pixels on the video M into position coordinates in the horizontal plane of the actual square grid. By performing keystone correction using this correction information, a video MD is obtained from the video MC shown in Fig. 5(A).

[0039] 5(B), the motion detector 12 generates correction information for performing the distortion correction shown in FIG. 5(B) on the moving image M (MD) that has been subjected to keystone correction, based on the distortion of the grid pattern G1 obtained by connecting the square patterns B and the distortion of each square pattern B. For example, the moving image ME is converted into a moving image MF by using this correction information.

[0040] By this keystone correction and distortion correction, the position on the video M calculated in pixel units is converted to a position on a horizontal plane in mm units in the actual space. That is, the motion detection unit 12 corrects the video M to convert the position coordinates of the pixel in the video M where the child T appears to the position coordinates on a horizontal plane in the actual space (room S).

[0041] Returning to FIG. 1, the motion detection unit 12 stores the correction information generated by the calibration in the storage device 3 as correction data 7.

[0042] Returning to FIG. 1, the motion detection unit 12 detects the motion of the child T based on the position of the child T converted by the correction data 7. Specifically, the motion detection unit 12 detects the motion of the child T based on the position in mm units in the horizontal plane of the actual space obtained by the above-mentioned correction. When multiple children T appear in the video M cut out by the video cut-out unit 10, the motion detection unit 12 detects the motion of each child T. Similarly to the subject identification unit 11, the motion detection unit 12 detects the motion of the child T from when the start is specified to when the end is specified by operation input. When the subject identification unit 11 detects the positions of the code figures in multiple videos M, the average of the positions in the horizontal plane determined from the respective positions may be regarded as the position in the horizontal plane of the child T.

[0043] In this embodiment, the state of child T to be determined is the concentration level during reading aloud, so the movement detection unit 12 detects the rocking speed of child T's head. The movement detection unit 12 detects the horizontal rocking speed of child T's head. The rocking speed is used as an index because when child T is concentrating, it is not uncommon for his or her head to move slowly. In addition, the rocking speed of the entire body is not used as an index because when child T is concentrating, it is not uncommon for the entire body to rock.

[0044] As described above, the positional relationship between pixels can be considered to correspond to the positional relationship of the actual lattice points in the room S by the keystone correction and the distortion correction. This actual position is a position in a horizontal plane at the height of the head of the child T. Here, the horizontal plane is assumed to be the XY plane of the XY Cartesian coordinate system. The subject identification unit 11 has already detected the code figure of the hat worn by the child T. The motion detection unit 12 determines, for example, the center position of the code figure as the XY position of the head of the child T in the images at each time point constituting the video M. As shown in FIG. 6(A), the determined XY position takes the trajectory of the XY position of the head, which continues as (X1, Y1), (X2, Y2), ... as time progresses. The motion detection unit 12 calculates the swing speed V of the head of the child T in the XY plane based on the relationship between the trajectory of the XY position and time. Specifically, the motion detection unit 12 divides the movement distance of the XY position over time by that time, and smoothes the result of the division through a low-pass filter or a moving average filter to obtain the oscillation velocity V.

[0045] The movement detection unit 12 stores the detected XY position and swing velocity V of the head of the child T in the storage device 3 as part of the log data 8.

[0046] [Situation Judgment Section] Returning to FIG. 1, the situation determination unit 13 determines the state of the child T based on the movement of the child T detected by the movement detection unit 12. When multiple children T appear in the video M extracted by the video extraction unit 10, the situation determination unit 13 determines the state of each child T. The situation determination unit 13 determines the state of the child T from when the start is specified to when the end is specified by an operation input. For example, in a storytelling session, a time for hand-playing is usually set aside first to attract the attention of the child T. During the time for hand-playing, head movements become large, so it is desirable to start the determination of the concentration level after the time for hand-playing ends. Therefore, the start and end of the determination of the concentration level is manually specified.

[0047] In this embodiment, the situation determination unit 13 determines the concentration level of the child T based on the rocking speed V detected by the movement detection unit 12. The concentration level is expressed as the ratio of the time during which the rocking speed V of the head of each child T is equal to or less than a threshold value to the total time during which the rocking speed V of the head of each child T is detected. For example, as shown in FIG. 6(B), assuming that the time from the start to the end of detection is the total time, the situation determination unit 13 calculates the ratio of the time during which the rocking speed V is equal to or less than the threshold value Th to the total time. Note that since the XY positions are detected in mm units, the unit of the rocking speed V is [mm / sec]. The threshold value Th is adjustable.

[0048] [Other components] Returning to FIG. 1, the observation system 1 further includes an activity content acquisition unit 14 that acquires activity content of the child T, a data integration unit 15 that integrates the data, and a data display unit 16 that displays the data.

[0049] [Activity content acquisition department] The activity content acquisition unit 14 acquires the activity content of the child T in the time period before or including the time when the observation condition is satisfied. The activity content includes various contents. For example, it may include anything such as the amount of sleep from the previous night, the time of waking up that day, the time of arrival at the kindergarten, health condition, breakfast contents, etc. The activity content from the previous day and before may also be included in this activity content. These activity contents may be input by operation input, or may be received by data communication.

[0050] The activity content may include the amount of physical activity of the child T measured by an activity meter 20 worn on the arm or body of the child T, as shown in Figs. 7(A) and 7(B). The activity meter 20 has a built-in acceleration sensor and a posture sensor, and measures the amount of activity of the child T, for example, at one-minute intervals, using the outputs of these sensors. The amount of activity measured by the activity meter 20 may include, for example, a METs value (total METs value, average METs value, maximum METs value) indicating the intensity of physical activity, exercise time, and calories burned through activity. The calories burned are calculated based on the intensity of physical activity and the weight of the child T, which is set in advance. The activity meter 20 may also measure biological information such as the pulse rate of the child T as the amount of activity.

[0051] The activity amount data measured by the activity meter 20 is acquired in a lump by the activity content acquisition unit 14. A commercially available one can be used as the activity meter 20. The activity meter 20 may be worn on the chest of the child T, and the place where it is worn is not limited.

[0052] [Data Integration Department] Returning to FIG. 1, the data integration unit 15 integrates the movements of the child T detected by the movement detection unit 12, the situation of the child T determined by the situation determination unit 13, and the activity details of the child T acquired by the activity details acquisition unit 14 in association with each other for each child T, and stores the log data 8 in the storage device 3. If the activity details include an activity amount measured by the activity meter 20, the activity amount and the movements are associated in a synchronized state using their respective time information. FIG. 8 shows, as an example, a state in which the attendance number, activity details (activity amount, exercise time, calories burned), and situation (concentration time, concentration level) are associated with each other and stored for each child T. The start time and end time of the concentration level determination are additionally stored in this data.

[0053] [Data display section] Returning to FIG. 1, the data display unit 16 displays the contents of the log data 8 stored in the storage device 3. Specifically, the data display unit 16 displays the status of the child T determined by the status determination unit 13 and the activity details of the child T acquired by the activity details acquisition unit 14 for each child T. Basically, this display is in the form of a table as shown in FIG. 8. This table can be sorted and displayed in ascending or descending order for each item. For example, it is possible to sort and display the children T in descending order of concentration level, calories burned, or exercise time.

[0054] The data display unit 16 can display information showing the time change in the movement of the child T detected by the movement detection unit 12. In this case, for example, as shown in Fig. 6(B), the data display unit 16 can display information for judging the state of the child T by superimposing it on the information showing the time change in the movement of the child T. For example, the background color of the time from the start to the end of the detection of the movement of the child T can be displayed in a different color from other background colors. Furthermore, the background color of the area below the threshold value Th can be displayed in a different color from other background colors.

[0055] As shown in Fig. 6(C), the data display unit 16 can also display a graph of the time change in the amount of activity (e.g., METs value) measured by the activity meter 20 as the activity content of the child T acquired by the activity content acquisition unit 14. In this graph, too, it is possible to change the background color of the time from the start to the end of detection of the child T's movement to another background color. By looking at the graph in Fig. 6(C), it is possible to grasp the change in the METs value before and after the reading aloud, i.e., the change in the amount of activity.

[0056] The data display unit 16 displays the video M that is played back from the video data 5 or directly input from the surveillance camera 2. For example, as shown in Fig. 2, in the displayed image, it is possible to superimpose the identification information (attendance number 002) of a particular child T among multiple children T as an index by operation input and display the identification information on the particular child T on the video M in a manner that points to the particular child T.

[0057] [Hardware configuration of Observation System 1] The observation system 1 shown in Fig. 1 is realized, for example, by a computer having the hardware configuration shown in Fig. 9 executing a software program. Specifically, the observation system 1 includes a CPU (Central Processing Unit) 31 that controls the entire device, a main memory 32 that operates as a work area or the like for the CPU 31, an external memory 33 that stores the operating program or the like for the CPU 31, an operation unit 34, a display 35, a communication unit 36, and an internal bus 38 that connects these together.

[0058] The main memory 32 is composed of a RAM (Random Access Memory) etc. A program 39 to be executed by the CPU 31 is loaded into the main memory 32 from an external memory 33. The main memory 32 is also used as a working area (temporary data storage area) of the CPU 31.

[0059] The external memory 33 is composed of a non-volatile memory such as a flash memory, a hard disk, etc. The external memory 33 stores in advance a program 39 to be executed by the CPU 31.

[0060] The operation unit 34 is composed of devices such as a keyboard and a mouse, and an interface device that connects these devices to an internal bus 38 .

[0061] The display 35 is composed of a display device such as a CRT (Cathode Ray Tube) or a liquid crystal monitor.

[0062] The communication unit 36 ​​is an interface for data communication. Data communication with external devices is possible via the communication unit 36. For example, via this communication unit 36, it is possible to receive video data 5 from the surveillance camera 2, obtain an amount of activity from the activity meter 20, and, if the storage device 3 is a cloud server, read and write data to and from the storage device 3.

[0063] The functions of the observation system 1 can be implemented in a computer system consisting of one or more computers including one or more processors and one or more storage devices including non-transitory storage media. The multiple computers realize the functions of the observation system 1 while communicating with each other via a communication network. For example, some of the multiple functions of the observation system 1 may be implemented in one computer, and the other parts may be implemented in other computers.

[0064] Next, an observation process executed by the observation system 1 according to the present embodiment will be described. It is assumed that the calibration has already been completed and the activity content of the child T has been acquired by the activity content acquisition unit 14.

[0065] In this observation process, as shown in FIG. 10, first, the video clipping unit 10 waits until a new frame image constituting the video M is acquired (step S1; No). When a new frame image is acquired (step S1; Yes), the video clipping unit 10 judges whether or not to start clipping (step S2). Here, clipping has not yet started, and it is judged whether or not the new frame image satisfies the observation condition, and if the observation condition is not satisfied, the judgment is negative. If the judgment is negative (step S2; No), the process returns to the processing of the video clipping unit 10 (step S1). After that, even if a new frame image is acquired (step S1; Yes), if clipping has not yet started, and the new frame image does not satisfy the observation condition (step S2; No), steps S1 to S2 are repeated.

[0066] If extraction has not yet started and a new frame image satisfies the viewing conditions and extraction is to start (step S2; Yes), the video extraction unit 10 judges whether or not to end extraction (step S3). Here, extraction has just started and the new frame image satisfies the viewing conditions, so the judgment is negative (step S3; No). Next, the video extraction unit 10 extracts the frame image (step S4).

[0067] Thereafter, the subject identification unit 11 judges whether or not a start command has been input (step S5). If a start command has not been input (step S5; No), the process returns to step S1. Thereafter, steps S1 → S2 → S3 → S4 → S5 are repeated, and the subject identification unit 11 waits until a start command is input (step S5; No). When a start command is input (step S5; Yes), the subject identification unit 11 judges whether or not an end command has been input (step S6). Here, since a start command has just been input, the judgment is negative (step S6; No).

[0068] Thereafter, the subject identification unit 11 refers to the identification data 6 for the child T (see FIG. 1) and identifies the child T appearing in the video M based on the video M cut out by the video cutout unit 10, and detects the position of the child T in the video M (step S7). The motion detection unit 12 converts the position of the child T's head in the video M to an actual XY position based on the correction data 7 (step S8). The motion detection unit 12 detects the motion of the child T based on the actual XY position of the child T (step S9). After step S9 is completed, the observation system 1 returns to step S1. In this way, steps S1 to S9 are repeated in the observation system 1, and each time a new frame image of the video M is acquired, cutout (step S4), subject identification (step S7), coordinate conversion (step S8), and motion detection (step S9) are performed.

[0069] In this repetition, when the extraction is completed (step S3; Yes) or an end command is input (step S4; Yes), the status determination unit 13 determines the status of child T (step S10). Then, the data integration unit 15 associates and integrates the status of child T determined by the status determination unit 13 and the activity content of child T acquired by the activity content acquisition unit 14 for each child T, and stores the data in the storage device 3 as log data 8 (step S11). After step S11 is completed, the observation system 1 ends the observation process.

[0070] In this embodiment, the head swing velocity V is detected as the movement of the child T. However, in addition to this, the movement detection unit 12 may be configured to detect the swing velocity of the entire body.

[0071] It is also conceivable that the head rocking speed V is small even when child T is sleeping. In this case, the situation determination unit 13 may determine that child T is sleeping by detecting whether or not the child's eyes are open based on the video M, or by detecting whether or not the head rocking speed V is changing impulsively.

[0072] Embodiment 2 Next, a second embodiment of the present invention will be described. The observation system 1 according to this embodiment is the same as the observation system 1 according to the above-mentioned embodiment in that the observed subject is a child T. However, in this embodiment, the surveillance camera 2 captures an image of a kindergarten playground U, not a kindergarten room S. The configuration of the observation system 1 according to this embodiment is the same as the configuration of the observation system 1 according to the first embodiment (see FIG. 1). In this embodiment, it is preferable that the code graphic shown in FIG. 4 is attached to the torso of the child T. Also, the subject identification unit 11 may perform pattern recognition of the torso of the child T without attaching the code graphic.

[0073] In the observation system 1 according to this embodiment, the motion detection unit 12 detects a trajectory TP of the body position P of child T captured in a video M, as shown in FIG. 11. In this case, the position P of child T indicates the body position of child T in a playground U. Meanwhile, play equipment is installed in the playground U as a facility or equipment. The situation determination unit 13 holds the position E of the play equipment in the playground U. The storage device 3 stores the position E of the play equipment in the playground U, and the situation determination unit 13 may refer to the position E stored in the storage device 3.

[0074] The situation determination unit 13 determines the degree of use of the play equipment by each child T as the situation of the child T based on the trajectory TP of the position P of the child T detected by the movement detection unit 12 and the position E of the play equipment. Specifically, if the position P of the child T and the position E of the play equipment are within a predetermined distance L, it is determined that the child T is using the play equipment. The degree of use of the play equipment is calculated as the ratio of the time that the position P of the child T and the position E of the play equipment were within the predetermined distance L with respect to the entire time from the start to the end of the movement detection.

[0075] In this way, if it is possible to determine the degree of use of play equipment by each child T, it is possible to evaluate each child T's preferences regarding play equipment. In this embodiment, the play equipment is facilities or equipment. However, this is not limited to this. If the location is indoors, the facilities or equipment may be musical instruments such as a piano that are provided. The facilities or equipment will vary depending on the subject of observation and the location of the observation.

[0076] Third embodiment Next, a third embodiment of the present invention will be described. The observation system 1 according to this embodiment is the same as the observation system 1 according to the above-mentioned embodiment in that the observed person is a child T. However, in this embodiment, as shown in FIG. 12, the surveillance camera 2 captures an image of a kindergarten playground U, not a kindergarten room S. The configuration of the observation system 1 according to this embodiment is the same as the configuration of the observation system 1 according to the first embodiment (see FIG. 1). In this embodiment, it is preferable that the code graphic shown in FIG. 4 is attached to the torso of the child T. Also, the subject identification unit 11 may perform pattern recognition of the torso of the child T without attaching the code graphic.

[0077] The motion detection unit 12 (see FIG. 1) detects a trajectory TP of a position P of the child T shown in the video M. The situation determination unit 13 (see FIG. 1) determines the amount of exercise (exercise state) of the child T as the situation of the child T based on the trajectory TP of the position P of the child T detected by the motion detection unit 12. The amount of exercise includes, for example, a moving distance and a moving speed during exercise.

[0078] Fourth embodiment Next, a fourth embodiment of the present invention will be described. The observation system 1 according to this embodiment is the same as the observation system 1 according to the above-mentioned embodiment in that the observation target is a child T. In this embodiment, the surveillance camera 2 may capture an image of a kindergarten room S or a kindergarten playground U. The configuration of the observation system 1 according to this embodiment is the same as the configuration of the observation system 1 according to the first embodiment (see FIG. 1). In this embodiment, it is preferable that the code graphic shown in FIG. 4 is attached to the torso of the child T. Also, the subject identification unit 11 may perform pattern recognition of the torso of the child T without attaching the code graphic.

[0079] In this embodiment, the motion detection unit 12 (see FIG. 1) detects a trajectory TP of the position P of each child T appearing in the video M. The situation determination unit 13 (see FIG. 1) determines the degree of interaction between the children T as the situation of the children T based on the trajectory TP of the position P of each child T detected by the motion detection unit 12. For example, as shown in FIG. 13, when the interval between the positions P of multiple children T is within a predetermined distance L, the children T are deemed to be interacting with each other. The degree of interaction is calculated as the ratio of the time that the children T were within the predetermined distance L of each other to the total time from the start to the end of detection.

[0080] In this way, by determining the degree of interaction between the T children, it is possible to group the T children together and understand their relationships.

[0081] In addition, in the observation system 1 according to the above-mentioned third and fourth embodiments, it is also possible to calculate the synchronization of the movements of multiple children T who move in unison while keeping the distance between them within a predetermined distance L.

[0082] [summary] As described in detail above, according to the observation system 1 of the above embodiment, it is possible to identify individual children T from video M containing multiple children T based on video M captured by the installed surveillance camera 2, detect their movements, and determine the status of the children T based on their movements, so that the status of each child T engaged in group activities can be determined individually without using dedicated imaging equipment.

[0083] According to the observation system 1 of the above embodiment, the situation of each child T can be judged, and the situation of each of the multiple children T can be detected collectively. Therefore, according to the observation system 1, it is also possible to evaluate the entire group of children T. That is, according to the observation system 1 of the above embodiment, when multiple children T appear in the video M, the subject identification unit 11 individually identifies each child T when multiple children T appear in the video M cut out by the video cutout unit 10. The movement detection unit 12 detects the movement of each child T. The situation determination unit 13 determines the situation of each child T. In this way, the movement of each of the multiple children T captured by the surveillance camera 2 can be detected and the situation can be judged, and the movement and situation of the entire group of children T can also be evaluated.

[0084] According to the observation system 1 of the above embodiment, various situations of the child T can be determined, such as the child T's concentration level, the degree of use of facilities or equipment, the physical activity level, the degree of interaction with other children T, and the degree of synchronization.

[0085] According to the observation system 1 of the above embodiment, it is possible to manually specify the time to start and end the motion detection depending on the situation of the child T. In this way, it is possible to limit the time for which the motion detection is performed to an appropriate time.

[0086] According to the observation system 1 of the above embodiment, it is possible to automatically cut out the moving image M in accordance with the observation conditions. In this way, it is possible to eliminate the need to manually cut out the moving image M that satisfies the observation conditions.

[0087] According to the observation system 1 of the above embodiment, it is possible to accurately detect the actual position of the child T by performing image correction through calibration.

[0088] According to the observation system 1 of the above embodiment, it is possible to accurately identify the child T and detect his / her position by the code graphic attached to the child T.

[0089] According to the observation system 1 of the above embodiment, the child T can be reliably identified based on the video M captured by the multiple monitoring cameras 2 that capture the child T from different directions.

[0090] According to the observation system 1 of the above embodiment, the situation of the child T and the contents of the previous activity are integrated and stored. In this way, the following effects are expected. (1) Evaluation and improvement of childcare methods For example, childcare methods can be evaluated based on whether there are many or few children T who are not concentrating when being read to. If there are many children T who are highly concentrated, the childcare methods at that time can be continued. Also, if there are many children T who are not concentrating, measures can be taken such as improving the schedule before and after the reading or the reading method. For example, if children spend more time exercising before reading to them, the exercise time can be increased. Also, if children who arrive earlier have higher levels of concentration, the concentration data can be shown to the parents to encourage them to arrive earlier. (2) It can be used as evidence of the growth of kindergartener T. If the concentration level of child T improves over time, this can be taken as evidence of the growth of child T. The relationship between the childcare method (the relationship between reading aloud and the exercise program) and the concentration level can be proven with evidence. Furthermore, it can be proven with evidence that it is possible to raise child T with high concentration levels even in kindergartens and nurseries that do not have playgrounds. (3) Early identification of children with biases and those who need intervention and support This will help to early identify a child T who is significantly less focused than other children T as a child with an imbalance, and will be useful in developing subsequent countermeasures.

[0091] According to the observation system 1 of the above embodiment, it is possible to display and understand the time change in the movement of the child T. In addition, the time change in the movement of the child T is displayed with information for judging the situation superimposed thereon, so that the movement and situation of the child T can be easily understood.

[0092] According to the observation system 1 of the above embodiment, a video M captured by a surveillance camera is displayed, and an index of each child T is displayed on the display, making it easier to grasp the condition of each child T through the video M.

[0093] In the above-mentioned first embodiment, the number of surveillance cameras 2 is four, but it may be three or less, or five or more. There may also be one surveillance camera 2. Furthermore, the camera that captures the video M is not limited to the surveillance camera 2. It may be a camera that is installed in a certain location, or it may be a mobile camera as long as the position of the subject to be captured can be obtained from the camera's position and angle of view.

[0094] In the above-mentioned embodiment 1, a code graphic is attached to the hat worn by the child T, and the child T is identified based on the code graphic, but this is not limited to the above. The code graphic may be attached to clothes. Also, the child T may be identified based on the physical features of the child T, such as face recognition, without using the code graphic.

[0095] In addition, distortion correction may be performed on the video M after it has been cut out by the video cut-out unit 10 before the subject is identified by the subject identification unit 11, and keystone correction may be used by the motion detection unit 12 to convert the pixels of the video M into coordinates of the actual positions.

[0096] In the above-mentioned embodiment 1, the keystone correction is performed by calibration limited to the horizontal plane (XY plane). However, this is not limited to this. When detecting three-dimensional movement, it is necessary to perform three-dimensional calibration using a three-dimensional figure. In this case, when there are multiple surveillance cameras 2, the parallax of the multiple surveillance cameras 2 may be used to detect the three-dimensional position.

[0097] There is no particular limitation on the movement to be detected or the situation to be determined. The movement of the subject includes the trajectory TP of the subject's position P, the speed, and the change in posture, and the situation of the subject may include anything that indicates the situation of the subject in addition to the above. For example, the synchronization of the postures of the subjects in a group activity, for example, the synchronization in radio calisthenics, may be determined as the situation to be determined.

[0098] Furthermore, the hardware configuration and software configuration of the observation system 1 are merely examples and can be changed and modified as desired.

[0099] The central part of the processing of the observation system 1, which is composed of the CPU 31, main memory 32, external memory 33, operation unit 34, display 35, communication unit 36, internal bus 38, etc., may be constructed as a dedicated system as described above, or may be realized by using a normal computer system. For example, the observation system 1 that executes the above processing may be configured by storing and distributing a computer-readable non-transient recording medium (flexible disk, CD-ROM, DVD-ROM, etc.) for executing the above operations, and installing the computer program on a computer. Also, the observation system 1 may be configured by storing the computer program in a storage device of a server device on a communication network such as the Internet, and downloading the computer program by a normal computer system.

[0100] When the functions of a computer are realized by sharing between an OS (operating system) and application programs, or by cooperation between the OS and application programs, only the application program portion may be stored in a recording medium or storage device.

[0101] It is also possible to superimpose a computer program on a carrier wave and distribute it via a communication network. For example, the computer program may be posted on a bulletin board system (BBS) on the communication network and distributed via the network. Then, the computer program may be started and executed under the control of the OS in the same way as other application programs, thereby enabling the above-mentioned processing to be performed.

[0102] Various embodiments and modifications of the present invention are possible without departing from the broad spirit and scope of the present invention. The above-described embodiments are for the purpose of explaining the present invention and do not limit the scope of the present invention. That is, the scope of the present invention is indicated by the claims, not the embodiments. Various modifications made within the scope of the claims and the scope of the invention equivalent thereto are considered to be within the scope of the present invention. [Industrial Applicability]

[0103] The present invention can be applied to observe a subject. [Explanation of symbols]

[0104] 1 Observation system, 2 Surveillance camera, 3 Storage device, 5 Video data, 6 Identification data (identification information), 7 Correction data (correction information), 8 Log data, 10 Video extraction unit, 11 Subject identification unit, 12 Motion detection unit, 13 Situation determination unit, 14 Activity content acquisition unit, 15 Data integration unit (information integration unit), 16 Data display unit (information display unit), 20 Activity meter, 31 CPU, 32 Main memory, 33 External memory, 34 Operation unit, 35 Display, 36 Communication unit, 38 Internal bus, 39 Program, B Square pattern, E Position, G1 Grid pattern, G2 Square grid, M, M1 to M4, MC, MD, ME, MF Video, P Position, S Room, T Child, TP Trajectory, U Playground

Claims

1. 1. An observation system for observing a subject, comprising: a video cutout unit that cuts out a portion of the video during which the behavior of a subject shown in the video satisfies an observation condition; a subject identification unit that identifies a subject appearing in the video based on the video cut out by the video cutout unit by referring to identification information of the subject, and detects a position of the subject within the video; a motion detection unit that detects a motion of the subject based on a position of the subject in the moving image detected by the subject identification unit; a situation determination unit that determines a situation of a subject based on the motion of the subject detected by the motion detection unit; An observation system comprising:

2. The subject identification unit is When a plurality of subjects appear in the video cut out by the video cutout unit, each subject is individually identified; The movement detection unit detects the movement of each subject, The situation determination unit determines the situation of each subject. The observation system of claim 1 .

3. The motion detection unit detects a head movement speed of a subject appearing in a video; The situation determination unit determines a concentration level of the subject as the situation of the subject based on the rocking speed detected by the motion detection unit. The observation system of claim 1 .

4. The concentration is The ratio of the time during which the subject's head sway velocity is equal to or less than the threshold value to the total time during which the subject's head sway velocity is detected is expressed as The observation system according to claim 3 .

5. The motion detection unit detects a trajectory of a position of a subject appearing in a video; The situation determination unit determines a utilization degree of a facility or equipment as a situation of the subject based on a trajectory of the subject's position detected by the movement detection unit and a position of the facility or equipment. The observation system of claim 1 .

6. The motion detection unit detects a trajectory of a position of a subject appearing in a video; The situation determination unit determines a motion state of the subject based on a trajectory of the subject's position detected by the motion detection unit. The observation system of claim 1 .

7. The motion detection unit detects a trajectory of a position of each subject appearing in a video; The situation determination unit determines a degree of interaction between the subjects as the situation of the subjects based on the trajectories of the positions of the subjects detected by the movement detection unit. The observation system according to claim 2 .

8. The situation determination unit determines a degree of synchronization between the movements of the subjects as the situation of the subjects based on the movements of the respective subjects detected by the movement detection unit. The observation system according to claim 2 .

9. An activity content acquisition unit that acquires activity content of the subject in a time period before or including a time period that satisfies the observation condition; an information integration unit that associates the subject's status determined by the status determination unit with the subject's activity content acquired by the activity content acquisition unit for each subject and stores the information in a storage device; Equipped with The observation system of claim 1 .

10. The activity content includes an amount of physical activity of the subject measured by an activity meter worn by the subject. The observation system according to claim 9.

Images