Information processing method, information processing system, and computer program

Abstract

To provide an information processing method, an information processing system and a computer program for analyzing motion patterns of an object.SOLUTION: Steps of processing for setting a monitoring area include: obtaining a photographed image photographed by a camera; detecting a specific object included in the photographed image; identifying a position of the detected object within the photographed image; clustering the positions of a plurality of objects identified from a plurality of the photographed images; and determining, as a monitoring area to be monitored for presence of the specific object, an area within the photographed image corresponding to a cluster obtained by the clustering.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to an information processing method, an information processing system, and a computer program for analyzing the movement pattern of an object. [Background technology]

[0002] Conventionally, it has been possible to detect objects such as people or vehicles from images captured using a camera. For example, it is possible to detect people from captured images and use the detection results for crime prevention. Patent Document 1 discloses an example of a technology for detecting suspicious people from images captured using a camera. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-12657 Summary of the Invention [Problem to be solved by the invention]

[0004] In facilities such as stores that are used by many people, there is a need to know people's movement patterns, such as the routes they take and the locations they stay in. For example, if it is possible to identify areas in a store where people tend to gather, it is possible to adjust the layout of merchandise so that more products can be sold to more people. For example, if it is possible to identify areas where people tend to gather, it is possible to efficiently perform air conditioning by focusing air conditioning on areas where people tend to gather. There is also a need to know the movement patterns of objects other than people, such as cars. Thus, there is a need for technology that not only detects objects from captured images but also analyzes the movement patterns of objects.

[0005] The present invention has been made in view of the above circumstances, and an object thereof is to provide an information processing method, an information processing system, and a computer program for analyzing the movement pattern of an object. [Means for solving the problem]

[0006] The information processing method according to the present invention is characterized in that it acquires an image captured by a camera, detects a specific object contained in the captured image, identifies the position of the detected object within the captured image, clusters the positions of multiple objects identified from multiple captured images, and determines an area within the captured image corresponding to the cluster obtained by clustering as a monitoring area in which the presence of the specific object should be monitored.

[0007] The computer program of the present invention is characterized in that it causes a computer to perform the following process: acquire an image captured by a camera, detect a specific object contained in the captured image, identify the position of the detected object within the captured image, cluster the positions of multiple objects identified from the multiple captured images, and determine an area within the captured image corresponding to the cluster obtained by clustering as a monitoring area in which the presence of the specific object should be monitored.

[0008] The information processing system of the present invention comprises a camera that photographs the outside world and an information processing device, and the information processing device acquires an image photographed by the camera, detects a specific object contained in the photographed image, identifies the position of the detected object within the photographed image, clusters the positions of the multiple objects identified from the multiple photographed images, and determines an area within the photographed image that corresponds to the cluster obtained by clustering as a monitoring area in which the presence of the specific object should be monitored.

[0009] In one aspect of the present invention, a specific object such as a person included in a photographed image is detected, the position of the object is identified, the identified positions are clustered, and a monitoring area within the photographed image corresponding to the cluster obtained by the clustering is determined. After the monitoring area is determined, the photographed image obtained is used to detect the object included in the monitoring area, thereby making it possible to analyze the movement pattern of the object. [Effects of the Invention]

[0010] The present invention has excellent effects such as making it possible to analyze the movement patterns of objects and appropriately adjust the facilities used by the objects in accordance with the movement patterns. [Brief explanation of the drawings]

[0011] [Figure 1] 1 is a schematic diagram illustrating an example of the configuration of an information processing system according to a first embodiment for analyzing a motion pattern of an object. [Figure 2] FIG. 2 is a block diagram illustrating an example of an internal functional configuration of the information processing device. [Figure 3] 10 is a flowchart illustrating an example of a procedure for setting a monitoring area. [Figure 4] FIG. 10 is a schematic diagram showing an example of a captured image that does not include a person. [Figure 5] FIG. 2 is a schematic diagram showing a first example of a captured image including a person. [Figure 6] FIG. 10 is a schematic diagram showing a second example of a captured image including a person. [Figure 7] FIG. 10 is a schematic diagram showing an example of the results of repeated person detection. [Figure 8] FIG. 2 is a schematic diagram showing an example of a monitoring area in a captured image. [Figure 9] 10 is a flowchart illustrating an example of a processing procedure for detecting a person included in a monitoring area. [Figure 10] 10 is a flowchart illustrating an example of a processing procedure for analyzing a movement pattern of a person. [Figure 11]10 is a diagram showing an example of identifying the time when a person was included in a monitored area. [Figure 12] 10 is a diagram showing an example of specifying the utilization rate of equipment. [Figure 13] FIG. 10 is a schematic diagram illustrating an example of the configuration of an information processing system according to a second embodiment. [Figure 14] FIG. 2 is a schematic diagram showing an example of the contents of relational data. [Figure 15] FIG. 10 is a schematic diagram showing an example of a power consumption plan. [Figure 16] FIG. 10 is a schematic diagram illustrating an example of the configuration of an information processing system according to a third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0012] The present invention will now be described in detail with reference to the drawings showing embodiments thereof. <Embodiment 1> FIG. 1 is a schematic diagram showing an example of the configuration of an information processing system 100 according to a first embodiment for analyzing the movement pattern of an object. The information processing system 100 executes an information processing method. FIG. 1 shows an example in which the information processing system 100 is installed in a sports gym. The sports gym is an example of a facility used by a person 4. A plurality of treadmills 3 are arranged in the sports gym. The person 4 trains using the treadmills 3. The information processing system 100 is equipped with a camera 2. The camera 2 captures an area inside the sports gym that includes the plurality of treadmills 3. The area captured by the camera 2 is fixed. For example, the camera 2 is configured using a wide-angle camera or a fisheye camera so as to be able to capture as wide an area as possible.

[0013] An information processing device 1 is connected to the camera 2. FIG. 2 is a block diagram showing an example of the internal functional configuration of the information processing device 1. The information processing device 1 is a computer such as a server device or a personal computer. The information processing device 1 includes a calculation unit 11, a memory 12, a drive unit 13, a storage unit 14, an operation unit 15, a display unit 16, and an interface unit 17. The calculation unit 11 is configured using, for example, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or a multi-core CPU. The calculation unit 11 may also be configured using a quantum computer. The memory 12 stores temporary data generated in conjunction with calculations. The memory 12 is, for example, a RAM (Random Access Memory). The drive unit 13 reads information from a recording medium 10 such as an optical disc or a portable memory. The storage unit 14 is non-volatile, for example, a hard disk or a non-volatile semiconductor memory.

[0014] The calculation unit 11 causes the drive unit 13 to read the computer program 141 recorded on the recording medium 10, and stores the read computer program 141 in the storage unit 14. The calculation unit 11 executes processing required for the information processing device 1 in accordance with the computer program 141. The computer program 141 may be a computer program product. The computer program 141 may be downloaded from outside the information processing device 1. Alternatively, the computer program 141 may be pre-stored in the information processing device 1. In these cases, the information processing device 1 does not need to be equipped with the drive unit 13.

[0015] The operation unit 15 receives input of information such as text by receiving operations from the user. The operation unit 15 is, for example, a touch panel, a pen tablet, a keyboard, or a pointing device. The display unit 16 displays images. The display unit 16 is, for example, a liquid crystal display or an EL display (Electroluminescent Display). The operation unit 15 and the display unit 16 may be integrated.

[0016] The camera 2 is connected to the interface unit 17. The camera 2 creates a captured image by capturing an image of a fixed area outside the camera 2 and inputs the captured image to the information processing device 1. The interface unit 17 accepts the captured image input from the camera 2. By accepting the captured image at the interface unit 17, the information processing device 1 acquires the captured image. The camera 2 continuously creates the captured images, or periodically repeatedly creates the captured images. The information processing device 1 continuously acquires the captured images, or periodically repeatedly acquires the captured images.

[0017] The information processing device 1 may be configured with multiple computers, and data may be stored and processed in a distributed manner by the multiple computers. The information processing device 1 may be realized using cloud computing, or may be realized by multiple virtual machines installed on a single computer. Alternatively, the information processing device 1 may be built into the camera 2.

[0018] The information processing device 1 includes a trained learning model 142. The learning model 142 is realized by the calculation unit 11 executing information processing in accordance with the computer program 141. The learning model 142 may be configured with hardware. The learning model 142 may be realized using a quantum computer. Alternatively, the learning model 142 may be provided outside the information processing device 1, and the information processing device 1 may execute processing using the external learning model 142. For example, the learning model 142 may be configured in the cloud.

[0019] The learning model 142 has been trained in advance by machine learning so that when a photographed image is input, it outputs a detection result that detects a person 4 included in the photographed image. The person 4 is a specific object. The detection result output by the learning model 142 includes the position of the person 4 in the photographed image. For example, a bounding box indicating the position of the person 4 in the photographed image and the center of the bounding box are output. For example, the learning model 142 is configured using YOLO (You Only Look Once). The learning model 142 may also be a model that uses a method other than YOLO, such as R-CNN (Region Based Convolutional Neural Networks) or segmentation.

[0020] The information processing device 1 executes processing to set a monitoring area in a captured image where the presence of a person 4 should be monitored. FIG. 3 is a flowchart showing an example of the processing procedure for setting the monitoring area. Hereinafter, step is abbreviated as S. The calculation unit 11 of the information processing device 1 executes processing in accordance with a computer program 141.

[0021] The camera 2 takes a photograph to create a photographed image, and inputs the photographed image to the information processing device 1. The information processing device 1 acquires the photographed image by receiving the photographed image input from the camera 2 via the interface unit 17 (S11). The calculation unit 11 stores the photographed image in the memory unit 14. The information processing device 1 inputs the photographed image to the learning model 142 (S12). In S12, the calculation unit 11 inputs the photographed image to the learning model 142 and causes the learning model 142 to execute processing. In response to the input of the photographed image, the learning model 142 outputs a detection result of detecting a person 4 included in the input photographed image. The information processing device 1 acquires the detection result of person 4 output by the learning model 142 (S13).

[0022] The information processing device 1 then determines whether person 4 has been detected by the learning model 142 (S14). In S14, the calculation unit 11 determines whether the detection result for person 4 output by the learning model 142 indicates that person 4 has been detected. For example, if the detection result includes a bounding box that encloses a part of the captured image and a probability that the part enclosed by the bounding box is person 4, and the probability is equal to or greater than a predetermined threshold, the calculation unit 11 determines that person 4 has been detected.

[0023] FIG. 4 is a schematic diagram showing an example of a captured image that does not include a person 4. FIG. 4 shows an example of a captured image of an empty gym. A camera 2 is installed on the ceiling of the gym, and a captured image is created by capturing multiple treadmills 3 from above. FIG. 5 is a schematic diagram showing a first example of a captured image that includes a person 4. The captured image shown in FIG. 5 includes the person 4. A bounding box 51 that surrounds the person 4 and a center point 52 of the bounding box 51 are shown. The center point 52 of the bounding box 51 is the position of the person 4 in the captured image. Note that the information processing device 1 may be configured to detect the person 4 included in the captured image using a method other than a method that uses a learning model, instead of steps S12 to S14. For example, the information processing device 1 may detect the person 4 included in the captured image according to a predetermined rule.

[0024] If person 4 is detected (S14: YES), the information processing device 1 stores the position of the detected person 4 in the captured image (S15). The storage unit 14 stores position data that records the position of the center point 52 of the bounding box 51. In S15, the calculation unit 11 stores the position of person 4 in the storage unit 14 by adding the position of the center point 52 of the bounding box 51 to the position data as the position of person 4. For example, the calculation unit 11 represents the position in the captured image using x and y coordinates, and records the x and y coordinate components corresponding to the center point 52 in the position data. A value other than the position of the center point 52, such as the center of gravity of person 4, may also be recorded as the position of person 4.

[0025] The information processing device 1 then determines whether to end detection of the person 4 (S16). For example, the calculation unit 11 measures the time that has elapsed since the camera 2 started capturing images, and determines to end detection of the person 4 when a predetermined time has elapsed, and determines not to end detection of the person 4 when the predetermined time has not yet elapsed. For example, the calculation unit 11 determines to end detection of the person 4 when the number of positions of the person 4 recorded in the position data is equal to or greater than a predetermined number, and determines not to end detection of the person 4 when the number of positions of the person 4 recorded in the position data is less than the predetermined number. For example, when the user operates the operation unit 15 to input an instruction to end detection of the person 4, the calculation unit 11 determines to end detection of the person 4.

[0026] When it is determined not to end the detection of person 4 (S16: NO), the information processing device 1 returns the process to S11 and repeats the processes of S11 to S16. By repeating the processes of S11 to S16, the information processing device 1 repeatedly acquires captured images and detects person 4 in the captured images. As a result, the information processing device 1 identifies the positions of multiple people 4 from multiple captured images. The camera 2 may create a moving image, and the information processing device 1 may acquire the moving image. In the processes of S11 to S16, the information processing device 1 identifies the position of person 4 from each of multiple captured images constituting the moving image. For example, the information processing device 1 performs the processes of S11 to S16 using one frame included in the moving image as the captured image, and when returning the process to S11, the information processing device 1 sets the next frame as the captured image and repeats the processes of S11 to S16 using each of the multiple frames included in the moving image as the captured image. Note that the process of the information processing device 2 is not limited to identifying the position of person 4 from all of the multiple captured images. The information processing device 2 may determine the position of person 4 from a portion of multiple captured images, such as determining the position of person 4 each time a predetermined number of captured images are obtained, or determining the position of person 4 from one frame each time multiple frames of a moving image are obtained.

[0027] Fig. 6 is a schematic diagram showing a second example of a captured image including person 4. Fig. 6 shows a captured image in which person 4 is included in a different position from the example shown in Fig. 5. A center point 52 of a bounding box 51 different from the example shown in Fig. 5 is obtained, and the position of center point 52 is added to the position data as the position of person 4 in the captured image. By repeating the processes of S11 to S16, person 4 is repeatedly detected, and the position of person 4 in a large number of captured images is recorded in the position data.

[0028] FIG. 7 is a schematic diagram showing an example of the results of repeated detection of person 4. In FIG. 7, bounding boxes 51 corresponding to repeatedly detected people 4 and center points 52 of the bounding boxes 51 are shown superimposed on the captured image. The rectangles and black dots shown in FIG. 7 are the bounding boxes 51 and center points 52. The captured image includes positions where many bounding boxes 51 are concentrated and positions where there are few bounding boxes 51.

[0029] Locations in the captured image where many bounding boxes 51 are concentrated are locations where person 4 has been detected many times and correspond to equipment where person 4 spends a long time. Locations in the captured image where few bounding boxes 51 are present are locations where person 4 has been detected few times and correspond to equipment where person 4 does not spend a long time. Many bounding boxes 51 are concentrated at locations in the captured image corresponding to the treadmill 3 used by person 4. There are fewer bounding boxes 51 at locations other than the location corresponding to the treadmill 3. By identifying an area where many bounding boxes 51 are concentrated, the position of the treadmill 3 in the captured image can be identified. Note that the information processing device 1 may be configured to acquire information other than the position of the center point 52 of the bounding box 51 as the position of person 4 in the captured image. For example, the information processing device 1 may acquire the position of person 4's face or the position of person 4's center of gravity as the position of person 4.

[0030] When it is determined that detection of the person 4 is to be ended (S16: YES), the information processing device 1 clusters the positions of the detected multiple people 4 (S17). In S17, the calculation unit 11 reads the position data from the storage unit 14 and clusters the positions of the center points 52 of the multiple bounding boxes 51 recorded in the position data. For example, clustering is performed so that multiple center points 52 that are close to each other in the captured image are included in the same cluster. In S17, the calculation unit 11 performs clustering using, for example, the k-means method. The user may input information necessary for clustering, such as the number of clusters, by operating the operation unit 15. The calculation unit 11 may perform clustering using a method other than the k-means method.

[0031] The information processing device 1 determines a monitoring area in which the presence of persons 4 should be monitored based on the clustering results (S18). In S18, the calculation unit 11 selects a cluster from among the multiple clusters obtained by clustering, the cluster including a number of positions of persons 4 equal to or greater than a predetermined threshold. The threshold is pre-stored in the storage unit 14 or is included in the computer program 141. The calculation unit 11 determines an area in the captured image corresponding to the selected cluster as the monitoring area. If multiple clusters are selected, multiple monitoring areas are determined. Note that the calculation unit 11 may perform processing to determine an area corresponding to a cluster including a number of positions of persons 4 exceeding the threshold as the monitoring area.

[0032] For example, in S18, the calculation unit 11 determines the average position of the people 4 included in the cluster as the center point of the monitoring area, and determines the range within a predetermined distance from the center point in the captured image as the monitoring area. The predetermined distance is pre-stored in the storage unit 14 or is included in the computer program 141. FIG. 8 is a schematic diagram showing an example of a monitoring area 53 in a captured image. The monitoring area 53 and a center point 54 of the monitoring area 53 are shown superimposed on the captured image. While FIG. 8 shows an example in which the shape of the monitoring area 53 is circular, the shape of the monitoring area 53 may be other than a circle, such as a square.

[0033] By setting the area corresponding to a cluster containing a number of person 4 positions equal to or greater than a predetermined threshold as the monitoring area 53, an area in which person 4 has been detected many times becomes the monitoring area 53, and an area in which person 4 has been detected few times does not become the monitoring area 53. In other words, the area in the captured image corresponding to the treadmill 3 becomes the monitoring area 53, and an area in the captured image corresponding to a location other than the treadmill 3 does not become the monitoring area 53. If multiple treadmills 3 are captured in the captured image, multiple monitoring areas 53 corresponding to the multiple treadmills 3 are defined. By detecting the people 4 included in the monitoring area 53, it is possible to investigate the movement patterns of the people 4 using the treadmills 3.

[0034] After S18 is completed, the information processing device 1 stores area data recording information indicating the monitoring area 53 in the storage unit 14, and ends the process of setting the monitoring area 53. The processes of S11 to S18 are repeatedly executed at a predetermined timing, and the monitoring area 53 is re-determined. For example, the processes of S11 to S18 are executed periodically, such as every time a predetermined time has elapsed. Alternatively, at any time, such as when the gym is redecorated, the user operates the operation unit 15 to input an instruction to execute the processes to the information processing device 1, and the processes of S11 to S18 are executed. By re-determining the monitoring area 53, it is possible to set the monitoring area 53 according to the latest equipment status. Note that in the processes of S11 to S18, person 4 is detected each time a photographed image is acquired; however, the information processing device 1 may also perform a process of detecting person 4 from photographed images obtained by continuous photographing over a long period of time or from multiple stored photographed images.

[0035] After determining the monitoring area 53, the information processing device 1 executes a process of detecting a person 4 included in the monitoring area 53 within the captured image. The process of detecting a person 4 included in the monitoring area 53 may be executed by an information processing device 1 different from the information processing device 1 that executed the process of setting the monitoring area 53. FIG. 9 is a flowchart showing an example of the procedure of the process of detecting a person 4 included in the monitoring area 53. The information processing device 1 acquires a captured image created by the camera 2 after determining the monitoring area 53 (S21). The calculation unit 11 stores the captured image in the storage unit 14. The information processing device 1 inputs the captured image to the learning model 142 (S22). In S22, the calculation unit 11 inputs the captured image to the learning model 142 and causes the learning model 142 to execute processing. In response to the input of the captured image, the learning model 142 outputs a detection result of detecting a person 4 included in the input captured image. The information processing device 1 acquires the detection result of the person 4 output by the learning model 142 (S23).

[0036] Note that, instead of the processes of S22 to S23, the information processing device 1 may perform a process of detecting a person 4 included in the monitoring area 53 using a method that does not use the learning model 142. For example, the information processing device 1 may detect a person 4 using a learning model different from the learning model 142 that has been trained in advance to output information indicating whether or not a person 4 is included in the monitoring area 53 when a captured image is input. By using a learning model different from the learning model 142, the calculation load of the information processing device 1 can be reduced. For example, the information processing device 1 may perform a process of detecting a person 4 using a method other than the method that uses a learning model.

[0037] The information processing device 1 then determines whether or not a person 4 has been detected within the monitoring area 53 (S24). In S24, the calculation unit 11 determines whether or not the person 4 detection result output by the learning model 142 indicates that the person 4 has been detected, and if the person 4 has been detected, determines whether or not the position of the detected person 4 is included in any of the monitoring areas 53. For example, if the center point 52 of the bounding box 51 surrounding the detected person 4 is located within the monitoring area 53, the calculation unit 11 determines that the person 4 has been detected within the monitoring area 53. For example, if the person 4 has not been detected, or if the center point 52 of the bounding box 51 surrounding the detected person 4 is located outside the monitoring area 53, the calculation unit 11 determines that the person 4 has not been detected within the monitoring area 53. If the person 4 has not been detected within the monitoring area 53 (S24: NO), the information processing device 1 ends the process of detecting the person 4.

[0038] If a person 4 is detected in a monitoring area 53 (S24: YES), the information processing device 1 identifies the density of the detected people 4 (S25). In S25, the calculation unit 11 identifies the density of the people 4 according to the number of monitoring areas 53 in which the person 4 is detected. For example, the calculation unit 11 identifies the density of the people 4 by dividing the number of monitoring areas 53 in which the person 4 is detected by the total number of monitoring areas 53. The information processing device 1 then determines whether the density of the identified people 4 is equal to or greater than a predetermined threshold density (S26). The threshold density is pre-stored in the storage unit 14 or is included in the computer program 141. In S26, the calculation unit 11 compares the density of the people 4 with the threshold density.

[0039] If the density of people 4 is equal to or greater than the critical density (S26: YES), the information processing device 1 outputs a warning indicating that the density of people 4 is high (S27). For example, in S27, the calculation unit 11 outputs the warning by displaying an image on the display unit 16 warning that the density of people 4 is high. Alternatively, the information processing device 1 may include an audio output unit, and the calculation unit 11 may output audio from the audio output unit warning that the density of people 4 is high. By being warned that the density of people 4 is high, it is possible to warn people 4 using the treadmill 3 and encourage them to disperse. For example, the warning can help prevent the spread of infectious diseases. The calculation unit 11 may also perform processing to output a warning when the density of people 4 exceeds the critical density.

[0040] If the density of people 4 is less than the threshold density (S26: NO), or after S27 is completed, the information processing device 1 associates the time with the monitored area 53 in which the person 4 was detected and stores the associated information (S28). The calculation unit 11 performs a process to acquire the time. For example, the calculation unit 11 measures the time. For example, the camera 2 measures the time, attaches the time of shooting to the captured image, and inputs the image to the information processing device 1, and the calculation unit 11 acquires the time attached to the captured image. The time includes the year, month, and day. The storage unit 14 stores detection result data in which the time is associated with information indicating the monitored area 53 in which the person 4 was detected. In S28, the calculation unit 11 adds the time the person 4 was detected or the time the captured image including the detected person 4 was created, and information indicating the monitored area 53 in which the person 4 was detected to the detection result data in association with each other, thereby storing the time and the monitored area 53 in which the person 4 was detected in the storage unit 14.

[0041] After S28 is completed, the information processing device 1 ends the process of detecting person 4 included in the monitoring area 53. The processes of S21 to S28 are repeatedly executed. For example, the processes of S21 to S28 are repeatedly executed throughout the day. The monitoring area 53 including person 4 is identified by the processes of S21 to S28. The treadmill 3 corresponding to the identified monitoring area 53 is the treadmill 3 used by person 4, and therefore the treadmill 3 used by person 4 is identified. By repeating the processes of S21 to S28, the number of times the treadmill 3 has been used by person 4 can be identified, and the time period during which the treadmill 3 has been used by person 4 can be identified based on the number of times.

[0042] The processes of S21 to S28 are repeated, and after the time and information on the monitoring area 53 in which the person 4 was detected are accumulated in the detection result data, the information processing device 1 performs a process of analyzing the movement pattern of the person 4 in the sports gym. The process of analyzing the movement pattern of the person 4 may be performed by an information processing device 1 different from the information processing device 1 that performed the process of setting the monitoring area 53 or the process of detecting the person 4 included in the monitoring area 53.

[0043] FIG. 10 is a flowchart showing an example of a processing procedure for analyzing the movement pattern of person 4. Based on the detection result data, the information processing device 1 identifies the time during which person 4 was included in the monitoring area 53 during a specific time period (S31). The specific time period is a period obtained by dividing a day into multiple periods. For example, the period from midnight to 1:00, the period from 1:00 to 2:00, etc. are each defined as a specific time period. In S31, the calculation unit 11 reads information indicating the time included in the specific time period and the monitoring area 53 in which person 4 was detected from the detection result data, and calculates the time during which person 4 was included in the monitoring area 53 based on the read information. For example, if person 4 is detected at one-minute intervals, the time during which person 4 was included in the monitoring area 53 is calculated by multiplying the number of times person 4 was detected in the monitoring area 53 by one minute. The calculation unit 11 calculates the time during which person 4 was included in each of the multiple monitoring areas 53.

[0044] In S31, the calculation unit 11 may total the time during which the person 4 was included in the monitored area 53 during a specific time period within a day, over multiple days. For example, the calculation unit 11 totals the time during which the person 4 was included in the monitored area 53 between midnight and 1 a.m., between 1 a.m. and 2 a.m., etc., over multiple days, such as a week or a month. The calculation unit 11 may total the time during which the person 4 was included in the monitored area 53 during each time period, by type of day. For example, the calculation unit 11 may total the time during which the person 4 was included in the monitored area 53 between midnight and 1 a.m., between 1 a.m. and 2 a.m., etc., by day of the week, or by distinguishing between weekdays and holidays.

[0045] FIG. 11 is a chart showing an example of identifying the time during which a person 4 was included in a monitoring area 53. The treadmills 3 corresponding to the multiple monitoring areas 53 are designated as treadmill 01, treadmill 02, treadmill 03, and treadmill 04. FIG. 11 shows the total time during one month during which a person 4 was included in each monitoring area 53 during each time period, such as between midnight and 1:00, between 1:00 and 2:00, and so on. The gym is open 25 days a month between 10:00 and 20:00, and 20 days a month between midnight and 10:00 and between 20:00 and 24:00. The numerical values ​​shown in FIG. 11 are in minutes. As shown in FIG. 11, the time during which a person 4 was included in each monitoring area 53 during each time period is identified.

[0046] The information processing device 1 then identifies a utilization rate indicating the proportion of time during which the equipment corresponding to the monitored area 53 is in use during a specific time period, based on the time identified in S31 (S32). The time during which the person 4 was included in the monitored area 53 is the time during which the equipment corresponding to the monitored area 53 was used by the person 4, and the utilization rate of the equipment is obtained by dividing this time by the length of the time period. An example of the equipment corresponding to the monitored area 53 is the treadmill 3. In S31, the calculation unit 11 calculates the utilization rate by dividing the time during which the person 4 was included in the monitored area 53 by the length of the time period. If the total time during which the person 4 was included in the monitored area 53 is the total time over multiple days, the utilization rate is calculated by dividing the time during which the person 4 was included in the monitored area 53 by the value obtained by multiplying the length of the time period by the number of days.

[0047] FIG. 12 is a chart showing an example of identifying the utilization rate of equipment. The equipment corresponding to the monitoring area 53 is treadmill 01, treadmill 02, treadmill 03, and treadmill 04. FIG. 12 shows the utilization rate of each treadmill 3 for one month in each time period, such as between midnight and 1:00, between 1:00 and 2:00, etc. The utilization rate is associated with each treadmill 3 and time period. FIG. 12 also shows the daily utilization rate of each treadmill 3 and the overall utilization rate of multiple treadmills 3 in each time period. The calculation unit 11 may calculate these utilization rates. The daily utilization rate is the average of the utilization rates for multiple time periods. The overall utilization rate of multiple treadmills 3 is the average of the utilization rates of multiple treadmills 3.

[0048] The length of the time period does not have to be one hour and may be other lengths. For example, the length of the time period may be one day. For example, the time that person 4 is included in the monitored area 53 on a particular day of the week may be summed over multiple weeks to calculate the utilization rate of the equipment on a particular day of the week over multiple weeks.

[0049] The information processing device 1 then outputs the utilization rate associated with the specific time period and equipment (S33). The memory unit 14 stores utilization rate data in which utilization rates are recorded in association with the time period and equipment. In S33, the calculation unit 11 adds the identified utilization rate to the utilization rate data in association with the specific time period and equipment, and displays the utilization rate associated with the specific time period and equipment on the display unit 16, thereby outputting the utilization rate. For example, the calculation unit 11 displays a chart on the display unit 16, as shown in FIG. 12, in which utilization rates are associated with each time period and equipment. The information processing device 1 may output the utilization rate by a method other than displaying it on the display unit 16. For example, the information processing device 1 is provided with a printer, and outputs a chart from the printer in which utilization rates are associated with each time period and equipment. For example, the information processing device 1 outputs the utilization rate data in the form of electronic data.

[0050] By outputting the equipment utilization rate, the user can know the utilization rate of the equipment for each time period. By clarifying the utilization rate of the equipment, the movement pattern of person 4, which indicates which equipment person 4 uses at which time period, becomes clear. By adjusting the equipment according to the revealed movement pattern of person 4, the efficiency of the facility where the equipment is located can be improved. For example, the efficiency of a sports gym can be improved by removing treadmills 3 with low utilization rates, stopping treadmills 3 during times when utilization rates are low, or adjusting the placement of treadmills 3 so that the utilization rate of all treadmills 3 increases.

[0051] The information processing device 1 then creates a power consumption plan based on the utilization rate associated with the specific time period and the equipment (S34). In S34, for example, the calculation unit 11 creates a power consumption plan that stops the treadmill 3 during a certain time period if the utilization rate of the treadmill 3 falls below a predetermined value. For example, the calculation unit 11 creates a power consumption plan that increases the power consumption of the air conditioning during time periods when the utilization rate is equal to or greater than the predetermined value, and decreases the power consumption of the air conditioning during time periods when the utilization rate is below the predetermined value.

[0052] The information processing device 1 then outputs the created power consumption plan (S35). In S35, the calculation unit 11 stores plan data recording the power consumption plan in the memory unit 14 and outputs the power consumption plan by displaying it on the display unit 16. The information processing device 1 may output the power consumption plan by a method other than displaying it on the display unit 16. For example, the information processing device 1 outputs the plan data in the form of electronic data. The user can refer to the output power consumption plan to appropriately adjust power consumption. For example, power consumption can be reduced by stopping a treadmill 3 that is underutilized. For example, by increasing the power consumption of the air conditioning during times of high utilization, the temperature in a sports gym can be appropriately adjusted, improving the comfort of the sports gym.

[0053] After S35 is completed, the information processing device 1 ends the process of analyzing the movement pattern of the person 4. The processes of S31 to S35 are performed as needed, with the time and information on the monitoring area 53 in which the person 4 was detected stored in the detection result data. For example, the processes of S11 to S18 are performed at a predetermined timing, and after the monitoring area 53 is re-determined, the processes of S21 to S28 are repeatedly performed, and then the processes of S31 to S35 are performed. Of the processes of S31 to S35, the processes of S31 to S33 and the processes of S34 to S35 may be performed at different timings. Alternatively, after the processes of S31 to S33 are performed, the processes of S34 to S35 may not be performed.

[0054] As described above, in this embodiment, a person 4 included in a captured image is detected, the position of the person 4 is identified, the identified positions are clustered, and a monitoring area 53 within the captured image corresponding to the cluster obtained by clustering is determined. Then, the person 4 included in the monitoring area 53 within the captured image is detected. The area within the captured image corresponding to the facility frequently used by the person 4 can be easily set as the monitoring area 53. By detecting the person 4 included in the monitoring area 53 within the captured image, it is possible to record a behavioral history indicating which facility the person 4 used and when, and to investigate the movement pattern of the person 4 based on the behavioral history. For example, the utilization rate of each treadmill 3 at a sports gym during each time period can be obtained. The facilities used by the person 4 can be appropriately adjusted according to the movement pattern of the person 4. For example, by creating a power consumption plan for the sports gym based on the utilization rate of the treadmill 3, it is possible to improve the efficiency of power consumption.

[0055] <Embodiment 2> In the first embodiment, an example was shown in which a single camera 2 was connected to the information processing device 1, but in the second embodiment, the information processing system 100 includes multiple cameras 2. FIG. 13 is a schematic diagram showing an example of the configuration of the information processing system 100 according to the second embodiment. The information processing system 100 includes multiple cameras 2, and the multiple cameras 2 are connected to the information processing device 1. The multiple cameras 2 capture images of different areas in a sports gym. The multiple cameras 2 each input the captured image to the information processing device 1. The information processing device 1 similarly executes the processes of S11 to S18 and S21 to S28. Monitoring areas 53 in the multiple captured images are determined, and people 4 included in each monitoring area 53 are detected.

[0056] The information processing device 1 stores relationship data in the storage unit 14 that indicates the relationship between multiple cameras, facilities, and power-consuming devices. FIG. 14 is a schematic diagram showing an example of the content of the relationship data. In the relationship data, the cameras, facilities, and power-consuming devices are associated with one another. FIG. 14 shows the relationship between a camera 2, a treadmill 3 that is a facility, and lighting and air conditioning that are power-consuming devices. The area photographed by camera 01, which is one camera 2, includes treadmills 01 to 04. Lighting 01 to 04 are located near treadmills 01 to 04, respectively. Air conditioning unit 01 is located near treadmills 01 and 02, and air conditioning unit 02 is located near treadmills 03 and 04. Similarly, the area photographed by camera 02 includes treadmills 05 to 08, lights 05 to 08 are respectively placed near treadmills 05 to 08, air conditioner 03 is placed near treadmills 05 and 06, and air conditioner 04 is placed near treadmills 07 and 08. Similarly, the relationships between cameras, facilities, and power-consuming devices are recorded in the relationship data. For example, in the relationship data, IDs for identifying each camera, facility, and power-consuming device are recorded in association with one another.

[0057] The information processing device 1 similarly executes the processes of S31 to S35. The utilization rate of treadmill 3 is obtained, and a power consumption plan is created. The information processing device 1 references the relational data and creates a power consumption plan according to the utilization rate of the equipment associated with the power-consuming devices. For example, the information processing device 1 creates a power consumption plan for each of lights 01 to 04 according to the utilization rate of each of treadmills 01 to 04. The information processing device 1 also creates a power consumption plan for air conditioner 01 according to the utilization rates of treadmills 01 and 02, and creates a power consumption plan for air conditioner 02 according to the utilization rates of treadmills 03 and 04.

[0058] FIG. 15 is a schematic diagram showing an example of a power consumption plan. FIG. 15 shows an example in which time periods are associated with the power consumption of each light. For example, the information processing device 1 creates a power consumption plan such that the associated light is turned on during time periods when the treadmill 3 is highly utilized and turned off during time periods when the treadmill 3 is less utilized. FIG. 15 also shows an example in which time periods are associated with the power consumption of each air conditioner. For example, the information processing device 1 creates a power consumption plan such that the associated air conditioner's power consumption is increased during time periods when the treadmill 3 is highly utilized, decreased during time periods when the treadmill 3 is less utilized, and turned off during time periods when the treadmill 3 is not utilized. In S35, the information processing device 1 outputs a power consumption plan such as that shown in FIG. 15. The output power consumption plan includes an ID for identifying each piece of equipment. The user can refer to the output power consumption plan to appropriately adjust power consumption.

[0059] As described above, even if it is not possible to photograph all of the treadmills 3 in a gym with a single camera 2, it is possible to investigate the movement patterns of the person 4 by using multiple cameras 2. In other words, it is possible to identify the utilization rate of each treadmill 3 and create a power consumption plan. Therefore, in the second embodiment, even if it is not possible to photograph all of the equipment with a single camera 2, it is possible to investigate the movement patterns of the person 4 by using multiple cameras 2.

[0060] <Embodiment 3> FIG. 16 is a schematic diagram showing an example of the configuration of an information processing system 100 according to a third embodiment. There are multiple facilities 6, and a camera 2 is installed in each facility 6. The facility 6 is, for example, a sports gym equipped with multiple treadmills 3. Multiple cameras 2 may be installed in one facility 6. Each camera 2 is connected to a communication network N, such as the Internet, using wired or wireless communication. An information processing device 1 is connected to the communication network N. The camera 2 captures images of the interior of the facility 6 in which the camera 2 is installed, and transmits the captured images to the information processing device 1 via the communication network N. The information processing device 1 acquires the captured images by receiving the captured images transmitted from the multiple cameras 2.

[0061] The information processing device 1 executes the processes of S11 to S18, S21 to S28, and S31 to S35 for each facility 6 based on the captured images acquired from the cameras 2 installed in each facility 6. This allows a monitoring area 53 to be determined for each facility 6, people 4 included in the monitoring area 53 to be detected, and the movement patterns of people 4 in each facility 6 to be investigated. For example, the utilization rate of treadmills 3 in each of a plurality of sports gyms can be obtained, and a power consumption plan for each of the plurality of sports gyms can be created.

[0062] In the first to third embodiments, the camera 2 captures a fixed area. However, the camera 2 may capture a wider area by changing the capture direction. In the first to third embodiments, a sports gym is cited as an example of a facility targeted by the information processing system 100. However, the facility targeted by the information processing system 100 may be a facility other than a sports gym. For example, the facility targeted by the information processing system 100 may be a store such as a supermarket. In this embodiment, the movement pattern of the person 4 in the store is obtained. For example, the monitored area 53 corresponds to the product display shelves or checkout area in the store, and the utilization rate of each facility is identified. By adjusting the product layout according to the utilization rate of the facilities, such as by displaying desired products on product display shelves with high utilization rates, the store's sales can be increased. Furthermore, by creating a power consumption plan according to the utilization rate, the store's power consumption can be improved by more appropriately adjusting the utilization of air conditioning or lighting, for example.

[0063] In the first to third embodiments, the object is a person 4, and the person 4 is detected from a captured image. However, the information processing system 100 may be configured to detect an object other than a person from a captured image. For example, the object may be an animal other than a human. The object may be an article. For example, the information processing system 100 may consider an article that is processed by a machine while being transported on a conveyor in a factory as an object, detect the article included in the monitoring area 53 corresponding to the machine, and determine the utilization rate of each machine. The object may be a vehicle. For example, a camera 2 may be installed in a parking lot, and vehicles included in the monitoring area 53 corresponding to a parking space in the parking lot may be detected, thereby determining the utilization rate of the parking space.

[0064] The present invention is not limited to the above-described embodiments, and various modifications are possible within the scope of the claims. In other words, embodiments obtained by combining technical means modified appropriately within the scope of the claims are also included in the technical scope of the present invention. [Explanation of symbols]

[0065] 100 Information Processing Systems 1. Information processing equipment 10 Recording media 141 Computer Programs 142 Learning Model 2 Cameras 3. Treadmill 4 People (objects) 51 Bounding Box 52 Bounding box center point 53 Monitoring area 54 Center point of the monitoring area 6 Facilities N Communication Network

Claims

1. Acquire the image captured by the camera, Detecting a specific object included in the captured image; Identifying the position of the detected object within the captured image; Clustering the positions of multiple objects identified from multiple captured images, determining an area in the captured image corresponding to the cluster obtained by the clustering as a monitoring area in which the presence of a specific object should be monitored; Detecting a specific object included in the monitoring area in a captured image captured after the monitoring area is determined; Based on the object detection result, a time during which the object was included in the monitoring area during a specific time period is identified; determining a utilization rate indicating a proportion of time during which the equipment corresponding to the monitoring area is in use during the specific time period based on the determined time period; Outputting the utilization rate associated with the equipment corresponding to the specific time period and the monitoring area An information processing method comprising:

2. If the number of object positions included in a cluster is equal to or greater than a threshold, the area corresponding to the cluster is determined as the monitoring area.

2. The information processing method according to claim 1,

3. Based on a plurality of images taken at a plurality of locations by a plurality of cameras, the utilization rate of the equipment at each of the plurality of locations is identified and output.

3. The information processing method according to claim 1 or 2.

4. outputting a power consumption plan for the facility based on the specific time period and the utilization rate associated with the facility; 4. The information processing method according to claim 1, wherein the information processing method comprises:

5. determining an object density based on the number of specific objects contained in the monitoring area; If the density is equal to or greater than a predetermined limit density, a warning is output.

5. The information processing method according to claim 1, wherein:

6. The monitoring area is redefined at a predetermined timing.

6. The information processing method according to claim 1, wherein:

7. Acquire the image captured by the camera, Detecting a specific object included in the captured image; Identifying the position of the detected object within the captured image; Clustering the positions of multiple objects identified from multiple captured images, determining an area in the captured image corresponding to the cluster obtained by the clustering as a monitoring area in which the presence of a specific object should be monitored; Detecting a specific object included in the monitoring area in a captured image captured after the monitoring area is determined; Based on the object detection result, a time during which the object was included in the monitoring area during a specific time period is identified; determining a utilization rate indicating a proportion of time during which the equipment corresponding to the monitoring area is in use during the specific time period based on the determined time period; Outputting the utilization rate associated with the equipment corresponding to the specific time period and the monitoring area A computer program that causes a computer to execute a process.

8. A camera that captures the exterior, an information processing device; The information processing device includes: Acquire a photographed image taken by the camera; Detecting a specific object included in the captured image; Identifying the position of the detected object within the captured image; Clustering the positions of multiple objects identified from multiple captured images, determining an area in the captured image corresponding to the cluster obtained by the clustering as a monitoring area in which the presence of a specific object should be monitored; Detecting a specific object included in the monitoring area in a captured image captured after the monitoring area is determined; Based on the object detection result, a time during which the object was included in the monitoring area during a specific time period is identified; determining a utilization rate indicating a proportion of time during which the equipment corresponding to the monitoring area is in use during the specific time period based on the determined time period; Outputting the utilization rate associated with the equipment corresponding to the specific time period and the monitoring area An information processing system comprising:

Images