System and program

Abstract

To provide a system capable of acquiring and outputting information on the number of people in a line and a waiting time in particular among information based on an image photographed by a camera.SOLUTION: A system for analyzing a video captured by a camera, the system comprising: a setting unit configured to set an area in the video; a setting unit configured to set a line corresponding to one or more sides of the area in association with the area; and a counting unit configured to count the number of targets exiting from the area excluding a target passing through the line.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] The present disclosure relates to a system and a program. [Background technology]

[0002] Conventionally, technology has been known that analyzes video captured by a camera and detects people in the video, and there have been attempts to use the video captured by such cameras for DX (digital transformation). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 7279241 [Patent Document 2] Patent No. 7302088 Summary of the Invention [Problem to be solved by the invention]

[0004] Patent Documents 1 and 2 describe systems that detect when a person has stayed in a specified area.

[0005] The inventions described in Patent Documents 1 and 2 notify users when a certain number of people remain in a specified area for a certain period of time, but they are not sufficient in terms of providing information regarding the waiting time for those staying in the area.

[0006] One of the problems that the present disclosure aims to solve is to provide a system that can acquire and output information based on images captured by a camera, particularly information regarding the number of people waiting in line and waiting times.

[0007] One example of a system of the present disclosure is a system that analyzes video captured by a camera, sets an area in the video, counts the number of objects that stay within the area for more than a threshold, and outputs the number of counted objects.

[0008] Furthermore, one example of a system according to the present disclosure is a system that analyzes video captured by a camera, sets areas in the video, counts the number of objects that stay within the area for a threshold or more, and outputs the number of counted objects. If multiple areas are set, the number of objects counted within each area can be output together.

[0009] Furthermore, one example of a system according to the present disclosure is a system that analyzes video captured by a camera, sets an area in the video, counts the number of objects that stay within the area for a threshold or more, and is capable of outputting the number of counted objects as well as one of the average number of people staying, the average length of stay, or the maximum length of stay.

[0010] Furthermore, one example of a system according to the present disclosure is a system that analyzes video captured by a camera, and is capable of setting areas and lines from the video and counting the number of objects that have left the area, excluding those that have passed through the line.

[0011] Furthermore, one example of a system of the present disclosure is a system that analyzes video captured by a camera, and is capable of setting areas and lines from the video and counting the number of objects that have passed through the lines among those that have exited the area.

[0012] Furthermore, one example of a program of the present disclosure is a program that analyzes video captured by a camera, sets an area in the video, counts the number of objects that stay within the area for more than a threshold, and outputs the number of counted objects.

[0013] Furthermore, one example of a method of the present disclosure is a method of analyzing video captured by a camera, in which an area is set in the video, the number of objects that stay within the area for a threshold or more is counted, and the number of counted objects is output. [Brief explanation of the drawings]

[0014] [Figure 1] FIG. 1 is a diagram illustrating an example of a system configuration of an information processing system. [Figure 2] FIG. 2 is a diagram illustrating an example of a hardware configuration of the server device. [Figure 3] FIG. 3 is a diagram illustrating an example of a hardware configuration of the client device. [Figure 4] FIG. 4 is a diagram illustrating an example of a hardware configuration of a camera. [Figure 5] FIG. 5 is a sequence diagram showing an example of the flow of processing executed in the system. [Figure 6] FIG. 6 is a diagram showing an example of area setting. [Figure 7] FIG. 7 is a diagram showing an example of a setting screen for thresholds to be counted. [Figure 8] FIG. 8 is a diagram showing an example of a display mode of a timeline. [Figure 9] FIG. 9 is a diagram showing an example of data output. [Figure 10] FIG. 10 is a diagram showing another example of data output. [Figure 11] FIG. 11 is a detailed explanation of the setting of conditions in step S501 of FIG. [Figure 12] FIG. 12 is a diagram showing an example of area setting. [Figure 13] FIG. 13 is a diagram showing an example of a pop-up that appears when setting a normal exit line. [Figure 14] FIG. 14 is a diagram showing an example of setting a normal exit line after setting an area. [Figure 15]FIG. 15 is a diagram showing an example in which a normal exit line that is set after an area is set is linked to the area and saved. [Figure 16] FIG. 16 is a diagram showing an example of ex post setting of a regular exit line to an area that has already been set. [Figure 17] FIG. 17 is a diagram showing an example in which a regular exit line that has been set afterwards for an area that has already been set is linked to the area and saved. [Figure 18] FIG. 18 is a detailed description of the detection process in S512 of FIG. [Figure 19] FIG. 19 is a diagram showing an example of the exit data table. [Figure 20] FIG. 20 is a flowchart showing an example of information processing when outputting target count information. [Figure 21] FIG. 21 is a diagram showing another example of data output. DETAILED DESCRIPTION OF THE INVENTION

[0015] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. The configurations of the embodiments described below, as well as the actions and effects brought about by the configurations, are merely examples and are not limited to the following description.

[0016] Fig. 1 is a diagram showing an example of the system configuration of an information processing system 1000. As shown in Fig. 1, the information processing system 1000 includes, as a system configuration, a server device 100, a client device 110, and a camera 160. The server device 100, the client device 110, and the camera 160 are communicably connected via a network 150. The network 150 may include either or both of a WAN (Wide Area Network) and the Internet. The network 150 is configured to enable communication between multiple devices connected to the network 150 via wired and wireless connections.

[0017] The information processing system 1000 is a system that counts the number of objects that stay in an area for a certain period of time.

[0018] The server device 100 is a device that provides the main functions of the information processing system 1000, and executes the main processes of the embodiment described below. The server device 100 is a device that outputs the video captured by the camera 160.

[0019] The camera 160 is a camera installed for the purpose of crime prevention and / or recording. In the description, it is assumed that the camera 160 is installed in each of multiple locations in a specific store. Examples of multiple locations include multiple entrances and exits to the store (e.g., an east entrance, a west entrance, a dedicated exit, etc.). Note that these are examples and do not limit the locations or number of cameras 160 installed. For simplicity, only three cameras 160 are shown in FIG. 1, but there may be one, four or more. It is sufficient that the camera 160 is included in the information processing system 1000. In addition, it is assumed that multiple cameras are installed in one store, but multiple stores may each have multiple cameras installed.

[0020] The client device 110 is a terminal device operated by the owner or manager of the camera 160 (hereinafter simply referred to as the user).

[0021] Here, the claimed information processing system may be composed of multiple devices or may be composed of a single device. When the claimed information processing system is composed of a single device, an example of the device is server device 100. When the claimed information processing system is composed of a plurality of devices, an example of the plurality of devices is at least one of server device 100 and camera 160, or a cloud server composed of a plurality of server devices that provide the functions of server device 100.

[0022] Fig. 2 is a diagram showing an example of the hardware configuration of server device 100. As shown in Fig. 2, server device 100 includes, as its hardware configuration, a control unit 210, a storage unit 220, a communication unit 230, and an internal bus 240. Control unit 210, storage unit 220, and communication unit 230 are electrically connected via internal bus 240.

[0023] The control unit 210 is a CPU (Central Processing Unit) or the like, and controls the entire server device 100 .

[0024] The storage unit 220 is any one of an HDD (Hard Disk Drive), a ROM (Read Only Memory), a RAM (Random Access Memory), an SSD (Solid State Drive), etc., or any combination thereof, and stores programs, data used when the control unit 210 executes processing based on the programs, etc. The storage unit 220 is an example of a storage medium. Examples of data used when the control unit 210 executes processing based on the programs include video data sent from the camera 160, a facial image of the detection target, data related to the detection target, etc.

[0025] In the present specification, the data used by control unit 210 when executing processing based on a program is described as being stored in storage unit 220, but the data may also be stored in a storage unit of another device that can communicate with server device 100. The data may be stored in a storage unit of any device as long as it can be referenced or acquired by control unit 210. When control unit 210 executes processing based on a program stored in storage unit 220, the functions of server device 100 and the processing of the sequence diagram shown in Figure 5, which will be described later, are realized.

[0026] The communication unit 230 connects the server device 100 to the network 150 and controls communication with other devices.

[0027] The hardware configuration of each of the control unit 210, the storage unit 220, and the communication unit 230 is not limited to one. For example, a plurality of control units may be included in the server device 100. The same applies to the client device 110 described below.

[0028] FIG. 3 is a diagram illustrating an example of the hardware configuration of the client device 110. As shown in FIG. 3, client device 110 includes, as its hardware configuration, a control unit 310, a storage unit 320, an input unit 330, an output unit 340, a communication unit 350, and an internal bus 360. Control unit 310, storage unit 320, input unit 330, output unit 340, and communication unit 350 are electrically connected via internal bus 360.

[0029] The control unit 310 is a CPU or the like, and controls the entire client device 110 .

[0030] The storage unit 320 is any one of an HDD, a ROM, a RAM, an SSD, etc., or any combination thereof, and stores programs, data used when the control unit 310 executes processing based on the programs, etc. The storage unit 320 is an example of a storage medium.

[0031] In the specification, the data used by control unit 310 when executing processing based on a program is described as being stored in storage unit 320, but the data may also be stored in a storage unit of another device that can communicate with client device 110. The data may be stored in a storage unit of any device as long as it can be referenced or acquired by control unit 310. The functions and the like of client device 110 are realized by control unit 310 executing processing based on the program stored in storage unit 320.

[0032] The input unit 330 is a device that inputs information to the client device 110 in response to an operation by an operator. The input unit 330 accepts an operation input made by a user. The operation input is transferred as a command signal to the control unit 310 via the internal bus 360. The control unit 310 may execute predetermined control and / or calculation based on the transferred command signal as necessary. The input unit 330 may be included in the housing of the client device 110 or may be externally attached. For example, the input unit 330 may be implemented as a touch panel integrated with the output unit 340. When the input unit 330 is implemented as a touch panel, the user can input tap operations, swipe operations, etc. to the input unit 330. Instead of a touch panel, a switch button, a mouse, a trackpad, a keyboard, etc. may be used as the input unit 330.

[0033] The output unit 340 is, for example, a display, and is a device that outputs information as a screen of a graphical user interface (GUI) that can be operated by a user. The output unit 340 may be included in the housing of the client device 110, or may be externally attached. More specifically, the output unit 340 may be implemented as a display device such as a liquid crystal display, an organic EL (Electron-Luminescence) display, or a plasma display. It is preferable that these display devices are implemented by selectively using them depending on the type of client device 110.

[0034] The communication unit 350 connects the client device 110 to the network 150 and controls communication with other devices.

[0035] In the description, a PC (Personal Computer) is used as an example of the client device 110. However, the client device 110 may be a smartphone, a tablet computer, or the like. The client device 110 may be any device that can display a screen as described below, accept user operations via the screen, or the like, and transmit information to the server device 100.

[0036] Fig. 4 is a diagram showing an example of the hardware configuration of camera 160. As shown in Fig. 4, camera 160 includes, as its hardware configuration, a control unit 410, a storage unit 420, an image capturing unit 430, a communication unit 440, and an internal bus 450. Control unit 410, storage unit 420, image capturing unit 430, and communication unit 440 are electrically connected via internal bus 450.

[0037] The control unit 410 is a CPU or the like, and controls the entire camera 160 .

[0038] The storage unit 420 is any one of an HDD, a ROM, a RAM, an SSD, etc., or any combination thereof, and stores programs, data used when the control unit 410 executes processing based on the programs, etc. The storage unit 420 is an example of a storage medium.

[0039] In the specification, the data used when control unit 410 executes processing based on a program is described as being stored in storage unit 420, but the data may also be stored in a storage unit of another device that can communicate with camera 160. The data may be stored in a storage unit of any device as long as control unit 310 can refer to or acquire the data. The functions of camera 160 are realized by control unit 410 executing processing based on the program stored in storage unit 420.

[0040] The photographing unit 430 is a camera that photographs a subject, and includes, for example, an image sensor, a lens, and an IR cut filter.

[0041] The communication unit 440 connects the camera 160 to the network 150 and controls communication with other devices.

[0042] FIG. 5 is an exemplary schematic sequence diagram showing the flow of processing executed in the system 1000 according to the embodiment.

[0043] First, a description will be given of a setting process for setting various conditions for the camera 160 using the client device 110. Details will be described later, but the conditions are, for example, conditions related to what criteria to use for detecting a person, which area within the shooting range of the camera 160 to target for detecting a person, and how to notify that a person has been detected.

[0044] In the setting process, first, in step S501, the input unit 330 of the client device 110 accepts the setting of conditions based on a user operation via a UI (user interface) such as those shown in FIGS. 6 to 10 below.

[0045] FIG. 6 is an exemplary schematic diagram for explaining area setting as an example of condition setting according to the embodiment.

[0046] An image 601 shown in FIG. 6 is an example of a UI that accepts operations related to setting areas and lines by the user.

[0047] In this image 601, a list of areas 602 with the names of areas that have already been set is displayed on the left side, along with a new button 603 for registering a new area. In each area 602, the icon on the left is linked to the design and color of the border of the area set in image 600 displayed in area 601A. By clicking this icon and selecting the border design and color, it is possible to set the desired design for each area. In addition, by clicking the delete button on the right side of each area 602, the corresponding area can be deleted. When one of the areas 602 is clicked to select it, its name and the corresponding area in image 600 displayed in area 601A are both highlighted, making it easy to identify that it has been selected. When an area is selected, the position and shape of the area can be changed by clicking or dragging the corresponding area in image 600, for example.

[0048] 6, there are two areas 611 and 612 that have been set. After clicking new button 603, the user can set an area of ​​a desired position, size, and shape by performing a drag-and-drop operation or the like on image 600 displayed in area 601A. For example, as shown in area 612, the user can set an area connecting the vertices by sequentially clicking the vertices of the area and double-clicking the last vertex. After setting an area, it is possible to set a line. In the example shown in FIG. 6, line 613 exists as a set line. Once an area has been set, the procedure for setting a line does not matter. For example, after setting an area, the user can set a line by following the instructions in a pop-up that asks, "Do you want to set a line next?", or by selecting an area that has already been set and switching on the line setting button. There is no restriction on the procedure for creating a line. A line may be created outside an area, or may be created so as to overlap the border of an area. For example, a user can set a line of a desired length by performing a drag-and-drop operation on image 600 displayed in area 601A. A user can also create a line by selecting at least one side of a set area or at least two or more vertices of a set area. The above settings are saved by clicking the save button 604.

[0049] FIG. 7 is an exemplary schematic diagram for explaining the setting of conditions according to the embodiment.

[0050] 7 is an example of a UI that accepts operations related to setting conditions by the user. This image 700 may include both of the two areas 611 and 612, or may include only one of them.

[0051] Area 710 displays, for the user's reference, the image currently being captured by camera 116. In Fig. 7, area 710 displays area 611, but it may also include both areas 611 and 612. Area 720 also displays an interface for accepting the setting of the area in which the above-mentioned target detection will be performed. Specifically, a list of pre-registered areas is displayed in a pull-down menu, and the user can select an area from the list and set it. Area registration is displayed, for example, by clicking "Register / Edit Area" in the upper right corner. Area 730 also displays an interface for accepting the input of a name to set the current settings.

[0052] An interface for accepting the setting of a threshold for the duration of stay to be counted is displayed in area 740. Also, an interface for accepting the setting of the timing (for example, time period, day of the week, date, etc.) for notification when the number of people in an area falls below the threshold is displayed in area 750. This means that even if the queue spans several locations or is winding, making it difficult to see the person in front of you, the queue status will be displayed on monitors inside and outside the store, allowing the queue to move forward without delay.

[0053] 5, when the setting of the conditions in step S501 is completed, in step S502, communication unit 350 of client device 110 transmits the set conditions to server device 100. Then, communication unit 230 of server device 100 receives the conditions transmitted by client device 110.

[0054] Then, in step S503, the communication unit 230 of the server device 100 transfers the conditions received from the client device 110 to the camera 160. The communication unit 440 of the camera 160 then receives the conditions sent by the server device 100.

[0055] Then, in step S504, camera 160 stores the conditions received from server device 100 for use in the photographing process described below. In this way, the setting process ends.

[0056] Next, the photographing process including photographing an image by the camera 160 and detecting various pieces of information based on the image will be described.

[0057] In the photographing process, first, in step S511, the photographing unit 430 of the camera 160 photographs an image. Then, in step S512, the camera 160 executes a detection process to detect a person from the image based on the settings made in the setting process described above. This detection process may be performed by either the camera 160 or the server device 100.

[0058] In the example shown in Figure 6, there is a customer 620 and a store clerk 621. Customers 620A and 620C are customers who are waiting in line at the cash register near the aisle where the products are displayed to purchase products, and customer 620B is a customer who is passing by the aisle to look at the products displayed on the shelves. Store clerk 621A is a store clerk who handles the checkout process for the products, and store clerk 621B is a store clerk who is in charge of displaying and replenishing the products. When a person enters the area, detection begins, and the number of people is counted when the stay time threshold set in area 740 is exceeded. The stay time is measured for each person who enters the area. Therefore, even if they enter the area, passersby who are not waiting in line at the cash register, such as customer 620B (people whose stay time is less than the threshold), are not counted. Also, if uniforms and facial images of store clerks have been registered in advance, store clerk 621B will not be counted even if he or she enters the area and the stay time exceeds the threshold.

[0059] Line 613 is a line that is set in association with area 612. This makes it possible to count only those who have passed through line 613 and exited from area 612. On the other hand, it is also possible to count those who have passed through line 613 and exited from area 612, excluding them. In the example shown in FIG. 6, people who have passed through line 613 and exited are counted, so it is possible to obtain data on both those who have exited area 612 and made a purchase at the cash register and those who left the queue before making a purchase at the cash register.

[0060] This system can also detect when an object moves from one area to another. For example, by detecting an object that moves from area 611 to area 612, it is possible to identify objects that have left the same area, such as those who have left the queue and stopped purchasing, or those who have lined up at a different cash register.

[0061] These features make it possible to quantify the number of people waiting in line or leaving the line when waiting to pay at a cash register, such as in the above-mentioned supermarket. The embodiments are not limited to waiting at a cash register in a supermarket, etc., but can be used in a variety of situations where lines occur, such as theme parks and events, ATMs in banks and convenience stores, smoking areas, waiting to order or pay at a food court, waiting to board public transportation, and baggage inspection at airports, etc.

[0062] Returning to FIG. 5, when step S512 is completed, in step S513, the communication unit 440 of the camera 160 transmits the video captured in step S511 and the results of the detection process in step S512 to the server device 100. Here, the video is associated with the time of capture, and the detection results include at least the time a person was detected, the time a person's presence in an area or line was detected, and the time a person's exit was detected. Note that the detection results may be transmitted only when the conditions set in step S501 are met, or may be transmitted at other times. Then, the communication unit 230 of the server device 100 receives the information transmitted by the camera 160.

[0063] Then, in step S514, the recording unit 220 of the server device 100 records the video and detection results received from the camera 160. Then, in step S515, the server device 100 generates image information called a timeline, for example, as shown in the following Fig. 8, as information indicating the detection results of human presence. In this way, the shooting process ends.

[0064] FIG. 8 is an exemplary schematic diagram showing an example of a display mode of a timeline according to the embodiment.

[0065] 8 is an example of a UI including a timeline generated by the server device 100. This UI can be provided to the client device 110 as information indicating the detection result of a person's presence in the confirmation process described below.

[0066] The image 800 shown in FIG. 8 includes an area 801 displaying a timeline. This timeline identifiably displays, in chronological order, video timings that satisfy various conditions set in the configuration process, such as when a person's exit from an area or line is detected by the above-described detection logic. "Identifiable" here refers to, for example, a mode in which the timeline displays timings that satisfy the above-described conditions with an object such as a colored band or flag. If colored, the color may correspond to the same color as the color set in the above-described area configuration (see FIG. 6). In FIG. 8, a case in which a person's exit is detected is classified as an "exit count" event, and a colored band is displayed on the timeline in a color that is distinguishable from other events (not shown). When the user selects a desired time from the timeline, the video corresponding to that time is displayed at the top. Providing such a timeline to the client device 110 makes it possible to appropriately narrow down the timings in the video to be focused on.

[0067] In the embodiment, the format of outputting the information indicating the result of the detection of a person's exit is not limited to the timeline format described above. For example, in the embodiment, thumbnails showing images at the time when the person's exit is detected may be arranged in chronological order and output as information indicating the result of the detection of a person's exit.

[0068] Furthermore, image information useful for DX is not limited to the timeline described above. For example, in an embodiment, the server device 100 may also generate image information such as the graphs shown in Figures 9 and 10 below.

[0069] FIG. 9 is an exemplary schematic diagram showing an example of a display mode of a graph according to the embodiment.

[0070] 9 is an example of a UI including a graph generated by the server device 100. This UI can be provided to the client device 110 as information useful for DX in the confirmation process described below.

[0071] Image 900 shown in Fig. 9 displays a graph showing the change over time in the detected number of visitors. In the upper left corner, a thumbnail including the area set in step S501 (see Fig. 5) and the name of that setting are displayed. Image 900 shown in Fig. 9 also includes an interface 901 for setting the period of time for which the graph is to be displayed, an interface 902 for adjusting the graph's aggregation interval (the interval of the horizontal axis corresponding to each bar), and an interface 903 for adjusting the duration of visit (the condition for aggregation). By changing these and clicking the "Update" button, the graph is updated according to the conditions.

[0072] Furthermore, image 900 shown in FIG. 9 displays the number of visitors by time in a graph. On the other hand, by checking condition 904, it is possible to switch from displaying the number of visitors per hour to displaying the average number of visitors per hour. Here, the time may be displayed as the number of visitors at a certain time point, or as the number of visitors between certain times. The average number of visitors can also be displayed over a similar time period. By providing such a graph to the client device 110, it becomes possible to statistically grasp the status of people staying in the target area.

[0073] 9, all of the set areas are displayed (not shown). Checking the output selection 905 allows the above-mentioned data to be output, and the output area can be selected to be only any area or all of the set areas.

[0074] In the embodiment, it is possible to simultaneously display the average value of the stay time and / or the maximum value of the stay time with the graph of FIG. 9, as shown in the following FIG.

[0075] FIG. 10 is an exemplary schematic diagram showing another example of a display mode of a graph according to the embodiment.

[0076] Image 1010 shown in Fig. 10 is an example of a UI that includes a graph generated by server device 100, different from that shown in Fig. 9. This image 1010 can display a graph of the average value of the stay time of the counted people when condition 1020 is checked, and a graph of the maximum value of the stay time of the counted people when condition 1021 is checked. Image 1010 displays an example where both condition 1020 and condition 1021 are checked, but it is also possible to display only one of them. By providing such a graph to the client device 110, it becomes possible to statistically grasp the average and maximum stay times of the objects that have been visited.

[0077] Returning to FIG. 5, a confirmation process for confirming the video captured and the information detected in the above-described shooting process will be described.

[0078] In the confirmation process, first, in step S521, the input unit 330 of the client device 110 accepts a confirmation request based on a user operation for confirming the video captured and the information detected in the above-described photographing process.

[0079] Then, in step S522, the communication unit 350 of the client device 110 transmits the received confirmation request to the server device 100. Then, the communication unit 230 of the server device 100 receives the confirmation request transmitted by the client device 110.

[0080] Then, in step S523, in response to the confirmation request received from the client device 110, the server device 100 reads out the image information (thumbnail, timeline, or graph) generated in the above-described photographing process.

[0081] Then, in step S524, the communication unit 230 of the server device 100 transmits the image information read in step S523 to the client device 110. Then, the communication unit 350 of the client device 110 receives the image information transmitted by the server device 100.

[0082] Then, in step S525, client device 110 displays on a display (not shown) the image information received from server device 100. The image information is displayed, for example, in the manner shown in FIGS. 6 to 8 described above.

[0083] Then, in step S526, the input unit 330 of the client device 110 accepts a selection instruction as an operation for the user to select (specify) a video at a desired timing based on the image information described above. Specifically, this corresponds to specifying a time on the timeline in Fig. 7, specifying a bar on the graphs in Fig. 9 and Fig. 10, etc.

[0084] Then, in step S527, the communication unit 350 of the client device 110 transmits the received selection instruction to the server device 100. Then, the communication unit 230 of the server device 100 receives the selection instruction transmitted by the client device 110.

[0085] Then, in step S528, the server device 100 reads out the video recorded in the above-described shooting process in response to the selection instruction received from the client device 110. That is, the server device 100 reads out the video at the time corresponding to the selection (designation) made in step S526.

[0086] Then, in step S529, the communication unit 230 of the server device 100 transmits the video read out in step S528 to the client device 110. Then, the communication unit 350 of the client device 110 receives the video transmitted by the server device 100.

[0087] Then, in step S530, client device 110 plays the video received from server device 100 on a display (not shown). For example, the video is played on the center screen of the UI of FIG. 8. Note that in step S526, selection (designation) is made via the UI of FIGS. 8 to 10, and if this is an operation via FIG. 8, the center screen is updated, and if this is an operation via another figure, a transition to FIG. 8 is made. In this way, the confirmation process is completed.

[0088] As described above, the system 1000 according to the embodiment displays video captured by the camera 160 connected to a network. The system 1000 may be configured to set a threshold for the duration of stay of objects staying in a target area in response to a user's instruction. The system 1000 may then be configured to measure (detect) the number of objects staying in the target area and the duration of time that people stay in the target area.

[0089] In some embodiments, the system 1000 may be configured to provide a configurable area of ​​interest, thereby providing information about various configurable areas of interest.

[0090] Also, in an embodiment, the system 1000 can set a line along with the area of ​​interest, and the system 1000 can be configured to detect an object exiting the line of interest or outside the line of interest.

[0091] In an embodiment, the system 1000 may be further configured to set a condition related to one or more of a desired time period, a day of the week, or a date, and generate thumbnails showing images when the condition is met. With this configuration, in addition to information related to the number of objects staying in the target area, the time people stay in the target area, and the number of objects that have left the target line, it is possible to provide information related to one or more of a time period, a day of the week, or a date from various perspectives.

[0092] In addition, in an embodiment, when multiple thumbnails are displayed, the system 1000 may be configured to select one of the displayed thumbnails in response to a user instruction and play the video corresponding to the selected thumbnail. With this configuration, by simply selecting a thumbnail, the user can easily check the video corresponding to the thumbnail as more detailed information.

[0093] In addition, in an embodiment, the system 1000 may be configured to measure the time that each object that stays in the target area or that exits the target line or other than the target line remains, and output the data.

[0094] Additionally, in an embodiment, the system 1000 may be configured to measure each object that exits the line of interest or outside the line of interest and generate thumbnails based on the measured time for each object.

[0095] Furthermore, in an embodiment, the system 1000 may be configured to generate a timeline that chronologically shows information on when a subject exited the subject line or a line other than the subject line, and to display the generated timeline.

[0096] Furthermore, the system 1000 according to the embodiment is configured to specify a time from the displayed timeline in response to a user instruction and play back the video corresponding to the specified time. With this configuration, by simply specifying a time from the timeline, the user can easily check the video as more detailed information at that time.

[0097] In the embodiment, the system 1000 is configured to specify a period for counting the number of people and generate a graph corresponding to the count results within the specified period. With this configuration, various graphs can be easily generated according to the specified period.

[0098] In an embodiment, the system 1000 is configured to specify a time interval for counting the number of people multiple times and generate a graph based on the count at the specified time interval. With this configuration, various graphs can be easily generated according to the specified time interval.

[0099] (Variation 1) Modification 1 of the embodiment will be described. Modification 1 is a detailed description of S501 to S512 in Fig. 5, and is not an embodiment different from the embodiment. The configuration and processing not described in Modification 1 are the same as those in the embodiment described above.

[0100] 11 is a flowchart showing an example of information processing when a condition setting based on a user operation via a UI such as those shown in the following Figures 12 to 18 is accepted, providing a detailed explanation of the condition setting in step S501 in Figure 5. This flowchart is implemented by accepting a condition setting based on a user operation at input unit 330 of client device 110 and executing a program stored in storage unit 220 of server device 100.

[0101] In step S1110, the client device 110 determines whether or not an area creation instruction has been issued by the control unit 310. Specifically, it determines that an area creation instruction has been issued when a new creation button 1203 in Fig. 12, which will be described later, is clicked. If client device 110 determines that control unit 310 has issued an instruction to create an area, it proceeds to step S1120, and if it determines that an instruction to create an area has not been issued, it repeats the process of step S1110.

[0102] In step S1120, the client device 110 creates an area using the input unit 330. A specific method for creating an area will be described later with reference to FIG.

[0103] In step S1130, client device 110 determines whether or not an instruction to complete area creation has been given in control unit 310. Specifically, it is determined that an instruction to complete area creation has been given when save button 604 in FIG. 6 is clicked. If client device 110 determines that control unit 310 has issued an instruction to complete area creation, it proceeds to step S1140, and if it determines that an instruction to complete area creation has not been issued, it repeats the process of step S1120.

[0104] In step S1140, client device 110 determines whether or not an instruction to set a regular exit line has been issued by control unit 310. Specifically, it is determined that an instruction to complete area creation has been issued when "Yes" button 1320 or "No" button 1330 in Fig. 13 (described later) is clicked. Here, the normal exit line is a line on the perimeter (line) of an area that is used to count the number of people who pass through a specific line and exit the area. For example, in a scene where people are waiting to pay at a cash register in a supermarket, people leaving the area can be divided into two groups: those who go to the cash register and pay, and those who stop waiting at the cash register and leave. By designating part of the area's line that users pass through when proceeding to the cash register as the normal exit line, users can count the number of people who lined up and paid, and those who left the line and stopped paying. If client device 110 determines that control unit 310 has issued an instruction to set a regular exit line, the process proceeds to step S1150; if control unit 310 determines that an instruction to set a regular exit line has not been issued, the process proceeds to step S502.

[0105] In step S1150, client device 110 selects a regular exit line using input unit 330. A specific method for selecting a regular exit line will be described later with reference to Fig. 14. When client device 110 determines that control unit 310 has selected the regular exit line setting, it proceeds to step S1160.

[0106] In step S1160, client device 110 determines whether or not an instruction to complete selection of the regular exit line has been given in control unit 310. Specifically, it is determined that an instruction to complete selection of the regular exit line has been given when save button 604 in FIG. 6 is clicked. When the control unit 310 of the client device 110 determines that an instruction to complete selection of the regular exit line has been issued, the process proceeds to step S502.

[0107] In step S502, the server device 100 determines in the control unit 210 whether an area has been set and whether a regular exit line has been selected along with the area setting, and stores the result in the storage unit 220. Specifically, if only an area has been set (more specifically, if the answer is NO in step S1140), the set area is stored in the storage unit 220, and if a regular exit line has been selected along with the area setting (more specifically, if the answer is YES in step S1160), the set area and the regular exit line are associated and stored in the storage unit 220.

[0108] In step S501, the client device 110 accepts, at the input unit 330, setting of conditions based on user operations via a UI such as those shown in FIGS. 12 to 18 below.

[0109] FIG. 12 is a diagram showing an example of area setting. The screen 1200 includes an area 1201A, a new creation button 1203, an area 1211, and an image 1220 captured by a camera. In step S1110, client device 110 determines via control unit 310 that the user has clicked new creation button 1203. Subsequently, in step S1120, client device 110 causes control unit 310 to recognize an area of ​​a desired position and size from a user operation via input unit 330 on screen 1220 displayed in area 1201A, and stores the area in storage unit 320. Although the area 1211 has a square shape, the shape of the area is not limited to this and can be circular, semicircular, or the like. For example, in the case of a circular area, the user can create the circle by adjusting the size of the circle by dragging the mouse on screen 1220 and then ending the dragging operation. The area creation method can be switched by providing a menu that is displayed when the mouse is right-clicked or a button for switching the area creation method on FIG. 12. In step S1130, when the user selects save button 604, control unit 310 displays screen 1310, which will be described later.

[0110] FIG. 13 is a diagram showing an example of a pop-up that appears when setting a normal exit line. Screen 1300 includes screen 1310 displaying the message "Area setting complete. Would you like to continue by setting a regular exit line?", a "Yes" button 1320, and a "No" button 1330, and also has a dark overlay enabled. The "Yes" button 1320 is a button that the user selects when they determine that it is necessary to set a regular exit line in association with an area. The "No" button 1330 is a button that the user selects when they determine that it is not necessary to set a regular exit line in association with an area (setting only the area is sufficient). In step S1140, when the "Yes" button 1320 is selected by the user, control unit 310 closes screen 1310 and subsequently displays screen 1400, which will be described later, and when the "No" button 1330 is selected, closes screen 1310.

[0111] FIG. 14 is a diagram showing an example of setting a normal exit line after setting an area. Screen 1400 includes visually highlighted area 1211, a procedure screen 1410 for setting up a regular exit line with the message "Area setting complete. Would you like to continue by setting up a regular exit line?", and also has a dark overlay enabled. In step S1150, when client device 110 recognizes via control unit 310 that the user has selected one or more sides from area 1211, it stores the selected side as a regular exit line. For example, when the mouse cursor is placed on or near any line segment, the corresponding line segment is highlighted (becomes thicker / changes color, etc.), and when the mouse is clicked in this state, the highlighted line segment is recognized and stored as a regular exit line. There are other methods for setting a regular exit line, such as selecting at least two vertices of the area, clicking one side or vertex of the area in sequence and then double-clicking the last side to set one or more sides as a regular exit line, or, in the case of a circular area, clicking a point on the circumference that will be the starting point will highlight the circumference that corresponds to the regular exit line, and then clicking any end point on the circumference will set the section from the highlighted start point to the end point as a regular exit line. In step S1160, when control unit 310 determines that the user has completed selection of the regular exit line, client device 110 terminates the dark overlay of screen 1400. Specifically, it is determined that an instruction to complete selection of the regular exit line has been given when save button 604 is clicked.

[0112] FIG. 15 is a diagram showing an example in which a normal exit line that is set after an area is set is linked to the area and saved. Screen 1500 includes a set regular exit line 1501 and a checkmarked regular exit line setting button 1505. After the dark overlay is finished by the control unit 310, the client device 110 adds a check mark to the regular exit line setting button 1505, and indicates that the regular exit line 1505 has been set in association with area A.

[0113] FIG. 16 is a diagram showing an example of ex post setting of a regular exit line to an area that has already been set. Screen 1600 includes an unchecked normal exit line setting button 1605 . When the control unit 310 of the client device 110 determines that the normal exit line setting button 1605 has been selected by the user, the client device 110 transitions to a screen 1400 shown in FIG. The subsequent steps are the same as steps S1150 to S1160 and thereafter.

[0114] FIG. 17 is a diagram showing an example in which a regular exit line that has been set afterwards for an area that has already been set is linked to the area and saved. Screen 1700 includes a checkmarked Set Regular Exit Line button 1705 . In step S502, the client device 110 ends the dark overlay in the control unit 310, and adds a check mark to the regular exit line setting button 1705, thereby indicating that a regular exit line has been set in association with area B.

[0115] 18 is a flowchart showing an example of information processing when a target is detected, which is a detailed explanation of S512 in Fig. 5. This flowchart is realized by accepting conditions set by a user's operation at input unit 330 of client device 110 and executing a program stored in memory unit 420 of camera 160.

[0116] In step S1810, camera 160 determines whether or not a target has entered the area using control unit 410. Specifically, control unit 410 determines that a target has entered the area when the target's feet or the center of its body have entered the area. If camera 160 determines in control unit 410 that the target has entered the area, the process proceeds to step S1820, and if it determines that the target has not entered the area, the process of step S1810 is repeated.

[0117] In step S1820, when control unit 410 determines that a target has entered the area, camera 160 assigns a detection ID and stores it in storage unit 420. Subsequently, in step S1830, camera 160 determines which camera detected the target using control unit 410 and stores the identified camera information in storage unit 420. Specifically, the camera information includes identification information for identifying camera 160 and the name of camera 160. Subsequently, in step S1840, camera 160 determines which area detected the target using control unit 410 and stores the identified area information in storage unit 420. Specifically, the area information includes identification information for identifying the area and the name of the area such as that in region 602. Subsequently, in step S1850, camera 160 determines the time when the target entered the area using control unit 410 as the detection start time and stores this in storage unit 420. Steps S1830 to S1850 are not limited to the above order, that is, the camera identification, area identification, and detection start time storage can be performed in any order.

[0118] In step S1860, camera 160 determines whether the subject has exited the area from the video using control unit 410. Specifically, control unit 410 determines that the subject has exited the area if the subject's feet or the center of the body have exited the area. If control unit 410 determines that the subject has exited the area, camera 160 proceeds to step S1870, and if it determines that the subject is still remaining in the area, it repeats the processing of step S1860.

[0119] In step S1870, camera 160 causes control unit 410 to store the time when the target exits the area as the detection end time in storage unit 420. Subsequently, in step S1880, camera 160 causes control unit 410 to calculate the target's stay time from the difference between the detection start time stored in step S1850 and the detection start and end time stored in step S1870, and stores the calculated time in storage unit 420.

[0120] In step S1890, camera 160 determines whether control unit 410 has set a regular exit line. If camera 160 determines that a regular exit line has been set by control unit 410, it stores "Set" in memory unit 420 in step S18100, and proceeds to step S18120. On the other hand, if camera 160 determines that a regular exit line has not been set by control unit 410, it stores "Not Set" in memory unit 420 in step S18110.

[0121] In step S18120, camera 160 determines in control unit 410 whether or not the subject has passed the regular exit line. If the camera 160 determines that the regular exit line has been set by the control unit 410, it stores "passed" in the memory unit 420 in step S18130, and if it determines that the regular exit line has not been set, it stores "not passed" in the memory unit 420 in step S18140.

[0122] When step S512 is completed, in step S513, the server device 100 stores the data stored in the storage unit 420 of the camera 160 in the storage unit 220.

[0123] 19 is a diagram showing an example of an exit data table, which shows a list of data stored in steps S1820 to S1850 and 1870 to 18140 in FIG. The exit data table includes the following data items: detection ID 1910, camera ID 1920, area ID 1930, detection start time 1940, detection end time 1950, stay time 1960, whether a regular exit line is set 1970, and whether the regular exit line was passed 1980. The detection ID 1910 is stored in the storage unit 220 in step S1820. Camera ID 1920 is stored in storage unit 220 in step S1830. Camera ID 1920 includes information for identifying camera 160 that captured the video in which the target was detected. The camera identification information includes the camera name and the like. Although not shown in the drawing, the camera ID is assigned by the control unit 210 when the camera 160 that captures the video is registered. The area ID 1930 is stored in the storage unit 220 in step S1840. The area ID 1930 includes identification information for identifying the area in which the target is detected. The area identification information includes the area name and the like. Although not shown in the drawing, the area ID is assigned by the control unit 210 when registering the area. As detection start time 1940, the time when the target entered the area in step S1850 is stored in storage unit 220. As detection end time 1950, the time when the target exited the area in step S1870 is stored in storage unit 220. Stay time 1960 is stored in storage unit 220 as the time that the target stayed in the area in step S1880. In the regular exit line setting status 1970, whether or not a part of the detection area has been set as a regular exit line in step S1890 is stored in the storage unit 220. This information is stored as "set" if a regular exit line has been set, and "not set" if not. In the regular exit line passing / not passing field 1980, whether or not the vehicle passed through the regular exit line to exit in step S18120 is stored in the storage unit 220. This information is stored as "passed" if the vehicle passed through the regular exit line to exit, "not passed" if not, and as a blank such as "-" if the regular exit line is not set.

[0124] FIG. 20 is a flowchart showing an example of information processing when outputting target count information.

[0125] In step S2010, the server device 100 determines whether the control unit 210 has set a normal exit line. If the control unit 210 of the server device 100 determines that a regular exit line is set, the process proceeds to step S2020; if the control unit 210 determines that a regular exit line is not set, the process proceeds to step S2030, which will be described later.

[0126] In step S2020, the server device 100 determines from the following A to C how to count objects when a regular exit line is set in the control unit 210. Specifically, it determines which of the three counting methods, namely, method A for counting all objects that have exited the area (hereinafter referred to as all counting), method B for counting objects that have exited the area by passing the regular exit line (hereinafter referred to as regular exit counting), and method C for counting objects that have exited the area without passing the regular exit line (hereinafter referred to as departure counting), has been set as the output condition in the counting condition display 2120. If the server device 100 determines that the all count has been selected by the control unit 210, the process proceeds to step S2030; if it determines that the normal exit count has been selected, the process proceeds to step S2040; and if it determines that the departure count has been selected, the process proceeds to step S2050.

[0127] In step S2030, the server device 100 calculates all count results in the control unit 210. Specifically, all targets (detection IDs 1 to 3) detected entering or leaving the area are counted, without taking into account whether or not the regular exit line is set in the regular exit line setting / non-setting field 1970 and whether or not the regular exit line has been passed in the regular exit line passing / non-passing field 1980 in FIG. The server device 100 calculates the count results of all targets that have left the area in the control unit 210, and proceeds to step S2060.

[0128] In step S2040, server device 100 calculates the result of the regular exit count in control unit 210. Specifically, the server device 100 counts the number of targets (detection ID1) for which the setting status of regular exit line setting 1970 in Fig. 19 is "Set" and the passing status of regular exit line 1980 is "Passed." The server device 100, in the control unit 210, calculates the count result of the objects that have passed through the regular exit line and exited the area, and proceeds to step S2060.

[0129] In step S2050, the server device 100 calculates the result of the exit count in the control unit 210. Specifically, the server device 100 counts the number of targets (detection ID3) for which the setting status of the regular exit line setting 1970 in Fig. 19 is "Set" and the passing status of the regular exit line 1980 is "Not passed through." The server device 100 calculates the count result of the objects that have exited the area without passing through the regular exit line in the control unit 210, and proceeds to step S2060.

[0130] In step S2060, the server device 100 calculates the counting result in the control unit 210 and displays it on the client device 110. Specifically, the counting result may be displayed as a bar graph like the counting result display 2130 in Fig. 21, which will be described later.

[0131] FIG. 21 is a diagram showing another example of data output. Here, the UI and functions described in FIG. 9 will not be described, but they can be displayed appropriately in this example as well. The screen 2100 includes a display 2110 indicating whether or not a regular exit line is set, a counting condition display 2120, and a counting result display 2130. In step S2010, if a regular exit line has been set by control unit 310, client device 110 highlights "Yes" in regular exit line setting status display 2110, and if a regular exit line has not been set, highlights "No." Specifically, if you want to highlight "Yes," you can surround "Yes" with a line or gray out "No." The client device 110 determines which counting method has been selected in the counting condition display 2120 by the control unit 310 (step S2020), and displays the selected counting result in the counting result display 2130 (step S2060). Specifically, if the normal exit line is set to "Yes," the counting condition display 2120 becomes active, allowing any counting condition to be selected. Only one counting condition may be selected, or multiple counting conditions may be selected. For example, if "exit count" is selected as the counting condition, the number of subjects who exited the area for each hour without crossing the regular exit line is output as a bar graph. The bar graph for 13:00 indicates that "the number of subjects who exited the area without crossing the regular exit line from after 12 o'clock until 13:00 is four." If multiple counting conditions are selected, the bar graphs for each hour may be displayed side by side.

[0132] Although the embodiments of the present disclosure have been described above, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be embodied in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and modifications thereof are included within the scope and spirit of the invention, and are also included in the inventions and their equivalents as defined in the claims.

[0133] For example, part of the processing of the server device 100 described above may be executed by the client device 110. Also, part of the processing of the server device 100 described above may be executed by the camera 160. [Explanation of symbols]

[0134] 100: Server device 110: Client device 150: Network 160: Camera 210: Control unit 220: Storage section 230: Communications Department 1000: Information Processing Systems

Claims

1. A system for analyzing video captured by a camera, Setting an area in the video; A line corresponding to one or more sides of the area is set in association with the area; Among those who left the area Counting the number of objects excluding those that have passed through the line. system.

2. A system for analyzing video captured by a camera, Setting an area in the video; A line corresponding to one or more sides of the area is set in association with the area; Among the subjects who have left the area, Counting the number of objects that pass through the line; system.

3. The setting of the line is performed by selecting one or more sides of the area.

3. The system according to claim 1 or 2.

4. The setting of the line is performed by selecting at least two or more vertices from the area.

3. The system according to claim 1 or 2.

5. A program for analyzing video captured by a camera, Setting an area in the video; Among those who left the area Counting the number of objects excluding those that have passed through the line. program.

6. A method for analyzing video captured by a camera, comprising: Setting an area in the video; Among those who left the area Counting the number of objects excluding those that have passed through the line. method.

Images