Information processing systems, information processing methods, and programs

Abstract

This invention provides an information processing system, information processing method, and program that analyze images captured by a camera and output data. [Solution] In an information processing system 1000 in which a server device 100, client devices 110 and 120, and a plurality of surveillance cameras 160 are connected to each other via a network 150, the control unit of the server device sets camera attributes for each surveillance camera, registers person attributes for each person, and when predetermined detection conditions are met in the video, stores the camera attributes related to the video and the person attributes related to the video as detection information, and outputs the detection information to the client device.

Description

Technical Field

[0001] The present invention relates to an information processing system, an information processing method, and a program.

Background Art

[0002] In recent years, with the spread of cloud cameras, initiatives to promote DX at the site by analyzing video data (hereinafter simply referred to as video) have increased in various industries. First and foremost in DX is visualizing the flow of people at the site. One of the key points at that time is to accurately count the number of people.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In a company, a plurality of cameras are installed for each store. When counting the number of people, for example, employees in the same store are not counted as people, but employees in different stores are counted as customers, etc. In some cases, flexible handling is required. Since the technology of Patent Document 1 does not consider such a situation, it is not user-friendly.

Means for Solving the Problems

[0005] According to one aspect of the present disclosure, an information processing system is provided that analyzes video captured by a camera and outputs data. The information processing system sets the attributes of each camera, registers the attributes of each person, and when a predetermined detection condition is satisfied in the video, stores, as detection information related to the detection, the attributes of the camera related to the video and the attributes of the person related to the video, and outputs the detection information. Furthermore, according to one aspect of this disclosure, an information processing system is provided that analyzes video footage captured by a camera and outputs data. The information processing system sets camera attributes for each camera, registers person attributes for each person, and when predetermined detection conditions are met in the video, stores detection information, including information indicating whether the camera attributes related to the video correspond to the person attributes related to the video, and outputs the detection information. Furthermore, according to one aspect of this disclosure, an information processing system is provided that analyzes video footage captured by a camera and outputs data. The information processing system sets camera attributes for each camera, registers person attributes for each person, and when predetermined detection conditions are met in the video, counts the number of people unless the person in the video is a registered person and the person's attributes correspond to the attributes of the camera related to the video, and outputs the count result. [Brief explanation of the drawing]

[0006] [Figure 1] Figure 1 shows an example of the system configuration of an information processing system. [Figure 2] Figure 2 shows an example of the hardware configuration of a server device. [Figure 3] Figure 3 shows an example of the hardware configuration of a client device. [Figure 4] Figure 4 shows an example of the hardware configuration of a surveillance camera. [Figure 5] Figure 5 shows an example of a surveillance camera table. [Figure 6] Figure 6 shows an example of a staff registration screen. [Figure 7] Figure 7 shows an example of a modal window. [Figure 8] Figure 8 shows an example of a registered staff table. [Figure 9] Figure 9 is a flowchart showing an example of the information processing performed by the server device for setting detection conditions and registering excluded individuals. [Figure 10]Figure 10 shows an example of a list screen for surveillance cameras. [Figure 11] Figure 11 shows an example of the detection condition setting screen. [Figure 12] Figure 12 is a flowchart showing an example of the process of counting the number of people. [Figure 13] Figure 13 shows an example of a detection result table. [Figure 14] Figure 14 shows an example of the count result list screen. [Figure 15] Figure 15 shows an example of the screen for creating measurement data (CSV). [Figure 16] Figure 16 shows an example of the detection group settings screen. [Figure 17] Figure 17 shows an example of a screen for selecting surveillance cameras to group. [Modes for carrying out the invention]

[0007] Embodiments of the present invention will be described below with reference to the drawings. The various features shown in the embodiments below (including modified examples; the same applies hereinafter) can be combined with each other.

[0008] <Embodiment 1> 1. System Configuration Diagram Figure 1 shows an example of the system configuration of the information processing system 1000. As shown in Figure 1, the information processing system 1000 includes, as a system configuration, a server device 100, a client device 110, a client device 120, and a plurality of surveillance cameras 160. The server device 100, the client device 110, the client device 120, and the surveillance cameras 160 are connected to each other via a network 150. The network 150 may include either or both a WAN (Wide Area Network) and / or the Internet. The network 150 is configured to enable communication between multiple devices connected to the network 150 via wired and wireless connections.

[0009] The information processing system 1000 is a system that analyzes videos captured by a plurality of surveillance cameras 160 and outputs data.

[0010] The server device 100 is a device that provides the main functions of the information processing system 1000 and executes the main processes of the following-described embodiments. The server device 100 is a device that analyzes videos captured by a plurality of surveillance cameras 160 and outputs (displays) data to the client device 110 or the client device 120.

[0011] The surveillance camera 160 is a camera installed for the purpose of surveillance and / or recording. In the specification, it is described that the surveillance cameras 160 are installed at each of a plurality of locations in a predetermined store. Examples of the plurality of locations include, for example, the store entrance (east entrance), the store entrance (north entrance), the employee entrance, the backyard, in front of the cash register, etc. Note that these are examples and do not limit the locations where the surveillance camera 160 is installed. In FIG. 1, for simplicity, only three surveillance cameras 160 are shown, but it may be two, or four or more. It is sufficient that a plurality of surveillance cameras 160 are included in the information processing system 1000. Also, in the specification, it is mainly described that a plurality of cameras are installed in one store, but a plurality of cameras may be installed in each of a plurality of stores.

[0012] The client device 110 is a terminal device operated by the owner or administrator of the surveillance camera 160.

[0013] The client device 120 is a terminal device operated by the administrator or the like of the store where the surveillance camera 160 is installed. On the client device 120, screens as shown in FIGS. 6 to 7, FIGS. 10 to 11, and FIGS. 14 to 17 described later are displayed. In the following, when referring to the client device 110 and the client device 120, they are simply referred to as the client device.

[0014] Here, the information processing system described in the claims may consist of multiple devices or of a single device. If the information processing system described in the claims consists of a single device, an example of such a device is a server device 100. If the information processing system described in the claims consists of multiple devices, examples of multiple devices include a server device 100, at least one of the multiple surveillance cameras 160, or a server device 100, at least one of the multiple surveillance cameras 160 and a client device 110 or client device 120, or a cloud server consisting of multiple server devices that provide the functions of the server device 100.

[0015] 2. Hardware Configuration (1) Hardware configuration of server device 100 Figure 2 shows an example of the hardware configuration of server device 100. As shown in Figure 2, the server device 100 includes, as a hardware configuration, a control unit 210, a storage unit 220, a communication unit 230, and an internal bus 240. The control unit 210, the storage unit 220, and the communication unit 230 are electrically connected via the internal bus 240.

[0016] The control unit 210 is a CPU (Central Processing Unit) or the like, and controls the entire server device 100.

[0017] The storage unit 220 is one of the following: HDD (Hard Disk Drive), ROM (Read Only Memory), RAM (Random Access Memory), SSD (Solid State Drive), or any combination thereof, and stores programs, data used by the control unit 210 when executing processing based on the programs, etc. The storage unit 220 is an example of a storage medium. Examples of data used by the control unit 210 when executing processing based on the programs include video data sent from the surveillance camera 160, data shown in Figures 5, 8, and 13 described later, data set via screens described later, data displayed on screens described later, etc.

[0018] In this specification, the data used by the control unit 210 when executing processing based on the program is described as being stored in the storage unit 220, but it may also be stored in the storage unit of another device that can communicate with the server device 100. The data may be stored in the storage unit of any device as long as the control unit 210 can access or retrieve it. The control unit 210 executes processing based on the program stored in the storage unit 220, thereby realizing the functions of the server device 100 and the processing of the flowcharts shown in Figures 9 and 12, which will be described later. Although this processing is mainly described as being executed by the server device 100, it may instead be executed by the client device 110 or client device 120, or by any of the multiple surveillance cameras 160.

[0019] The communications unit 230 connects the server device 100 to the network 150 and manages communication with other devices.

[0020] Note that the hardware configurations of the control unit 210, storage unit 220, and communication unit 230 are not limited to one. For example, multiple control units may be included in the server device 100. The same applies to the client devices 110 and 120 shown below.

[0021] (2) Hardware configuration of the client device Figure 3 shows an example of the hardware configuration of the client device 110. As shown in Figure 3, the client device 110 includes, as a hardware configuration, a control unit 310, a storage unit 320, an input unit 330, an output unit 340, a communication unit 350, and an internal bus 360. The control unit 310, the storage unit 320, the input unit 330, the output unit 340, and the communication unit 350 are electrically connected via the internal bus 360.

[0022] The control unit 310 is a CPU or the like, and controls the entire client device 110.

[0023] The storage unit 320 is one of the following: HDD, ROM, RAM, SSD, etc., or any combination thereof, and stores programs, data used by the control unit 310 when executing processing based on the programs, etc. The storage unit 320 is an example of a storage medium.

[0024] In this specification, the data used by the control unit 310 when executing processing based on the program is described as being stored in the storage unit 320, but it may also be stored in the storage unit of another device that can communicate with the client device 110. The data may be stored in the storage unit of any device as long as the control unit 310 can access or retrieve it. The functions of the client device 110 are realized when the control unit 310 executes processing based on the program stored in the storage unit 320.

[0025] The input unit 330 is a device that inputs information to the client device 110 in response to the operator's actions. The input unit 330 receives operation inputs made by the user. The operation inputs are transferred to the control unit 310 via the internal bus 360 as command signals. The control unit 310 can, if necessary, perform predetermined controls and / or calculations based on the transferred command signals. The input unit 330 may be included in the housing of the client device 110 or it may be external. For example, the input unit 330 may be implemented as a touch panel integrated with the output unit 340. When the input unit 330 is implemented as a touch panel, the user can input tap operations, swipe operations, etc. to the input unit 330. Instead of a touch panel, the input unit 330 can be a switch button, mouse, trackpad, keyboard, etc.

[0026] The output unit 340 is, for example, a display unit, which outputs (displays) information as a screen of a graphical user interface (GUI) that can be operated by the user. The output unit 340 may be included in the housing of the client device 110 or it may be externally attached. More specifically, the output unit 340 may be implemented as a display device such as a liquid crystal display, an organic EL (Electron-Luminescence) display, or a plasma display. It is preferable that these display devices be used according to the type of client device 110.

[0027] The communications unit 350 connects the client device 110 to the network 150 and manages communication with other devices.

[0028] The hardware configuration of client device 120 is the same as that of client device 110.

[0029] In this specification, a PC (Personal Computer) is used as an example for client devices 110 and 120. However, client devices 110 and 120 may be smartphones, tablet computers, etc. The client device can be any device that can display a screen as described later, accept user operations via the screen, and transmit information to the server device 100.

[0030] (3) Hardware configuration of surveillance camera 160 Figure 4 shows an example of the hardware configuration of the surveillance camera 160. As shown in Figure 4, the surveillance camera 160 includes, as a hardware configuration, a control unit 410, a storage unit 420, an imaging unit 430, a communication unit 440, and an internal bus 450. The control unit 410, the storage unit 420, the imaging unit 430, and the communication unit 440 are electrically connected via the internal bus 450.

[0031] The control unit 410 is a CPU or the like, and controls the entire surveillance camera 160.

[0032] The storage unit 420 is one of the following: HDD, ROM, RAM, SSD, etc., or any combination thereof, and stores programs, data used by the control unit 410 when executing processing based on the program, etc. The storage unit 420 is an example of a storage medium.

[0033] In this specification, the data used by the control unit 410 when executing processing based on the program is described as being stored in the storage unit 420, but it may also be stored in the storage unit of another device that can communicate with the surveillance camera 160. The data may be stored in the storage unit of any device as long as the control unit 310 can access or retrieve it. The functions of the surveillance camera 160 are realized when the control unit 410 executes processing based on the program stored in the storage unit 420.

[0034] The shooting unit 430 is a camera that photographs the subject. The camera includes, for example, an image sensor, a lens, and an IR cut filter.

[0035] The communications unit 440 connects the surveillance camera 160 to the network 150 and manages communication with other devices.

[0036] 3. Information Processing The information processing of Embodiment 1 will be described below.

[0037] (1) Overview of the process (1-1) The control unit 210 sets the attributes of each surveillance camera 160. The control unit 210 registers the attributes of each person. When predetermined detection conditions are met in the video, the control unit 210 stores detection information related to the detection, including the attributes of the surveillance camera 160 related to the video and the attributes of the person related to the video. The control unit 210 outputs detection information. By performing this type of processing, it is possible to output detection information flexibly and accurately.

[0038] (1-2) The control unit 210 sets the attributes of each surveillance camera 160. The control unit 210 registers the attributes of each person. When predetermined detection conditions are met in the video, the control unit 210 stores detection information, including information indicating whether or not the attributes of the surveillance camera 160 related to the video correspond to the attributes of the person related to the video. The control unit 210 outputs detection information. By performing this type of processing, it is possible to output detection information flexibly and accurately.

[0039] (1-3) The control unit 210 sets the attributes of each surveillance camera 160. The control unit 210 registers the attributes of each person. The control unit 210 counts the number of people when predetermined detection conditions are met in the video, except when the person in the video is a registered person and the person's attributes correspond to the attributes of the surveillance camera 160 related to the video. The control unit 210 outputs the count result. By performing this type of processing, it is possible to output flexible and accurate count results.

[0040] (1-4) The control unit 210 sets a department for the surveillance camera 160. The department of the surveillance camera 160 is an example of an attribute of the surveillance camera 160. As a more specific example of a department, the specification mainly uses a store as an example. However, the department is not limited to stores, and may be, for example, the floor of the store (1st floor, 2nd floor, basement, etc.) or the area related to the store (the name of the region that includes multiple stores in the location where the store is installed). Examples of areas related to stores include, for example, Shinagawa Ward, Shibuya Ward, Meguro Ward, etc., or Tokyo, Kanagawa Prefecture, Chiba Prefecture, etc., or Kanto, Tohoku, Hokuriku, etc. Note that information may be a combination of any two or more of the following: store, store floor, area related to the store, etc. The control unit 210 sets information related to the conditions for counting the number of people in the video footage captured by the surveillance camera 160. The control unit 210 registers individuals to be excluded when counting people, associating them with their respective departments. A person's department is an example of a person's attribute. Similar to the departments of the surveillance camera 160, the specification primarily uses stores as an example to explain the person's department. However, similar to the departments of the surveillance camera 160, a person's department is not limited to stores; for example, it could be a store floor or an area related to the store.

[0041] The control unit 210 counts the person in the video as a person if the conditions are met in the video and the person in the video is not a registered person. The control unit 210 counts a person as a person if the conditions are met in the video, the person included in the video is a registered person, and the department of the person associated with the registered person is not the same as the department of the surveillance camera 160 related to the video. The control unit 210 does not count a person as part of the number of people if the conditions are met in the video and the person included in the video is a registered person, and the department of the person associated with the registered person is the same as the department of the surveillance camera 160 related to the video. The control unit 210 outputs the result of the number of people counted.

[0042] By performing this process, it is possible to count the number of people flexibly and accurately.

[0043] (2) Details of the process (2-0) Pretreatment First, as a preliminary setting, the angle to be determined as the frontal angle of the face, the angle difference to be determined as sufficiently different from the frontal angle, the reliable number of image pixels, the specified time for acquiring the best shot, etc., are set. These settings are configured, for example, according to user instructions via the client device 110, and stored in the memory unit of each surveillance camera 160 (or server device 100).

[0044] The information processing system 1000 basically performs the process of each surveillance camera 160 capturing video and transmitting it to the server device 100 for storage, while simultaneously performing a process to extract the best shot. Here, the best shot is the optimal facial image (or full-body image) of the same person taken from different predetermined angles. This process is mainly performed on the surveillance camera 160 side, but it may also be performed on the server device 100 side. The advantage of performing the processing on the surveillance camera 160 side is that the processing load and usage fees on the server device 100 side are reduced accordingly, but on the other hand, the performance of the control unit 410 and / or storage unit 420 etc. on the surveillance camera 160 side will be required to perform the relevant processing.

[0045] The details of the best shot extraction process are as follows: First, the control unit 410 of the surveillance camera 160 tracks the entire body of a person from the camera's video feed. Next, the control unit 410 detects and tracks the face from the full-body image data. Then, the control unit 410 detects the angle from the tracked face and selects the image that falls within the face's frontal angle range as the 1st best shot, recording the face angle in the memory unit 420 or the like. The 1st best shot may also be selected from the video feed using an AI model that detects, for example, frontal images of faces, and the image with the highest score is adopted.

[0046] Furthermore, the control unit 410 continuously detects angles from the tracked faces, and if there are faces whose angle difference from the frontal angle is greater than or equal to the angle of the face in Best Shot 1st, it saves them as Best Shot 2nd in the storage unit 420 or the like. For Best Shot 2nd, an AI model that detects, for example, images of faces facing left (or right) relative to the camera from the video may be used to select the one with the highest score.

[0047] Furthermore, the control unit 410 saves an image that has an angle difference from both Best Shot 1st and 2nd as Best Shot 3rd. Best Shot 3rd may be selected using an AI model that detects images of faces facing right (or left) relative to the camera, and the image with the highest score is adopted. Furthermore, for at least one of Best Shots 1st, 2nd, and 3rd, the control unit 410 may save a full-body image (or bust-up image, etc.) of that person as Best Shot 4th. A bust-up image is an image showing the upper body of a person. Best Shot 4th may be selected using an AI model that detects full-body images (or bust-up images, etc.) from the video, and the image with the highest score is adopted.

[0048] Furthermore, the control unit 410 may replace the first best shot if it continues to detect the tracked face and finds an image within the frontal angle range that exceeds a reliable number of image pixels. Similarly, the control unit 410 may also replace the second, third, and fourth best shots. In addition, the control unit 410 may stop extracting the best shot if the specified time for acquiring the best shot is exceeded. Moreover, although the control unit 410 selects the best shot 1st, 2nd, 3rd, and 4th based on the highest score in each respective category, it may also select multiple best shots, including those with lower scores. Finally, when tracking is complete, the control unit 410 transmits the first, second, third, and 4th best shots, along with metadata indicating the camera and date / time of each image, to the server device 100.

[0049] The control unit 210 of the server device 100 receives Best Shot 1st, 2nd, 3rd, and 4th, along with metadata indicating the camera and date / time used to take each image, and then stores these images in association with each other and with their respective metadata. At this stage, the control unit 210 may also perform averaging processing on the face images based on Best Shot 1st, 2nd, and 3rd, calculate facial feature quantities, and store these in association with the images and metadata. Hereinafter, this set of associated information (Best Shot 1st, 2nd, 3rd, and 4th / metadata indicating the camera and date / time used to take each image / facial feature quantities) will be referred to as Best Shot information. The Best Shot information is stored in a predetermined storage area such as the storage unit 220 as multiple different Best Shot entries for each person and each scene.

[0050] (2-1) Setting up the 160 surveillance cameras in each department The control unit 210 sets the department of the surveillance camera 160. For example, the control unit 210 sets data in the surveillance camera table 500, which includes the department of the surveillance camera 160, in response to instructions from a store manager or the like via the client device 120. The surveillance camera table 500 is stored in a predetermined storage area, such as the storage unit 220 of the server device 100.

[0051] Figure 5 shows an example of a surveillance camera table 500. As shown in Figure 5, the surveillance camera table 500 includes camera ID 510, camera name 520, and camera department 530 as data items. Camera ID 510 stores identification information (camera ID) that identifies surveillance camera 160. Camera name 520 stores the name (camera name) of surveillance camera 160. The camera section 530 includes a camera section (camera section). This section includes information such as stores, store floors, areas related to stores, or a combination of two or more of these, as mentioned above.

[0052] (2-2) Staff Registration Furthermore, the control unit 210 registers staff members with the information processing system 1000. Staff members can also be considered individuals. For example, the control unit 210 generates and transmits a staff registration screen 600, as shown in Figure 6, in response to a request from a store manager or the like via the client device 120. Figure 6 shows an example of the staff registration screen 600. The staff registration screen 600 includes a navigation menu and an area 620. When the staff registration button 610 of the navigation menu is selected, information about the registered staff (list of registered staff, each staff member's photo, various basic information, etc.) and the staff registration button 630 are displayed in area 620.

[0053] When the control unit 210 detects that the staff registration button 630 has been selected, it generates a modal window 700 as shown in Figure 7 and displays it superimposed on the staff registration screen 600. Figure 7 shows an example of a modal window 700. The modal window 700 is an example of a screen for registering new staff. Store managers can add photos (images) of store staff via the modal window 700. More preferably, store managers can register photos (images) of store staff in the information processing system 1000 by selecting photos of store staff stored in a predetermined storage area such as the storage unit 220 via the modal window 700. Furthermore, store managers and other personnel can register staff members by entering their names in the input area 710 and their departments in the input area 720 via the modal window 700. The concept of a staff member's department is the same as the camera department mentioned earlier.

[0054] Furthermore, store managers can add staff photos by selecting the "Add Photo" button 730 included in the modal window 700. In this case, store managers should select photos of the same staff member taken from various angles. Furthermore, the control unit 210 may, for example, if one of the staff's photos is selected as a cover photo in the modal window 700 and the "Add Photo" button 730 is selected, extract best shot information related to the image selected as the cover photo from the storage unit 220, etc., and display the best shot image included in the best shot information as a suggested image for the staff's face photo. The control unit 210 may then register the best shot image selected from the suggestion screen as an additional face image for the staff. The cover photo image selected in Modal Window 700, or the cover photo image selected in Modal Window 700 and the best shot image selected from the suggestion screen, will hereafter also be referred to simply as the staff image.

[0055] As another example, when a staff member's photograph is selected as a cover photograph in the modal window 700, the control unit 210 may extract the selected photograph (image) and the best shot information related to the selected image from the storage unit 220, etc., and register the best shot image included in the best shot information as a staff member's image in the storage unit 220, etc.

[0056] Figure 8 shows an example of a registered staff table 800. As shown in Figure 8, the registered staff table 800 includes the following data items: person ID 810, person name 820, person department 830, and person image 840. Person ID 810 stores identification information to identify the person. This identification information is assigned by the control unit 210 each time a new staff member is registered via the modal window 700. The name of the staff member is stored in person name 820. The name entered in input field 710 is stored there. The personnel section 830 stores the staff's department. The department entered in the input area 720 is stored there. This department refers to information such as the store, the store's floor, the area related to the store, or a combination of two or more of these, as mentioned above. Person image 840 stores the staff member's photo. If multiple photos are registered in the modal window 700, person image 840 stores multiple photos of the person. Alternatively, if a photo is selected in the modal window 700, person image 840 stores the selected photo along with the staff member's photo associated with that photo.

[0057] (2-2) Setting detection conditions and registering excluded individuals Figure 9 is a flowchart showing an example of information processing for setting detection conditions and registering excluded individuals, which is performed by the server device 100. In step S910, the control unit 210 determines whether or not it has received a request to display a list screen from the client device 120 or the like. When a predetermined operation is performed by the manager of the store where the surveillance camera 160 is installed, the client device 120 sends a request to display a list screen, as shown in Figure 10, which will be described later, to the server device 100. If the control unit 210 determines that it has received a request to display the list screen, it proceeds to step S920. If it determines that it has not received a request to display the list screen, it repeats the process in step S910.

[0058] In step S920, the control unit 210 generates a list screen and sends it to the requesting client device 120. Upon receiving the list screen, the client device 120 displays the list screen on the output unit 340.

[0059] Figure 10 shows an example of a list screen 1010 for surveillance cameras 160. The list screen 1010 displays a list of surveillance cameras 160 managed by the relevant administrator as devices. When the administrator selects one of the surveillance cameras 160 from the list (for example, the store entrance (east exit) 1020 in Figure 7), device selection information, including identification information that identifies the selected surveillance camera 160, is sent from the client device 120 to the server device 100.

[0060] In step S930, the control unit 210 determines whether a device (surveillance camera 160) has been selected in the client device 120. When the control unit 210 receives device selection information from the client device 120, it determines that the device has been selected. If the control unit 210 determines that the device has been selected, it proceeds to step S940; if it determines that the device has not been selected, it repeats the process in step S920 (or step S930).

[0061] In step S940, the control unit 210 generates a detection condition setting screen and sends it to the requesting client device 120. Upon receiving the detection condition setting screen, the client device 120 displays the detection condition setting screen on the output unit 340.

[0062] Figure 11 shows an example of the detection condition setting screen 1100. The detection condition setting screen 1100 is a screen for setting detection conditions. The detection condition setting screen 1100 is configured to allow setting the area where people are detected (hereinafter also referred to as the detection area) while the video is being played. That is, the information related to the conditions for counting the number of people in the video includes information about the area in the video. In addition, the detection condition setting screen 1100 is configured to allow setting lines (hereinafter also referred to as the passage lines) for detecting the passage of people IN and / or OUT while the video is being played. That is, the information related to the conditions for counting the number of people in the video includes information about the lines in the video.

[0063] Areas 1115, 1125, 1135, and 1145 are detection areas set on the detection condition setting screen 1100. Different areas are displayed on the detection condition setting screen 1100 in different ways. Examples of different ways include different colors and different arrangements of explanatory text. The "Back of Store" button 1110 on the navigation menu corresponds to area 1125. The bench seat button 1120 in the navigation menu corresponds to area 1115. The button 1130 for the two seats by the wall in the navigation menu corresponds to area 1135. The in-store center button 1140 on the navigation menu corresponds to area 1145. Lines 1155 and 1165 are the lines that the detection condition setting screen 1100 has set to pass through. Different lines are displayed on the detection condition setting screen 1100 in different ways. Examples of different ways include different colors and different arrangements of explanatory text. The "Toilet Front_Up" button 1150 in the navigation menu corresponds to line 1155. Button 1160 next to the large table in the navigation menu corresponds to line 1165.

[0064] In step S950, the control unit 210 determines whether or not detection conditions have been set on the detection condition setting screen 1100. The control unit 210 determines that detection conditions have been set if a detection area and / or passing lines are set in the video on the detection condition setting screen 1100. The control unit 210 determines that detection conditions have not been set if a detection area and / or passing lines are not set in the video on the detection condition setting screen 1100. If the control unit 210 determines that detection conditions have been set, it proceeds to step S960. If the control unit 210 determines that detection conditions have not been set, it returns to step S940.

[0065] In step S960, the control unit 210 sets information such as the position and size on the video for one or more detection areas (in the example in Figure 11, areas 1115, 1125, 1135, and 1145) and / or one or more passing lines (in the example in Figure 11, lines 1155 and 1165) as detection conditions via the detection condition setting screen 1100. The control unit 210 stores the detection conditions in a predetermined storage area such as the storage unit 220, associating them with the camera ID that identifies the camera that captured the video displayed on the detection condition setting screen 1100.

[0066] In step S970, the control unit 210 determines whether or not an excluded person has been set on the detection condition setting screen 1100. For example, the administrator of the surveillance camera 160 can set a person to be excluded from the count by right-clicking on the detection condition setting screen 1100 and displaying a list of registered staff from the context menu that appears, and then selecting the person (staff) to be excluded from the count from the list. A person to be excluded from the count is also referred to as an excluded person below. If an excluded person is set via the detection condition setting screen 1100, the control unit 210 proceeds to step S980. If no excluded person is set via the detection condition setting screen 1100, the control unit 210 terminates the process shown in Figure 9.

[0067] In step S980, the control unit 210 sets data of excluded persons to be excluded from the count, set via a predetermined screen such as the detection condition setting screen 1100, as excluded person data in the storage unit 220, etc. The excluded person data includes, in association with the detection conditions associated with the camera ID (hereinafter also simply referred to as detection conditions) set on the detection condition setting screen 1100, and information of excluded persons to be excluded from the detection of persons to be counted based on the detection conditions set on the detection condition setting screen 1100. The excluded person information may be detection conditions relating to all detection areas and passage lines relating to a predetermined video set on the detection condition setting screen as shown in Figure 11, or it may be detection conditions relating to each of the detection areas and passage lines set on the detection condition setting screen as shown in Figure 11. That is, the control unit 210 may accept settings for excluded persons to be excluded for all detection conditions set on the screen as shown in Figure 11, or it may accept settings for excluded persons to be excluded for each of the detection conditions set on the screen as shown in Figure 11. However, for the sake of simplicity, the following explanation will assume that the detection conditions relate to all detection areas and passage lines related to a predetermined video (i.e., related to the camera ID that identifies the surveillance camera 160 that captured the video displayed on the detection condition setting screen) as set on the detection condition setting screen as shown in Figure 11.

[0068] The excluded person information includes the registered staff records of selected staff members from the staff registered in the registered staff table 800. The registered staff record is the data stored in the person ID 810, person name 820, person department 830, and person image 840 of the relevant staff member.

[0069] (2-3) Counting the number of people Figure 12 is a flowchart showing an example of the process of counting the number of people. In step S1200, the control unit 210 determines whether or not a person has been detected from the video footage sent from each of the multiple surveillance cameras 160. For example, the control unit 210 inputs the video to a trained model. This trained model (hereinafter also referred to as the person detection trained model) is a person detection trained model that takes the video as input data and outputs whether or not a person is included in the video, if a person is included in the video, an image of the person (a facial image of the person), the detected location (the smallest rectangular area encompassing the detected person), and person movement data obtained from the video before and after in chronological order as output data. As another example, the control unit 210 may perform image analysis on the video to output as analysis results whether or not a person is included in the video, if a person is included in the video, an image of the person (a facial image of the person), the detected location, and person movement data obtained from the video before and after in chronological order. However, for the sake of simplicity in the following explanation, the control unit 210 will be described as using the person detection trained model to obtain whether or not a person is included in the video, if a person is included in the video, the detected location, and person movement data obtained from the video before and after in chronological order. The control unit 210 acquires output data from the trained model for person detection. If the control unit 210 determines that a person has been detected, it proceeds to step S1210; if it determines that no person has been detected, it repeats the process in step S1200.

[0070] In step S1210, the control unit 210 determines whether the detected person is a person who satisfies the detection conditions. As described above, the camera ID is associated with the detection conditions. The control unit 210 obtains the detection conditions associated with the camera ID of the camera that was capturing the video in which the person was detected, and determines whether the detected person is a person who satisfies the detection conditions based on the detection conditions and the detected location and movement data of the person included in the output data output from the person detection trained model described above. For example, if the detected location included in the output data (i.e., the smallest rectangular area encompassing the detected person) is within the detection area of ​​the detection conditions, the control unit 210 determines that a person who satisfies the detection conditions has been detected. For example, if the detected location included in the output data (i.e., the smallest rectangular area encompassing the detected person) is not within the detection area of ​​the detection conditions, the control unit 210 determines that a person who satisfies the detection conditions has not been detected. Furthermore, for example, the control unit 210 determines that a person satisfying the detection conditions has been detected if the detected location included in the output data (i.e., the smallest rectangular area encompassing the detected person) passes through the detection line of the detection conditions, along with the direction of passage (IN / OUT, etc.). For example, the control unit 210 determines that no person satisfying the detection conditions has been detected if the detected location included in the output data (i.e., the smallest rectangular area encompassing the detected person) does not pass through the detection line of the detection conditions.

[0071] If the control unit 210 determines that the detected person meets the detection criteria, it proceeds to step S1220. If the control unit 210 determines that the detected person does not meet the detection criteria, it repeats step S1210.

[0072] In step S1220, the control unit 210 determines whether the detected person is a registered person or not. The control unit 210 determines whether the detected person is a registered person or not based on the face image of the person included in the output data output from the trained person detection model and the person images and staff images stored in the person image 840 of the registered staff table 800. For example, the control unit 210 uses a feature point detection algorithm such as SIFT (Scale-Invariant Feature Transform) or SURF (Speeded Up Robust Features) to extract feature points from the face image and determines the consistency by comparing the position and shape of the feature points. Alternatively, the control unit 210 inputs the face image into a trained model. This trained model is trained using the face image as input data and the feature quantities of the face image as output data. The control unit 210 may also determine the consistency of the face image by comparing the output feature quantities. If the control unit 210 determines that there is a match, it determines that the detected person is a registered person; if it determines that there is no match, it determines that the detected person is not a registered person. If the control unit 210 determines that the detected person is a registered person, it proceeds to step S1230; if it determines that the detected person is not a registered person, it proceeds to step S1240.

[0073] In step S1230, the control unit 210 determines whether the department of the registered person determined to be the same as the detected person matches (corresponds to) the department of the corresponding surveillance camera 160. This match (correspondence) includes not only an exact match, but also a partial match, similarity, etc. Based on the person ID of the registered person (registered staff) determined to be the same as the detected person, the control unit 210 obtains the department of the registered staff from the corresponding person department 830. Also, based on the camera ID of the surveillance camera 160 that captured the video in which the person was detected, the control unit 210 obtains the department of the surveillance camera 160 from the corresponding camera department 530. If the departments are the same, the control unit 210 determines that the attributes are the same; if the departments are not the same, it determines that the attributes are different. If the control unit 210 determines that the attributes are the same, it returns the process to step S1200; otherwise, it proceeds to step S1240.

[0074] In step S1240, the control unit 210 stores predetermined information in the detection result table 1300.

[0075] Figure 13 shows an example of the detection result table 1300. This table shows a list of data stored in S1240 or S1250 when the answer in S1210 is YES. The detection result table 1300 includes the following data items: detection ID 1310, camera ID 1315, camera name 1320, camera department 1325, detection date and time 1330, detection type 1335, detection name (area / line) 1340, duration of stay 1345, direction of passage 1350, registered person 1355, person ID 1360, person name 1365, and person department 1370. Detection ID 1310 stores identification information that identifies the detection, which is assigned by the control unit 210 when a person is detected in the video. Camera ID 1315 stores identification information that identifies the surveillance camera 160 that was recording the video in which a person was detected. Camera name 1320 stores the name (camera name) of surveillance camera 160. In the camera category (1325), a camera category (camera division) is established. The detection date and time 1330 records the date and time when a person was detected by the surveillance camera 160. Detection type 1335 stores information indicating whether a person was detected in the detection area or along the passage line. The detection name (area / line) 1340 stores the name of the detection area or the name of the line to be passed, as set in the detection condition setting screen 1100, etc. The dwell time 1345 stores the amount of time a person stayed in the detection area. The dwell time is stored when a person is detected in the detection area. This information is stored when the detection type 1335 is an area; otherwise (for example, a line), information indicating a blank space such as "-" is stored. The passage direction 1350 stores information indicating whether a person entered or exited the passage line. The passage direction is stored when a person enters or exits the passage line. This information is stored when the detection type 1335 is a line; otherwise (for example, when it is an area), information indicating a blank space such as "-" is stored. The registered person 1355 stores information indicating whether the detected person is registered in the staff registration ("○" or "×"). Person ID 1360 stores identification information to identify the person (registered staff). The name of a staff member is stored in the name of person 1365. The personnel category 1370 stores the staff department. Person ID 1360, Person Name 1365, and Person Category 1370 are stored when "○" is found in Registered Person 1355, meaning the detected person is a registered staff member. On the other hand, if "×" is found in Registered Person 1355, information indicating a blank space, such as "-", is stored. Alternatively, for example, a column called "Department Match (Not Illustrated)" could be created. If registered person 1355 is marked with "○", then based on the correspondence between camera department 1325 and person department 1370, "○" could be stored if the two correspond (match / similar), and "×" if they do not correspond (match / similar).

[0076] The detection result records stored in the detection result table 1300 in step S1240 are records concerning detected persons who are not registered persons (registered staff), and records concerning persons who are registered persons (registered staff) but belong to different camera departments and personnel departments. In step S1260, the control unit 210 counts up the number of records in the detection result table 1300 for detected persons who are not registered persons (registered staff), and for persons who are registered persons (registered staff) but whose camera department and person department are different. This count is performed for each camera's detection condition (area / line). If the detection condition is a line, the IN / OUT of the same line may be counted separately. The control unit 210 may also count up the number of records for a pre-set period (data for the most recent month), etc. The control unit 210 may also count up the number of records for a pre-set period (data for the most recent month), etc., for each detection name. The control unit 210 stores the count-up result in a predetermined storage area such as the storage unit 220. The counted number is stored in association with the camera / detection condition (area / line) information. If the detection condition is a line, the IN / OUT of the same line may be stored separately. The count-up result can be said to represent the number of people detected in each area / line for each camera. Although the count in step S1260 was described as being performed each time step S1240 is executed, it may be counted all at once later. For example, when the user instructs the creation of a graph, as described later, the detection result records in the detection result table 1300 in Figure 13 that correspond to the report period (measurement period), device (camera ID / camera name / camera department), and detection conditions (area or line) specified in the instruction may be picked out and their number counted.

[0077] In step S1250, the control unit 210 stores predetermined information in the detection result table 1300. The detection result record stored in the detection result table 1300 in step S1250 is a registered person (registered staff), and the camera department and the person department record pertain to the same person.

[0078] (2-4) Output processing of measurement results The control unit 210 generates a count result list screen in response to a request from the client device 120 and sends it to the client device 120. Figure 14 shows an example of the count result list screen 1400. The count results list screen 1400 displays a list of data showing the count-up results for each predetermined period (measurement period). This data is created when the measurement data (CSV) creation button 1410 is selected. When the measurement data (CSV) creation button 1410 is selected, the control unit 210 generates a measurement data (CSV) creation screen 1500 as shown in Figure 15. Figure 15 is a diagram showing an example of the measurement data (CSV) creation screen 1500. CSV stands for Comma Separated Values. The control unit 210 then sends the measurement data (CSV) creation screen 1500 to the client device 120. The measurement data (CSV) creation screen 1500 is a screen for creating a CSV file of measurement data.

[0079] On the measurement data (CSV) creation screen 1500, the manager of the store where the surveillance camera 160 is installed can set various information such as the data name (file name), report period (measurement period), and device (camera ID / camera name / camera department). Here, regarding device settings, cameras can be set individually by camera ID and camera name, and it is also possible to set multiple devices at once on an attribute basis by grouping them into departments as described later. When the measurement data (CSV) creation screen 1500 receives the settings of various information from the user and the selection of the button to create, the control unit 210 creates a CSV file of measurement data with the set data name. As a result, measurement data (CSV) is created for the results detected in the video captured by the desired camera during the desired measurement period. For example, the image is a filtered version of the data stored in the detection result table 1300 shown in Figure 13, which corresponds to the device (camera ID / camera name / camera department) with camera ID 1315 / camera name 1320 / camera department 1325 set, and the data corresponding to the report period (measurement period) with detection date and time 1330 set. The control unit 210 may also make the created CSV file downloadable to the client device 120. For example, the download method could involve displaying a list of measurement data (CSV) as shown in Figure 14, allowing the user to select one or more from the list and press a download button (not shown). The control unit 210 then accepts this selection and executes the download (one form of output) of the corresponding measurement data (CSV). In other words, the control unit 210 outputs the measurement results to a file in CSV format. This allows users to perform their desired data analysis by filtering and displaying various information using this measurement data (CSV), as they see fit.

[0080] Another example of outputting measurement results is that the control unit 210 may generate a screen containing the count-up results and send the generated screen to the requesting client device. The screen includes a graph of the count-up results. As for how to create the graph, for example, a graph creation button 1420 (not shown) is provided in Figure 14, and when it is pressed by the user, a graph creation screen is displayed as in Figure 15, where various information such as data name (file name), report period (measurement period), device (camera ID / camera name / camera department), and detection conditions (area or line) can be set. This creates a graph of the results detected under the desired detection conditions (area or line) in the video captured by the desired camera during the desired measurement period. The image is that of the number counted in S1260, the information of the target camera / measurement period is picked up, and for example, the horizontal axis is time and the vertical axis is the count, and the count under the target detection conditions (area or line) is expressed in chronological order. If the detection condition is a line, the IN / OUT of the same line may be represented separately. Furthermore, multiple different detection conditions for the same camera, as well as detection results from multiple different cameras, may be superimposed on a single graph. The generated graph may be made downloadable, similar to the measurement data (CSV), or it may be displayed on the UI of the client device 120. In other words, the control unit 210 may display the measurement results as a graph.

[0081] As described above, according to Embodiment 1, it is possible to easily find a person in a video and register the face image of the person of interest in the information processing system 1000. Furthermore, according to Embodiment 1, it is possible to detect and output the video of the person whose face image is selected from the registered face images.

[0082] As described above, Embodiment 1 allows for accurate counting of the number of people. For example, flexible counting is possible, such as not counting staff from the same store but counting staff from different stores as customers.

[0083] (Variation 1) Modification 1 of Embodiment 1 will now be described. Modification 1 is included in Embodiment 1 and is not a different embodiment from Embodiment 1. Configurations and processes not described in Modification 1 are the same as those in Embodiment 1 described above.

[0084] In the modified example 1, the control unit 210 may generate a detection group setting screen for setting detection groups in response to a request from the client device 120 or the like, and send it to the client device 120 or the like. Figure 16 shows an example of the detection group setting screen 1600. The detection group setting screen 1600 is a screen used to identify the same person across multiple devices when there are multiple surveillance camera 160 installation locations within one department (in the example in the specification, it is assumed to be on a store basis, assuming a company that operates multiple stores, but it can also be on a country / region basis, for example, for a company that operates in multiple countries or regions, or on a brand / company basis, for a company that has multiple brands or group companies).

[0085] If a department is selected in Figure 16, the control unit 210 generates a screen containing a list of one or more surveillance cameras 160 installed in the selected department and transmits it to the client device 120 or the like. Figure 17 shows an example of a screen 1700 for selecting surveillance cameras 160 to be grouped. Through this screen, users can edit (add / delete, etc.) cameras corresponding to departments, and when instructions are received from the user, the information (information that associates camera ID 510, camera name 520 with camera department 530 in Figure 5) is saved / updated.

[0086] According to Modification 1, the surveillance cameras 160 related to each department can be grouped together. When setting up personnel departments (input area 720 in Figure 7), a list of grouped departments may be displayed, allowing the user to select a group from among them. Upon receiving instructions from the user, the information (information linking personnel ID 810 and personnel name 820 to personnel department 830 in Figure 8) may be saved / updated. This makes setting up personnel departments easy and reliable. Furthermore, when selecting a device for creating measurement data (CSV) (the device addition screen in the lower right of Figure 15), a list of grouped departments may be displayed, allowing the user to select a group. Once instructions are received from the user, measurement data (CSV) may be created using that information (video footage captured by cameras corresponding to the selected department). This makes it easy and reliable to select cameras when creating measurement data (CSV).

[0087] <Note> The product may be provided in any of the following embodiments. (Note 1) An information processing system that analyzes images captured by a camera and outputs data, Set the camera attributes for each camera. For each person, register their attributes. When predetermined detection conditions are met in the aforementioned video, detection information related to the detection is stored, including the attributes of the camera related to the video and the attributes of the person related to the video. Outputting the aforementioned detection information, Information processing system. (Note 2) An information processing system that analyzes images captured by a camera and outputs data, Set the camera attributes for each camera. For each person, register their attributes. When predetermined detection conditions are met in the aforementioned video, detection information related to the detection is stored, including information indicating whether or not the attributes of the camera related to the video correspond to the attributes of the person related to the video. Outputting the aforementioned detection information, Information processing system. (Note 3) An information processing system that analyzes images captured by a camera and outputs data, Set the camera attributes for each camera. For each person, register their attributes. When predetermined detection conditions are met in the aforementioned video, the number of people is counted unless the person in the video is a registered person and the person's attributes correspond to the attributes of the camera related to the video. Output the result of the count mentioned above. Information processing system. (Note 4) An information processing system described in any one of the appendices 1 to 3, The detection conditions are set as follows: Information processing system. (Note 5) An information processing system described in any one of the appendices 1 to 3, The aforementioned attribute is either the store, the store floor, or the area related to the store. Information processing system. (Note 6) The information processing system described in Appendix 5, Group the aforementioned attributes, Information processing system. (Note 7) An information processing system described in any one of the appendices 1 to 6, The aforementioned detection conditions include information about the area of ​​the video. Information processing system. (Note 8) An information processing system described in any one of the appendices 1 to 6, The aforementioned detection conditions include line information in the video. Information processing system. (Note 9) The information processing system described in Appendix 3, Output the count results to a file in CSV format. Information processing system. (Note 10) The information processing system described in Appendix 3, The results of the aforementioned count are displayed in a graph. Information processing system. (Note 11) An information processing method performed by an information processing system that analyzes images captured by a camera and outputs data, Set the camera attributes for each camera. For each person, register their attributes. When predetermined detection conditions are met in the aforementioned video, detection information related to the detection is stored, including the attributes of the camera related to the video and the attributes of the person related to the video. Outputting the aforementioned detection information, Information processing methods. (Note 12) An information processing method performed by an information processing system that analyzes images captured by a camera and outputs data, Set the camera attributes for each camera. For each person, register their attributes. When predetermined detection conditions are met in the aforementioned video, detection information related to the detection is stored, including information indicating whether or not the attributes of the camera related to the video correspond to the attributes of the person related to the video. Outputting the aforementioned detection information, Information processing methods. (Note 13) An information processing method performed by an information processing system that analyzes images captured by a camera and outputs data, Set the camera attributes for each camera. For each person, register their attributes. When predetermined detection conditions are met in the aforementioned video, the number of people is counted unless the person in the video is a registered person and the person's attributes correspond to the attributes of the camera related to the video. Output the result of the count mentioned above. Information processing methods. (Note 14) It is a program, Computers, A program to function as an information processing system as described in any one of the appendices 1 through 10.

[0088] While various embodiments of the present invention have been described, these are presented as examples only and are not intended to limit the scope of the invention. Novel embodiments can be implemented in a variety of other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. Embodiments and variations thereof are included in the scope and spirit of the invention, as well as in the claims and their equivalents.

[0089] For example, the client device 110 and / or client device 120 may perform some of the processing performed by the server device 100 described above. Alternatively, the surveillance camera 160 may perform some of the processing performed by the server device 100 described above. [Explanation of Symbols]

[0090] 100: Server device 110: Client device 150: Network 160: Surveillance camera 210: Control Unit 220: Storage section 230: Communications Department 1000: Information Processing System

Claims

1. An information processing system that analyzes images captured by a camera and outputs data, Set the camera attributes for each camera. For each person, register their attributes. When predetermined detection conditions are met in the aforementioned video, detection information related to the detection is stored, including the attributes of the camera related to the video and the attributes of the person related to the video. Outputting the aforementioned detection information, Information processing system.

2. An information processing system that analyzes images captured by a camera and outputs data, Set the camera attributes for each camera. For each person, register their attributes. When predetermined detection conditions are met in the aforementioned video, detection information related to the detection is stored, including information indicating whether or not the attributes of the camera related to the video correspond to the attributes of the person related to the video. Outputting the aforementioned detection information, Information processing system.

3. An information processing system that analyzes images captured by a camera and outputs data, Set the camera attributes for each camera. For each person, register their attributes. When predetermined detection conditions are met in the aforementioned video, the number of people is counted unless the person in the video is a registered person and the person's attributes correspond to the attributes of the camera related to the video. Output the result of the count mentioned above. Information processing system.

4. An information processing system according to any one of claims 1 to 3, The detection conditions are set as follows: Information processing system.

5. An information processing system according to any one of claims 1 to 3, The aforementioned attribute is either the store, the store floor, or the area related to the store. Information processing system.

6. The information processing system according to claim 5, Group the aforementioned attributes, Information processing system.

7. An information processing system according to any one of claims 1 to 3, The aforementioned detection conditions include information about the area of ​​the video. Information processing system.

8. An information processing system according to any one of claims 1 to 3, The aforementioned detection conditions include line information in the video. Information processing system.

9. The information processing system according to claim 3, Output the count results to a file in CSV format. Information processing system.

10. The information processing system according to claim 3, The results of the aforementioned count are displayed in a graph. Information processing system.

11. An information processing method performed by an information processing system that analyzes images captured by a camera and outputs data, Set the camera attributes for each camera. For each person, register their attributes. When predetermined detection conditions are met in the aforementioned video, detection information related to the detection is stored, including the attributes of the camera related to the video and the attributes of the person related to the video. Outputting the aforementioned detection information, Information processing methods.

12. An information processing method performed by an information processing system that analyzes images captured by a camera and outputs data, Set the camera attributes for each camera. For each person, register their attributes. When predetermined detection conditions are met in the aforementioned video, detection information related to the detection is stored, including information indicating whether or not the attributes of the camera related to the video correspond to the attributes of the person related to the video. Outputting the aforementioned detection information, Information processing methods.

13. An information processing method performed by an information processing system that analyzes images captured by a camera and outputs data, Set the camera attributes for each camera. For each person, register their attributes. When predetermined detection conditions are met in the aforementioned video, the number of people is counted unless the person in the video is a registered person and the person's attributes correspond to the attributes of the camera related to the video. Output the result of the count mentioned above. Information processing methods.

14. It is a program, Computers, A program for functioning as an information processing system according to any one of claims 1 to 3.

Images