System, method and program

Abstract

To provide a technology that enables more accurate settings for people flow analysis. [Solution] System (1) is a system that analyzes video captured by a camera, detects people from the video, displays the trajectory of the people's movements, and while the trajectory is displayed, displays a settings screen for configuring settings related to people flow analysis, accepts input of the settings from the user via the settings screen, and analyzes the video based on the accepted settings.

Description

[Technical Field]

[0001] The present invention relates to a system, a method, and a program. [Background technology]

[0002] Techniques have been proposed for performing people flow analysis using video captured by a camera. For example, Patent Document 1 discloses a system that displays video captured by a camera connected to a network, in which a first threshold value related to the number of people remaining in a target area and a second threshold value related to the amount of time people remain in the target area are set in response to a user's instruction, the system measures the number of people remaining in the target area and the amount of time people remain in the target area, generates thumbnails showing video when the number of people who have remained in the target area for a time equal to or greater than the second threshold value reaches or exceeds the first threshold value, and displays the generated thumbnails. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2024-21191 Summary of the Invention [Problem to be solved by the invention]

[0004] In people flow analysis, it is difficult for users to properly set the target area and the lines indicating the boundaries. Therefore, if the user performs such settings intuitively, the detection results may not be as accurate as the user desires. Even with the technology described in Patent Document 1, there is room for improvement in terms of allowing users to properly set the settings related to people flow analysis.

[0005] An object of one aspect of the present invention is to realize a technology that enables more accurate settings related to people flow analysis. [Means for solving the problem]

[0006] In order to solve the above problems, a system according to one aspect of the present disclosure is a system that analyzes video captured by a camera, detects people from the video, displays the trajectory of the people's movement, and while the trajectory is displayed, displays a settings screen for configuring settings related to people flow analysis, accepts input of the settings from a user via the settings screen, and analyzes the video based on the accepted settings.

[0007] Furthermore, a method according to one aspect of the present disclosure is a method for analyzing video captured by a camera, which includes detecting a person from the video, displaying a trajectory of the person's movement, displaying a settings screen for configuring settings related to people flow analysis while the trajectory is displayed, accepting input of the settings from a user via the settings screen, and analyzing the video based on the accepted settings.

[0008] Furthermore, a program according to one aspect of the present disclosure is a program for analyzing video captured by a camera, which causes a computer to perform the following operations: detect people from the video; display the trajectory of the people's movement; display a setting screen for configuring settings related to people flow analysis while the trajectory is displayed; accept input of the settings from a user via the setting screen; and analyze the video based on the accepted settings.

[0009] The system according to each aspect of the present invention may be realized by a computer. In this case, the control program for the system that realizes the system on a computer by causing the computer to operate as each part (software element) of the system, and the computer-readable recording medium on which the program is recorded, also fall within the scope of the present invention. [Effects of the Invention]

[0010] According to one aspect of the present invention, settings related to people flow analysis can be made with greater accuracy. [Brief explanation of the drawings]

[0011] [Figure 1] 1 is a block diagram showing an example of a system configuration according to an embodiment; [Figure 2] FIG. 1 is a block diagram showing an example of the configuration of a camera according to an embodiment. [Figure 3] FIG. 2 is a block diagram illustrating an example of a configuration of a server according to the embodiment. [Figure 4] FIG. 1 is a block diagram illustrating an example of a configuration of an information terminal according to an embodiment. [Figure 5] FIG. 1 is a sequence diagram showing an example of the flow of an imaging and detection method according to an embodiment. [Figure 6] FIG. 10 is a sequence diagram showing an example of the flow of a setting screen display method according to the embodiment. [Figure 7] FIG. 10 is a diagram illustrating an example of an area setting list screen according to the embodiment. [Figure 8] FIG. 10 is a diagram illustrating an example of an area setting screen according to the embodiment. [Figure 9] FIG. 10 is a diagram illustrating an example of an area detail setting screen according to the embodiment. [Figure 10] FIG. 10 is a diagram illustrating an example of a line detail setting screen according to the embodiment. [Figure 11] FIG. 10 is a diagram illustrating an example of an area detail setting screen according to the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0012] [Embodiment] (System Configuration) FIG. 1 is a block diagram showing an example of the configuration of a system 1 according to the present disclosure. The system 1 is a system that analyzes video captured by a camera 10. In the present disclosure, the system 1 particularly performs people flow analysis. People flow analysis is the analysis of people's movement or behavior patterns. People flow analysis can visualize, for example, the flow of people in a specific area, congestion levels, stay times, movement routes, and the like, which can be used for efficient space design and service improvement.

[0013] 1, system 1 includes camera 10, server 20, and information terminal 30. Camera 10, server 20, and information terminal 30 are communicably connected via communication line N. Although the specific configuration of communication line N does not limit the present embodiment, examples of communication line N include a wireless LAN (Local Area Network), a wired LAN, a WAN (Wide Area Network), a public line network, a mobile data communication network, or a combination thereof.

[0014] The camera 10 is installed, for example, in a store. The camera 10 captures images and transmits data representing the captured images to the server 20. Hereinafter, the data representing the images will also be simply referred to as "images." The server 20 is a device having a function for analyzing images, and is, for example, a general-purpose computer. The information terminal 30 is a terminal used by a user of the system 1, and is, for example, a personal computer such as a laptop computer, a tablet terminal, or a smartphone. While FIG. 1 illustrates one camera 10 and one information terminal 30, the system 1 may include multiple cameras 10 and multiple information terminals 30. The server 20 may also be configured to be realized by multiple devices working together.

[0015] (Camera configuration) 2 is a block diagram showing an example of the configuration of camera 10. Camera 10 includes control unit 110, storage unit 120, communication unit 130, and image capture unit 140. Image capture unit 140 captures images and generates video data representing the captured images. Control unit 110 executes instructions of a computer program stored in storage unit 120. Communication unit 130 communicates with server 20 via communication line N. Communication unit 130 transmits video data to server 20 and provides data received from server 20 to control unit 110.

[0016] The storage unit 120 stores instructions of a computer program executed by the control unit 110. The storage unit 120 also stores various information referenced by the control unit 110. The control unit 110 attaches a timestamp to the video data obtained by the imaging unit 140, and also attaches a camera ID (identification information) that identifies the control unit 110, and transmits the data to the server 20 via the communication unit 130.

[0017] (Server configuration) 3 is a block diagram showing an example of the configuration of the server 20. The server 20 includes a control unit 210, a storage unit 220, a communication unit 230, an input unit 240, and an output unit 250. The control unit 210 executes instructions of a computer program stored in the storage unit 220. The communication unit 230 communicates with devices external to the server 20 via a communication line N. The communication unit 230 transmits data supplied from the control unit 210 to other devices, and supplies data received from other devices to the control unit 210.

[0018] The input unit 240 is configured to receive input to the server 20, and includes, for example, input devices such as a keyboard, mouse, touch panel, camera, and microphone. The input unit 240 may also be configured to receive data from the input devices via an interface such as a USB (Universal Serial Bus). The output unit 250 is configured to perform output from the server 20, and includes, for example, output devices such as a display, printer, and speaker. The output unit 250 may also be a display with a touch panel that also has the functions of the input unit 240. The output unit 250 may also be configured to include, for example, an interface such as a USB, and to output data to the output device via the interface.

[0019] The storage unit 220 stores instructions of a computer program executed by the control unit 210. The storage unit 220 is an example of a storage device according to the present disclosure. The storage unit 220 also stores various types of information referenced by the control unit 210. Examples of such information include video data D1 and trajectory data D2. The video data D1 is data representing video captured by the camera 10. As an example, the video data D1 is stored in association with a camera ID that identifies the camera 10. The trajectory data D2 is data representing the trajectory of a person's movement, obtained by the video analysis unit 211, which will be described later, analyzing the video.

[0020] The control unit 210 includes a video analysis unit 211, a display control unit 212, and an input reception unit 213. Each unit of the control unit 210 is realized by the control unit 210 reading and executing instructions of a computer program stored in the storage unit 220. The video analysis unit 211 acquires video captured by the camera 10 and detects people from the video. As an example, the video analysis unit 211 detects people from the video using an existing object detection method. Examples of object detection methods include, but are not limited to, YOLO (You Only Look Once), Faster R-CNN (Region-based Convolutional Neural Network), and SSD (Single Shot MultiBox Detector).

[0021] Furthermore, the video analysis unit 211 generates trajectory data indicating the trajectory of the person's movement, and stores the trajectory data indicating the trajectory of the person detected from the video in association with time in the storage unit 220. As an example, the video analysis unit 211 detects an area including a person in each frame constituting the video, and identifies a representative point of the detected area. In this case, the video analysis unit 211 stores a set of coordinates of the identified representative point in each frame as trajectory data in the storage unit 220. At this time, the video analysis unit 211 associates the coordinates of the identified representative point in each frame with the time corresponding to that frame and stores them in the storage unit 220. Note that if multiple people are included in each frame constituting the video, the video analysis unit 211 performs the above process for each of the multiple people.

[0022] The representative point of an area (area including a person) detected by the video analysis unit 211 is, for example, the center point of that area. However, the representative point is not limited to this, and any point included inside or on the boundary of the area detected by the video analysis unit 211 can be used as the representative point of that area. As an example, if the area detected by the video analysis unit 211 is a rectangular area (bounding box), the center point of the detected rectangular area may be used as the representative point, or one of the four vertices of the detected rectangular area may be used as the representative point. In the former case, the video analysis unit 211 calculates the center point of the detected area by, for example, averaging the coordinates of the four vertices of the detected area.

[0023] The display control unit 212 transmits the trajectory data to the information terminal 30. The information terminal 30 displays the trajectory indicated by the received trajectory data on a display or the like. In this specification, the display control unit 212 transmitting the trajectory data to the information terminal 30 and displaying the trajectory indicated by the trajectory data on a display or the like of the information terminal 30 is also referred to as "the display control unit 212 displaying the trajectory."

[0024] The display control unit 212 also transmits screen data representing a setting screen for making settings related to people flow analysis to the information terminal 30, along with the trajectory data. The information terminal 30 displays the trajectories of people's movements and also displays a screen for making settings related to people flow analysis, based on the screen data received from the server 20. In other words, the display control unit 212 displays a setting screen for making settings related to people flow analysis on the display or the like of the information terminal 30, while displaying the trajectories.

[0025] The settings related to people flow analysis include, for example, setting an area to be used for people flow analysis. Examples of areas include, but are not limited to, an area in front of a cash register or an area in front of a vending machine in a store. The set area is used in the analysis process of the video analysis unit 211. For example, the video analysis unit 211 counts the inflow of people into an area and / or the outflow of people from an area to perform people flow analysis.

[0026] Additionally, the settings related to people flow analysis include, for example, the setting of lines to be used in people flow analysis. The lines indicate the boundaries of areas. For example, the video analysis unit 211 performs processing to identify the number of people who have passed a set line and the direction in which the people have passed the set line. For example, to perform people flow analysis, the video analysis unit 211 counts the number of people passing from one side of the line to the other and / or the number of people passing from the other side of the line to one side.

[0027] The input receiving unit 213 receives settings input by the user via the setting screen. The video analysis unit 211 analyzes the video captured by the camera 10 based on the settings received by the input receiving unit 213.

[0028] (Configuration of information terminal) 4 is a block diagram showing an example of the configuration of information terminal 30. Information terminal 30 includes a control unit 310, a storage unit 320, a communication unit 330, an input unit 340, and an output unit 350. Control unit 310 executes instructions of a computer program stored in storage unit 320. Storage unit 320 stores instructions of the computer program executed by control unit 310. Storage unit 320 also stores various types of information referenced by control unit 310. Communication unit 330 communicates with devices external to information terminal 30 via communication line N.

[0029] The input unit 340 is configured to receive input to the information terminal 30, and includes, for example, input devices such as a keyboard, mouse, touch panel, camera, and microphone. The input unit 340 may also be configured to receive data from the input device via an interface such as USB. The output unit 350 is configured to perform output from the information terminal 30, and includes, for example, output devices such as a display, printer, and speaker. The output unit 350 may also be a display with a touch panel that also has the functions of the input unit 240. The output unit 350 may also be configured to include, for example, an interface such as USB, and to output data to the output device via the interface.

[0030] The control unit 310 includes an application execution unit 311. The application execution unit 311 is realized by the control unit 310 reading and executing instructions of an application program stored in the storage unit 320. The application execution unit 311 executes processes such as receiving screen data transmitted from the server 20 and transmitting setting data input by the user to the server 20. An example of an application implemented by the application execution unit 311 is, but is not limited to, a general-purpose web browser. The application implemented by the application execution unit 311 may also be, for example, a dedicated application for communicating with the server 20 and transmitting and receiving data.

[0031] (Shooting method flow) 5 is a sequence diagram showing an example of the flow of the imaging and detection method by system 1. In step S101, camera 10 captures an image and generates video data representing the captured image. In step S102, camera 10 transmits the generated video data to server 20. In step S103, server 20 receives the video data and stores the received video data in storage unit 220.

[0032] In step S104, the video analysis unit 211 detects a person from the video represented by the video data, and generates trajectory data representing the trajectory of the person's movement. The video analysis unit 211 also stores the generated trajectory data in the storage unit 220.

[0033] (Flow of how to display the settings screen) 6 is a sequence diagram showing an example of the flow of a setting screen display method according to the present disclosure. When a user uses the input unit 340 of the information terminal 30 to perform an operation to display a setting screen for performing area setting or line setting, in step S111, the application execution unit 311 of the information terminal 30 accepts an instruction to display the setting screen. In step S112, the application execution unit 311 transmits a request for the setting screen to the server 20. In step S113, the display control unit 212 transmits data for an area list screen to the information terminal 30. In step S114, the application execution unit 311 receives screen data from the server 20 and displays the area list screen on a display or the like based on the received screen data.

[0034] FIG. 7 is a diagram showing an example of an area setting list screen. In the example of FIG. 7, the area setting list screen SC10 includes a display area A11 and a display area A12. An interface for accepting the selection of a camera 10 is displayed in the display area A11. Specifically, a pull-down menu for selecting from a list of cameras 10 is displayed in the display area A11. A list of pre-registered areas for a camera 10 selected by a user from the cameras 10 displayed in the pull-down menu is displayed in the display area A12. In the example of FIG. 7, a list of three areas, "next to the cash register," "in front of the vending machine," and "in front of the front shelf," is displayed in the display area A12.

[0035] Display area A12 includes display areas A121, A122, and A123. Display area A121 displays information about the area "next to the cash register." Display area A122 displays information about the area "in front of the vending machine." Display area A123 displays information about the area "in front of the front shelf." Display areas A121, A122, and A123 each display a thumbnail, area name, detection conditions, notification settings, etc.

[0036] The thumbnails are thumbnails of the video captured by the camera 10. The thumbnails have superimposed thereon the trajectory of the person's movement detected by the video analysis unit 211. However, instead of thumbnails, the display areas A121, A122, and A123 may display images in which the trajectory is superimposed on the video captured by the camera 10.

[0037] The area name is the name of the area, such as "next to the cash register," "in front of the vending machine," or "in front of the front shelf." The detection condition is the condition for detecting people in each area, such as "detection when one or more people remain in the area for five seconds or more." The notification setting is a setting for what kind of notification is to be sent to the user. The area name, detection condition, and notification setting are information that is set in advance by the user.

[0038] Furthermore, a settings button C1211 and a results list button C1212 are displayed in the display areas A121, A122, and A123, respectively. The settings button C1211 is a button for confirming and changing the settings for that area. The results list button C1212 is a button for displaying a list of results detected under the detection conditions. The user can use the input unit 340 to perform an operation to select the settings button C1211 or the results list button C1212.

[0039] When the user performs an operation to select the setting button C1211 on the screen of Fig. 7, the application execution unit 311 transmits a request for an area setting screen to the server in step S115 of Fig. 6. In step S116, the server 20 transmits data of the area setting screen to the information terminal 30 as a response to the request received from the information terminal 30. The application execution unit 311 of the information terminal 30 displays the area setting screen on a display or the like based on the received data of the area setting screen.

[0040] Fig. 8 is a diagram showing an example of an area setting screen. In the example of Fig. 8, area setting screen SC20 includes display areas A21 to A26. Information for identifying camera 10 is displayed in display area A21. An image in which a trajectory is superimposed on a video captured by camera 10 is displayed in display area A22. However, display area A22 may also display an image in which a trajectory is superimposed on a thumbnail of the video captured by camera 10.

[0041] An interface for registering and editing areas is displayed in the display area A23. Specifically, a hyperlink for registering and editing areas is displayed in the display area A23. When the user clicks on this hyperlink using the input unit 340, the application execution unit 311 transmits a request for an area setting screen to the server 20.

[0042] A pull-down menu for selecting an area is displayed in display area A24. A text box for editing the setting name of an area selected by the user from the areas displayed in the pull-down menu of display area A24 is displayed in display area A25. The user can edit the setting name of the area by editing the text displayed in the text box of display area A25.

[0043] A text box for editing the detection conditions is displayed in the display area A26. The user can edit the detection conditions by editing the text displayed in the text box in the display area A26.

[0044] When an operation of clicking on the hyperlink displayed in display area A23 in Fig. 8 is performed, application execution unit 311 transmits a request for a detailed setting screen for the area to server 20 in step S118 in Fig. 6. Upon receiving the request from information terminal 30, display control unit 212 of server 20 superimposes the trajectory indicated by the trajectory data on the image received from camera 10 in step S119, and transmits a detailed setting screen including the image data on which the trajectory has been superimposed to information terminal 30 in step S120.

[0045] Fig. 9 is a diagram showing an example of an area detail setting screen. The application execution unit 311 displays the detail setting screen on a display or the like based on the data of the detail setting screen received from the server 20. As shown in Fig. 9, the application execution unit 311 displays a setting screen for making settings related to people flow analysis while displaying the trajectories of people's movements.

[0046] 9, the detail setting screen SC30 includes a display area A31 and a display area A32. In the display area A31, an image is displayed in which figures A311 to A314 indicating an area and trajectories A315, A316, ... of people's movements are superimposed on the video captured by the camera 10. In other words, the application execution unit 311 displays the trajectories A315, A316, ... superimposed on the video captured by the camera 10 based on the data received from the server 20. Here, the video displayed is, for example, live video captured by the camera 10. Alternatively, for example, the video displayed may be recorded video captured by the camera 10 at a time specified by the user.

[0047] Graphic A311 is a graphic that indicates the area "in front of the register," and graphic A312 is a graphic that indicates the area "in front of the vending machine." Graphic A313 is a graphic that indicates the area "in front of the front shelf," and graphic A314 is a graphic that indicates the area "in front of the right shelf." The user can edit the position and shape of each area by using input unit 340 to perform operations such as clicking or dragging on graphics A311 to A314 that indicate each area.

[0048] Trajectories A315, A316, ... are trajectories of the movement of a person detected from the video, and as an example, are trajectories of the representative point of the area (bounding box) of the person detected from the video. In other words, the application execution unit 311 detects an area including a person from the video based on the data received from the server 20, and displays the trajectory of the movement of the representative point of the area as the trajectory of the person's movement. More specifically, as an example, the application execution unit 311 displays the trajectory of the movement of the center point of the area based on the data received from the server 20 as the trajectory of the person's movement.

[0049] 9, each portion of the trajectory A315, A316, ... is displayed in a display mode corresponding to the state of the person corresponding to that portion. In other words, the application execution unit 311 displays each portion of the trajectory in a display mode corresponding to the state of the person corresponding to that portion, based on the data received from the server 20. More specifically, as an example, the application execution unit 311 displays, based on the data received from the server 20, a portion of the trajectory corresponding to a state in which the person is moving and a portion of the trajectory corresponding to a state in which the person is standing still, in different display modes. As an example, based on the data received from the server 20, the application execution unit 311 displays, in different display modes, a portion of the trajectory corresponding to a state in which the person is moving at a speed above a threshold and a portion of the trajectory corresponding to a state in which the person is moving at a speed below the threshold. As another example, the application execution unit 311 may change the display mode depending on the acceleration of the person's movement.

[0050] Here, displaying in a display mode according to the person's state means changing the appearance (color, thickness, pattern, etc.) of the curve representing the trajectory depending on the person's state. In the example of Fig. 9, the trajectory in the section where the moving speed is equal to or greater than a predetermined threshold is displayed as a solid line, and the trajectory in the section where the moving speed is less than the threshold is displayed as a dotted line.

[0051] Display area A32 displays a list A321 and a new button A322. List A321 is a list of areas set by the user. New button A322 is a button for creating a new area. For example, the user can use input unit 340 to select the new button A322, and then click the vertices of the area in order and double-click the last vertex to set an area connecting those vertices.

[0052] 6, if the information terminal 30 transmits a request for a line detail setting screen instead of an area detail setting screen, the server 20 transmits data for the line detail setting screen to the information terminal 30 in step S120. Fig. 10 is a diagram showing an example of a line detail setting screen. In the example of Fig. 10, the detail setting screen SC40 includes a display area A41 and a display area A42.

[0053] Display area A41 displays an image in which trajectories A315, A316, ... of people's movements and a graphic A411 representing a line are superimposed on the video captured by camera 10. The user can edit the position and shape of the line by using input unit 340 to perform operations such as clicking or dragging on graphic A411 representing the line.

[0054] Display area A42 displays a list A421 and a new button A422. List A421 is a list of lines set by the user. New button A422 is a button for creating a new line. For example, the user can set a desired line by selecting new button A422 and then setting the start and end points of the line using a click operation or the like. More specifically, for example, the start point is determined with a first click and the end point with a second click, and a line connecting the two points and an arrow indicating IN / OUT in a direction perpendicular to the line are set in opposite directions.

[0055] 11 is a diagram showing another example of the area detail setting screen. In the example of FIG. 11, the area detail setting screen SC50 includes a display area A33 in addition to the display areas A31 and A32 of FIG. 9. A timeline for adjusting the time of day to be displayed in the display area A31 is displayed in the display area A33. The user can use the input unit 340 to perform an operation to move a slider A331 on the timeline. When the slider A331 is moved by a user operation, the application execution unit 311 displays the video of the time corresponding to the position of the slider A331 in the display area A31.

[0056] 9, 10, or 11, when the user edits / adds area or line settings on the setting screen, the application execution unit 311 accepts the setting input via the setting screen in step S122 of FIG. 6, and transmits setting data indicating the accepted settings to the server 20 in step S123. Upon receiving the setting data from the information terminal 30, the server 20 saves the settings indicated by the received setting data in step S124. In step S125, the video analysis unit 211 of the server 20 analyzes the video based on the settings received from the information terminal 30.

[0057] (Example of how to display the trajectory) As described above, the area setting list screen SC10 of FIG. 7, the area setting screen SC20 of FIG. 8, the area detail setting screen SC30 of FIG. 9, the line detail setting screen SC40 of FIG. 10, and the area detail setting screen SC50 of FIG. 11 display trajectories A315, A316, ... of people's movements superimposed on the video captured by the camera 10. At this time, the application execution unit 311 may display only the most recent few trajectories rather than all trajectories detected so far based on the data received from the server 20. In this case, the application execution unit 311 may, for example, display a predetermined number of trajectories going back from the time the video was captured. More specifically, for example, the application execution unit 311 may, based on the data received from the server 20, display trajectories indicated by each of a predetermined number of trajectory data items in order of most recent associated time from among a plurality of trajectory data items stored in the storage unit 220 in association with times earlier than the time the video was captured. At this time, the number of trajectories to be displayed may be, for example, a predetermined number, or may be specified by the user.

[0058] Further, as an example, the application execution unit 311 may display a trajectory within a predetermined time period going back from the time when the video was captured. More specifically, as an example, the application execution unit 311 may display, based on data received from the server 20, trajectories indicated by each of the trajectory data associated with a time period after the predetermined time period going back from the time when the video was captured, among a plurality of trajectory data stored in the storage unit 220 in association with a time period before the time when the video was captured. In this case, the predetermined time period may be, for example, a predetermined time period or may be specified by the user. Further, as an example, the application execution unit 311 may display the latest trajectory and a non-latest trajectory in a distinguishable manner.

[0059] Furthermore, the application execution unit 311 may display the trajectory of a person's movement in a display mode according to the attributes of the person corresponding to the trajectory, based on the data received from the server 20. Here, the attributes may be, for example, any of age, sex, clothing, race, nationality, and employee-related attributes. Examples of clothing-related attributes include, but are not limited to, the type of clothing, whether or not the person is wearing a hat, whether or not the person is wearing a mask, and whether or not the person is wearing glasses (especially sunglasses). Examples of employee-related attributes include, but are not limited to, the employee's job title, affiliation, etc.

[0060] In this case, the video analysis unit 211, for example, identifies the attributes of people detected from the video, generates setting screen data in which the display mode of the trajectory varies depending on the identified attributes, and transmits the generated setting screen data to the information terminal 30. Examples of methods for identifying attributes include a method using a trained model generated by machine learning, a method using the output of an object detection model, and a method performing image analysis processing. For example, when an object detection model is used to detect people, the video analysis unit 211 may use the object classification results output from the object detection model as the attributes of the people. In this case, for example, the object detection model is a model that detects people from the video and outputs the area of ​​the detected person and the attributes of the person. Furthermore, the video analysis unit 211 may estimate the attributes of the people using a trained model generated by machine learning. In this case, for example, the trained model is a model that receives an image of the area of ​​the person detected by the object detection model as input and outputs the attributes of the person. Furthermore, for example, the video analysis unit 211 may detect faces in the video and estimate attributes such as age or gender by analyzing facial features (such as wrinkles, bone structure, and texture). Furthermore, the video analysis unit 211 may analyze facial feature points (eyes, mouth, etc.) and estimate attributes based on the feature points.

[0061] Furthermore, the video analysis unit 211 may identify a person detected from the video and specify the person's attributes by referring to a database in which personal identification information and attributes are linked. More specifically, the video analysis unit 211 may identify an individual using a technique such as face recognition, and specify the attributes corresponding to the identified individual by referring to a database in which personal identification information and attributes (nationality, race, department, etc.) are linked.

[0062] (Another example of area settings) In the example of FIG. 9 , the area is set by the user editing the position and shape of the area using the input unit 340, but the method for setting the area is not limited to the above example. For example, an area including a section of the trajectory where a person is stopped may be set. In this case, as an example, the application execution unit 311 receives from the user a designation of a portion of the trajectory corresponding to the stopped state, and transmits data indicating the received designation to the server 20. The input reception unit 213 of the server 20 sets an area including the above-mentioned portion based on the received designation.

[0063] The server 20 may also perform settings related to people flow analysis based on a trajectory indicated by the generated trajectory data. For example, the input receiving unit 213 may set an area to be targeted for people flow analysis based on a trajectory selected by a user from among multiple displayed trajectories. More specifically, for example, the input receiving unit 213 may set an area including a section where a person stopped on the trajectory selected by the user as the target area for people flow analysis. For another example, the input receiving unit 213 may set a line to be targeted for people flow analysis based on a trajectory selected by a user from among multiple displayed trajectories. More specifically, for example, the input receiving unit 213 may set a line intersecting the trajectory selected by the user as the target line for people flow analysis.

[0064] (Problems of the embodiment) In recent years, digital transformation (DX) has been promoted in various workplaces by analyzing video captured by cameras. One of the main technologies is to visualize people flow by setting areas and lines in the video. With this type of technology, accurate detection results cannot be obtained unless the areas and lines are set appropriately. However, when users set the areas and lines intuitively, there are many cases where the desired detection accuracy cannot be achieved. In such cases, the user's experience is required to set the settings, which is not practical for a system used by many users.

[0065] Generally, various detections are often performed using the center point of a bounding box. This is because many object detection models use an output format based on the center point of the bounding box. Furthermore, the center point is relatively stable even when pose or posture changes, has high compatibility with tracking algorithms, and is easy to implement. In such cases, the gap with many users' intuition (setting the bounding box with the feet in mind) becomes large, which can significantly reduce detection accuracy. One of the objectives of this embodiment is to provide a system that is easy for users to use while ensuring detection accuracy in setting up people flow analysis.

[0066] (Effects of the embodiment) System 1 according to the present disclosure is a system that analyzes video captured by camera 10, detects people from the video, displays the trajectories of the people, and while the trajectories are displayed, displays a setting screen for configuring people flow analysis settings. The system accepts settings input from a user via the setting screen and analyzes the video based on the accepted settings. At this time, as illustrated in FIG. 9 , the setting screen displays area-setting graphics A311-A314 and people's movement trajectories A315, A316, etc., superimposed on the video captured by camera 10. By visually checking the trajectories superimposed on the video, the user can grasp where people are actually moving, which facilitates configuring people flow analysis settings such as area and line settings. In other words, system 1 according to the present disclosure allows users to configure people flow analysis settings with greater accuracy.

[0067] Furthermore, the system 1 according to the present disclosure displays each part of a person's trajectory on the setting screen in a display mode that corresponds to the state of the person corresponding to that part. This allows the user to set areas and lines that take into account the state of people's behavior, such as setting an area to include a location where a person is stopped, and allows for more accurate settings related to people flow analysis.

[0068] Furthermore, the system 1 according to the present disclosure displays the trajectory of a person's movement in different display modes for the portion of the trajectory where the person is moving and the portion of the trajectory where the person is stopped, and displays in different display modes for the portion of the trajectory where the person is moving at a speed above a threshold and the portion of the trajectory where the person is moving at a speed below the threshold. This allows the user to set areas and lines that take into account the behavioral state of people, for example, by setting an area to include a section where people are moving slowly, and allows for more accurate settings related to people flow analysis.

[0069] Furthermore, when displaying trajectories superimposed on video, the system 1 according to the present disclosure displays trajectories indicated by each of a predetermined number of trajectory data items among a plurality of trajectory data items associated with times before the time the video was captured and stored in the storage unit 220, in order of most recent associated time, or displays trajectories indicated by each of a plurality of trajectory data items associated with times after a predetermined time prior to the time the video was captured, among a plurality of trajectory data items associated with times before the time the video was captured and stored in the storage unit 220. In this way, selecting trajectories to display can prevent the video displayed on the setting screen from becoming cluttered, thereby enabling the user to more accurately configure people flow analysis.

[0070] Furthermore, the system 1 according to the present disclosure displays the latest trajectories and non-latest trajectories in a distinguishable manner on the setting screen. By checking the trajectories displayed in a distinguishable manner, the user can perform settings related to people flow analysis with greater accuracy.

[0071] Furthermore, the system 1 according to this embodiment displays the trajectory of the center point of the area of ​​a person detected from the video as the trajectory of the person's movement. The center point of the bounding box is relatively stable even when the person's pose or posture changes, has high compatibility with tracking algorithms, and is relatively easy to implement.

[0072] [Modification] The functions of the server 20 described above may be shared and implemented by a plurality of devices. For example, the server 20 described above may be realized as a system in which two or more devices are connected via a communication network. In this case, the system may include, as an example, a first device including a display control unit 212 and an input receiving unit 213, and a second device including a video analysis unit 211. In this case, the functions of the server 20 are realized by the first device and the second device working together.

[0073] Furthermore, at least one of the functions of the camera 10, the server 20, and the information terminal 30 according to the above-described embodiments may be implemented in different devices. For example, in the above-described embodiment, as shown in step S104 of Fig. 5, the video analysis unit 211 detects a person from the video represented by the video data and generates trajectory data representing the trajectory of the person's movement. However, the camera 10, rather than the server 20, may execute the process of step S104 of Fig. 5. As another example, the camera 10, rather than the server 20, may execute at least one of the processes of step S119 (processing of superimposing a trajectory on the video), step S124 (processing of saving settings), and step S125 (processing of analyzing the video) of Fig. 6.

[0074] [Software implementation example] The functions of the camera 10, the server 20, and the information terminal 30 (hereinafter referred to as "devices") can be realized by a program that causes a computer to function as the device, and a program that causes a computer to function as each control block of the device (particularly each part included in the control units 110, 210, 310).

[0075] In this case, the device includes a computer having at least one control device (e.g., a processor) and at least one storage device (e.g., a memory) as hardware for executing the program. The control device and storage device execute the program, thereby realizing the functions described in each of the above embodiments.

[0076] The program may be non-transitory and may be recorded on one or more computer-readable recording media. The recording media may or may not be included in the device. In the latter case, the program may be supplied to the device via any wired or wireless transmission medium.

[0077] Furthermore, some or all of the functions of the control blocks can be realized by logic circuits. For example, an integrated circuit in which a logic circuit that functions as each of the control blocks is formed is also included in the scope of the present invention. In addition, the functions of the control blocks can also be realized by, for example, a quantum computer.

[0078] Furthermore, each process described in each of the above embodiments may be executed by AI (Artificial Intelligence). In this case, the AI ​​may run on the control device or on another device (for example, an edge computer or a cloud server).

[0079] The present invention is not limited to the above-described embodiments, and various modifications are possible within the scope of the claims. Embodiments obtained by appropriately combining the technical means disclosed in different embodiments are also included in the technical scope of the present invention.

[0080] 〔summary〕 The system according to aspect 1 of the present invention is a system that analyzes video captured by a camera, detects people from the video, displays the trajectory of the people's movements, and while the trajectory is displayed, displays a settings screen for configuring settings related to people flow analysis, accepts input of the settings from a user via the settings screen, and analyzes the video based on the accepted settings.

[0081] Furthermore, a system according to a second aspect of the present disclosure is the system according to the first aspect, in which each part of the trajectory is displayed in a display mode according to the state of the person corresponding to that part.

[0082] In addition, a system according to aspect 3 of the present disclosure is the system described in aspect 2, in which the portion of the trajectory corresponding to the state in which the person is moving and the portion of the trajectory corresponding to the state in which the person is stopped are displayed in different display modes.

[0083] In addition, a system according to aspect 4 of the present disclosure is a system according to aspect 2 or 3, in which a portion of the trajectory corresponding to a state in which the person is moving at a speed above a threshold and a portion of the trajectory corresponding to a state in which the person is moving at a speed below the threshold are displayed in different display modes.

[0084] Furthermore, a system according to a fifth aspect of the present disclosure is the system according to any one of the first to fourth aspects, in which the trajectory is displayed in a display mode according to the attributes of the person corresponding to the trajectory.

[0085] Furthermore, a system according to a sixth aspect of the present disclosure is the system according to the fifth aspect, wherein the attribute is any one of age, sex, clothing, race, nationality, and employee.

[0086] Furthermore, a system according to a seventh aspect of the present disclosure is the system according to any one of the first to sixth aspects, in which the trajectory is displayed superimposed on the image.

[0087] Furthermore, a system according to an eighth aspect of the present disclosure is the system according to the seventh aspect, wherein the video is live video captured by the camera.

[0088] Furthermore, a system according to a ninth aspect of the present disclosure is the system according to the seventh aspect, wherein the video is a recorded video taken by the camera at a time designated by a user.

[0089] Furthermore, a system according to a tenth aspect of the present disclosure is the system according to any one of the seventh to ninth aspects, and displays a predetermined number of trajectories going back from the time when the video was captured.

[0090] Furthermore, a system according to an eleventh aspect of the present disclosure is the system according to any one of the seventh to ninth aspects, and displays a trajectory within a predetermined time period going back from the time when the video was captured.

[0091] Furthermore, a system according to a twelfth aspect of the present disclosure is a system according to any one of the first to eleventh aspects, wherein the settings include settings of an area to be used for the people flow analysis.

[0092] Furthermore, a system according to aspect 13 of the present disclosure is a system according to any one of aspects 1 to 12, wherein the settings include settings of lines to be used in the people flow analysis.

[0093] Furthermore, a system according to a fourteenth aspect of the present disclosure is a system according to any one of the first to thirteenth aspects, in which the latest trajectory and the non-latest trajectory are displayed in a distinguishable manner.

[0094] In addition, a system according to aspect 15 of the present disclosure is a system according to aspect 3, which receives a designation from the user of a portion of the trajectory that corresponds to the stopped state, and sets an area including the portion based on the designation.

[0095] In addition, a system according to aspect 16 of the present disclosure is a system described in any one of aspects 1 to 15, which detects an area containing a person from the video and displays the trajectory of the movement of a representative point of the area as the trajectory of the movement of the person.

[0096] In addition, a system according to aspect 17 of the present disclosure is the system described in aspect 16, in which the trajectory of the movement of the center point of the area is displayed as the trajectory of the movement of the person.

[0097] Furthermore, a method according to aspect 18 of the present disclosure is a method for analyzing video captured by a camera, and includes detecting a person from the video, displaying a trajectory of the person's movement, displaying a setting screen for configuring settings related to people flow analysis while the trajectory is displayed, accepting input of the settings from a user via the setting screen, and analyzing the video based on the accepted settings.

[0098] Furthermore, the program according to aspect 19 of the present disclosure is a program for analyzing video captured by a camera, and causes a computer to perform the following operations: detect people from the video; display the trajectory of the people's movement; while the trajectory is displayed, display a setting screen for configuring settings related to people flow analysis; accept input of the settings from a user via the setting screen; and analyze the video based on the accepted settings. [Explanation of symbols]

[0099] 1 System 10 Camera 20 servers 30 Information terminal 211 Video Analysis Department 212 Display control unit 213 Input Reception Unit 311 Application Execution Unit

Claims

1. A system for setting up people flow analysis for images captured by a camera, Detecting a person from the video; Displaying the trajectory of the person's movement; displaying a setting screen for setting the trajectory to be used in the people flow analysis while the trajectory is displayed; receiving input of the settings from a user via the setting screen; The received input is used as a setting for the people flow analysis of the video. system.

2. Each part of the trajectory is displayed in a display mode corresponding to the state of the person corresponding to that part. The system of claim 1 .

3. a portion of the trajectory corresponding to a state in which the person is moving and a portion of the trajectory corresponding to a state in which the person is standing still are displayed in different display modes; The system of claim 2 .

4. a portion of the trajectory corresponding to a state in which the person is moving at a speed exceeding a threshold and a portion of the trajectory corresponding to a state in which the person is moving at a speed below the threshold are displayed in different display modes; The system of claim 2 .

5. displaying the trajectory in a display mode according to the attribute of the person corresponding to the trajectory; The system of claim 1 .

6. The attribute is any one of age, sex, clothing, race, nationality, and employee. The system of claim 5.

7. displaying the trajectory superimposed on the image; The system of claim 1 .

8. The video is a live video captured by the camera. The system of claim 7.

9. The video is a recorded video taken by the camera at a time designated by the user. The system of claim 7.

10. Displaying a predetermined number of trajectories going back from the time when the video was taken; The system of claim 7.

11. Displaying a trajectory within a predetermined time period going back from the time when the video was taken; The system of claim 7.

12. The setting includes setting an area to be used for the people flow analysis. The system of claim 1 .

13. The setting includes setting a line to be used in the people flow analysis. The system of claim 1 .

14. Displaying the latest trajectory and non-latest trajectory in a distinguishable manner; The system of claim 1 .

15. receiving, from a user, a designation of a portion of the trajectory corresponding to a state where the person is stopped; setting an area including the location based on the designation; The system of claim 3.

16. Detecting an area including a person from the video; The trajectory of the movement of the representative point of the area is displayed as the trajectory of the movement of the person. The system of claim 1 .

17. The trajectory of the movement of the center point of the area is displayed as the trajectory of the movement of the person.

17. The system of claim 16.

18. A method for setting up people flow analysis for video captured by a camera, comprising: detecting a person from the video; Displaying a trajectory of movement of the person; displaying a setting screen for setting the people flow analysis while the trajectory is displayed; receiving an input of the setting from a user via the setting screen; setting the received input as a setting for the people flow analysis for the video; A method comprising:

19. A program for setting up people flow analysis for images captured by a camera, detecting a person from the video; Displaying a trajectory of movement of the person; displaying a setting screen for setting the people flow analysis while the trajectory is displayed; receiving an input of the setting from a user via the setting screen; setting the received input as a setting for the people flow analysis for the video; A program that causes a computer to execute the following.

Images