Flow line analysis device, flow line analysis method, and computer program

Abstract

Provided is a line of movement analysis device that uses detection results from at least one sensor that is provided in an environment in which there is a moving body to extract lines of movement along which the moving body has moved. The line of movement analysis device comprises: a storage unit that stores partial line of movement data that indicates a plurality of partial lines of movement that have been pre-extracted from detection results, connection parameter information that indicates connection conditions for connecting the partial lines of movement, and partial area data that defines each of a plurality of partial areas; and a control unit that connects pairs of the partial lines of movement that each belong to one of the plurality of partial areas on the basis of the connection parameter information and thereby extracts a line of movement for the moving body for every partial area. The connection parameter information is set for every partial area.

Description

[Technical Field]

[0001] The present disclosure relates to a flow line analysis device, a flow line analysis method, and a computer program that extract the flow line of a moving object from the detection results of a sensor installed in an environment where the moving object exists. [Background technology]

[0002] Patent Document 1 provides a technology for tracking people in consecutive image frames captured by a camera. More specifically, Patent Document 1 discloses a technology for robustly continuing person tracking by complementing tracking information even when inter-person occlusion occurs, where people overlap each other in an image frame, or when person detection fails. In Patent Document 1, the connection destination of an interrupted flow line is determined based on the score between frames. Flow lines with an evaluation cost equal to or greater than a preset threshold are adopted as candidate connection destinations. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2020-091664 Summary of the Invention [Problem to be solved by the invention]

[0004] The technology of Patent Document 1 requires that a threshold value for the evaluation cost be set in advance, and setting an optimal threshold value requires trial and error, which is time-consuming.

[0005] The present disclosure provides a technology for automatically determining connection parameter information indicating connection conditions for connecting multiple partial traffic lines into one when an original single traffic line is interrupted and recognized as multiple partial traffic lines. [Means for solving the problem]

[0006] A movement trajectory analysis device according to one aspect of the present disclosure extracts a movement trajectory of a moving object based on detection results from one or more sensors installed in an environment in which the moving object exists. The movement trajectory analysis device includes a memory unit that stores partial movement trajectory data indicating a plurality of partial movement trajectories previously extracted from the detection results, connection parameter information indicating connection conditions for connecting the partial movement trajectories, and partial area data defining each of a plurality of partial areas, and a control unit that connects two partial movement trajectories, each of which belongs to one of the plurality of partial areas, based on the connection parameter information, to extract a movement trajectory of the moving object for each partial area. The connection parameter information is set for each partial area.

[0007] A movement trajectory analysis method according to one aspect of the present disclosure is a method for extracting a movement trajectory of a moving object from detection results of one or more sensors installed in an environment where the moving object exists, using a movement trajectory analysis system. The movement trajectory analysis device of the movement trajectory analysis system includes a control unit and a memory unit that stores partial movement trajectory data indicating multiple partial movement trajectories extracted in advance from the detection results, connection parameter information indicating connection conditions for connecting the partial movement trajectories, and partial area data that defines each of multiple partial areas. The control unit executes the following processes: reading the partial movement trajectory data, connection parameter information, and partial area data from the memory unit; setting the connection parameter information for each partial area; and connecting two partial movement trajectories, each of which belongs to one of the multiple partial areas, based on the connection parameter information, to extract the movement trajectory of the moving object for each partial area. [Effects of the Invention]

[0008] According to the present disclosure, it is possible to provide a technology for automatically determining connection parameter information indicating connection conditions for connecting disconnected flow lines. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram illustrating an overview of a traffic path analysis system according to an exemplary embodiment of the present disclosure. [Figure 2] FIG. 1 is a block diagram illustrating a configuration of a movement line analysis device according to an exemplary embodiment of the present disclosure. [Figure 3A]FIG. 10 is a diagram showing a frame image including an image of a worker. [Figure 3B] FIG. 10 is a diagram showing a frame image including an image of a worker. [Figure 3C] FIG. 10 is a diagram showing a frame image including an image of a worker. [Figure 3D] A diagram showing an example of a flow line generated by connecting a series of centroids [Figure 4] A diagram showing an example of a flow line generated by focusing on a specific person [Figure 5] Diagram showing the relationship between the fields of view of the two cameras [Figure 6] A diagram showing the time series of changes in the position of a worker moving across the field of view of two cameras. [Figure 7] A diagram showing the movement paths generated from video data from two cameras. [Figure 8A] Diagram showing the worker's position before entering the camera's blind spot [Figure 8B] Diagram showing the worker's position while in the camera's blind spot [Figure 8C] Diagram showing the worker's position after entering the camera's blind spot [Figure 8D] A diagram showing an example of the flow lines of the same worker being generated as different flow lines. [Figure 9] FIG. 10 is a diagram showing an example of flow line data defining a plurality of flow lines including four independently generated flow lines. [Figure 10] A diagram showing an example in which two partial flow lines are connected via an interpolated flow line. [Figure 11] FIG. 10 is a diagram showing a flow of partial flow line connection processing performed by the flow line analysis device. [Figure 12] A diagram showing the relationship between the connection distance and connection time between two partial flow lines when the connection conditions of the two partial flow lines are changed. [Figure 13] A diagram showing an example of a range surrounding a predetermined percentage of the correct point group. [Figure 14] A diagram for explaining a method for determining flow line connection parameter information while changing the connection conditions of two partial flow lines. [Figure 15] A diagram showing subregions determined from the feature values ​​of each subregion using edge detection technology. [Figure 16] A diagram showing a plurality of partial regions set for the entire video of the environment. [Figure 17] A diagram showing an example of two ranges representing connection conditions set for each of different partial areas, and a range representing connection conditions when a partial area is not set and the connection conditions are set for the entire environment. [Figure 18] A flowchart showing the procedure for dividing an entire area into a plurality of partial areas and setting connection conditions, and for applying the set connection conditions to each partial area to connect one flow line from the partial flow lines. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments will be described in detail with reference to the drawings as appropriate. However, more detailed description than necessary may be omitted. For example, detailed description of already well-known matters or redundant description of substantially identical configurations may be omitted. This is to avoid unnecessary redundancy in the following description and to facilitate understanding by those skilled in the art. Note that the inventor(s) provide the accompanying drawings and the following description to enable those skilled in the art to fully understand the present disclosure, and are not intended to limit the subject matter described in the claims.

[0011] 1. Configuration The movement line analysis system according to this embodiment will be described with reference to Fig. 1. Fig. 1 is a diagram showing an overview of a movement line analysis system 100 according to this embodiment. Hereinafter, the movement line analysis system 100 will be abbreviated as "this system 100."

[0012] 1-1. Overview of the traffic flow analysis system As shown in FIG. 1, the present system 100 includes multiple cameras 20a and 20b and a movement line analysis device 50. The present system 100 makes it possible to extract movement lines, which are the trajectories of the movements of each worker 11 and 12, from video footage of a workplace environment 200, such as a factory or store, where the workers 11 and 12 are present. In this specification, people, objects, etc., such as the workers 11 and 12, that are photographed and from which movement lines are extracted are referred to as "moving bodies." In this embodiment, the process of extracting the movement lines of the moving bodies is mainly performed by the movement line analysis device 50.

[0013] The system 100 may also store the extracted movement lines. The stored movement lines of the workers 11 and 12, etc., can be used by a user 30, such as a manager or analyst of the workplace environment 200, when analyzing the layout of equipment and work efficiency.

[0014] In this system 100, cameras 20a and 20b are each an example of a sensor that detects the position of a mobile object installed in a workplace environment 200. Another example of a sensor is a beacon signal receiver when the mobile object moves with a beacon signal transmitter, or a GPS (Global Positioning System) signal receiver when the mobile object has a receiver for receiving the GPS signal. As a beacon signal, for example, a signal using high-frequency electromagnetic waves such as radio waves conforming to the Bluetooth (registered trademark) standard or infrared rays can be used. Note that the number of sensors installed may be one or more.

[0015] The cameras 20a and 20b are, for example, box cameras. The cameras 20a and 20b are arranged so as to capture the entire range in which the workers 11 and 12, etc., move in the workplace environment 200. The cameras 20a and 20b each capture video of the workplace 200 to generate video data D0. The video data D0 is, for example, a collection of frame image data at a frame rate of 5 frames per second (5 fps).

[0016] It should be noted that the system 100 does not necessarily have to have multiple cameras 20a and 20b. For example, a single omnidirectional camera with a fisheye lens that can capture the entire environment 200 may be provided. The entire range in which the workers 11 and 12, etc., move does not need to be completely included in the field of view of the cameras 20a and 20b. For example, the area hidden by the pillars P1 and P2 in the environment 200 may be a blind spot for the cameras 20a and 20b. However, the entire range in which the workers 11 and 12, etc., move may include such a blind spot. A camera is sometimes called an imaging device.

[0017] The flow line analysis device 50 of the present system 100 collects or generates various data and stores it in a storage device (described later). For example, the flow line analysis device 50 collects video data D0 obtained by capturing images using the cameras 20a and 20b. The flow line analysis device 50 also stores flow line data D1, area data D2, and flow line connection parameter information D3 in the storage device. The data D0 to D3 will be described in detail later.

[0018] The movement line analysis device 50 may present the movement line data of each moving object, etc., to the user 30 so that the data can be analyzed. In Fig. 1, two movement lines La and Lb are shown on a display unit such as a display.

[0019] 1-2.Configuration of the traffic flow analysis device The configuration of the flow line analysis device 50 in the present system 100 will be described below with reference to Fig. 2. Fig. 2 is a block diagram illustrating the configuration of the flow line analysis device 50 of this embodiment.

[0020] The flow line analysis device 50 is configured with an information processing device such as a PC or a tablet computer. The flow line analysis device 50 illustrated in Fig. 2 includes a device interface (I / F) 51 for connecting to external devices such as cameras 20a and 20b, an operation unit 52 for accepting user operations, a display unit 53 for displaying screen images, a control unit 54 for controlling the operation of the flow line analysis device 50, and a storage unit 57 for storing data and programs.

[0021] The device I / F 51 is a connection circuit that connects devices and communicates data between devices in accordance with communication standards such as USB, IEEE 802.11, or Bluetooth (registered trademark). The system 100 transfers video data from the cameras 20a and 20b to the movement line analysis device 50 via the device I / F 51. The device I / F 51 may also constitute an input unit that inputs user operations from an external device.

[0022] The operation unit 52 is configured by, for example, any one of a keyboard, a mouse, a trackpad, a touch panel, etc., or a combination thereof. The operation unit 52 acquires various information input by user operations. The operation unit 52 is an example of an input unit of the movement line analysis device 50 in this embodiment.

[0023] The display unit 53 is configured by, for example, a liquid crystal display or an organic EL display. The display unit 53 may display various information such as various icons for operating the operation unit 52 and information input from the operation unit 52. Various display devices such as a projector and a head-mounted display may be used for the display unit 53. Furthermore, for example, when an external display device is used, the display unit 53 of the movement line analysis device 50 may be an output interface circuit for a video signal or the like that complies with, for example, the HDMI (registered trademark) standard.

[0024] The control unit 54 is configured, for example, by a CPU or MPU that works in cooperation with software to realize predetermined functions. The control unit 54 reads data and programs stored in the storage unit 57 and performs various arithmetic processing to realize various functions. For example, the control unit 54 executes a program including a group of instructions for realizing various processes of the movement line analysis device 50. The above program is, for example, an application program, and may be provided via a communication network such as the Internet, or may be stored in a portable storage medium.

[0025] 2, the control unit 54 executes, for example, a flow line generation process 55 and a flow line analysis process 56 as functional components. The flow line generation process 55 is a process of applying image recognition technology to video data D0 stored in the storage unit 57, for example, to generate flow line data D1 and output the data to the storage unit 57. The flow line analysis process 56 is a process of filtering and displaying information such as flow lines based on the flow line data D1 in response to user operations on the operation unit 52. The flow line analysis process 56 will be described in detail later.

[0026] The control unit 54 may be a dedicated electronic circuit designed to realize a predetermined function or a hardware circuit such as a reconfigurable electronic circuit. The control unit 54 may be configured with a semiconductor integrated circuit such as a CPU, MPU, GPU, GPGPU, TPU, microcomputer, DSP, FPGA, or ASIC. The control unit 54 may also include an internal memory as a temporary storage area for storing various data and programs.

[0027] The storage unit 57 is a recording medium for storing data and programs, and is configured, for example, by a hard disk drive (HDD), an optical disk drive, or a semiconductor storage device (SSD). The storage unit 57 stores various data, such as video data D0, flow line data D1, area data D2, and flow line connection parameter information D3. The storage unit 57 may include a temporary storage element configured, for example, by a RAM such as a DRAM or an SRAM, and may function as a work area for the control unit 54. It should be noted that it is not essential for the flow line analysis device 50 to have the operation unit 52 and the display unit 53, as long as the processing described below can be realized.

[0028] Here, the moving image data D0 and the flow line data D1 will be explained while explaining the flow line generation process 55. As described above, the moving image data D0 is a collection of data of frame images having a predetermined frame rate.

[0029] 3A to 3C show how a worker image 31 moves in frame images 30, 32, and 34. The control unit 54 is able to recognize the worker image 31 in each frame image by utilizing a trained model in which machine learning has been performed on the shape of a moving object in the image, in this example, a person, using a known machine learning algorithm. Such machine learning can be performed, for example, by using various images of people and frames surrounding the images of people as training data.

[0030] For example, the control unit 54 calculates the position of the center of gravity G0 of the image 31 of a person present in the frame image 30 shown in FIG. 3A as the representative point of that image 31. Similarly, for the frame image 32 of FIG. 3B and the frame image 34 of FIG. 3C, the control unit 54 calculates the positions of the centers of gravity G2 and G4 of the image 31 of the same person as the person shown in FIG. 3A, respectively. FIG. 3D shows an example of a traffic line L generated by connecting the centers of gravity G0, G2, and G4. The traffic line generation process 55 generates the traffic line L using the above-described procedure. Note that while only the positions of the centers of gravity are mentioned in the examples of FIGS. 3A to 3C, information on the time at which each frame image was captured is also acquired along with the position of each center of gravity. In other words, the traffic line L indicates not only changes in the position of the center of gravity but also changes over time. The control unit 54 assigns an identifier ID to the generated traffic line to uniquely identify that traffic line. The example of FIG. 3D shows that the control unit 54 assigns ID=k.

[0031] If multiple people are present in a frame image, a movement line is calculated for each person. Therefore, the movement line data D1 is a collection of movement line L data acquired from images of the same person moving across multiple frame images. FIG. 4 shows an example of a movement line La generated by focusing on a certain person. Note that the movement line La in FIG. 4 indicates the movement line of the worker 11 moving within the environment 200 shown in FIG. 1. For reference, the positions of the cameras 20a and 20b and the positions of the pillars P1 and P2 are also shown in FIG. 4.

[0032] 1-3. Reasons for interruptions in traffic flow In the example shown in FIG. 4, the flow line La is drawn as a single line. Ideally, the control unit 54 of the flow line analysis device 50 would generate a single, uninterrupted, correct flow line. However, in reality, what should be a single flow line may be generated as two or more partial flow lines. Below, with reference to FIGS. 5 to 8D, two examples will be described in which a flow line is detected as being interrupted due to the presence of a blind spot that is not within the camera's field of view.

[0033] First, a first example will be described in which a line of movement is detected as being interrupted by a blind spot. FIG. 5 shows the relationship between the field of view 22a of camera 20a and the field of view 22b of camera 20b. Each field of view 22a and 22b corresponds to the range of an image captured as a frame image by each camera 20a and 20b. As shown in the figure, a blind spot area 24 exists in the environment 200, which is a blind spot that is not included in either field of view 22a or 22b. Such a blind spot area 24 can occur when setting the capture ranges of multiple cameras. For ease of understanding, FIG. 5 illustrates the position of the blind spot area 24 as a clearly defined rectangular area, but in reality, the shape and size may vary as appropriate.

[0034] If a moving object is in blind spot area 24, the moving object will not be photographed by both cameras 20a and 20b. Therefore, when the moving object moves from field of view 22a or field of view 22b into blind spot area 24, the flow line that had been detected up to that point is considered to have ended and is interrupted. Then, as the moving object continues to move, if it enters field of view 22b or field of view 22a from blind spot area 24, the flow line is detected at that point.

[0035] 6(a) to 6(d) show the time-series changes in the position of worker 11 as he moves across fields of view 22a and 22b. As shown in FIGS. 6(a) and 6(b), worker 11 moves across field of view 22a from the left to the right of the drawing. Until he approaches blind spot area 24, worker 11 is only photographed by camera 20a.

[0036] Thereafter, worker 11 leaves field of view 22a and enters blind spot area 24, continuing to move. At this time, worker 11 disappears from field of view 22a of camera 20a. Eventually, as shown in (c) of FIG. 6, worker 11 leaves blind spot area 24 and enters field of view 22b of camera 20b, and is photographed by camera 20b. Thereafter, as shown in (d) of FIG. 6, worker 11 is photographed by camera 20b as he moves.

[0037] When the video data D is obtained by the cameras 20a and 20b, the control unit 54 uses the frame images taken by the cameras 20a and 20b to recognize the image of the worker 11 and independently generate a line of movement.

[0038] FIG. 7 shows a flow line L0 (ID: 0) generated from the video data of camera 20a and a flow line L1 (ID: 1) generated from the video data of camera 20b. Because the flow lines L0 and L1 represent the flow lines of the same worker 11, the control unit 54 should assign the same ID and treat them as the same flow line. However, the control unit 54 assigns different IDs to the flow lines L0 and L1. This is because the control unit 54 generated the flow lines L0 and L1 from video data captured by different cameras 20a and 20b. Note that the flow line L0 ends at time T0 and coordinates (x0, y0) when the worker 11 leaves the field of view 22a of camera 20a and enters the blind spot 24. The flow line L1 starts at time T1 and coordinates (x1, y1) when the worker 11 leaves the blind spot 24 and enters the field of view 22b of camera 20b. In this way, when there is a blind spot that is not included in the field of view of multiple cameras, the movement lines of the same worker are treated as different movement lines. Note that in Figure 7, the time T0 and the coordinates (x0, y0) are collectively expressed as (T0, x0, y0). This will be used in the rest of this specification.

[0039] Next, a second example will be described in which a movement line is detected as being interrupted due to a blind spot. In the second example, an obstruction is present in part of the camera's field of view, and the movement line is interrupted because the time and position when the worker enters the shadow of the obstruction are physically separated from the time and position when the worker emerges from the shadow of the obstruction. 8A to 8C respectively show the change in position of worker 11 before entering the blind spot of the camera, while in the blind spot, and after leaving the blind spot. The blind spot corresponds to a position where the field of view 22a of camera 20a is blocked by, for example, a pillar P1.

[0040] As shown in Fig. 8A, before the worker 11 enters the blind spot, the control unit 54 can generate a series of movement lines L2 that represent changes in the position of the worker 11. However, as shown in Fig. 8B, when the worker 11 disappears from the frame image, the control unit 54 ends generation of the movement line L2 at that point and assigns "ID: 2" to the movement line L2.

[0041] If the worker 11 continues to move in the blind spot where the field of view is obscured by the pillar P1, the worker 11 will again appear in the camera's field of view as shown in FIG. 8C. That is, the image of the worker 11 will appear on the frame image. At this point, the control unit 54 recognizes the image of the worker 11 on the frame image as the image of a new moving object, and starts generating a different flow line L3. When the flow line is completed, the control unit 54 assigns it "ID:3".

[0042] 8D shows an example in which the movement lines of the same worker 11 are generated as different movement lines L2 and L3. The end point (x2, y2) of movement line L2 at time T2 and the viewpoint (x3, y3) of movement line L3 at time T3 are separated by a distance D equivalent to the blind spot. In this way, when a moving object enters the blind spot of the camera, the movement lines before and after entering the blind spot are treated as different movement lines, even if they are the movement lines of the same worker.

[0043] The two cases where a movement line is detected as being interrupted can occur independently. Therefore, if a moving object enters a camera's blind spot and moves across the field of view of multiple cameras, multiple movement lines can be generated independently.

[0044] FIG. 9 shows exemplary flow line data D1 defining multiple flow lines, including four independently generated flow lines with IDs 0 to 3. Multiple entries with the same ID represent points that make up the flow line with that ID. The first and last points with that ID are the start and end points, respectively, of the flow line with that ID. One or more points with the same ID that exist between them are waypoints between the start point and the end point. Connecting the start point, one or more waypoints, and the end point forms the flow line with that ID. For example, in the case of a flow line with ID 0, the start point is acquired at time 0 and its position is (10,20), and the end point is acquired at time 10 and its position is (25,40). As an example of one or more waypoints, FIG. 9 shows the position (9,22) acquired at time 1. Entries in the flow line data D1 are added one after another as time passes, and no entries with the same ID occur after the last point of each ID.

[0045] In this specification, the flow line data generated in this way is called "partial flow line data." Partial flow line data means data on flow lines that can constitute one flow line. When such partial flow lines are connected to each other, one flow line is formed. The process of connecting flow lines corresponds to the "flow line analysis process 56" shown in Figure 2.

[0046] The flow line analysis process 56 analyzes partial flow lines and automatically sets connection parameter information that indicates the connection conditions for connecting partial flow lines. Examples of connection parameter information include the time (connection time) allowed from when a partial flow line ends until it connects to the next partial flow line, and the distance (connection distance) allowed from when a partial flow line ends until it connects to the next partial flow line. In other words, the connection parameter information is the time (connection time) from the end time of a partial flow line to the start time of the next partial flow line, and the distance (connection distance) allowed from the end position of a partial flow line to the start position of the next partial flow line.

[0047] For example, FIG. 10 shows an example in which partial flow lines L2 and L3 are connected via an interpolated flow line L23. Let "Tth" be the threshold for connection time, "Dth" be the threshold for connection distance, P be the position (x21, y21), and Q be the position (x30, y30). The end time of partial flow line L2 and the start time of partial flow line L3 are T21 and T30, respectively. Time T30 is later in time than time T21, and is represented by a larger numerical value. In this case, if T30-T21≦Tth and / or |QP|≦Dth, partial flow lines L2 and L3 are connected. The thresholds Tth and Dth that define the connection conditions at this time are connection parameter information. The connection distance and connection time thresholds indicate the threshold or tolerance for the physical separation between two partial traffic lines or the temporal separation between the times at which each of the two partial traffic lines was acquired, and can be determined based on data on multiple partial traffic lines previously collected in each partial area. Note that the "physical separation between two partial traffic lines" here can be understood as either the two partial traffic lines detected in the first example or the two partial traffic lines detected in the second example. In other words, the "physical separation between two partial traffic lines" can be defined in cases where a single traffic line is detected as two partial traffic lines due to the presence of a blind spot that is not captured by any of the multiple cameras, or in cases where a single traffic line is detected as two partial traffic lines due to a worker being hidden behind an obstruction within the field of view.

[0048] The speed and direction of movement of a moving object may be used as connection parameter information. For example, if there are a relatively large number of installations that obstruct movement, the speed of movement may naturally be limited. More specifically, it is physically and empirically impossible for a partial flow line of a worker moving on foot to connect with another partial flow line 2 meters ahead in 0.1 seconds. In order to eliminate such impossible movements, it is useful to use the speed of movement as connection parameter information. Furthermore, if the movement direction can only be considered in a straight line in relation to the installations, movement directions that draw an arc can be eliminated. In this way, connection parameter information may be prepared for each partial area that is part of the environment 200.

[0049] The inventor further proposes that such connection parameter information is not uniformly set for the environment 200, but is set for each partial region constituting the environment 200. This is because the environment 200 may have various blind spots, and the position, size, etc. of the common region 24 where the fields of view of multiple cameras overlap may vary. Setting connection parameter information for each location where a break occurs, i.e., for each partial region, is believed to improve the accuracy of connecting multiple partial flow lines into a single flow line. For example, in a factory, the work content often differs depending on the work location. By setting connection parameter information for each work location, which has different tendencies in the movement and breaks of moving objects, as an individual partial region, it is possible to improve the accuracy of connecting multiple partial flow lines.

[0050] If two partial flow lines satisfy the connection conditions indicated by the connection parameter information, the two partial flow lines are determined to be a pair of partial flow lines that are connected to each other. Note that in reality, partial flow lines of the same moving object are required to be connected correctly, but in this specification, if the connection parameter information is satisfied, the two partial flow lines are treated as being connected correctly.

[0051] 2. Operation The operation of the system 100 and the traffic flow analysis device 50 configured as above will be described below.

[0052] 2-1. Overview of operation FIG. 11 is a diagram showing the flow of partial flow line connection processing performed by the flow line analysis device 50. First, each of the multiple cameras 20a-20n, each having a sensor such as an image sensor, generates video data D0. As described above, a "sensor" refers to a device capable of detecting the position of a moving object. The control unit 54 of the movement line analysis device 50 extracts movement lines from the movement line data generated by each camera. At this time, partial movement lines are generated for the reasons described above. In this embodiment, the movement lines extracted from the movement line data D0 are considered to be partial movement lines. That is, the control unit 54 generates partial movement line data D1 and stores it in the storage unit 57.

[0053] The control unit 54 performs analysis and connection processing of the partial flow lines. At this time, flow line connection parameter information D3 is used. The flow line connection parameter information D3 is set for each of a plurality of partial areas obtained by dividing the entire area of ​​the environment 200 prepared in advance in the area data D2. The setting is made based on partial flow line data previously collected for each partial area. The control unit 54 connects the partial flow lines in accordance with the flow line connection parameter information D3 and updates the flow line data D1.

[0054] 2-2.Details of operation The operation of the traffic line analysis device 50 will be described in detail below. First, a method for setting flow line connection parameter information to be applied to a certain area will be described.

[0055] 2-2-1. First setting method for flow line connection parameter information FIG. 12 shows the relationship between the connection distance and connection time between two partial flow lines when the connection conditions for the two partial flow lines are changed. The horizontal axis represents the connection distance D, and the vertical axis represents the connection time T. Referring back to FIG. 10, the following explanation will be given. The two partial flow lines L2 and L3 are separated by a distance D and a time T. The two partial flow lines L2 and L3 will not connect unless the connection conditions related to distance or time, i.e., T30-T21≦Tth and / or |QP|≦Dth, are satisfied. However, as the thresholds Dth and Tth are increased, the thresholds Dth and Tth eventually exceed the distance D and the time T, respectively. As a result, the connection between the two partial flow lines L2 and L3 is permitted. The pair of distance D and time T when the two partial flow lines L2 and L3 are connected can be depicted as a single point, such as point 40, on the DT graph in FIG. 12.

[0056] As described above, by gradually increasing the threshold Dth or Tth, which is the connection condition, the number of combinations of connected partial flow lines increases. Therefore, the number of points on the DT graph also increases. Eventually, when the threshold Dth or Tth becomes very large, all partial flow lines will be connected. However, among these connections between partial flow lines, there will be combinations that are clearly incorrect.

[0057] Of the countless points drawn in Figure 12, we will focus on point 42. Point 42 shows the relationship between two partial flow lines that were finally connected under very relaxed connection conditions. Specifically, it shows that there is a considerable gap in the acquisition times of the two partial flow lines.

[0058] Such a connection between two partial flow lines is likely to be an incorrect connection. Therefore, the present inventors considered it appropriate to set connection conditions to surround a point cloud where two partial flow lines are considered to be correctly connected (hereinafter referred to as the "correct point cloud"). Specifically, the present inventors set the connection conditions using a range 44 surrounding the correct point cloud. Assuming that the range 44 is a rectangle, the connection conditions are set as follows: a connection distance threshold Dth, which is the maximum value in the D-axis direction, and a connection time threshold Tth, which is the maximum value in the T-axis direction. For example, in the environment 200, the threshold Dth is 1.5 m, and the threshold Tth is 2 seconds. By setting the connection conditions in this manner, the control unit 54 of the flow line analysis device 50 can determine two partial flow lines that fall within the range of the connection conditions as a pair of mutually connected partial flow lines.

[0059] Note that the distance between partial flow lines is not limited to a distance defined within the environment 200, i.e., a distance measured in real space. For example, the distance between partial flow lines may be defined by the number of pixels between the partial flow lines on the video. In this specification, the distance may be expressed as the "physical distance between two partial flow lines" since it includes both the distance defined within the environment 200 and the distance defined on the video. Furthermore, the time difference between when each partial flow line was acquired may be expressed as the "temporal distance between the times when each of the two partial flow lines was acquired."

[0060] The range 44 including the correct point cloud may be various shapes other than a rectangle. For example, Fig. 13 shows an example of a range 46 that surrounds 90% of the correct point cloud. The range 46 may be, for example, an ellipse, and for the distance D and the time T, allowable ranges Dra and Tra for the distance and time within which a connection is allowed are defined. It can be seen that the range 44 shown in Fig. 12 includes a relatively large number of areas where no points exist, whereas the range 46 includes significantly fewer areas where no points exist.

[0061] Here is another example of how to determine the range. First, the center of gravity of the point cloud is found, and the area within a predetermined distance from that center may be determined as the acceptable range. For example, the "predetermined distance" may be defined using Euclidean distance or Mahalanobis distance, and the closure that contains 90% of the correct point cloud may be determined as the acceptable range.

[0062] 2-2-2. Second setting method for flow line connection parameter information FIG. 14 is a diagram illustrating a method for determining flow line connection parameter information while changing the connection conditions of two partial flow lines. The vertical and horizontal axes are the same as those in FIGS. 12 and 13. As in FIGS. 12 and 13, each black dot in the graphs of (a) to (d) in FIG. 14 indicates the relationship between the connection distance and connection time between two partial flow lines when they are connected. However, the black dots represent pairs of partial flow lines that are disconnected and not connected because the connection conditions are not met. On the other hand, the white dots indicate the range of two partial flow lines that are connected because the connection conditions are met when the connection conditions regarding time are changed.

[0063] FIG. 14(a) shows an example where the threshold value Tth for the time-related connection condition is set to 0 seconds. Since the time-related connection condition is never satisfied, there are no pairs of partial flow lines that are actually connected. Therefore, there are no white dots. In this case, it is assumed that there are 100 black dots, for example.

[0064] Figure 14(b) shows an example where the threshold value Tth for the time-related connection condition is set to 1 second. Pairs that satisfy the time-related connection condition appear and are shown as white dots. Pairs that are still not connected are shown as black dots, and there are, for example, 50 of them.

[0065] Figure 14(c) shows an example where the threshold value Tth for the time-related connection condition is set to 2 seconds. The number of pairs that satisfy the time-related connection condition increases, and the number of white dots also increases. Unconnected pairs remain as black dots, and there are, for example, 20 of them.

[0066] In this way, the connection time can be increased until the number of black dots representing the number of interruptions falls below a certain level, for example, until it is 5% or less of the connection time of 0 seconds (a), and the connection time when it falls below 5% can be set as the threshold value Tth ((d) in Figure 14). In a similar manner, the distance threshold Dth can also be set.

[0067] 2-2-3. Setting partial areas Next, a method for dividing the entire area of ​​the environment 200 into a plurality of partial areas and setting them will be described. As explained above, a single flow path is interrupted for various reasons and split into multiple partial flow paths. The tendency for flow path interruptions to occur varies for each partial region depending on the presence or absence of obstructions, differences in movement patterns, and the like. For example, some partial regions are often interrupted because they are hidden in the shadows of other moving objects, while other partial regions are easily hidden by obstructions such as pillars. It is considered appropriate to determine connection conditions for connecting interrupted flow paths in accordance with such circumstances. In other words, if the same connection conditions are set uniformly throughout the entire environment 200, they may be appropriate for some parts but not for other parts, and therefore may not necessarily be appropriate connection conditions for the entire region of the environment 200.

[0068] Therefore, the inventors divided the entire environment 200 into multiple partial regions and set parameters appropriate for each partial region, thereby improving the accuracy of connecting multiple partial flow lines into a single flow line. In order to divide the entire area of ​​the environment 200 into multiple partial areas, we first introduce the concept of a "unit area." A "unit area" may be, for example, one pixel on an image that includes part or all of the environment 200, or M pixels by N pixels (M and N are natural numbers). It can be said that an image of the environment 200 is composed of multiple unit areas. Each of the multiple partial areas that are set can be said to be a collection of one or more unit areas. In the following, a unit area is defined as one pixel.

[0069] The tendency for traffic lines to be interrupted varies for each partial area. This tendency can be determined based on connection conditions related to the time and / or distance required for multiple partial traffic lines to connect with each other. Therefore, the control unit 54 of the traffic line analysis device 50 extracts partial traffic lines from the video data D0 acquired on a trial basis for setting the partial areas, sets the same connection conditions for the entire environment 200, and calculates the number and frequency of interrupted partial traffic lines, the connection time between interrupted partial traffic lines, the connection distance, the movement direction, and / or speed for each unit area.

[0070] For example, the number of interrupted partial traffic lines refers to the number of times that traffic lines have been interrupted in the unit area in the past. Specifically, the number of partial traffic lines that have ended within the unit area can be counted and calculated as the number of interrupted partial traffic lines. The method of counting the number of interrupted partial traffic lines is one example, and the number of partial traffic lines may be counted using other methods. Furthermore, with regard to the movement direction, for example, if an obstruction is present in the unit area, the way in which a moving body enters the camera's blind spot is thought to be different depending on whether the moving body is moving toward the obstruction or not, and this can be an indicator of the tendency for interruptions to occur. In this specification, these calculated values ​​are referred to as "feature values." The calculated feature values ​​represent the characteristics of each unit area calculated from past partial flow lines. The feature value does not have to be selected from the number of interrupted partial flow lines, the number of times, the connection time between interrupted partial flow lines, the connection distance, the movement direction, and the speed, but rather multiple values ​​may be selected from them.

[0071] In this embodiment, an edge detection technique in image processing is applied to set partial regions based on the obtained feature values ​​of each unit region. The edge detection technique in image processing is a technique that detects a pixel as the edge of an object in an image when the difference in pixel value between two adjacent pixels, that is, the grayscale value or RGB value of each pixel, is equal to or greater than a predetermined threshold.

[0072] In this embodiment, the control unit 54 calculates the feature value for each unit region using the above-mentioned procedure, and classifies the two adjacent unit regions into different partial regions if the difference between the feature values ​​of two adjacent unit regions is equal to or greater than a predetermined threshold. Below, an example will be described in which only the number of discontinuities is used as the feature value of a unit region.

[0073] FIG. 15 shows a partial region 60 determined from the feature values ​​of each partial region using edge detection technology. A blind spot of the camera 20a caused by an obstruction P1 in the environment 200 increases the number of discontinuities and the feature values. As a result, the difference between adjacent pixels increases and is detected as an edge. The thick lines defining the partial region 60 correspond to the edges. Alternatively, instead of the straight lines defining each unit region 60, the edges may be defined by curves 62 passing through each unit region 60.

[0074] 16 shows a plurality of partial regions 60a, 60b, 60c, etc. that are set for the entire video of environment 200. Each of the set partial regions 60a, 60b, 60c, etc. has a feature value that differs from the other adjacent partial regions. Because the feature value is related to the connection condition, by individually setting the connection condition for each partial region according to factors such as the likelihood of discontinuity, it becomes possible to accurately connect a plurality of partial traffic lines together as a single traffic line.

[0075] FIG. 17 shows an example of two ranges 62a and 62b representing connection conditions set for different partial areas 60a and 60b, respectively, and a range 64 representing connection conditions when no partial areas are set and the connection conditions are set for the entire environment 200. If range 64 is set, for example, a pair of partial traffic lines included in circular area 66 will be allowed to connect as a single traffic line, which may reduce the connection accuracy of the partial traffic lines. On the other hand, because connection conditions reflecting the feature values ​​of each partial area are set for ranges 62a and 62b, a pair of partial traffic lines included in circular area 66 will not be connected as a single traffic line. This makes it possible to improve the connection accuracy of multiple partial traffic lines.

[0076] 18 is a flowchart showing steps S11 to S13 for dividing the entire area into a plurality of partial areas and setting connection conditions, and steps S21 to S24 for applying the set connection conditions to each partial area and connecting a plurality of partial flow lines into a single flow line. This process is executed by the control unit 54 of the flow line analysis device 50.

[0077] The control unit 54 acquires overall area data of the environment (S11). The control unit 54 divides the overall area into multiple partial areas based on past partial flow lines, and generates area data D2 (S12). The control unit 54 sets / updates connection conditions for each of the multiple partial areas, and generates flow line connection parameter information D3 (S13).

[0078] The control unit 54 acquires sensor data S0 (S21). An example of the sensor data S0 is video data D0. The control unit 54 extracts partial flow lines from the sensor data S0 (S22). The control unit 54 reads the area data D2 generated in steps S12 and S13 and the flow line connection parameter information D3 (S23). The control unit 54 connects two partial flow lines into one flow line based on connection conditions according to the partial area (S24).

[0079] If a shelf in environment 200 is moved and the blind spots of cameras 20a and 20b change, it is preferable to reset multiple partial areas for environment 200 after the shelf is moved. This is because the situation in which a single flow line is detected as a partial flow line may change. Therefore, steps S11 to S13 can be preferably performed periodically or when a predetermined change occurs in environment 200.

[0080] Note that steps S11 to S13 and steps S21 to S24 do not always need to be executed by a single movement line analysis device 50. For example, steps S11 to S13 may be executed by another device, which may generate area data D2 defining multiple partial areas and movement line connection parameter information D3 indicating connection conditions to be applied to each partial area. The control unit 54 of the movement line analysis device 50 may receive the area data D2 and the movement line connection parameter information D3 from the other device and perform connection processing of the partial movement lines.

[0081] The above processing provides a flow path connection technology that connects interruptions in flow paths caused by erroneous object recognition results or the presence of obstructions to the correct flow path when extracting an object's flow path using a sensor device such as a camera. Conventional technologies require manual setting of parameters necessary for flow path connection, which requires analysis of the causes and trends of interruptions and trial and error in parameter setting. Considering that flow path interruptions tendencies vary depending on the presence or absence of obstructions and the object's movement patterns, manual setting is a very time-consuming task. Furthermore, setting connection conditions uniformly across the entire system poses a problem in that it is difficult to improve the accuracy of connecting multiple partial flow paths into a single flow path.

[0082] According to the technology disclosed herein, instead of using the same parameters for the entire environment from which traffic lines are extracted, the entire area is subdivided into multiple partial areas according to individual trends, and individual connection conditions are set for each partial area, thereby improving the accuracy of connecting multiple partial traffic lines. According to the technology disclosed herein, connection conditions calculated from past partial traffic lines can be automatically set for each partial area, thereby improving the accuracy of connecting partial traffic lines without manual work. [Industrial Applicability]

[0083] The present disclosure is applicable to a flow line analysis device and method for analyzing the flow line of a moving object and connecting a plurality of partial flow lines that are disconnected from one flow line into a single flow line.

Claims

1. A flow line analysis device that extracts a flow line of a moving object by utilizing detection results of one or more sensors installed in an environment where the moving object exists, a storage unit that stores partial flow line data indicating a plurality of partial flow lines extracted in advance from the detection results, connection parameter information indicating connection conditions for connecting the partial flow lines, and partial area data that defines each of a plurality of partial areas; a control unit that connects two partial flow lines that belong to the same partial area based on the connection parameter information and extracts the flow line of the moving object for each partial area; Equipped with The connection parameter information is set for each partial region.

2. The flow line analysis device according to claim 1 , wherein the control unit sets connection parameter information for each partial area based on partial flow line data indicating a plurality of partial flow lines previously collected in each partial area.

3. As a process of setting connection parameter information of a predetermined partial region among the plurality of partial regions, the control unit determining a set of mutually connected partial flow lines while changing connection conditions using the past partial flow line data collected in the predetermined partial area; A connection condition that satisfies a predetermined threshold or tolerance range is determined as connection parameter information of the predetermined partial region. The traffic line analysis device according to claim 2 .

4. the connection condition is a maximum value of a physical distance between two partial flow lines or a time distance between two partial flow lines at which the respective partial flow lines are acquired; The control unit determines two partial flow lines that fall within the range of the connection condition as a pair of partial flow lines that are connected to each other. The traffic line analysis device according to claim 3 .

5. The traffic line analysis device according to claim 4 , wherein the control unit determines a predetermined magnitude of the physical distance and / or the temporal distance as the threshold value.

6. The traffic line analysis device according to claim 4 , wherein the control unit determines a predetermined range of the physical distance and / or the temporal distance as the allowable range.

7. The traffic path analysis device according to claim 5 or 6, wherein the physical distance is a distance defined in the environment.

8. Each partial flow line is assigned time information indicating the time at which it was acquired. the two partial flow lines include a first partial flow line acquired at a relatively early time and a second partial flow line acquired at a relatively late time, The traffic flow analysis device according to claim 7 , wherein the distance defined in the environment is a distance along a straight line or a curve connecting an end position of the first partial traffic flow and a start position of the second partial traffic flow.

9. the one or more sensors are one or more imaging devices installed in the environment and configured to capture images of the environment; The traffic line analysis device according to claim 5 or 6, wherein the physical distance is a distance defined on the video.

10. Each partial flow line is assigned time information indicating the time at which it was acquired. The traffic flow analysis device according to claim 5 , wherein the time interval is a difference between times when the two partial traffic flows are acquired.

11. the area representing the environment is composed of a plurality of unit areas, each of the plurality of partial regions is a collection of one or a plurality of unit regions; The control unit Calculating a feature value of each unit area from the partial flow line data; 11. The traffic path analysis device according to claim 1, wherein when a change amount between feature values ​​of two adjacent unit areas is equal to or greater than a predetermined threshold, the two adjacent unit areas are classified as different partial areas.

12. The flow path analysis device according to claim 11, wherein the control unit determines the feature value based on one or more selected from the number of times the flow path has been interrupted in the unit area in the past, the size of the physical or temporal distance required to connect the interrupted partial flow paths, and the direction of movement of the moving object.

13. A flow line analysis method for extracting a flow line of a moving object from detection results of one or more sensors installed in an environment where the moving object exists, using a flow line analysis system, comprising: The movement line analysis device of the movement line analysis system includes a control unit and a storage unit that stores partial movement line data indicating a plurality of partial movement lines extracted in advance from the detection result, connection parameter information indicating connection conditions for connecting the partial movement lines, and partial area data that defines each of a plurality of partial areas, The control unit a process of reading out partial flow line data, the connection parameter information, and the partial area data from the storage unit; a process of setting the connection parameter information for each partial area; a process of connecting two partial flow lines belonging to the same partial area based on the connection parameter information and extracting the flow line of the moving object for each partial area; A flow analysis method that performs the following.

14. A computer program for causing a control unit of a computer system to execute the flow line analysis method according to claim 13.

Images