Object movement measuring device, system, method, and program

Abstract

To provide an object movement measurement apparatus, or the like, which can contribute to high-speed data processing to easily measure movement of an object in an arbitrary place.SOLUTION: An object movement measurement apparatus is configured to: detect a plane from three-dimensional point cloud data to determine a reference plane; extract a point cloud of an object from the three-dimensional point cloud data to perform two-dimensional projection on a plane parallel to the reference plane; expand pixels of the two-dimensionally projected pixel group to label adjacent contact pixels with the same label; convert the pixel group three-dimensionally to generate point clouds labeled with the same label; detect an object related to a point cloud that meets a detection condition out of the labeled point clouds; determine the identity with a past object and the type for the latest object; add common identification information to the object which is identical with and has the same type as the past object; specify a moving direction of the object with the identification information; and count the number of moving objects with a count point, on the basis of the specified moving direction of the object.SELECTED DRAWING: Figure 9

Description

[Technical field]

[0001] The present invention relates to an object movement measuring device, a system, a method, and a program. [Background technology]

[0002] There is a demand in various fields such as marketing and location surveys in the real estate industry to grasp the number of people in a certain indoor or outdoor space, and to grasp and investigate the flow rate of people there, that is, the direction of movement (for example, how many people moved to the right and how many moved to the left).As a technology that meets such a demand, there is a technology that uses machine learning to analyze video data (2D data, video, still images) from a surveillance camera to detect the direction of movement of an object (for example, see Patent Documents 1 and 2). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-86467 [Patent Document 2] International Publication No. 2010 / 098024 [Patent Document 3] JP 2019-207653 A Summary of the Invention [Problem to be solved by the invention]

[0004] The following analysis is provided by the present inventors.

[0005] However, in the technology described in Patent Documents 1 and 2, the learning data must be acquired in an environment equivalent to the installation position and angle in which the device will actually be used; otherwise, the learning results will differ and cause malfunctions.

[0006] Furthermore, human flow surveys are often conducted spot for a certain period of time rather than being conducted continuously. However, with technologies such as those disclosed in Patent Documents 1 and 2, it is difficult to adjust the surveillance cameras and their installation positions and angles to match the machine learning, and it is necessary to start from scratch learning again to match the positions and angles at which the surveillance cameras can be installed, which is wasteful.

[0007] In view of this, it is conceivable to use a technique for detecting the moving direction of an object by analyzing three-dimensional point cloud data from a three-dimensional sensor without performing machine learning (see, for example, Patent Document 3).

[0008] However, as in Patent Document 3, a technique for directly connecting nearby point clouds from 3D point cloud data to find a certain amount or more of clusters (groups) (i.e., clustering or labeling a plurality of point clouds capturing people, etc.) places a large calculation load on the computer and requires a significant processing time. In other words, since 3D point cloud data requires more dimensions for calculating distance than 2D data, processing can be simplified using a simple tree structure when processing 2D data, but when processing 3D point cloud data, the structure becomes complicated, increasing the processing load and processing time.

[0009] In addition, when detecting the movement of an object, information on how the object moved must be pieced together between frames (moments) recognized by a computer, but in a technology such as that in Patent Document 3, the processing load and processing time can affect the number of frames that the computer can recognize, which can impede the process of pieced together information between frames and make it difficult to perform accurate measurement. Therefore, even if a person is captured in a space at a certain moment, by the time the next moment is processed, that person may have already walked away, and it may be impossible to obtain information on which direction the person moved in, which is important for grasping the flow rate.

[0010] A main object of the present invention is to provide an object movement measuring device, system, method, and program that can contribute to increasing the speed of data processing and easily measuring the movement of an object at any location. [Means for solving the problem]

[0011] The object movement measuring device according to the first viewpoint includes a reference plane determination unit configured to detect one or more planes from 3D point cloud data of a shooting site photographed by a 3D sensor and determine a reference plane from the detected planes, an object extraction unit configured to extract a point cloud of an object from the 3D point cloud data, a 2D projection unit configured to create a pixel group by 2D projecting the point cloud of the extracted object onto a plane parallel to the reference plane with each point being one pixel, an expansion unit configured to create a pixel group by expanding each pixel of the 2D projected pixel group so that adjacent pixels are in contact with each other, a labeling unit configured to create a pixel group by labeling pixels of the expanded pixel group where adjacent pixels are in contact with each other with the same label, and a pixel group obtained by 2D projection. the labeling processing unit is configured to convert the labeled pixel group into three dimensions based on a relationship with a group, and create a point cloud labeled with the same label; a matching unit configured to detect an object associated with the point cloud that satisfies a detection condition from the point cloud converted into three dimensions and labeled with the same label; an identification information assignment unit configured to determine the identity and type of the most recently detected object with respect to a previously detected object, and to assign common identification information to objects that are identical and of the same type as the previously detected object; a movement direction identification unit configured to identify the movement direction of the object to which the identification information has been assigned; and a counting unit configured to count the number of objects moving at a count point based on the movement direction of the identified object.

[0012] The object movement measuring system relating to the second viewpoint includes an object movement measuring device relating to the first viewpoint, and a three-dimensional sensor communicably connected to the object movement measuring device and configured to capture an image of a shooting site.

[0013] An object movement measurement method according to a third aspect is a method for measuring object movement using hardware resources, comprising the steps of: detecting one or more planes from 3D point cloud data of a shooting site photographed by a 3D sensor, and determining a reference plane from the detected planes; extracting a point cloud of an object from the 3D point cloud data; creating a pixel group by two-dimensionally projecting the extracted point cloud of the object onto a plane parallel to the reference plane, with each point representing one pixel; creating a pixel group by expanding each pixel of the two-dimensionally projected pixel group so that adjacent pixels are in contact with each other; and creating a pixel group by labeling, with the same label, pixels of the expanded pixel group where adjacent pixels are in contact with each other. a step of converting the labeled pixel group into three dimensions based on the relationship between the point group and the pixel group when projected into two dimensions, and creating a point group labeled with the same labels; a step of detecting an object associated with the point group that meets a detection condition from the point group converted into three dimensions and labeled with the same labels; a step of determining the identity and type of the most recent detected object with respect to previously detected objects, and assigning common identification information to objects that are identical and of the same type as the previously detected object; a step of specifying the movement direction of the object to which the identification information has been assigned; and a step of counting the number of objects moving at a count point based on the movement direction of the specified object.

[0014] A program according to a fourth aspect is a program for causing hardware resources to execute a process of measuring the movement of an object, the program including a process of detecting one or more planes from 3D point cloud data of a shooting site photographed by a 3D sensor and determining a reference plane from the detected planes, a process of extracting a point cloud of an object from the 3D point cloud data, a process of creating a pixel group by two-dimensionally projecting the point cloud of the extracted object onto a plane parallel to the reference plane, with each point representing one pixel, a process of creating a pixel group by expanding each pixel of the two-dimensionally projected pixel group so that adjacent pixels are in contact with each other, a process of labeling pixels of the expanded pixel group where adjacent pixels are in contact with each other with the same label, and a process of creating a pixel group by labeling the pixels of the expanded pixel group by labeling the pixels with the same label, and The hardware resource is caused to execute the following processes: converting the labeled pixel group into three dimensions based on the relationship between the point group and the pixel group when projected onto the original surface, and creating a point group labeled with the same labels; detecting an object associated with the point group that meets detection conditions from the point group converted into three dimensions and labeled with the same labels; determining the identity and type of the most recent detected object with respect to previously detected objects, and assigning common identification information to objects that are identical and of the same type as the previously detected object; specifying the movement direction of the object to which the identification information has been assigned; and counting the number of objects moving at a count point based on the movement direction of the specified object.

[0015] The program can be recorded in a computer-readable storage medium. The storage medium can be a non-transient medium such as a semiconductor memory, a hard disk, a magnetic recording medium, or an optical recording medium. In addition, the present disclosure can be embodied as a computer program product. The program is input to the computer device from an input device or an external device via a communication interface, stored in a storage device, drives the processor according to a predetermined step or process, and can display the processing result, including an intermediate state as necessary, at each stage via a display device, or can communicate with the outside via the communication interface. For example, a computer device for this purpose typically includes a processor, a storage device, an input device, a communication interface, and a display device as necessary, which are connectable to each other via a bus. Effect of the Invention

[0016] According to the first to fourth aspects, it is possible to contribute to facilitating measurement of the movement of an object at any location by speeding up data processing. [Brief description of the drawings]

[0017] [Figure 1] 3 is an image diagram showing an example of detecting an object using the object movement measuring system according to the first embodiment. FIG. [Diagram 2] 1 is a block diagram showing a schematic configuration of an object movement measuring system according to a first embodiment. [Diagram 3] 4 is an image diagram showing a schematic diagram of processing from when a site is photographed to when an object detection processing image is created in the object movement measuring system according to the embodiment 1. FIG. [Figure 4] 4 is a conceptual diagram showing a schematic representation of processing from projecting a point cloud onto a reference plane to labeling in the object movement measuring system according to the first embodiment. FIG. [Diagram 5] 4 is an image diagram showing a schematic representation of an identification-information-added image when identification information is added to an object detected in the object movement measuring system according to the embodiment 1. FIG. [Figure 6] 6 is an image diagram showing a schematic representation of an image in which information relating to the moving direction of each object is added to an identification-information-added image after the object has moved from the state in FIG. 5 in the object movement measuring system according to the first embodiment. FIG. [Figure 7] 4 is a flowchart showing a schematic operation of an object movement measuring device of the object movement measuring system according to the first embodiment. [Figure 8] 5 is a flowchart showing in schematic detail an object detection process of the object movement measuring device of the object movement measuring system according to the first embodiment. [Figure 9] FIG. 11 is a block diagram showing a schematic configuration of an object movement measuring device according to a second embodiment. [Figure 10] FIG. 2 is a block diagram illustrating a schematic configuration of hardware resources. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] Hereinafter, the embodiments will be described with reference to the drawings. In addition, when drawing reference symbols are used in this application, they are intended to aid understanding only and are not intended to limit the present invention to the illustrated embodiment. In addition, the following embodiments are merely examples and do not limit the present invention. In addition, the connection lines between blocks in the drawings and the like referred to in the following description include both bidirectional and unidirectional. A unidirectional arrow is a schematic representation of the flow of a main signal (data) and does not exclude bidirectionality. Furthermore, although not explicitly shown in the circuit diagrams, block diagrams, internal configuration diagrams, connection diagrams, and the like shown in the present disclosure, an input port and an output port exist at the input end and output end of each connection line. The same is true for input / output interfaces. A program is executed via a computer device, which includes, for example, a processor, a storage device, an input device, a communication interface, and a display device as necessary, and which is configured to be able to communicate with an internal or external device (including a computer) via the communication interface, regardless of whether it is wired or wireless.

[0019] [Embodiment 1] The object movement measuring system according to the first embodiment will be described with reference to the drawings. FIG. 1 is an image diagram showing an example of detecting an object using the object movement measuring system according to the first embodiment. FIG. 2 is a block diagram showing a schematic configuration of the object movement measuring system according to the first embodiment. FIG. 3 is an image diagram showing a schematic process from taking a picture of a site to creating an object detection processing image in the object movement measuring system according to the first embodiment. FIG. 4 is an image diagram showing a schematic process from projecting a point cloud onto a reference plane to labeling in the object movement measuring system according to the first embodiment. FIG. 5 is an image diagram showing an image with identification information added when identification information is added to an object detected in the object movement measuring system according to the first embodiment. FIG. 6 is an image diagram showing an image with information related to the movement direction of each object added to the identification information added image after the object moves from the state of FIG. 5 in the object movement measuring system according to the first embodiment.

[0020] The object movement measuring system 1 is a system that measures the movement of objects (e.g., person 10, car 11, etc.) at a shooting site (see Figs. 1 to 3). The object movement measuring system 1 can be used for traffic volume surveys to determine the need for installing signals and signs, traffic volume surveys for real estate properties, especially store tenants, counting the number of people entering and leaving places where personal identification is unnecessary or prohibited or avoided, investigating the number of users of a specific floor or toilet in a workplace, grasping the traffic volume of drones, grasping the frequency of intrusion of heavy machinery with long arms such as cranes into a specific area, counting the number of people entering and leaving between a pasture and a barn for livestock, investigating the direction of travel at a fork in a mountain trail, etc. The object movement measuring system 1 includes an object movement measuring device 200 and a three-dimensional sensor 300.

[0021] The object movement measuring device 200 is a device that measures the movement of an object to be inspected based on the three-dimensional point cloud data 100 of the shooting site photographed by the three-dimensional sensor 300 (see FIG. 2). The object movement measuring device 200 may be a terminal (computer terminal) equipped with functional units (e.g., a processor, a storage device, an input device, a communication interface, and a display device) that constitute a computer, and may be, for example, a personal computer, a server, a tablet terminal, or a smartphone. The object movement measuring device 200 is directly or indirectly connected to the three-dimensional sensor 300 so as to be able to communicate (wirelessly or by wire) and may be connected to the three-dimensional sensor 300 so as to be able to communicate via a network (not shown). The object movement measuring device 200 executes a predetermined program to realize a configuration equipped with a preprocessing unit 210, a monitoring target area designation unit 220, a reference plane determination unit 230, an object detection unit 240, a path monitoring unit 250, a user interface unit 260, a result output unit 270, and a result database 280.

[0022] The pre-processing unit 210 is a functional unit that performs pre-processing (data shaping) on ​​the input three-dimensional point cloud data 100 (see FIG. 2). The three-dimensional point cloud data 100 processed by the pre-processing unit 210 can be displayed on a display unit 261 of a user interface unit 260. The pre-processing unit 210 includes a format conversion unit 211 and a noise removal unit 212.

[0023] The format conversion unit 211 is a functional unit that converts the format of the input three-dimensional point cloud data 100 into a common format that can be commonly used in the object movement measuring device 200 as pre-processing (see FIG. 2). The format conversion unit 211 outputs the converted three-dimensional point cloud data 100 in the common format to the noise removal unit 212. Note that if the format of the three-dimensional point cloud data 100 is originally a common format, the format conversion process in the format conversion unit 211 may be skipped.

[0024] The noise removal unit 212 is a functional unit that removes noise (points unnecessary for measurement) from the points in the three-dimensional point cloud data 100 from the format conversion unit 211 as pre-processing (see FIG. 2). The noise removal unit 212 outputs the three-dimensional point cloud data 100 from which noise has been removed to the monitoring target area designation unit 220, the reference plane determination unit 230, and the user interface unit 260. Examples of noise removal methods include smoothing processing, filtering (e.g., moving average filter processing, median filter processing, etc.), and outlier removal processing (e.g., outlier removal processing by chi-square test). If there is almost no noise, the noise removal processing in the noise removal unit 212 may be skipped.

[0025] The monitored area designation unit 220 is a functional unit that designates a monitored area for the three-dimensional point cloud data 100 (see FIG. 2). A plurality of monitored areas may be designated. The monitored area designation unit 220 designates the monitored area based on designation information input by a user operation from an input unit 262 of a user interface unit 260. When the entire area of ​​the three-dimensional point cloud data 100 is the monitored area, designation of the monitored area can be skipped. The monitored area designated by the monitored area designation unit 220 can be displayed on a display unit 261 of the user interface unit 260. The monitored area designation unit 220 includes a range designation unit 221 and an angle designation unit 222.

[0026] The range designation unit 221 is a functional unit that designates the range (width, depth, height) of the area to be monitored in the captured panoramic view of the 3D point cloud data 100 (see FIG. 2). The range designation unit 221 designates the range of the area to be monitored based on designation information input by a user operation from an input unit 262 of a user interface unit 260. The range designation unit 221 outputs information related to the designated range of the area to be monitored to the reference plane determination unit 230 and the object detection unit 240.

[0027] The angle designation unit 222 is a functional unit that designates an angle for rotating the point cloud of the three-dimensional point cloud data 100 around a predetermined coordinate point (or any designated coordinate point) (see FIG. 2). The angle designation unit 222 designates an angle for rotating the point cloud based on designation information from the input unit 262 of the user interface unit 260 input by a user operation. The designation of the angle for rotating the point cloud by the angle designation unit 222 assists the designation of the range of the monitoring target area by the range designation unit 221. If the rotation of the point cloud is not necessary, the angle designation unit 222 may be omitted or the designation of the angle may be skipped. The angle designation unit 222 outputs information related to the designated angle to the reference plane determination unit 230 and the object detection unit 240. Note that the information related to the designated angle is independent of the information related to the range of the designated monitoring target area.

[0028] The reference plane determination unit 230 is a functional unit that determines a reference plane (e.g., a floor surface) in the monitored area (see FIG. 2). The reference plane may be not only a horizontal floor surface or the ground, but also a plane on a slope, stairs, or a ladder. The reference plane may be set in the air. This makes it possible to monitor not only cars and people on the ground, but also birds, drones, etc. The reference plane determined by the reference plane determination unit 230 can be displayed on the display unit 261 of the user interface unit 260. The reference plane determination unit 230 includes a plane detection unit 231, a reference plane selection unit 232, and a reference plane storage unit 233.

[0029] The plane detection unit 231 is a functional unit that detects a plane (for example, a plane having a preset area or more; in FIG. 3(A), the ground 12) from among the points in the three-dimensional point cloud data 100 that are in the monitored area (see FIG. 2 and FIG. 3). The plane detection unit 231 acquires the three-dimensional point cloud data 100 preprocessed by the preprocessing unit 210, and acquires information (range, angle) related to the monitored area designated by the monitored area designation unit 220. The plane detection unit 231 may set the monitored area in the three-dimensional point cloud data 100 within a designated range, and create an image by rotating the point cloud of the three-dimensional point cloud data 100 (including the monitored area) at a designated angle. If there are multiple planes in the point cloud in the monitored area, the plane detection unit 231 detects the multiple planes. The plane detection unit 231 may set the type of plane to be detected (for example, a horizontal plane, a vertical plane, an inclined plane, a boundary surface of the range of the monitored area, etc.) in advance, and selectively detect the set type of plane. Depending on the characteristics of the 3D sensor 300, there may be cases where the actual plane appears as a wavy point cloud in the image due to variations in the errors in the acquired coordinates. In such a case, the plane detection unit 231 can be configured to increase the specified range of the error threshold between each point cloud to be recognized as a plane, thereby reducing the effect of the waviness and detecting the plane.

[0030] The reference plane selection unit 232 is a functional unit that selects a reference plane from among the planes detected by the plane detection unit 231 (see FIG. 2). The reference plane selection unit 232 selects a reference plane based on selection information from an input unit 262 of a user interface unit 260 that is input by a user operation.

[0031] The reference plane storage unit 233 is a functional unit that stores information related to the reference plane selected by the reference plane selection unit 232 (see FIG. 2). The information related to the reference plane stored in the reference plane storage unit 233 becomes available to the object detection unit 240.

[0032] The object detection unit 240 is a functional unit that performs image processing on a cloud of points in a monitored area to detect objects (detection targets; people 10 and cars 11 in FIG. 3(A)) that satisfy specified detection conditions (see FIG. 2). A screen processed by the object detection unit 240 can be displayed on a display unit 261 of a user interface unit 260. The object detection unit 240 includes a detection condition designation unit 241, an object extraction unit 242, a two-dimensional projection unit 243, an expansion unit 244, a labeling unit 245, a three-dimensional conversion unit 246, and a matching unit 247.

[0033] The detection condition designation unit 241 is a functional unit that designates detection conditions for a detection target based on designation information input by a user operation from an input unit 262 of a user interface unit 260 (see FIG. 2). Examples of detection conditions include a size range and a volume range of a detection target, as well as a time series check to discover a moving object. It is not necessary to designate detection conditions, and when no detection conditions are designated, a preset detection condition may be used, and designation of the detection conditions may be skipped.

[0034] The object extraction unit 242 is a functional unit that extracts a point cloud of a monitoring target area from the 3D point cloud data 100 and deletes the point cloud of the reference plane determined by the reference plane determination unit 230 (corresponding to the point cloud of the ground 12 in FIG. 3(A)) to extract a point cloud of an object (person 10 and car 11 in FIG. 3(A)) (see FIGS. 2 and 3). The object extraction unit 242 may extract a point cloud of an object using an image created by the plane detection unit 231. When there are multiple planes detected by the plane detection unit 231, the object extraction unit 242 may also delete planes other than the reference plane when deleting the point cloud of the reference plane.

[0035] The two-dimensional projection unit 243 is a functional unit that two-dimensionally projects (projects from a direction perpendicular to the reference plane) the point clouds of the object extracted by the object extraction unit 242 (the person point cloud 10a and the vehicle point cloud 11a in FIG. 3B) as one pixel each onto a plane (the plane including the projection reference plane 12b in FIG. 3C) parallel to the reference plane (the reference plane 12a in FIG. 3B) determined by the reference plane determination unit 230 (see FIG. 2 and FIG. 3). The two-dimensional projection unit 243 creates a projected pixel group image related to a projected pixel group (the person projected pixel group 10b and the vehicle projected pixel group 11b in FIG. 3C) obtained by two-dimensionally projecting the point clouds of the detection object (the person point cloud 10a and the vehicle point cloud 11a in FIG. 3B). The two-dimensional projection unit 243 holds information related to the correspondence between the point clouds and the pixel groups during two-dimensional projection. In addition, the projected pixel group (human projected pixel group 10b and car projected pixel group 11b in Figure 3 (C)) is a two-dimensional projection of the point group (human point group 10a and car point group 11a in Figure 3 (B)) of the surface of the detected object photographed by the 3D sensor 300, and since it does not reflect the back surface of the detected object (occlusion part that is not visible from the 3D sensor 300), it is also possible to predict the point group or pixel group of the occlusion part and create a projected pixel group image.

[0036] The expansion unit 244 is a functional unit that expands each projection pixel so that adjacent pixels contact each other (there may be overlapping parts) based on the projection pixel group image created by the two-dimensional projection unit 243 (see FIG. 2). The expansion unit 244 creates an expanded pixel group image related to an expanded pixel group (41a, 41b, 41c in FIG. 4) in which each projection pixel of the projection pixel group (40a, 40b, 40c in FIG. 4) of the projection pixel group image is expanded to the same size (area; for example, the size of a combination of pixels adjacent to each projection pixel on the top, bottom, left, and right sides) (see FIG. 4). The expansion of each projection pixel may be performed by expanding to a size set in advance, expanding to a size set by the user, or expanding by automatically adjusting the size so that adjacent projection pixels contact each other.

[0037] The labeling unit 245 is a functional unit that performs labeling (assigning the same label (number or code)) processing to each expanded pixel of an expanded pixel group where adjacent expanded pixels are in contact with each other, based on an expanded pixel group image related to the expanded pixel group (41a, 41b, 41c in FIG. 4) created by the expansion unit 244 (see FIG. 2). The labeling unit 245 assigns different labels (1, 2, 3 in FIG. 4) to each detection object. The labeling unit 245 creates a labeling pixel group image related to a labeling pixel group (42a, 42b, 42c in FIG. 4; human labeling expanded pixel group 10c and car labeling expanded pixel group 11c in FIG. 3(D)) in which each expanded pixel is labeled (see FIGS. 3 and 4). In addition, the labeling unit 245 may delete (exclude) a labeling pixel group (42c in FIG. 4) having less than a preset number of pixels (e.g., 10 pixels) from among the labeling pixel groups (42a, 42b, 42c in FIG. 4), and may delete (exclude) an expanded pixel group (41c in FIG. 4) having less than a preset number of pixels (e.g., 10 pixels) before performing the labeling process.

[0038] The 3D conversion unit 246 is a functional unit that converts the labeling pixel group image created by the labeling unit 245 into a labeling point group image related to the labeled 3D point group (human labeling point group 10d and vehicle labeling point group 11d in FIG. 3(E)) based on information related to the correspondence between pixels and point groups held by the 2D projection unit 243 (see FIGS. 2 and 3). During conversion, the 3D conversion unit 246 assigns the same label as the label assigned to the labeling pixel group to the labeling point group corresponding to the labeling pixel group.

[0039] The matching unit 247 is a functional unit that detects only objects (detection targets) related to the labeling point clouds in the labeling point cloud image that meet the detection conditions based on the labeling point cloud image converted by the three-dimensional conversion unit 246 and the detection conditions specified by the detection condition specification unit 241 (see FIG. 2). The matching unit 247 may assign detection frames (human detection frame 20, car detection frame 21 in FIG. 3E) to the labeling point clouds to which the same label is assigned (human labeling point cloud 10d, car labeling point cloud 11d in FIG. 3E) for the labeling point clouds related to each detected object. The matching unit 247 may delete the labeling point clouds that do not meet the detection conditions. The matching unit 247 outputs the detection results (for example, an object detection processing image in which a detection frame is added to the labeling point cloud related to each detected object; an object detection processing image such as that shown in FIG. 3(E)) to the route monitoring unit 250, and also stores the results in a result database 280 and can be displayed on the display unit 261 of the user interface unit 260.

[0040] The path monitoring unit 250 is a functional unit that monitors the moving path of an object (a labeling point group related to each detection target object) (see FIG. 2). The path monitoring unit 250 includes an identification information assigning unit 251, a moving direction specifying unit 252, and a counting unit 253.

[0041] The identification information assigning unit 251 is a functional unit that assigns common identification information to objects in the latest detection result that are the same as and of the same type as objects in past results (detection results, identification results). When there are no past results, the identification information assigning unit 251 assigns new identification information to each object in the latest detection result. The identification information assigning unit 251 calculates identity elements (object position, dimensions, shape, etc.) of each object in the latest detection result based on the latest detection result and past results (detection results, identification results) stored in the result database 280, and determines the identity of each object between the latest detection result and past detection results. For each object in the latest detection result that is determined to be the same, the identification information assigning unit 251 calculates type elements (object volume, moving direction, moving speed, etc.) of each object in the latest detection result based on the latest detection result and past results (detection results, identification results) stored in the result database 280, and determines the type of each object in the latest detection result. The identification information assigning unit 251 assigns common identification information (e.g., object type and number; person identification information 50, car identification information 51 in FIG. 5 and FIG. 6) to the same object (e.g., detection frame of the labeling point cloud related to each detection target object; person detection frame 20, car detection frame 21 in FIG. 5 and FIG. 6) as the object in the past result for each object in the latest detection result. The identification information assigning unit 251 can store the identification results (e.g., an identification information-assigned image in which identification information is assigned to the detection frame of the labeling point cloud related to each object as in FIG. 5 and FIG. 6, and the calculated value of each element) in the result database 280 and display it on the display unit 261 of the user interface unit 260. The identification information is used for tracking the object.

[0042] The moving direction identifying unit 252 is a functional unit that identifies the moving direction of each object to which identification information has been assigned (see FIG. 2). The moving direction identifying unit 252 may identify the moving direction of the object using the moving direction calculated by the identification information assigning unit 251. The moving direction identifying unit 252 may assign information related to the moving direction of each identified object (for example, an arrow; in FIG. 6, an arrow indicating a person moving direction 60 and a vehicle moving direction 61) to the identification information assigned image (see FIG. 6).

[0043] The counting unit 253 is a functional unit that counts the number of objects moving (passing) through count points (for example, virtual gates provided on the moving path, multiple gates are possible) based on the moving direction of each object identified by the moving direction identification unit 252 (for example, rightward or leftward as viewed from the three-dimensional sensor 300 side) (see FIG. 2). The counting unit 253 may count according to the moving direction of the object at the count point (not only one direction but also two directions are possible). The count points may be set by the user before the measurement, or may be automatically set by learning the moving pattern of the object based on the three-dimensional point cloud data 100 captured in a rehearsal before the measurement, or may be selected or modified by the user from among several count points presented by learning the moving pattern of the object based on the three-dimensional point cloud data 100 captured in a rehearsal before the measurement.

[0044] The user interface unit 260 is a functional unit that mediates between the user and the object movement measuring device 200 (see FIG. 2). The user interface unit 260 includes a display unit 261 and an input unit 262.

[0045] The display unit 261 is a functional unit that displays information (see FIG. 2). In addition to the three-dimensional point cloud data 100 preprocessed by the preprocessing unit 210, the display unit 261 can display the monitoring target area specified by the monitoring target area designation unit 220, the reference plane determined by the reference plane determination unit 230, the object detection processing image processed by the object detection unit 240, and the like. The display unit 261 displays the results of the detected flow rate measurement according to an instruction from the result output unit 270.

[0046] The input unit 262 is a functional unit that inputs information operated by a user to the object movement measuring device 200 (see FIG. 2). The input unit 262 mainly inputs information to the monitoring target area designation unit 220, the reference plane determination unit 230, and the detection condition designation unit 241 of the object detection unit 240.

[0047] The result output unit 270 is a functional unit that, when a count is performed by the counting unit 253, outputs the result to the display unit 261 of the user interface unit 260 (see FIG. 2).

[0048] The result database 280 is a database that associates and stores the detection results from the object detection unit 240, the identification results from the identification information assignment unit 251, the calculation results, and the like (including the prerequisite designated conditions, setting conditions, etc.) (see FIG. 2).

[0049] The three-dimensional sensor 300 is a device that three-dimensionally senses and photographs the surface of a subject (a person 10, a car 11, and a ground surface 12 in FIG. 1) (see FIGS. 1 and 2). The three-dimensional sensor 300 is communicably connected to the object movement measuring device 200. The three-dimensional sensor 300 generates three-dimensional point cloud data 100 in a predetermined format by photographing the subject, and outputs the generated three-dimensional point cloud data 100 to the object movement measuring device 200. The three-dimensional point cloud data 100 may be generated by the object movement measuring device 200 instead of by the three-dimensional sensor 300. For the three-dimensional sensor 300, for example, a ToF (Time of Flight) camera, a stereo camera, a three-dimensional LIDAR (Laser Imaging Detection And Ranging), a depth sensor, a distance measuring sensor, a distance camera, etc. may be used. Here, the three-dimensional point cloud data 100 is data generated in a predetermined format by the three-dimensional sensor 300, and is point cloud data depicted as a point cloud (a collection of many points having XYZ coordinate (three-dimensional coordinate) information) (see FIG. 2). The three-dimensional sensor 300 can be changed to a sensor device with various output formats according to customer requests. There may be multiple three-dimensional sensors 300, and multiple three-dimensional point cloud data 100 may be synthesized by the object movement measurement device 200 to perform wide-range measurement that is not possible with a single three-dimensional sensor 300.

[0050] Next, the operation of the object movement measuring device in the object movement measuring system according to embodiment 1 will be described with reference to the drawings. Fig. 7 is a flow chart showing a schematic operation of the object movement measuring device in the object movement measuring system according to embodiment 1. Please refer to Fig. 2 and its description for the detailed configuration of the object movement measuring system.

[0051] First, preprocessing unit 210 of object motion measuring device 200 acquires three-dimensional point cloud data 100 (point cloud data of an area including a monitoring target area) of a subject captured by three-dimensional sensor 300 (step A1).

[0052] Next, as preprocessing, the format conversion unit 211 of the preprocessing unit 210 of the object movement measurement device 200 converts the format of the acquired 3D point cloud data 100 into a common format that can be commonly used in the object movement measurement device 200 (step A2).

[0053] Next, the noise removal section 212 of the preprocessing section 210 of the object movement measuring device 200 removes noise from the point cloud in the 3D point cloud data 100 whose format has been converted by the format conversion section 211, as preprocessing (step A3).

[0054] Next, the monitoring target area designation unit 220 of the object movement measurement device 200 designates a monitoring target area (range, rotation angle) in the captured panoramic view of the 3D point cloud data 100 based on designation information input by a user operation from the input unit 262 of the user interface unit 260 (step A4). Note that after the monitoring target area is initially designated, step A4 can be skipped.

[0055] Next, the reference plane determination unit 230 of the object movement measuring device 200 detects a plane from among the points in the three-dimensional point cloud data 100 that are in the monitoring target area, and determines (selects) a reference plane (e.g., a floor surface) from among the detected planes based on selection information input by a user operation from the input unit 262 of the user interface unit 260, and stores information related to the determined reference plane (step A5). Note that after the reference plane is determined for the first time, step A5 can be skipped.

[0056] Next, the detection condition designation unit 241 of the object detection unit 240 of the object movement measurement device 200 designates the detection conditions (e.g., size range, volume range, etc.) of the detection target object based on designation information input by the user's operation from the input unit 262 of the user interface unit 260 (step A6). Note that after the detection conditions are initially designated, step A6 can be skipped. Also, it is not necessary to designate the detection conditions, and if no detection conditions are designated, step A6 can be skipped using the detection conditions set in advance.

[0057] Next, the object detection unit 240 of the object movement measurement device 200 performs image processing on the point cloud in the monitored area to detect an object that satisfies the specified detection condition (step A7). Details of step A7 will be described later (see FIG. 8).

[0058] Next, the identification information assignment unit 251 of the path monitoring unit 250 of the object movement measuring device 200 determines the identity and type of each object in the latest detection result with respect to each object in the past results (detection results, identification results), and assigns common identification information to objects that are identical and of the same type as objects in the past results (detection results, identification results) (step A8).

[0059] Next, the movement direction identification unit 252 of the path monitoring unit 250 of the object movement measurement device 200 identifies the movement direction of each object to which identification information has been assigned (step A9). Note that the movement direction of an object can be identified based on the movement direction of the object calculated when determining the identity and type of the object (see above for details).

[0060] Next, the counting unit 253 of the path monitoring unit 250 of the object movement measuring device 200 counts the number of objects moving (passing) the count point based on the identified movement direction of each object (step A10). If there is no object moving (passing) the count point, it is skipped.

[0061] Next, the result output unit 270 of the object movement measuring device 200 outputs the latest count result to the display unit 261 of the user interface unit 260 (step A11), and then the process returns to step A1 to repeatedly execute the flow from step A1 to step A11.

[0062] Next, details of the object detection process (step A7) of the object movement measuring device in the object movement measuring system according to embodiment 1 will be described with reference to the drawings. Fig. 8 is a flow chart showing in schematic form the details of the object detection process of the object movement measuring device in the object movement measuring system according to embodiment 1. Please refer to Fig. 2 and its description for the detailed configuration of the object movement measuring system.

[0063] In the object detection process (step A7) after step A6 in Figure 7, first, the object extraction unit 242 of the object detection unit 240 of the object movement measurement device 200 extracts the point cloud of the object by extracting the point cloud of the monitored area from the 3D point cloud data 100 and deleting the point cloud of the reference plane (step B1).

[0064] Next, the two-dimensional projection unit 243 of the object detection unit 240 of the object movement measurement device 200 two-dimensionally projects the point cloud of the extracted detection object onto a plane parallel to the reference plane, with each point representing one pixel, to create a projected pixel group image, and retains information related to the correspondence between the point cloud and the pixel group during two-dimensional projection (step B2).

[0065] Next, the expansion unit 244 of the object detection unit 240 of the object movement measurement device 200 expands each pixel based on the created projected pixel group image so that adjacent pixels contact each other, to create an expanded pixel group image (step B3).

[0066] Next, the labeling unit 245 of the object detection unit 240 of the object movement measurement device 200 performs labeling processing on each expanded pixel of an expanded pixel group in which adjacent expanded pixels are in contact with each other based on the created expanded pixel group image, to create a labeled pixel group image (step B4).

[0067] Next, the 3D conversion unit 246 of the object detection unit 240 of the object movement measurement device 200 converts the created labeling pixel group image into a labeling point cloud image based on the retained information related to the correspondence between pixels and point clouds when projected in 2D (step B5).

[0068] Next, the matching unit 247 of the object detection unit 240 of the object movement measuring device 200 detects (matches) only objects (detection targets) associated with the labeling point groups in the labeling point cloud image that meet the detection conditions based on the converted labeling point cloud image and the specified detection conditions (step B6), and then proceeds to step A8 in Figure 7.

[0069] According to the first embodiment, the labeling process (clustering process) can be accelerated by projecting the three-dimensional point cloud data 100 into two dimensions and expanding the pixels, so that as many moments as possible can be captured and continuity can be confirmed. The clustering process in Patent Document 3 is a process for connecting neighboring point clouds, and is a process that cannot perform appropriate noise removal or extraction of appropriate connected parts. In addition, the process of searching for neighboring point clouds in Patent Document 3 takes too much processing time, so the processing interval is too long, and there is a possibility that the correct result cannot be obtained.

[0070] Furthermore, according to embodiment 1, there are fewer installation constraints on the 3D sensor 300, and it is possible to go to the site, install the sensor at any location, and immediately start an investigation and obtain measurement results with only minimal settings, which contributes to easily measuring the movement of objects at any location.

[0071] Moreover, according to the first embodiment, object detection processing can be performed by simple settings (designation of a monitoring target area, selection of a reference plane, designation of detection conditions) without using machine learning or AI (Artificial Intelligence), so that even an untrained user can operate it and contribute to easily measuring the movement of an object at any location. In addition, when AI and machine learning are used, if the device is installed at a mounting position or angle different from when the learned data was acquired, the possibility of not obtaining a correct result increases. In particular, when three-dimensional information is determined by machine learning, there is more one-dimensional information than two-dimensional information, and therefore malfunctions become more noticeable. In addition, when three-dimensional information is used as it is to be determined by machine learning, the installation position and angle are irrelevant, but the amount and type of learning data become large, and calculation costs are very high, which is not realistic.

[0072] Moreover, according to the first embodiment, the speed of the individual segmentation process by labeling the three-dimensional point cloud can be increased, and as many segmentation processes as possible can be performed before the flow measurement target has completely traversed the space.

[0073] Moreover, according to the first embodiment, since object types are classified by volume, moving speed, degree of change in moving direction, etc. in addition to dimensions, classification of people, motorcycles, etc. can be flexibly handled, and tracking is also possible when occlusion occurs. In addition, in Patent Document 3, classification of vehicles and non-vehicles is performed only by the dimensions of objects, and there is a possibility that classification of people, motorcycles, etc. cannot be flexibly handled, and tracking is also difficult when occlusion occurs.

[0074] Furthermore, according to the first embodiment, the three-dimensional point cloud data 100 obtained by the three-dimensional sensor 300 is sparse point cloud data, so flow detection can be performed while protecting personal information. In recent years, there has been a trend to avoid the use of surveillance cameras due to consideration of privacy, but according to the first embodiment, sparse three-dimensional point cloud data 100 captured by the three-dimensional sensor 300 is used, so flow detection can be performed without identifying individuals.

[0075] Furthermore, according to the first embodiment, by setting the projection plane to a plane other than the floor surface (ground surface), it is possible to utilize the projection plane to count vertical movements, for example.

[0076] Furthermore, according to the first embodiment, even if multiple 3D sensors 300 are used, it is easy to align the coordinates of corresponding points in multiple 3D point cloud data 100, so that synthesis between 3D point cloud data can be easily performed, enabling flow rate surveys in a wide range of locations. Note that when multiple surveillance cameras such as RGB (Red-Green-Blue) cameras are used, the installation position and angle change for each spot, making it difficult to map between the respective images (video).

[0077] [Embodiment 2] An object movement measuring device according to the second embodiment will be described with reference to the drawings. Fig. 9 is a block diagram showing a schematic configuration of the object movement measuring device according to the second embodiment.

[0078] The object movement measuring device 200 is a device that measures the movement of an object to be inspected based on 3D point cloud data 100 of a shooting site photographed by a 3D sensor 300. The object movement measuring device 200 includes a reference plane determining unit 230, an object extracting unit 242, a 2D projecting unit 243, an expanding unit 244, a labeling unit 245, a 3D converting unit 246, a matching unit 247, an identification information assigning unit 251, a moving direction identifying unit 252, and a counting unit 253.

[0079] The reference plane determination unit 230 is configured to detect one or more planes from the three-dimensional point cloud data 100 of the shooting site photographed by the three-dimensional sensor 300, and determine the reference plane from among the detected planes. The object extraction unit 242 is configured to extract a point cloud of an object from the three-dimensional point cloud data 100. The two-dimensional projection unit 243 is configured to create a pixel group by two-dimensionally projecting the point cloud of the extracted object onto a plane parallel to the reference plane, with each point being one pixel. The expansion unit 244 is configured to create a pixel group by expanding each pixel of the two-dimensionally projected pixel group so that adjacent pixels are in contact with each other. The labeling unit 245 is configured to create a pixel group by labeling pixels of the expanded pixel group, in which adjacent pixels are in contact with each other, with the same label. The three-dimensional conversion unit 246 is configured to convert the labeled pixel group into a three-dimensional pixel group based on the relationship between the point cloud and the pixel group when projected in two dimensions, and create a point cloud by labeling with the same label. The matching unit 247 is configured to detect an object related to a point group that meets a detection condition among the point groups that have been converted into three dimensions and labeled with the same label. The identification information assigning unit 251 is configured to determine the identity and type of the most recently detected object with respect to a previously detected object, and assign common identification information to objects that are the same and of the same type as the previously detected object. The moving direction identifying unit 252 is configured to identify the moving direction of the object to which the identification information has been assigned. The counting unit 253 is configured to count the number of objects moving at the count point based on the identified moving direction of the object.

[0080] According to the second embodiment, the labeling process can be speeded up by projecting the three-dimensional point cloud data 100 into two dimensions and expanding the pixels, which contributes to speeding up data processing and making it easier to measure the movement of an object at any location.

[0081] The object movement measuring device according to the first and second embodiments can be configured by so-called hardware resources (information processing device, computer), and can use a configuration exemplified in Fig. 10. For example, the hardware resources 1000 include a processor 1001, a memory 1002, a network interface 1003, etc., which are mutually connected by an internal bus 1004.

[0082] 10 is not intended to limit the hardware configuration of the hardware resource 1000. The hardware resource 1000 may include hardware (e.g., an input / output interface) that is not shown. Furthermore, the number of units such as the processor 1001 included in the device is not intended to be limited to the example shown in FIG. 10, and for example, multiple processors 1001 may be included in the hardware resource 1000. For example, a CPU (Central Processing Unit), an MPU (Micro Processor Unit), a GPU (Graphics Processing Unit), etc. can be used as the processor 1001.

[0083] The memory 1002 may be, for example, a random access memory (RAM), a read only memory (ROM), a hard disk drive (HDD), or a solid state drive (SSD).

[0084] The network interface 1003 may be, for example, a LAN (Local Area Network) card, a network adapter, a network interface card, or the like.

[0085] The functions of the hardware resource 1000 are realized by the above-mentioned processing module. The processing module is realized, for example, by the processor 1001 executing a program stored in the memory 1002. The program can be updated by downloading it via a network or by using a storage medium on which the program is stored. Furthermore, the processing module may be realized by a semiconductor chip. That is, it is sufficient that the functions performed by the processing module are realized by executing software on some kind of hardware.

[0086] A part or all of the above-described embodiments may be described as, but is not limited to, the following supplementary notes.

[0087] [Appendix 1] a reference plane determination unit configured to detect one or more planes from 3D point cloud data of a shooting site captured by a 3D sensor and determine a reference plane from among the detected planes; an object extractor configured to extract an object point cloud from the three-dimensional point cloud data; a two-dimensional projection unit configured to create a pixel group by two-dimensionally projecting the extracted point group of the object onto a plane parallel to the reference plane, with each point representing one pixel; an expansion unit configured to create an expanded pixel group for each pixel of the two-dimensionally projected pixel group such that adjacent pixels are in contact with each other; a labeling unit configured to create a pixel group by labeling adjacent pixels that are in contact with each other with the same label among the expanded pixel group; a three-dimensional conversion unit configured to convert the labeled pixel group into a three-dimensional image based on a relationship between the point group and the pixel group when projected in two dimensions, and generate a point group labeled with the same labels; A matching unit configured to detect an object associated with a point cloud that satisfies a detection condition from the point cloud that has been converted into three dimensions and labeled with the same label; an identification information assigning unit configured to determine the identity and type of a most recently detected object with respect to a previously detected object, and to assign common identification information to objects that are identical and of the same type as the previously detected object; a movement direction identification unit configured to identify a movement direction of the object to which the identification information is assigned; A counting unit configured to count the number of objects moving at a count point based on the identified moving direction of the object; An object movement measuring device comprising: [Appendix 2] A monitoring target area designation unit configured to designate a monitoring target area in the three-dimensional point cloud data, The reference plane determination unit is configured to detect one or more planes from a point cloud in the specified monitoring target area. 2. An object movement measuring device as described in appendix 1. [Appendix 3] The monitoring target area designation unit is a range designation unit configured to designate a range of the monitoring target area in the captured panoramic view of the three-dimensional point cloud data; an angle designation unit configured to designate an angle for rotating the point cloud of the three-dimensional point cloud data around a predetermined coordinate point; Equipped with the reference plane determination unit is configured to set the monitoring target area in the three-dimensional point cloud data within the specified range, and to create an image by rotating the point cloud of the three-dimensional point cloud data including the monitoring target area around the specified coordinate point at the specified angle. 3. An object movement measuring device as described in appendix 2. [Appendix 4] A detection condition designation unit configured to designate the detection condition, The matching unit is configured to detect an object related to a point group that satisfies the specified detection condition from among the point group that has been converted into three dimensions and labeled with the same label. 4. An object movement measuring device according to any one of claims 1 to 3. [Appendix 5] The identification information assignment unit, determining the identity of the object by calculating a predetermined identity element of the detected latest object and comparing it with the calculated identity element of the detected previous object to determine the identity of the detected latest object and the previous object; In the determination of the type of the object, a type element of the latest object is calculated to determine the type of the latest object. It is configured as follows: 5. An object movement measuring device according to any one of claims 1 to 4. [Appendix 6] the object extraction unit is configured to extract the point cloud of the object by deleting the point cloud of the determined reference plane from the three-dimensional point cloud data. 6. An object movement measuring device according to any one of claims 1 to 5. [Appendix 7] an input unit configured to input information related to the determination of the reference surface and information related to the detection conditions by a user operation; 7. An object movement measuring device according to any one of claims 1 to 6. [Appendix 8] An object movement measuring device according to any one of appendix 1 to 7; A three-dimensional sensor that is communicably connected to the object movement measuring device and captures an image of a shooting site; An object movement measurement system comprising: [Appendix 9] An object movement measurement method for measuring movement of an object using hardware resources, comprising: A step of detecting one or more planes from 3D point cloud data of the shooting site photographed by the 3D sensor, and determining a reference plane from among the detected planes; extracting a point cloud of an object from the three-dimensional point cloud data; creating a pixel group by two-dimensionally projecting the extracted point group of the object onto a plane parallel to the reference plane, with each point representing one pixel; creating an expanded pixel group for each pixel of the two-dimensionally projected pixel group such that adjacent pixels are in contact with each other; A step of creating a pixel group by labeling adjacent pixels that are in contact with each other with the same label among the expanded pixel group; converting the labeled pixel group into a three-dimensional image based on the relationship between the point group and the pixel group when projected in two dimensions, and creating a point group labeled with the same labels; A step of detecting an object associated with a point cloud that satisfies a detection condition from the point cloud that has been converted into three dimensions and labeled with the same label; determining the identity and type of each of the most recently detected objects with respect to previously detected objects, and assigning common identification information to objects that are identical and of the same type as the previously detected objects; Identifying a direction of movement of the object to which the identification information is assigned; Counting the number of objects moving at a count point based on the identified moving direction of the object; A method for measuring object movement, comprising: [Appendix 10] A program for causing hardware resources to execute a process for measuring the movement of an object, A process of detecting one or more planes from 3D point cloud data of the shooting site photographed by a 3D sensor, and determining a reference plane from among the detected planes; Extracting a point cloud of an object from the three-dimensional point cloud data; A process of creating a pixel group by two-dimensionally projecting the point group of the extracted object onto a plane parallel to the reference plane, with each point representing one pixel; A process of creating an expanded pixel group for each pixel of the two-dimensionally projected pixel group such that adjacent pixels are in contact with each other; A process of creating a pixel group by labeling adjacent pixels that are in contact with each other with the same label among the expanded pixel group; A process of converting the labeled pixel group into a three-dimensional image based on the relationship between the point group and the pixel group when projected in two dimensions, and creating a point group labeled with the same labels; A process of detecting an object related to a point cloud that satisfies a detection condition from the point cloud that has been converted into three dimensions and labeled with the same label; a process of determining the identity and type of a most recently detected object with respect to a previously detected object, and assigning common identification information to objects that are identical and of the same type as the previously detected object; A process of identifying a moving direction of the object to which the identification information is assigned; A process of counting the number of objects moving at a count point based on the identified moving direction of the object; A program for causing the hardware resource to execute the above.

[0088] The disclosures of the above patent documents are incorporated herein by reference and may be used as the basis or part of the present invention as necessary. Within the framework of the entire disclosure of the present invention (including the claims and drawings), and further based on the basic technical idea, modifications and adjustments of the embodiments and examples are possible. Furthermore, within the framework of the entire disclosure of the present invention, various combinations or selections (or non-selection as necessary) of various disclosed elements (including each element of each claim, each element of each embodiment or example, each element of each drawing, etc.) are possible. In other words, the present invention naturally includes various modifications and corrections that a person skilled in the art would be able to make according to the entire disclosure, including the claims and drawings, and the technical idea. Furthermore, with regard to the numerical values ​​and numerical ranges described in this application, any intermediate value, lower numerical value, and small range are deemed to be described even if not specified. Furthermore, the disclosures of the above cited documents may be used in part or in whole in combination with the descriptions in this document as part of the disclosure of the present invention as necessary, in accordance with the spirit of the present invention, and are deemed to be included (belong) in the disclosures of this application. [Explanation of symbols]

[0089] 1. Object movement measurement system 10 people 10a Human point cloud 10b Human projection pixel group 10c Human labeling expanded pixel group 10d human labeling point cloud 11 cars 11a Car point group 11b Car projection pixel group 11c Car labeling expansion pixel group 11d car labeling point cloud 12 Ground 12a Reference plane 12b Projection reference plane 20 people detection slots 21 Car detection frame 40a, 40b, 40c Projected pixel groups 41a, 41b, 41c Expanded pixel group 42a, 42b, 42c Labeling pixel groups 50 Personal Identification Information 51 Vehicle Identification Information 60 people moving direction 61 Car travel direction 100 3D point cloud data 200 Object movement measurement device 210 Pretreatment section 211 Format conversion unit 212 Noise removal section 220 Monitoring area designation section 221 Range specification section 222 Angle designation section 230 Reference plane determination section 231 Plane detector 232 Reference plane selection section 233 Reference plane memory section 240 Object detection unit 241 Detection condition specification section 242 Object extraction part 243 2D Projection Section 244 Expansion section 245 Labeling Department 246 3D conversion section 247 Matching Department 250 Route Monitoring Department 251 Identification information assignment unit 252 Movement direction identification part 253 Counting Department 260 User Interface Section 261 Display section 262 Input section 270 Result output section 280 Results Database 300 3D Sensor 1000 Hardware Resources 1001 Processor 1002 Memory 1003 Network Interface 1004 Internal Bus

Claims

1. a reference plane determination unit configured to detect one or more planes from three-dimensional point cloud data of a shooting site photographed by a three-dimensional sensor and determine a reference plane from among the detected planes; an object extraction unit configured to extract an object point cloud from the three-dimensional point cloud data; a two-dimensional projection unit configured to create a pixel group by two-dimensionally projecting the extracted point group of the object onto a plane parallel to the reference plane, with each point being one pixel; an expansion unit configured to create an expanded pixel group for each pixel of the two-dimensionally projected pixel group such that adjacent pixels are in contact with each other; a labeling unit configured to create a pixel group by labeling adjacent pixels that are in contact with each other with the same label among the expanded pixel group; a three-dimensional conversion unit configured to convert the labeled pixel group into a three-dimensional image based on a relationship between the point group and the pixel group when projected in two dimensions, and generate a point group labeled with the same labels; A matching unit configured to detect an object associated with a point cloud that satisfies a detection condition from the point cloud that has been converted into three dimensions and labeled with the same label; an identification information assigning unit configured to determine the identity and type of a most recently detected object with respect to a previously detected object, and to assign common identification information to objects that are identical and of the same type as the previously detected object; a movement direction identification unit configured to identify a movement direction of the object to which the identification information is assigned; A counting unit configured to count the number of objects moving at a count point based on the identified moving direction of the object; An object movement measuring device comprising:

2. a monitoring target area designation unit configured to designate a monitoring target area in the three-dimensional point cloud data, The reference plane determination unit is configured to detect one or more planes from a point cloud in the specified monitoring target area.

2. The object movement measuring device according to claim 1.

3. The monitoring target area designation unit is a range designation unit configured to designate a range of the monitoring target area in an imaged panoramic view of the three-dimensional point cloud data; an angle designation unit configured to designate an angle for rotating a point cloud of the three-dimensional point cloud data around a predetermined coordinate point; Equipped with the reference plane determination unit is configured to set the monitoring target area in the three-dimensional point cloud data within the specified range, and to create an image by rotating a point cloud of the three-dimensional point cloud data including the monitoring target area around the predetermined coordinate point at the specified angle.

3. The object movement measuring device according to claim 2.

4. A detection condition designation unit configured to designate the detection condition, The matching unit is configured to detect an object related to a point cloud that satisfies the specified detection condition from the point cloud that has been converted into three dimensions and labeled with the same label.

4. The object movement measuring device according to claim 1.

5. The identification information assignment unit, determining the identity of the object by calculating a predetermined identity element of the detected latest object and comparing it with the calculated identity element of the detected previous object to determine the identity of the detected latest object and the previous object; In the determination of the type of the object, a type element of the latest object is calculated to determine the type of the latest object. It is configured as follows: The object movement measuring device according to any one of claims 1 to 4.

6. the object extraction unit is configured to extract a point cloud of the object by deleting a point cloud of the determined reference plane from the three-dimensional point cloud data. The object movement measuring device according to any one of claims 1 to 5.

7. an input unit configured to input information related to the determination of the reference surface and information related to the detection conditions by a user operation; The object movement measuring device according to any one of claims 1 to 6.

8. The object movement measuring device according to any one of claims 1 to 7, A three-dimensional sensor that is communicably connected to the object movement measuring device and captures an image of a shooting site; An object movement measurement system comprising:

9. An object movement measurement method for measuring movement of an object using hardware resources, comprising: A step of detecting one or more planes from three-dimensional point cloud data of a shooting site photographed by a three-dimensional sensor, and determining a reference plane from among the detected planes; extracting a point cloud of an object from the three-dimensional point cloud data; creating a pixel group by two-dimensionally projecting the extracted point group of the object onto a plane parallel to the reference plane, with each point representing one pixel; creating an expanded pixel group for each pixel of the two-dimensionally projected pixel group such that adjacent pixels are in contact with each other; A step of creating a pixel group by labeling adjacent pixels that are in contact with each other with the same label among the expanded pixel group; converting the labeled pixel group into a three-dimensional image based on a relationship between the point group and the pixel group when projected in two dimensions, and creating a point group labeled with the same labels; A step of detecting an object associated with a point cloud that satisfies a detection condition from the point cloud that has been converted into three dimensions and labeled with the same label; determining the identity and type of each of the most recently detected objects with respect to previously detected objects, and assigning common identification information to objects that are identical and of the same type as the previously detected objects; Identifying a direction of movement of the object to which the identification information is assigned; Counting the number of objects moving at a count point based on the identified moving direction of the object; A method for measuring object movement, comprising:

10. A program for causing hardware resources to execute a process for measuring the movement of an object, A process of detecting one or more planes from three-dimensional point cloud data of a shooting site photographed by a three-dimensional sensor, and determining a reference plane from among the detected planes; A process of extracting a point cloud of an object from the three-dimensional point cloud data; a process of creating a pixel group by two-dimensionally projecting the point group of the extracted object onto a plane parallel to the reference plane, with each point representing one pixel; A process of creating an expanded pixel group by expanding each pixel of the two-dimensionally projected pixel group so that adjacent pixels are in contact with each other; A process of creating a pixel group by labeling adjacent pixels that are in contact with each other with the same label among the expanded pixel group; A process of converting the labeled pixel group into a three-dimensional image based on the relationship between the point group and the pixel group when projected in two dimensions, and creating a point group labeled with the same labels; A process of detecting an object related to a point cloud that satisfies a detection condition from the point cloud that has been converted into three dimensions and labeled with the same label; a process of determining the identity and type of a most recently detected object with respect to a previously detected object, and assigning common identification information to objects that are identical and of the same type as the previously detected object; A process of identifying a moving direction of the object to which the identification information is assigned; A process of counting the number of objects moving at a count point based on the identified moving direction of the object; A program for causing the hardware resource to execute the above.

Images