Information Processing Apparatus, Information Processing Method, and Information Processing Program

Abstract

The present invention captures images of a plurality of mobile objects by using a first camera and acquires an identification-including partial traffic line of each mobile object. The present invention captures images of the mobile objects by using a second camera and acquires an identification-lacking partial traffic line of each mobile object. The present invention individually identifies the mobile object to which an identification-lacking partial traffic line can be attributed, on the basis of individual identification of the mobile objects by the identification-including partial traffic lines, and integrates the identification-lacking partial traffic lines and the identification-including partial traffic lines together to create traffic lines of individually identified mobile objects.

Description

Technical Field

[0001] [Description of Related Applications] This invention is based on a claim of priority from Japanese Patent Application: Japanese Patent Application No. 2021-205562 (filed on December 17, 2021), and the entire contents of the application are incorporated herein by reference. This invention relates to an information processing apparatus, an information processing method, and an information processing program. In particular, it relates to an information processing apparatus, an information processing method, and an information processing program for analyzing the movement line of a moving object.

Background Art

[0002] For example, Patent Document 1 discloses a technique for analyzing the movement line of a moving object (such as a person or an animal) using distance measurement techniques such as LiDAR (Light Detection and Ranging) and ToF (Time Of Flight) cameras.

[0003] With only the above distance measurement techniques, it is not possible to identify individuals of multiple moving objects. Therefore, when multiple moving objects cross or pass by each other, the movement lines of individual moving objects cannot be specified. For this reason, techniques such as a "marker location system" have been developed to analyze the movement lines of moving objects while identifying individuals of the moving objects.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] The following analysis is made from the perspective of the present invention. It should be noted that each disclosure of the above prior art documents is incorporated herein by reference.

[0006] In the technology of analyzing the movement path of a moving object while identifying the moving object, the area that can be covered by a single camera is narrow. Therefore, there is a problem that the number of cameras increases in order to cover a wide area.

[0007] Therefore, an object of the present invention is to provide an information processing apparatus, an information processing method, and an information processing program that acquire the movement path of a moving object while identifying the moving object with a small number of cameras.

Means for Solving the Problem

[0008] According to a first aspect of the present invention, a first movement path acquisition unit that images a plurality of moving objects with a first camera and acquires a partial movement path with identification for each moving object; a second movement path acquisition unit that images the moving object with a second camera and acquires a partial movement path without identification for each moving object; a movement path creation unit that identifies which moving object each partial movement path without identification belongs to based on the individual identification of the moving object by the first movement path acquisition unit, and integrates the partial movement path without identification and the partial movement path with identification to create a movement path of the individually identified moving object; An information processing apparatus including the above is provided.

[0009] According to a second aspect of the present invention, a first movement path acquisition step of imaging a plurality of moving objects with a first camera and acquiring a partial movement path with identification for each moving object; a second movement path acquisition step of imaging the moving object with a second camera and acquiring a partial movement path without identification for each moving object; a movement path creation step of identifying which moving object each partial movement path without identification belongs to based on the individual identification of the moving object in the first movement path acquisition step, and integrating the partial movement path without identification and the partial movement path with identification to create a movement path of the individually identified moving object; An information processing method including the above is provided.

[0010] According to a third aspect of the present invention, A first trajectory acquisition process that captures a plurality of moving objects with a first camera and acquires a partial trajectory with identification for each moving object, A second trajectory acquisition process that captures the moving object with a second camera and acquires a partial trajectory without identification for each moving object, Based on the individual identification of the moving object by the first trajectory acquisition process, identify which moving object each partial trajectory without identification belongs to, and integrate the partial trajectory without identification and the partial trajectory with identification to create a trajectory of the individually identified moving object. A trajectory creation process, An information processing program is provided that causes a computer to execute the above.

Advantages of the Invention

[0011] According to each aspect of the present invention, an information processing apparatus, an information processing method, and an information processing program are provided that contribute to acquiring the trajectory of a moving object while individually identifying the moving object with a small number of cameras.

Brief Description of the Drawings

[0012]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Embodiments for Carrying Out the Invention

[0013] Preferred embodiments of the present invention will be described in detail with reference to the drawings. Note that the reference numerals added to the following description are for convenience of each element as an example to assist understanding, and are not intended to limit the present invention to the illustrated embodiments. Also, the connection lines between blocks in each figure include both bidirectional and unidirectional ones. Further, in the block diagrams shown in the present application disclosure, although not explicitly shown, input ports and output ports exist at the input ends and output ends of each connection line, respectively. The same applies to input / output interfaces.

[0014] First, an overview of the present invention will be described. As shown in FIG. 1, the information processing apparatus 100 of the present invention includes a first trajectory acquisition unit 10, a second trajectory acquisition unit 20, and a trajectory creation unit 30.

[0015] The first trajectory acquisition unit 10 captures a plurality of moving objects with a first camera and acquires partial trajectories of each moving object with identification. Further, the second trajectory acquisition unit 20 captures a plurality of moving objects with a second camera and acquires partial trajectories of each moving object without identification. The trajectory creation unit 30 identifies which moving object each partial trajectory without identification belongs to based on the individual identification of the moving objects by the first trajectory acquisition unit 10. Then, the trajectory creation unit 30 integrates the partial trajectory without identification and the partial trajectory with identification to create a trajectory of the individually identified moving object.

[0016] Here, the moving object is exemplified by a human. The first camera is exemplified by a visible light camera or an RGB camera, and the first trajectory acquisition unit 10 acquires the "trajectory with identification" shown in FIG. 2. Note that since a "marker location system" can be applied as the first trajectory acquisition unit 10, a visible light camera or an RGB (Red-Green-Blue) camera may also be referred to as a "marker camera". Also, the second camera is exemplified by a 3D (three-dimensional) point cloud camera, LiDAR (Light Detection and Ranging), or a ToF (Time Of Flight) camera (which may also be referred to as a "3D camera"), and the second trajectory acquisition unit 20 acquires the "trajectory without identification" shown in FIG. 2.

[0017] To explain with a specific example, as shown in FIG. 3, the first trajectory acquisition unit 10 holds the position of the human with marker ID: 3 as a marker position history, and acquires a partial trajectory with identification (also referred to as a marker partial trajectory) based on the marker position history. In FIG. 3, the partial trajectory with identification is represented as a partial trajectory with an ID (identification) within the imaging area of the marker camera.

[0018] The second trajectory acquisition unit 20 holds the position of the human as a moving object position history without identifying the human, and acquires a partial trajectory without identification (also referred to as a moving object partial trajectory) based on the moving object position history. In FIG. 3, the partial trajectory without identification is represented as a partial trajectory without an ID within the imaging area of the 3D camera.

[0019] The trajectory creation unit 30 identifies which human each partial trajectory without identification is by based on the individual identification of the human by the first trajectory acquisition unit 10. In the example of FIG. 3, an ID will be attached to the partial trajectory (moving object position history) within the imaging area of the 3D camera.

[0020] Then, the trajectory creation unit 30 integrates the partial trajectory without identification and the partial trajectory with identification to create the trajectory of the moving object with individual identification. That is, the integrated trajectory as shown in FIG. 3 is created by the trajectory creation unit 30.

[0021] According to the present invention, it is possible to acquire the movement path of a moving object while identifying the moving object with a small number of cameras. That is, generally, a single 3D camera can cover a relatively wide area, but it has the disadvantage that it cannot identify a moving object. In addition, the 3D camera also has the disadvantage of high installation costs. In addition, the marker camera can acquire the movement path of the moving object while identifying the moving object, but has the disadvantage of a narrow coverage area. In the present invention, while taking advantage of the advantage of the 3D camera that it can cover a relatively wide area, it is possible to cover the disadvantage that the moving object cannot be identified. In addition, in the present invention, it is not necessary to cover all areas with marker cameras. Furthermore, in the present invention, it is not necessary to cover all areas with 3D cameras. In any case, according to the present invention, it is possible to acquire the movement path of a moving object while identifying the moving object with a small number of cameras.

[0022] [Embodiment 1] Next, as Embodiment 1, the information processing apparatus 100 described above will be described in more detail. As shown in FIG. 1, the information processing apparatus 100 includes a first movement path acquisition unit 10, a second movement path acquisition unit 20, and a movement path creation unit 30. More specifically, the information processing apparatus 100 has the configuration shown in FIG. 4.

[0023] <First Movement Path Acquisition Unit 10> The first movement path acquisition unit 10 captures a plurality of moving objects with a first camera and acquires a partial movement path with identification of each moving object. In particular, the first movement path acquisition unit 10 acquires a partial movement path with identification of each moving object in a first area. Specifically, the first movement path acquisition unit 10 includes a visible light camera 11, a marker identification and positioning unit 12, a marker position history holding unit 13, a movement path analysis unit 14, and a marker partial movement path holding unit 15.

[0024] The visible light camera 11 (first camera) is a camera that captures visible light, and can also be referred to as an RGB camera, a marker camera, or a marker positioning camera. The visible light camera 11 is a camera for performing position measurement and individual identification of a moving object using a marker attached to the moving object.

[0025] The marker identification and positioning unit 12 decodes the pattern (ID) of the marker shown in the video captured by the visible light camera 11, and based on the size, inclination, and shape distortion thereof, calculates in the real space the direction and the distance from the camera where the marker is located.

[0026] The operating principles of the visible light camera 11 and the marker identification and positioning unit 12 will be described. In the following description, there may be expressions such as "the visible light camera 11 mounted on the marker positioning camera", but the marker positioning camera and the visible light camera 11 may be regarded as the same.

[0027] A marker is a means for attaching or printing on an object, person, etc., and transmitting ID information to a camera. As the marker, an ArUco (registered trademark) marker, a QR (Quick Response) code (registered trademark) of Denso can be applied, but ordinary two-dimensional barcodes, barcodes, color codes, etc. may also be used. What corresponds to the marker here is something that expresses the ID with a pattern of geometric figures such as a combination of a predetermined shape and color. Also, if there is a certain pattern that can uniquely identify an object, such as a human face, fingerprint, livestock nose print, body pattern, physique, etc., these can also be used as markers.

[0028] The visible light camera 11 preferably performs continuous shooting, that is, shoots a video. When the marker is imaged by the visible light camera 11, information on the size of the imaged marker in the imaging range can be obtained. From this information, the marker identification and positioning unit 12 can determine the distance from the camera to the marker (and the moving object to which the marker is attached). If the distance from the marker positioning camera to the tag is short, the marker will appear large, and if the distance from the marker positioning camera to the marker is long, the marker will appear small. That is, since the size at which the marker appears is inversely proportional to the distance from the visible light camera 11 to the marker, the distance from the camera to the marker (and the object to which the marker is attached) can be determined. Also, since it is known where the marker appears in the scene of the captured video of the visible light camera 11, it is known in which direction the marker (and the object to which the marker is attached) is offset from the direction in which the visible light camera 11 is facing. Further, for example, when a marker with a predetermined shape and size, such as a square marker or a circular marker, is used, there will be a difference in shape between when imaged from the front and when imaged from diagonally in front. Therefore, the position of the marker relative to the camera can also be obtained from this difference in shape. For example, if a marker that appears as a square when the camera and the marker are facing each other is imaged, when the marker is displaced from the front of the camera, it will appear deformed into a shape close to a trapezoid or a rhombus.

[0029] In this way, when the marker is imaged by the visible light camera 11 mounted on the marker positioning camera, the distance from the camera can be determined from the size of the marker when it is photographed, and the direction of the marker as seen from the camera can be known from the position and shape of the marker on the captured image. Therefore, the position of the marker (and the object to which the marker is attached) can be determined. The obtained position (positioning target position) information of the marker is mainly added to the marker position history holding unit 13 together with the attached attribute information at any time.

[0030] The marker position history holding unit 13 is data holding means having functions such as addition, deletion, and search of information like a relational database, and is means for holding the position history of the marker (marker position history). Regarding the information to be stored, tabular information will be described as an example, but it may be in a format such as JSON format, a structure in C language, or an object in an object-oriented language. Also, this information only needs to hold the core information necessary for processing, and may be held on a persistent storage such as a hard disk or an SD memory card, or may be held on a volatile memory such as RAM or a temporary memory.

[0031] For example, as illustrated in FIG. 5, the marker position history holding unit 13 holds a marker position history in which a processed flag, a timestamp, a camera ID, a marker ID, and a marker position are associated.

[0032] The processed flag is represented by, for example, two values of true / false. False is set when the marker identification and positioning unit 12 adds an information row, and true is set for the row processed by the flow analysis unit 14.

[0033] The timestamp is registered by the marker identification and positioning unit 12 with the timestamp of the timing when the visible light camera 11 captures an image. If there are implementation issues such as the visible light camera 11 not being able to be equipped with a clock, only the integrated time from the start of shooting can be obtained, or only video data can be sent, the marker identification and positioning unit 12 may add a timestamp instead of the visible light camera 11. For example, the marker identification and positioning unit 12 may calculate and add a timestamp from the clock mounted on itself, the number of frames of video data, etc.

[0034] The camera ID is registered by the marker identification and positioning unit 12 as ID information that uniquely represents the visible light camera 11 that captured the information. Based on this ID, it is possible to distinguish which camera the information row is based on the information captured by. The camera ID may be information that can uniquely identify the visible light camera that captured the video that is the source of the information in that row, such as a number, name, IP address if it is a network camera, or location information of the camera installation location. For example, the camera ID can be determined in advance among each functional unit and used.

[0035] The marker ID is registered by the marker identification and positioning unit 12 with the marker ID information detected and identified by the marker identification and positioning unit 12.

[0036] The marker position is registered by the marker identification and positioning unit 12 with the marker position calculated by the marker identification and positioning unit 12, that is, the position information of the position of the moving object to which the marker is attached.

[0037] The traffic flow analysis unit 14 is a means for obtaining the movement trajectory of the marker in the real space by integrating the position of the marker in the real space obtained by the marker identification and positioning unit 12 along the time axis. The traffic flow analysis unit 14 acquires the history of the position of the marker, that is, the position of the object, from the marker position history holding unit 13, divides it for each marker ID, and arranges it in time series to create traffic flow information within the range that can be tracked by one camera. This traffic flow information can be referred to as "partial traffic flow with identification" or "marker partial traffic flow". The created partial traffic flow is added to the marker partial traffic flow holding unit 15 by the traffic flow analysis unit 14.

[0038] The marker partial trajectory holding unit 15 is data holding means having functions such as addition, deletion, and search of information like a relational database, and is means for holding the partial trajectory of a marker (the trajectory for each marker ID within the range that can be captured by one camera). Regarding the information to be stored, tabular information will be described as an example, but it may also be in a format such as JSON (JavaScript (registered trademark) Object Notation), a structure in C language, or an object in an object-oriented language. Further, this information only needs to hold the core information necessary for processing, and may be held on a persistent storage such as a hard disk or an SD memory card, or may be held on a volatile memory such as RAM (Random Access Memory) or a temporary memory.

[0039] For example, as illustrated in FIG. 6, the marker partial trajectory holding unit 15 holds information in which a marker partial trajectory ID, a processed flag, a timestamp, a camera ID, a marker ID, and a marker position are associated. Note that the information held by the marker partial trajectory holding unit 15 has a data structure in which a marker partial trajectory ID is added to the marker position history held by the marker position history holding unit 13.

[0040] Regarding the marker partial trajectory ID, the same marker partial trajectory ID is input to the marker partial trajectory ID column for each row of the marker position history constituting the same partial trajectory.

[0041] The processed flag is represented by, for example, two values of true / false. False is set at the time when the trajectory analysis unit 14 adds an information row, and true is set by the trajectory integration unit 32 for the row processed by the trajectory integration unit 32.

[0042] Regarding the timestamp, the camera ID, the marker ID, and the marker position, they are the same as those of the marker position history holding unit 13.

[0043] The above is the description of the first movement path acquisition unit 10. Basically, the first movement path acquisition unit 10 can adopt any known technology as long as it can acquire the movement path while identifying the moving object. For example, as the first movement path acquisition unit 10, a "marker location system" can be adopted. Also, it is sufficient if the movement path can be acquired while identifying the moving object without imaging the moving object with a camera. For example, a person may be identified by a terminal ID transmitted from a mobile terminal (such as a smartphone) carried by a person (moving object). It is also conceivable to calculate the distance between the mobile terminal and the access point based on the intensity of the radio wave transmitted from the mobile terminal to the access point, specify the position of the person (moving object), and acquire the movement path.

[0044] <Second movement path acquisition unit 20> The second movement path acquisition unit 20 images the moving object with a second camera and acquires the partial movement paths of each moving object without identification. In particular, the second movement path acquisition unit 20 acquires the partial movement paths of each moving object without identification in a second area adjacent to the first area. Specifically, the second movement path acquisition unit 20 includes a 3D point cloud camera 21, a 3D point cloud positioning unit 22, a moving object position history holding unit 23, a movement path analysis unit 24, and a moving object partial movement path holding unit 25.

[0045] The 3D point cloud camera 21 (the second camera) is typically represented by those such as LiDAR (Light Detection and Ranging) and ToF (Time Of Flight) cameras. It is a mechanism that measures the time from when an invisible light beam such as an infrared laser irradiates a moving object until it is reflected and returns, and determines the distance to the moving object. By scanning the laser beam or irradiating multiple lasers into space, it is a means of obtaining the depth within a certain space and the shape of the moving object as point cloud data. It is desirable for the 3D point cloud camera 21 to perform continuous shooting. In short, it is desirable to shoot a video of the 3D point cloud. When shooting with the 3D point cloud camera 21, 3D point cloud information with the distance (depth) from the camera to the object across the entire shooting range can be obtained. By performing processing such as clustering on the point cloud closer to the object from the camera in the point cloud, a group of 3D point clouds can be obtained, which represent the object and can be detected.

[0046] The 3D point cloud positioning unit 22 performs clustering etc. on the point cloud data obtained by the 3D point cloud camera 21 to group the point clouds. Then, the 3D point cloud positioning unit 22 estimates the object from the shape, size, etc. of the point cloud group, and is a means of calculating from which direction and at what distance from the 3D point cloud camera 21 in the real space the moving object is located. Among the multiple detected objects, the distance of the object that meets the conditions for the moving object to be detected can be obtained as the distance of the moving object. That the conditions are met for the moving object to be detected means that, for example, if a human is to be detected, a 3D point cloud cluster with a size close to the height and width of a human is selected. Representative values of the points constituting the 3D point cloud cluster, such as the average value of the distances of the points, the distance of the point closest to the camera, or the distance of the point closest to the centroid position of the cluster shape, can be considered, and this can be used as the distance of the moving object. The obtained information on the position of the moving object (the position of the positioning target) is mainly added to the moving object position history holding unit 23 along with the attached attribute information at any time.

[0047] The mobile object position history holding unit 23 is data holding means having functions such as addition, deletion, and search of information like a rational database, and is means for holding the position history of the mobile object (history of the mobile object position). Regarding the information to be stored, tabular information will be described as an example, but it may be in a format such as JSON format, a structure in C language, or an object in an object-oriented language. Further, this information only needs to hold the core information necessary for processing, and may be held on a persistent storage such as a hard disk or an SD memory card, or may be held on a volatile memory such as RAM or a temporary memory.

[0048] For example, as illustrated in FIG. 7, the mobile object position history holding unit 23 holds a mobile object position history in which a processed flag, a timestamp, a camera ID, a mobile object ID, and a mobile object position are associated.

[0049] The processed flag is represented by, for example, two values of true / false. False is set at the time when the 3D point cloud positioning unit 22 adds an information row, and true is set by the traffic flow analysis unit 24 for the row processed by the traffic flow analysis unit 24.

[0050] The timestamp is registered by the 3D point cloud positioning unit 22 with the timestamp at the timing of imaging by the 3D point cloud camera 21. When there are implementation circumstances such as the 3D point cloud camera 21 not being able to be equipped with a clock, only the integrated time from the start of shooting can be obtained, and only video data can be sent, the 3D point cloud positioning unit 22 may add a timestamp instead of the 3D point cloud camera 21. For example, the 3D point cloud positioning unit 22 may calculate and add a timestamp from the clock mounted on itself, the number of frames of video data, etc.

[0051] The 3D point cloud positioning unit 22 registers the camera ID, which is ID information uniquely representing the 3D point cloud camera 21 that captured the information. Based on this ID, it is possible to distinguish from which camera the information row is based on the captured information. The ID information uniquely representing the captured 3D point cloud camera 21 may be any information that can uniquely identify the visible light camera that captured the video serving as the source of the information in that row, such as a number, name, IP address if it is a network camera, or location information of the camera installation location. For example, it can be pre-determined among each functional unit and used accordingly.

[0052] Regarding the mobile object ID, since the detected object in the 3D point cloud positioning unit 22 has no ID, "Null" is set.

[0053] Regarding the mobile object position, the 3D point cloud positioning unit 22 registers the position information of the object calculated by the 3D point cloud positioning unit 22.

[0054] The traffic flow analysis unit 24 acquires the history of the mobile object's position from the mobile object position history holding unit 23 and arranges it in time series to create traffic flow information within the range that can be tracked by one camera. This traffic flow information can be referred to as "undetermined partial traffic flow" or "mobile object partial traffic flow". The created partial traffic flow is added to the mobile object partial traffic flow holding unit 25 by the traffic flow analysis unit 24.

[0055] The mobile object partial traffic flow holding unit 25 is data holding means equipped with functions such as addition, deletion, and search of information like a relational database, and is means for holding the partial traffic flow of the mobile object (traffic flow within the range that can be captured by one camera). Regarding the information to be stored, tabular information will be described as an example, but it may also be in a format such as JSON format, a structure in C language, or an object in an object-oriented language. Also, this information only needs to hold the core information necessary for processing and may be held on a persistent storage such as a hard disk or SD memory card, or may be held on a volatile memory such as RAM or temporary memory.

[0056] For example, as illustrated in FIG. 8, the moving object partial trajectory holding unit 25 holds information in which a processed flag, a moving object partial trajectory ID, a timestamp, a camera ID, a moving object ID, and a moving object position are associated. Note that the information held by the moving object partial trajectory holding unit 25 has a data structure in which the moving object partial trajectory ID is added to the position history of the moving object held by the moving object position history holding unit 23.

[0057] The processed flag is represented by, for example, two values of true / false. False is set at the time when the traffic flow analysis unit 24 adds an information row, and true is set by the traffic flow integration unit 32 for the row processed by the traffic flow integration unit 32.

[0058] Regarding the moving object partial trajectory ID, the same moving object partial trajectory ID is input to the moving object partial trajectory ID column for each row of the moving object position history that constitutes the same partial trajectory.

[0059] The timestamp, camera ID, moving object ID, and moving object position are the same as those of the moving object position history holding unit 23.

[0060] The above is the description of the second traffic flow acquisition unit 20. Basically, the second traffic flow acquisition unit 20 can adopt any known technique as long as it can acquire the traffic flow. For example, as the second traffic flow acquisition unit 20, a technique (such as Japanese Patent Application Laid-Open No. 2019-144941) for analyzing the traffic flow of a moving object (such as a person or an animal) using a distance measurement technique called LiDAR (Light Detection and Ranging), ToF (Time Of Flight) camera, etc. can be adopted. Also, it is sufficient if the traffic flow of the moving object can be acquired without imaging the moving object with a camera. For example, it is also conceivable to calculate the distance between the mobile terminal and the access point based on the intensity of the radio wave transmitted from the mobile terminal to the access point to specify the position of the person (moving object) and acquire the traffic flow.

[0061] <Traffic flow creation unit 30> The route creation unit 30 identifies which moving object each partial route without identification belongs to based on the individual identification of the moving object by the first route acquisition unit 10, and integrates the partial route without identification and the partial route with identification to create the route of the individually identified moving object. In particular, the route creation unit 30 selects the partial route with identification and the partial route without identification to be integrated based on the position history of the partial route with identification and the position history of the partial route without identification. Specifically, the route creation unit 30 includes a camera position management unit 31, a route integration unit 32, an integrated route holding unit 33, and a route browsing unit 34.

[0062] The camera position management unit 31 is a means for managing the positional relationship and orientation information of a plurality of cameras installed for grasping the route. The positional relationship and orientation of the cameras may be measured and input by a person at the time of installation by providing a registration means. Alternatively, the cameras may be equipped with position measurement means such as an attitude sensor or GPS, and the results may be automatically registered by aggregating them via an arbitrary communication means. This information is used by the route integration unit 32 when integrating the routes. Note that the camera position management unit 31 may have a configuration separate from that of the route creation unit 30.

[0063] The route integration unit 32 is a means for connecting the routes obtained by the route analysis unit 14 and the route analysis unit 24, using the positional relationship of each camera, the detection position of the end point of the route in the real space, and the detection time of the end point as clues, and connecting them into a series of routes. Here, the routes obtained by the route analysis unit 24 may be connected to each other. That is, as shown in FIG. 3, the route integration unit 32 may connect the partial routes without IDs (i.e., moving object partial routes) within the imaging area of the 3D camera.

[0064] The integrated route holding unit 33 is a means for holding the routes for each moving object, which are integrated and obtained by the route integration unit 32, in a searchable form such as a database. For example, as shown in FIG. 9, the integrated route holding unit 33 holds information in which the integrated route ID, the moving object partial route ID, the marker partial route ID, the timestamp, the camera ID, the moving object ID, the marker ID, and the moving object position are associated with each other.

[0065] The integrated flow line ID is ID information for uniquely identifying the integrated flow line. The same ID is set for each row constituting the same integrated flow line.

[0066] The mobile body partial flow line ID is the same as that of the mobile body partial flow line holding unit 25.

[0067] The marker partial flow line ID is the same as that of the marker partial flow line holding unit 15.

[0068] The time stamp and camera ID are the same as those of the marker position history holding unit 13 and the mobile body position history holding unit 23.

[0069] The mobile body ID is the same as that of the mobile body position history holding unit 23.

[0070] The marker ID is the same as that of the marker position history holding unit 13.

[0071] The mobile body position is the same as that of the mobile body position history holding unit 23.

[0072] The flow line browsing unit 34 is a means for browsing, searching, displaying, etc. the flow lines held by the integrated flow line holding unit 33. A person may search and display from a terminal such as a PC, or project it onto something like a digital signage (electronic billboard) based on defined search rules, etc., or search and provide the flow line information as input data for another system. Note that the flow line browsing unit 34 can also have a configuration separate from the flow line creation unit 30.

[0073] For example, the traffic line browsing unit 34 has a display mechanism for the user and an information input mechanism from the user (in short, a mechanism equivalent to a PC (personal computer), tablet, operation panel, etc.), and displays the integrated traffic line stored in the integrated traffic line storage unit 33 according to the user's request. The traffic line browsing unit 34 provides real-time monitoring of the traffic line by searching and displaying the traffic line information at any time, or by specifying the period or object ID for which the traffic line is to be confirmed and performing a search for visualization. An example of visualization is shown in FIG. 10. Note that the list display is information that lists the moving object ID, time stamp, moving object position, etc. for a plurality of points specified by the user, and is information created by the traffic line browsing unit 34 referring to the moving object position history holding unit 23.

[0074] The information shown in FIG. 10 shows, for example, the traffic lines of three people moving on the floor. This information is obtained, for example, by the arrangement of the imaging area (dashed line frame and filled) by the visible light camera 11 and the imaging area (solid line frame and unfilled) by the 3D point cloud camera 21 as shown in FIG. 11. Here, focusing on the imaging area by the central 3D point cloud camera 21, it can be seen that the person identified by 〇 and the person identified by △ are passing by each other. Since the second traffic line acquisition unit 20 alone does not perform individual identification of people, it is not possible to determine which traffic line is by 〇 (or by △), but it can be determined based on the identified partial traffic line acquired by the first traffic line acquisition unit 10. Therefore, it is desirable to arrange the imaging area by the visible light camera 11 and the imaging area by the 3D point cloud camera 21 adjacent to each other. In other words, it is desirable that the first traffic line acquisition unit 10 acquires the identified partial traffic line of each moving object in the first area, and the second traffic line acquisition unit 20 acquires the unidentified partial traffic line of each moving object in the second area adjacent to the first area.

[0075] Further, it is not necessarily limited to being displayed as a flow line. As shown in FIG. 12, it is also possible to display the position of the object by focusing on a certain moment. Further, in order to make the position easily understandable against the background of the flow line display and the snapshot display, a map, a floor plan, a drawing, a grid line, a square pattern, etc. may be displayed together. Further, the visualization method is not limited to two dimensions. Since the coordinate information is held in three dimensions, the same information may be displayed in three-dimensional computer graphics. In order to make the positional relationship easily understandable, in addition to the object, other objects, for example, if imagining an office floor, the position of the flow line and the object may be displayed together with three-dimensional CG such as chairs and desks.

[0076] The above is the description of the flow line creation unit 30.

[0077] Next, the processing by the information processing apparatus 100 will be described. FIGS. 13 to 15 are diagrams showing the flow of the overall processing by the information processing apparatus 100. According to the notation of UML 2.0, solid arrows represent asynchronous requests, and broken arrows represent responses to requests. When only returning that a request has been received and no particular response is to be returned, the response is omitted according to this notation.

[0078] FIG. 13 is a sequence diagram (UML 2.0 notation) showing the flow of processing and the flow of information between the functional units from the visible light camera 11 to the marker partial flow line holding unit 15. As shown in FIG. 13, the marker identification and positioning unit 12 acquires a time stamp etc. from the image / video at the timing captured by the visible light camera 11 (step S1-1), and registers it in the marker position history holding unit 13 as the marker position history (step S1-2). Note that steps S1-1 and S1-2 are in a loop and the processing is repeated. The flow line analysis unit 14 requests the marker position history holding unit 13 for the unprocessed marker position history (step S1-3), and acquires the unprocessed marker position history list (step S1-4). Then, the flow line analysis unit 14 creates a marker partial flow line and adds it to the marker partial flow line holding unit 15 (step S1-5), and sets the unprocessed flag of the processed line to false (step S1-6). Note that steps S1-3 to S1-6 are in a loop and the processing is repeated.

[0079] FIG. 14 is a sequence diagram (UML 2.0 notation) showing the processing flow and information flow between each functional unit from the 3D point cloud camera 21 to the moving object partial trajectory holding unit 25. As shown in FIG. 14, the 3D point cloud positioning unit 22 acquires a time stamp or the like from the point cloud data obtained by the 3D point cloud camera 21 (step S2-1), and registers it in the moving object position history holding unit 23 as a moving object position history (step S2-2). Note that steps S2-1 and S2-2 are in a loop, and the processing is repeated. The trajectory analysis unit 24 requests the moving object position history holding unit 23 for an unprocessed moving object position history (step S2-3), and acquires an unprocessed moving object position history list (step S2-4). Then, the trajectory analysis unit 24 creates a moving object partial trajectory and adds it to the moving object partial trajectory holding unit 25 (step S2-5), and sets the unprocessed flag of the processed row to false (step S2-6). Note that steps S2-3 to S2-6 are in a loop, and the processing is repeated.

[0080] FIG. 15 is a sequence diagram (UML2.0 notation) showing the processing flow and information flow centered around the flow line integration unit 32 and the flow line browsing unit 34. As shown in FIG. 15, the flow line integration unit 32 requests unprocessed marker partial flow lines from the marker partial flow line holding unit 15 (step S3-1) and acquires an unprocessed marker partial flow line history list (step S3-2). Also, the flow line integration unit 32 requests the camera position management unit 31 to search for camera position information regarding the marker partial flow line (step 3-3) and acquires the camera position information (step S3-4). Then, the flow line integration unit 32 requests unprocessed moving body partial flow lines from the moving body partial flow line holding unit 25 (step S3-5) and acquires an unprocessed moving body partial flow line history list (step S3-6). Also, the flow line integration unit 32 requests the camera position management unit 31 to search for camera position information regarding the moving body partial flow line (step 3-7) and acquires the camera position information (step S3-8). Further, the flow line integration unit 32 requests the integrated flow line holding unit 33 to search for integrated flow lines (step S3-9) and acquires an integrated flow line list (step S3-10). Then, the flow line integration unit 32 integrates each partial flow line and adds the flow line for each moving body as an integrated flow line to the integrated flow line holding unit 33 (step S3-11). Further, the flow line integration unit 32 sets the unprocessed flag of the processed row in the marker partial flow line holding unit 15 to false (step S3-12) and sets the unprocessed flag of the processed row in the moving body partial flow line holding unit 25 to false (step S3-13). Note that steps S3-1 to S3-13 are looped and the processing is repeated. The flow line browsing unit 34 requests the integrated flow line holding unit 33 to search for integrated flow lines (step S3-14) and acquires an integrated flow line list (step S3-15). Note that steps S3-14 to S3-15 can be performed at an arbitrary timing (for example, when receiving a browsing instruction from the user) when wanting to browse the flow line.

[0081] FIGS. 16 to 20 are diagrams showing details of the processing flow by the information processing apparatus 100.

[0082] FIG. 16 is a diagram showing details of a series of processes performed by each functional unit from imaging by the visible light camera 11 to saving the processing result in the marker position history holding unit 13. When the information processing apparatus 100 starts processing (step S4-1), it starts an infinite loop (step S4-2). The visible light camera 11 captures an image to obtain an image, and assigns a camera ID and a time stamp (step S4-3). Note that the marker identification and positioning unit 12 may assign the camera ID and the time stamp in the next step. The marker identification and positioning unit 12 detects a marker from the captured image and decodes it into a marker ID (step S4-4). The marker identification and positioning unit 12 measures the position of the detected marker (step S4-5). Then, the marker identification and positioning unit 12 adds a marker position history (a set of a time stamp, a camera ID, a marker ID, and a marker position) to the marker position history holding unit 13 (step S4-6). Note that the image captured by the visible light camera 11 may be saved in the marker position history holding unit 13 together. Unless the system stops, the information processing apparatus 100 returns to step S4-2 and continues the loop (step S4-7). When the system stops, the information processing apparatus 100 stops the processing (step S4-8).

[0083] FIG. 17 is a diagram showing details of a series of processes performed in each functional unit from imaging by the 3D point cloud camera 21 to saving the processing result in the moving object position history holding unit 23. When the information processing apparatus 100 starts the process (step S5-1), it starts an infinite loop (step S5-2). The 3D point cloud camera 21 performs imaging to acquire an image and acquire point cloud data representing the spatial shape, and assigns a camera ID and a time stamp (step S5-3). Note that the 3D point cloud positioning unit 22 may assign the camera ID and the time stamp in the next step. The 3D point cloud positioning unit 22 performs clustering or the like on the imaged point cloud data to group point clouds that are close to each other (step S5-4). The 3D point cloud positioning unit 22 determines whether the grouped point cloud fits the object for which the size, shape, etc. of the point cloud group are to be detected (such as the size and shape of a person or the size and shape of a vehicle). Then, the 3D point cloud positioning unit 22 obtains the position of the point cloud group corresponding to the moving object (when it is desired to obtain the movement paths of multiple types of moving objects such as people and vehicles, the type of moving object is also determined here from the size and shape) (step S5-5). Then, the 3D point cloud positioning unit 22 adds a moving object position history (a set of a time stamp, a camera ID, and a moving object position) to the moving object position history holding unit 23 (step S5-6). Note that when it is desired to obtain the movement paths of multiple types of moving objects such as people and vehicles, the type of moving object is also saved. The information processing apparatus 100 returns to step S5-2 and continues the loop unless the system stops (step S5-7). When the system stops, the information processing apparatus 100 stops the process (step S5-8).

[0084] FIG. 18 is a diagram showing details of a series of processes from when the traffic line analysis unit 14 acquires information from the marker position history holding unit 13 until the processing result is stored in the marker partial traffic line holding unit 15. Along with the start of processing (step S6-1), the information processing apparatus 100 starts an infinite loop (step S6-2). The traffic line analysis unit 14 acquires a list of marker position histories (sets of time stamps, camera IDs, and object positions) without a processed flag from the marker position history holding unit 13 (step S6-3). The traffic line analysis unit 14 examines the marker IDs in each row of the acquired list of marker position histories, and creates a temporary list in which rows with the same ID are grouped and arranged in time series (step S6-4). The traffic line analysis unit 14 compares adjacent rows in the temporary list, and if the difference in time and the difference in distance are greater than or equal to a certain value, determines that it is not a series of traffic lines, and divides the temporary list there (step S6-5). Note that even when the coordinates are on the edge of the observation area, the temporary list is divided. The determination of being on the edge may have a margin. The divided temporary list becomes a partial traffic line with a marker ID. The traffic line analysis unit 14 assigns a partial traffic line ID to each of the divided temporary lists (step S6-6). The traffic line analysis unit 14 saves the partial traffic lines with the partial traffic line ID assigned to the marker partial traffic line holding unit 15 (step S6-7). Then, the traffic line analysis unit 14 sets a processed flag in the row corresponding to the list of marker position histories without a processed flag acquired in step S6-3 held in the marker position history holding unit 13 (step S6-8). The information processing apparatus 100 returns to step S6-2 and continues the loop unless the system stops (step S6-9). When the system stops, the information processing apparatus 100 stops the processing (step S6-10).

[0085] FIG. 19 is a diagram showing details of a series of processes from when the traffic line analysis unit 24 acquires information from the moving object position history holding unit 23 until the processing result is stored in the moving object partial traffic line holding unit 25. When the information processing apparatus 100 starts processing (step S7-1), it starts an infinite loop (step S7-2). The traffic line analysis unit 24 acquires a list of moving object position histories (a set of time stamps, camera IDs, and object positions) without a processed flag from the moving object position history holding unit 23 (step S7-3). The traffic line analysis unit 24 compares adjacent rows of the list of moving object position histories (a set of time stamps, camera IDs, and moving object positions). If the difference in time and the difference in distance are greater than or equal to a certain value, it determines that they are not a series of traffic lines and splits the temporary list there (step S7-4). The traffic line analysis unit 24 also splits this list when the coordinates cross the edge of the observation area, but the determination of crossing the edge may have a width. The split temporary list becomes the moving object partial traffic line. The traffic line analysis unit 24 assigns a partial traffic line ID to each of the split temporary lists (step S7-5). The traffic line analysis unit 24 saves the partial traffic lines with the partial traffic line IDs assigned to the moving object partial traffic line holding unit 25 (step S7-6). Then, the traffic line analysis unit 24 sets a processed flag in the row corresponding to the list of moving object position histories without the processed flag acquired in S7-3 held in the moving object position history holding unit 23 (step S7-7). The information processing apparatus 100 returns to step S7-2 and continues the loop unless the system stops (step S7-8). When the system stops, the information processing apparatus 100 stops the processing (step S7-9).

[0086] FIG. 20 is a diagram showing details of a series of processes until the flow line integration unit 32 acquires information from the marker partial flow line holding unit 15, the moving body partial flow line holding unit 25, the camera position management unit 31, and the integrated flow line holding unit 33 and stores the processing result in the integrated flow line holding unit 33. The information processing apparatus 100 starts an infinite loop (step S8-2) together with the start of processing (step S8-1). The flow line integration unit 32 acquires an unprocessed marker partial flow line list from the marker partial flow line holding unit 15 (step S8-3). The flow line integration unit 32 acquires camera position information of the camera ID included in the unprocessed marker position history list from the camera position management unit 31 (step S8-4). The flow line integration unit 32 converts the coordinate information of the marker position in the unprocessed marker partial flow line list acquired in S8-3 from camera coordinates to common coordinates (step S8-5). The flow line integration unit 32 acquires an unprocessed moving body partial flow line list from the moving body partial flow line holding unit 25 (step S8-6). The flow line integration unit 32 acquires camera position information of the camera ID included in the unprocessed moving body position history list from the camera position management unit 31 (step S8-7). The flow line integration unit 32 converts the coordinate information of the moving body position in the unprocessed moving body partial flow line list acquired in S8-6 from camera coordinates to common coordinates (step S8-8). The flow line integration unit 32 integrates, as a continuous flow line, partial flow lines at both ends of which the times and positions are close to each other among the partial flow lines of the marker partial flow line list and the moving body partial flow line list that have been converted to common coordinates (step S8-9). The flow line integration unit 32 searches, based on the times and positions at both ends of the continuous flow line obtained in S8-9, for a flow line in the integrated flow line holding unit 33 whose end times and positions are close to those of the continuous flow line obtained in S8-9 and can be regarded as a continuous flow line, and further integrates it with the continuous flow line (step S8-10). The flow line integration unit 32 assigns the same integrated flow line ID to each row constituting the same integrated flow line (step S8-11). At this time, in the case of a flow line integrated with the integrated flow line holding unit 33, the integrated flow line ID of the integrated flow line held by the integrated flow line holding unit 33 is used. The flow line integration unit 32 stores the integrated flow line integrated in S8-11 in the integrated flow line holding unit 33 (step S8-12).The flow line integration unit 32 sets the processed flag for the row corresponding to the unprocessed marker partial flow line list acquired in step S8-3 of the marker partial flow line holding unit 15 to true. Further, the flow line integration unit 32 sets the processed flag for the row corresponding to the unprocessed moving body partial flow line list acquired in S8-6 of the moving body partial flow line holding unit 25 to true (step S8-13). The information processing apparatus 100 returns to step S8-2 and continues the loop unless the system stops (step S8-14). When the system stops, the information processing apparatus 100 stops the processing (step S8-15).

[0087] As described above, according to the information processing apparatus 100 of the present invention, the problems in the following prior art are solved. · When trying to obtain each flow line after identifying an individual using a marker and a visible light camera, although the flow line can be obtained after individual identification, there is a problem that the flow line can only be obtained in a narrow area. · In any method, when trying to capture the flow lines in a wider and arbitrary range without interruption, there is a limit to the range that can be covered by a single camera, and a method of integrating the flow lines captured by multiple cameras becomes a problem. · Also, the 3D point cloud camera used when obtaining the flow line is generally much more expensive than the visible light camera used for marker positioning. Therefore, the problem is how to configure the number of 3D point cloud cameras required to obtain the flow line to be as small as possible, that is, how to complement the expensive camera with an inexpensive visible light camera. · When observing the flow line without identifying the individual, when multiple moving bodies cross in front of the camera, there is an inconvenience that it is impossible to distinguish which moving body the flow line after the crossing belongs to. When trying to obtain each flow line after identifying the individual using a single 3D point cloud camera, a wider range of flow lines can be obtained compared to the marker and the visible light single camera, but since the individual cannot be identified, the problem is to integrate the information obtained from the 3D point cloud camera and the information obtained from the visible light camera so that the flow line can be obtained and the individual can be identified.

[0088] Further, according to the present invention, in addition to solving the above problems, there is an effect that the number of 3D point cloud cameras, which are generally particularly expensive, can be supplemented and reduced by marker cameras using inexpensive ordinary cameras.

[0089] [Modifications] Hereinafter, modifications of Embodiment 1 will be described.

[0090] In Embodiment 1, the description was given by imaging the movement trajectory of a person. However, the object for tracing the movement trajectory is not limited to a person, and may be a vehicle such as an automobile or a heavy machine, a railway vehicle, a movable facility such as a door or a shutter, a luggage, etc. Further, as illustrated in FIG. 21, like a joint of a robot arm, not only the movement trajectory of the entire object but also a part of the movable part may be the object for tracing the movement trajectory. Naturally, fine movement trajectories such as the movement of a human arm or finger, the movement of the head, and the movement of the eyes may be traced, and it can be applied regardless of the length of the movement trajectory. Further, the movement trajectories of animals such as livestock, pets, and wild animals may be traced. That is, the present invention can be applied to applications for tracing the movement trajectories of all moving objects.

[0091] The visible light camera 11 and the marker identification and positioning unit 12 can be replaced with other means as long as they can identify an individual and measure the position of an object. In recent years, there has also appeared a product commonly called a fusion camera that integrates both the visible light camera 11 and the 3D point cloud camera 21. For such a camera, for example, with a function equivalent to that of the visible light camera 11, face authentication, face recognition, face identification, gait authentication, etc. are performed to identify an individual, and the position of the object is captured by the 3D point cloud camera 21, so that the same effect can be obtained and it can be replaced. Further, although face identification was given as an example, for animals such as livestock, instead of individual identification by a marker using the body size, nose pattern, body surface pattern, tag attached to the ear or nose, gait of the animal, etc., it may be used.

[0092] The visible light camera 11 and the marker identification and positioning unit 12 can be replaced with other means as long as they can identify individuals and measure the position of the object. For example, Wi-Fi radio waves emitted by terminals such as smartphones and PCs held by people can be used for position and individual identification. The position can be determined by the field strength received by multiple access points installed with Wi-Fi radio waves, and smartphones and PCs can send their identification information and address information (such as IP addresses and MAC addresses) over the Wi-Fi radio waves, so individual identification can be performed and replacement is possible. Not limited to Wi-Fi, other radio waves such as radio beacons and other wireless communication methods can be used for position measurement and individual identification as long as they can do so.

[0093] The visible light camera 11 and the 3D point cloud camera 21 are not limited to this method as long as they provide equivalent functions. For example, a marker can be created with ink that reacts only to a specific wavelength, and it can be configured using a camera that captures only that specific wavelength. For example, an infrared camera, an ultraviolet camera, etc. A geometric pattern can be made of metal as a marker and embedded inside the object, and it can be photographed by transmitting through an X-ray camera for positioning and individual identification.

[0094] Knowing the movement path of the object means not only understanding how the object moved, but also being able to grasp when and where it stopped, whether it did not move, and for how long it did not move. Therefore, the mechanism of the present invention can also detect and illustrate traffic jams in automobiles, congestion, crowding, and clustering of livestock, etc. Also, if the object is a package, it can be applied to detecting packages that have not moved in the warehouse for a long time, finding expired products, and detecting defective inventory. Also, if the object that does not move for a long time is, for example, livestock, it may be dead or sick, so it can also be used to detect this. Also, the congestion situation in a certain space, such as the degree of crowding in a vehicle or a waiting room, etc., can also be detected and illustrated.

[0095] Even without changing the basic configuration, the above can be done visually. For example, as shown in FIG. 22, if a moving object density detection unit 35, a stationary moving object detection unit 36, and an abnormal behavior moving object detection unit 37 are added, these can be automatically detected to issue a warning to the user, or an operation instruction can be issued to another control device or the like for handling the situation. FIG. 23 shows a display example.

[0096] For example, when the moving object density detection unit 35 is added to the configuration, the information of the integrated traffic line held by the integrated traffic line holding unit 33 is periodically checked. When a certain number or more of objects pass within a certain range of a certain width within a certain time width, a warning can be displayed on the traffic line browsing unit 34 or the like to notify the user that the objects are densely packed. Regarding the specific values of the certain time, certain range, and certain number, the user can set them in advance via the traffic line browsing unit 34 according to the application, or give them to the moving object density detection unit 35 in advance in the form of a definition file or the like.

[0097] For example, when the stationary moving object detection unit 36 is added to the configuration, the information of the integrated traffic line held by the integrated traffic line holding unit 33 is periodically checked. When there is an object that does not move within a certain range of a certain width for a certain time width or more, a warning can be displayed on the traffic line browsing unit 34 or the like to notify the user that the object has stopped moving. Regarding the specific values of the certain time and certain range, the user can set them in advance via the traffic line browsing unit 34 according to the application, or give them to the stationary moving object detection unit 36 in advance in the form of a definition file or the like.

[0098] For example, when the abnormal behavior moving object detection unit 37 is added to the configuration, the integrated movement line information of the integrated movement line holding unit 33 is periodically checked. When a characteristic movement of the object is observed, an operation corresponding to the operation condition is observed, or a movement deviating from the past tendency is observed, a warning is displayed on the movement line browsing unit 34 or the like, and the user can be notified that the object has stopped moving. Regarding specific values such as the determination conditions for characteristic movements, operation conditions, and deviated movements, the user can set them in advance via the movement line browsing unit 34 according to the application, or give them to the abnormal behavior moving object detection unit 37 in advance in the form of a definition file or the like.

[0099] Furthermore, the determination processes and judgment processes performed in the above basic embodiment and other embodiments may be implemented by a program or the like, or may be determined and judged by a learning model such as machine learning.

[0100] Some or all of the above embodiments may be described as follows in the appended claims, but are not limited thereto.

[0101] (Appended Note 1) A first movement line acquisition unit that captures a plurality of moving objects with a first camera and acquires the identified partial movement lines of each moving object, A second movement line acquisition unit that captures the moving objects with a second camera and acquires the unidentified partial movement lines of each moving object, Based on the individual identification of the moving objects by the first movement line acquisition unit, identify which moving object each unidentified partial movement line belongs to, and integrate the unidentified partial movement line and the identified partial movement line to create the movement line of the individually identified moving object. A movement line creation unit, An information processing apparatus comprising:

[0102] (Appended Note 2) The first movement line acquisition unit acquires the identified partial movement lines of each moving object in a first area, The second movement line acquisition unit acquires the unidentified partial movement lines of each moving object in a second area adjacent to the first area, The information processing apparatus according to Appended Note 1.

[0103] (Appendix 3) The flow line creation unit selects the identified partial flow line and the non-identified partial flow line to be integrated based on the position history of the identified partial flow line and the position history of the non-identified partial flow line. The information processing apparatus according to Appendix 1 or 2.

[0104] (Appendix 4) The flow line creation unit selects the identified partial flow line and the non-identified partial flow line to be integrated based on the time history of the identified partial flow line and the time history of the non-identified partial flow line. The information processing apparatus according to any one of Appendices 1 to 3.

[0105] (Appendix 5) A first flow line acquisition step of imaging a plurality of moving objects by a first camera and acquiring an identified partial flow line of each moving object; A second flow line acquisition step of imaging the moving object by a second camera and acquiring a non-identified partial flow line of each moving object; An individual identification of which moving object each non-identified partial flow line is from based on the individual identification of the moving object in the first flow line acquisition step, and a flow line creation step of integrating the non-identified partial flow line and the identified partial flow line to create a flow line of the individually identified moving object. An information processing method including the above.

[0106] (Appendix 6) A first flow line acquisition process of imaging a plurality of moving objects by a first camera and acquiring an identified partial flow line of each moving object; A second flow line acquisition process of imaging the moving object by a second camera and acquiring a non-identified partial flow line of each moving object; An individual identification of which moving object each non-identified partial flow line is from based on the individual identification of the moving object in the first flow line acquisition process, and a flow line creation process of integrating the non-identified partial flow line and the identified partial flow line to create a flow line of the individually identified moving object. An information processing program for causing a computer to execute the above.

[0107] In addition, each disclosure of the above-cited patent documents and the like is incorporated herein by reference and can be used as the basis or a part of the present invention as necessary. Within the scope of the entire disclosure of the present invention (including the claims), modifications and adjustments of the embodiments or examples can be made based on the basic technical idea. Also, within the scope of the entire disclosure of the present invention, various combinations or selections (including partial deletion) of various disclosure elements (including each element of each claim, each element of each embodiment or example, each element of each drawing, etc.) are possible. That is, the present invention naturally includes all modifications and corrections that could be made by those skilled in the art according to the entire disclosure including the claims and the technical idea. In particular, regarding the numerical ranges described in this document, any numerical value or small range included within the range should be construed as specifically described even without separate description. Furthermore, each disclosure item of the above-cited documents can be used in combination with the description items of this document as a part of the disclosure of the present invention in accordance with the gist of the present invention as necessary, and is considered to be included in the disclosure items of this application.

Explanation of Reference Numerals

[0108] 10: First traffic flow acquisition unit 11: Visible light camera 12: Marker identification and positioning unit 13: Marker position history holding unit 14: Traffic flow analysis unit 15: Marker partial traffic flow holding unit 20: Second traffic flow acquisition unit 21: 3D point cloud camera 22: 3D point cloud positioning unit 23: Moving body position history holding unit 24: Traffic flow analysis unit 25: Moving body partial traffic flow holding unit 30: Traffic flow creation unit 31: Camera position management unit 32: Traffic flow integration unit 33: Integrated traffic flow holding unit 34: Traffic flow browsing unit 35: Mobile Object Dense Detection Unit 36: Stationary Mobile Object Detection Unit 37: Abnormal Behavior Mobile Object Detection Unit 100: Information Processing Device

Claims

1. A first trajectory acquisition unit that captures a plurality of moving objects with a first camera and acquires a partial trajectory with identification for each moving object; A second trajectory acquisition unit that captures the moving objects with a second camera and acquires a partial trajectory without identification for each moving object; Based on the individual identification of the moving objects by the first trajectory acquisition unit, the partial trajectory without identification is individually identified as being caused by which moving object, and a trajectory creation unit that integrates the partial trajectory without identification and the partial trajectory with identification to create a trajectory of the individually identified moving object; comprising: The first trajectory acquisition unit includes: A marker identification and positioning unit that calculates the angle and distance of a marker existing with respect to the first camera based on the size, inclination, and shape distortion of the marker shown in the captured video; A marker position history holding unit that holds a position history of the marker indicated by the angle and distance of the marker existing with respect to the first camera; A trajectory analysis unit that obtains a moving trajectory of the marker in the real space by integrating the position history held by the marker position history holding unit along the time axis, and an information processing apparatus.

2. The first trajectory acquisition unit acquires a partial trajectory with identification for each moving object in a first region, The second trajectory acquisition unit acquires a partial trajectory without identification for each moving object in a second region adjacent to the first region, The information processing apparatus according to claim 1.

3. The trajectory creation unit selects the partial trajectory with identification and the partial trajectory without identification to be integrated based on the position history regarding the partial trajectory with identification and the position history regarding the partial trajectory without identification, The information processing apparatus according to claim 1 or 2.

4. The moving path creation unit selects the identified partial moving path and the unidentified partial moving path to be integrated based on the time history of the identified partial moving path and the time history of the unidentified partial moving path. The information processing apparatus according to claim 1.

5. A first moving path acquisition step of imaging a plurality of moving objects with a first camera and acquiring an identified partial moving path of each moving object; A second moving path acquisition step of imaging the moving object with a second camera and acquiring an unidentified partial moving path of each moving object; Based on the individual identification of the moving object in the first moving path acquisition step, identify which moving object each unidentified partial moving path belongs to, and integrate the unidentified partial moving path and the identified partial moving path to create a moving path of the individually identified moving object, including: The first moving path acquisition step includes: Calculating an angle and a distance at which a marker exists with respect to the first camera based on the size, inclination, and shape distortion of the marker shown in the captured video; Holding a position history of the marker indicated by the angle and the distance at which the marker exists with respect to the first camera; Including obtaining a moving path of the marker in the real space by integrating the position history along the time axis. Information processing method.

6. A first moving path acquisition process of imaging a plurality of moving objects with a first camera and acquiring an identified partial moving path of each moving object; A second moving path acquisition process of imaging the moving object with a second camera and acquiring an unidentified partial moving path of each moving object; Based on the individual identification of the moving object in the first moving path acquisition process, identify which moving object each unidentified partial moving path belongs to, and integrate the unidentified partial moving path and the identified partial moving path to create a moving path of the individually identified moving object, and An information processing program for causing a computer to execute. The first flow line acquisition process includes: a process of calculating the angle and distance at which a marker exists with respect to the first camera based on the size, inclination, and shape distortion of the marker shown in the captured video; a process of holding a position history of the marker indicated by the angle and distance at which the marker exists with respect to the first camera; a process of obtaining the movement flow line of the marker in the real space by integrating the position history along the time axis. An information processing program.

Images