Traffic line estimation method, and traffic line estimation system

Abstract

To provide a traffic line estimation method which can more precisely estimate a position of a user based on inertia information measured by a user terminal in an actual environment.SOLUTION: A computer system executes a step of generating a first movement locus 30 based on inertia information output by an inertia measurement apparatus 10, a step of acquiring floor information 40b describing a layout of a movement space 40, a visit position acquiring step of acquiring visit positions (42, 43) where a user stops by in the movement space, a step of setting a floor plan 40a including the floor information and the visit position, and a locus correction step of outputting a second movement locus 60 obtained by correcting the first movement locus based on the floor plan. In the user position acquiring step, the visit position with unclear visit time and the visit position with visit time are acquired. In the locus correction step, the first movement locus is corrected based on a region where the user can move in the movement space, a position of an obstacle 41 where the user cannot move, and the visit position.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a flow line estimation method and an information processing system, and more particularly to a flow line estimation method using an inertial measurement unit and a computer system that performs information processing related to flow line estimation. [Background technology]

[0002] In a real environment (also called "real society" or "real world"), measuring or estimating people's movement lines (sometimes called "movement trajectories") can be used to evaluate the effectiveness of advertising, for example, by determining whether customers who actually visit a retail store purchase a product advertised in the store or stay in front of the shelf where the product is placed. Alternatively, estimating the movement lines of customers and store clerks can be used to develop and improve the layout (floor plan) of the store. In patrol inspections of facilities such as factories, understanding the time that inspectors spend at each inspection location, the time taken from the start to the end of the inspection and the relationship between the route, etc. can be used to improve the inspection route.

[0003] As a technology for measuring or estimating a person's movement line (hereinafter referred to as "movement line estimation technology"), one that uses an AI camera or a beacon is known. As is well known, an AI camera estimates a movement line by detecting a user who is the subject of movement line estimation and the direction of movement of the user based on images captured by a surveillance camera or the like. However, with a movement line estimation technology using an AI camera, it is difficult to estimate a movement line in a wide area beyond the camera's shooting range. In addition, all people who enter the camera's field of view are recorded regardless of whether they have consented to being photographed by the camera. In particular, there are issues with regard to people who do not consent, such as violation of privacy and portrait rights, or not being able to enter the environment where the movement line estimation is being performed or not being able to enjoy the services provided in that environment.

[0004] A beacon is a wireless signal that uses Bluetooth Low Energy (BLE) technology to communicate with a user terminal such as a smartphone carried by a user. When the user terminal detects a beacon, it reads the identifier of the transmitting device contained in the beacon and executes a predetermined process. In this movement line estimation technology using a beacon, application software that executes a predetermined process to estimate a movement trajectory when a beacon is received is installed in the user terminal. When the user terminal receives a beacon, it acquires information such as the signal strength of the beacon and the distance to the transmitting device, and identifies the reception position. Then, the user's smartphone receives the beacon at each location where a transmitting device is installed, and the movement line of the user is estimated. However, there is a limit to the range that a beacon can reach, and it is difficult to estimate movement outside the receivable range of the beacon. In addition, if it is desired to estimate the movement line with high accuracy, it is necessary to install more beacon transmitting devices. Note that the following Patent Document 1 describes one estimation technology using a beacon.

[0005] On the other hand, the technology of estimating a line of movement using an inertial measurement unit (hereinafter sometimes referred to as "IMU"), which is the subject of the present invention, is called inertial navigation or inertial localization (hereinafter sometimes collectively referred to as "inertial localization technology"). The IMU is composed of an accelerometer and a gyroscope, and measures acceleration and angular velocity in three axial directions (three dimensions). The inertial localization technology has been researched mainly in the field of robotics, for example, but it can also be applied to estimating a user's line of movement by using an IMU mounted on a user terminal (smartphone, smart watch, tablet terminal, etc.). Therefore, the inertial localization technology does not require the installation of devices necessary for line of movement estimation, such as a surveillance camera or a beacon transmitter, in the real environment. In the inertial localization technology, the measurement of acceleration and angular velocity continues as long as the user carries the user terminal, so that the user's movement trajectory can be estimated over a wide area. In recent years, a method has been attracting attention in which the IMU of the user terminal is used to estimate the position and movement line with higher accuracy by using a neural network to learn the relationship between the inertial information measured by the user terminal and the actual movement trajectory (for example, see Non-Patent Documents 1 and 2).

[0006] The following non-patent document 3 describes "Scaled Dot-Product Attention" related to an embodiment of the present invention. In addition, in non-patent document 4, in relation to an application example of the present invention, a method of using a service in which a user can scan the barcode of a product with a smartphone in a store and pay at a dedicated register is described. In non-patent documents 5 and 6, application software installed on a smartphone to use the service is described. In non-patent document 7, an example of a shopping cart called a "smart cart" equipped with a barcode reader and eliminating the need for checkout at a register is described. In non-patent document 8, the current trend of smart carts is described. In addition, the following patent document 2 describes a method of determining the current position of a pedestrian by performing map matching based on the context using the detection value of a sensor unit such as an IMU and map information. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] Patent No. 6695064 [Patent Document 2] JP 2020-085783 A [Non-patent literature]

[0008] [Non-Patent Document 1] Yan, H., Herath, S. and Furukawa, Y.: RoNIN: Robust neural inertial navigation in the wild: Benchmark, evaluations, & new methods, ICRA (2020) [Non-Patent Document 2] Herath, S., Caruso, D., Liu, C., Chen, Y. and Furukawa, Y.: Neural inertial localization, CVPR (2022) [Non-Patent Document 3] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, AN, Kaiser, L. and Polosukhin, I.: Attention is all you need, NeurIPS (2017). [Non-Patent Document 4] AEON Co., Ltd., "How does "RegiGo" work, where you don't have to wait in line at the cash register? A detailed explanation of how to use it and how to pay!", [online], [searched June 23, 2023], Internet<URL:https: / / www.aeon.co.jp / column / 20230203 / > [Non-Patent Document 5] AEON Co., Ltd., "Rejigo", [online], [searched June 23, 2023], Internet<URL:https: / / www.regigo.jp / > [Non-Patent Document 6] Bussan Food Service Co., Ltd., "Business Contents", [online], [searched June 23, 2023], Internet<URL:https: / / www.bussanfood.jp / business / > [Non-Patent Document 7] Trial Holdings Co., Ltd., "Trial Regi Cart", [online], [searched June 23, 2023], Internet<URL:https: / / www.trial-net.co.jp / prepaid / regi-cart / > [Non-Patent Document 8] Frontier Management Co., Ltd., "The revolution brought about by Caper's smart cart (Retail Tech) Notable technologies from overseas startup companies 1," [online], [Retrieved June 23, 2023], Internet <URL:https: / / frontier-eyes.online / caper_retail-tech-series_featured_technology_of_abroad_start-up-companies01 / > Summary of the Invention [Problem to be solved by the invention]

[0009] A user terminal equipped with an IMU inevitably measures unnecessary inertial information (noise) due to the user's walking and various operations on the terminal. Inertial position estimation technology is based on the basic principle of estimating a movement trajectory from a starting point to a current position based on a distance obtained by integrating twice the acceleration measured during movement with the measurement start position of the inertial information as the starting point, and an attitude obtained by integrating a displacement angle obtained by integrating the angular velocity. Therefore, even if the noise is minute, there is a problem that an error between a true movement trajectory and an estimated movement trajectory is accumulated. This problem is also present in the method described in Patent Document 2. In addition, in the method described in Patent Document 2, the position of a pedestrian is determined using map information assuming that the pedestrian is present on a walkable passage, so it is difficult to accurately determine the position of a pedestrian in an open area where there is no passage or the passage is wide.

[0010] In the inertial position estimation technology described in the cited documents 1 and 2, a model of a neural network that estimates a movement line from inertial information containing noise is obtained by supervised learning, and then the inertial information measured by an IMU is applied to the model. This reduces errors caused by noise. However, the conditions of the movement space (layout, area, etc.) are different between the environment when the neural network is learning and the real environment. In addition, naturally, the unspecified number of users who are the targets of the actual movement estimation are not test subjects who are aware of the intention of the test at the time of learning. Therefore, the actual movement line of a user in the real environment and the movement line estimated by the neural network based on the inertial information measured by the user's terminal are actually significantly different.

[0011] Therefore, the present invention aims to provide a movement line estimation method that can more accurately estimate a user's position and movement line based on inertial information measured by the user's terminal even in a real environment, and a movement line estimation system that is a computer system for estimating movement lines based on the method. [Means for solving the problem]

[0012] In order to achieve the above object, one aspect of the present invention is a method for estimating a user's movement line in a moving space using a computer system, comprising: The computer system comprises: a trajectory generating step of generating a first movement trajectory based on inertial information outputted by a user terminal equipped with an inertial measurement device and moving in the movement space together with the user; a floor information acquisition step of acquiring floor information describing a layout of the movement space; a visiting position acquiring step of acquiring visiting positions where the user has stopped in the moving space; a floor plan setting step of setting a floor plan including the floor information and the visiting location; a trajectory correction step of correcting the first movement trajectory based on the floor plan to output a second movement trajectory; Run In the visiting position acquisition step, the visiting position with an unknown time and the visiting position with a time are acquired, In the trajectory correction step, the first movement trajectory is corrected based on a movable area in the movement space included in the floor plan, a position of an obstacle that cannot be moved, and the visited position. This is a method for estimating traffic flow.

[0013] The moving space is a store, The computer system is linked to a POS system of the store; A product information acquisition step of acquiring information on a product purchased by a user via the POS system, the floor information includes the locations of product shelves in the store, information on products displayed on each product shelf, and the location of a place where a predetermined operation for a POS system is performed when making a payment; The visiting location acquisition step can also be a traffic flow estimation method in which the visiting location with an unknown time based on the product information acquired in the product information acquisition step and the visiting location with a time attached based on a specified operation time for the checkout are acquired.

[0014] Furthermore, the floor information may be a traffic flow estimation method that includes a recording start position at which the user terminal starts recording inertial information, and obtains the visited position with a time attached based on the time when an instruction to start recording the inertial information is received for the user terminal and the recording start position.

[0015] The moving space is a facility where a patrol inspection is performed, The floor information includes an inspection location in the facility, The user position acquisition step may be a traffic flow estimation method in which the inspection position is acquired as the visiting position, and the visiting position includes the visiting position accompanied by a time based on user input to a specified device at a specified inspection position.

[0016] In the trajectory generating step, a trained neural network may estimate the first movement trajectory.

[0017] In any of the above-described traffic flow estimation methods, the visited locations accompanied by time stamps may include those obtained based on at least one of images captured by an AI camera and beacons received by the user terminal.

[0018] The scope of the present invention also includes a computer system that is configured with one or more information processing devices and estimates a user's movement line in a moving space, and the computer system includes: The information processing device includes a user terminal equipped with an inertial measurement device and moving in the moving space together with the user; storing floor information describing the layout of the travel space; a trajectory generating step of generating a first movement trajectory based on inertial information output by the user terminal; a visiting position acquiring step of acquiring visiting positions where the user has stopped in the moving space; a floor plan setting step of setting a floor plan including the floor information and the visiting location; a trajectory correction step of correcting the first movement trajectory based on the floor plan to output a second movement trajectory; Run In the visiting position acquisition step, the visiting position with an unknown time and the visiting position with a time are acquired, In the trajectory correction step, the first movement trajectory is corrected based on a movable area in the movement space included in the floor plan, a position of an obstacle that cannot be moved, and the visited position. It is a traffic flow estimation system. Effect of the Invention

[0019] According to the present invention, there is provided a method and system for estimating a user's movement line, which can estimate a user's position more accurately based on inertial information measured by the user's terminal even in a real environment. Other advantages will be made clear in the following description. [Brief description of the drawings]

[0020] [Figure 1] FIG. 2 is a diagram illustrating a concept of a flow line estimation method according to an embodiment of the present invention. [Diagram 2] 11A and 11B are diagrams illustrating results of a comparative experiment between a flow line estimation method according to an embodiment and a flow line estimation method according to a comparative example. [Diagram 3] 11A and 11B are diagrams illustrating results of a comparative experiment between a flow line estimation method according to an embodiment and a flow line estimation method according to a comparative example. [Figure 4] 11A and 11B are diagrams illustrating results of a comparative experiment between a flow line estimation method according to an embodiment and a flow line estimation method according to a comparative example. [Diagram 5] 1 is a diagram showing a procedure for estimating a movement trajectory of a user in a store using a flow line estimation method according to an embodiment. [Figure 6] 11 is a diagram specifically illustrating information processing, communication flow, and the like when the traffic line estimation method according to the embodiment is applied to a store. FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0021] An embodiment of the present invention will be described with reference to the accompanying drawings. === Example === The technical idea of ​​the present invention is to use a computer system to correct inaccurate movement trajectories obtained by existing inertial position estimation technologies (such as Inertial Navigation described in Non-Patent Document 1 above) without relying on machine learning, based on information describing the situation (size, shape, arrangement of obstacles, etc.) of the user's movement space, which is an individual test environment including a real environment, and the positions of places actually visited by the user in that information. Note that, hereinafter, information describing the situation of the movement space will be referred to as "floor information," and information in which the floor information is accompanied by the positions of places actually visited by the user (hereinafter sometimes referred to as "visited positions") will be referred to as a "floor plan" or "floor plan information." <Concept of the flow line estimation method according to the embodiment> FIG. 1 is a diagram showing the concept of a flow line estimation method according to an embodiment based on the above technical idea. As shown in FIG. 1, the basic principle of the inertial position estimation technology is to estimate a user's movement trajectory (flow line) based on inertial information measured by an IMU 10 mounted on a user terminal 1 such as a smartphone carried by the user. In the flow line estimation method according to the embodiment, first, a movement trajectory 30 is obtained using an existing inertial position estimation technology, for example, a trained neural network 20 described in the above-mentioned Non-Patent Documents 1 and 2. However, the movement trajectory 30 estimated using the existing inertial position estimation technology in a test environment different from that during learning has many errors compared to the actual trajectory.

[0022] Therefore, the movement trajectory 30 obtained using the neural network 20 is corrected based on a floor plan 40a of the movement space 40 of the user that is the target of the traffic line estimation, such as an arbitrary test environment or an actual environment. The floor information 40b constituting the floor plan 40a is data that is a so-called "flooring" of the movement space 40, and is information including the planar shape and size of the movement space 40, and obstacles 41 (pillars, walls, fixed furniture and fixtures, etc.) installed in the movement space 40. The area where the obstacles 41 are located is a position where the user cannot be present, and the actual movement trajectory is within the area of ​​the movement space 40 and avoids the obstacles 41.

[0023] In summary, the movement path estimation method of the embodiment estimates an accurate movement path 60 of a user in a movement space 40 by applying inertial information continuously measured by an IMU 10 to a neural network 20 described in Reference 1 to correct an (inaccurate) movement path 30 estimated by applying the inertial information continuously measured by an IMU 10 to a neural network 50 which corrects the (inaccurate) movement path 30 using discrete information as clues such as a visited location 42 which is a place where the user actually stopped (can stop, will stop) at an unknown time, shown by a white circle in the figure, a visited location 43 which has a known time, shown by a black circle in the figure, and the location of an obstacle 41 which cannot be stopped at.

[0024] The traffic line estimation method according to the embodiment can be applied to a case where, for example, the sales floor of a store is treated as the movement space 40, and a movement trajectory 60 of a user (customer) moving within the store is estimated. Floor information 40b corresponding to the sales floor of the store includes, in addition to structures such as pillars and walls, product shelves and cash registers as obstacles 41. This floor information 40b is updated from time to time in response to product replacement, renovation, and the like, and the latest floor information 40b is used when estimating the movement trajectory 60.

[0025] Meanwhile, the POS (Point of Sales) system in the store can obtain the purchased items of the user who purchased the product at the store. Then, by checking the purchased items against the floor information 40b, the location of the product shelf where the product purchased by the user is placed can be specified as the location (visited location) 42 actually visited by the user, and information on the visited location with unknown time can be obtained. Furthermore, since the store involves a process of payment, the location and time of which can be specified, a visited location 43 with a known time can be specified based on the location (POS terminal, cash register, etc.) where a specific operation is performed on the POS system during payment and the time of payment, and a visited location with a time attached can be generated. This makes it possible to set a floor plan 40a consisting of the floor information 40b and the visited locations (42, 43) of the user. <Calculation method for estimating traffic flow> The inertial position estimation technology estimates the movement trajectory of the user based on the inertial information measured by the IMU 10. The calculation method for the estimation is as follows: First, a normalized finite two-dimensional space Ω=[0,1] is calculated as the movement space 40, which is the environment in which the user is active. 2 Consider the user's movement trajectory P in a certain time interval [1,T] as a sequence of two-dimensional positions, P=(P1, ,P T ),P t In contrast, the six-dimensional inertial information M consisting of three-dimensional acceleration and three-dimensional angular velocity obtained in time synchronization from a device (smartphone, smartwatch, etc.) equipped with an IMU10 that is freely operated by the user is expressed as M = (m1, ,M T ),m t ∈R 6 This is expressed as:

[0026] In the conventional method of estimating the movement line based on the existing inertial position estimation technology, a neural network that estimates the movement trajectory P from the inertial information M is used to estimate the movement trajectory P from the previously collected data D={M i ,P i} N i=1 The neural inertial localization is obtained by supervised learning from the neural inertial information M obtained by the neural inertial navigation described in non-patent document 1. For example, non-patent document 2 describes Neural Inertial Localization, which obtains a neural network (hereinafter, sometimes referred to as a "correction network") that corrects the error Δ between the movement trajectory Q obtained by the Neural Inertial Navigation described in non-patent document 1 and the actual movement trajectory P. In this Neural Inertial Localization, the movement speed of the user in the world coordinate system is estimated from the inertial information M using the trained neural network shown in non-patent document 1, and the movement trajectory Q=(q1, , q T ) coincides with the actual trajectory P, the error Δ=(δ1, , δ T ),δ t =P t -q t is predicted (corrected).

[0027] The Neural Inertial Localization described in Non-Patent Document 2 has the advantage that, once a correction network for predicting the error Δ is obtained, the movement trajectory P can be estimated from only the inertial information M. However, as described above, conventional movement line estimation methods including Non-Patent Document 2 have a problem that the estimation accuracy of the estimated movement trajectory P decreases when the movement space 40, the situation within the movement space, the user, etc. are different between the learning environment in which the data D was collected and the test environment or the actual environment. Therefore, in the movement line estimation method according to the embodiment, instead of collecting learning data D in advance, floor plan information A=(X, X timed ,Ω obs ) to optimize the correction network for the movement trajectory Q based on the inertial information M.

[0028] In addition, X in the floor plan information A is a visited location 42 of unknown time. Although the specific time is unknown, it is an absolute location x k For example, in the above-mentioned application to a store, the location of a product shelf where a purchased item is placed can be determined by comparing the purchase history from the POS system with the floor information 40b, and the location of an inspection target in the above-mentioned patrol inspection work can be given. A visited location X with an unknown time is given by X=(x j ) J j=1 ,x j ∈Ω.

[0029] X in Floor Plan Information A timed is a visited location 43 with a time stamp, and is the absolute location x of the place that the user has definitely visited. k and the visit time t k The location of the user terminal 1 equipped with the IMU 10 operated by the user and the time of use are paired together. The location of the user terminal 1 is a self-checkout, a checkout counter when electronic payment is made, an automatic ticket gate, etc., and the location of the user terminal 1 is a pair of the location of the user terminal 1 equipped with the IMU 10 operated by the user and the time of use. timed X timed ={(x k ,t k )} K k=1 ,(xk ,t k )∈Ω×N.

[0030] Ω in floorplan information A obs is a set of positions where the user cannot move regardless of time, that is, the above-mentioned obstacle area (hereinafter, sometimes referred to as the "obstacle layout"). Then, the obstacle layout Ω obs is Ω obs This is expressed as ⊂Ω.

[0031] Here, visit position X timed It is considered that only a few points can be obtained for a movement of several hundred meters. Inertial information M from the IMU 10 can generally be obtained at any time with a period of several tens to several hundreds of Hz. In addition to the estimated result of the movement trajectory Q obtained by the trained model described in Non-Patent Document 1, the visited positions X, X in the test environment are also obtained. timed is used as input, and the correction value Δ=(δ1, , δ T The estimation result Q is calculated by integrating the velocity component estimated from the inertial information M, and since errors in the velocity estimation accumulate, a correction value δ t is considered to depend on time t. Also, the estimated position q t Some of the visited positions X∈X timed should be corrected toward the direction of X and X timed It is desirable to be able to describe the relationship in the correction network. Based on the above assumptions, we designed a correction network expressed by the following equations (1) to (3) in "Mathematical formula 1".

[0032]

number

[0033] In the compensation network designed as above, the movement trajectory Q is P'=Q+Δ,p' t =q t +δ t The objective function L(P',A) is defined using the floor plan information A for this correction result, and the parameters of the correction network are minimized using the gradient descent method. Specifically, L is given as in the following formula (4) in "Mathematical formula 2". In addition, L included in formula (4) vp , L obs , L bd , L reg are respectively expressed by the following equations (5) to (8) in “Mathematical Expression 2”.

[0034]

number

[0035] In the above formula (4) of "Mathematical formula 2", L shown in formula (5) vp is P' t and visited location X∈X∪X timed This is a term about the positional relationship with L vp By minimizing the time, we can estimate the time p' for each visited location x if the corresponding time is known. t approach x, and if the time is unknown, the position p' closest to the current position x is selected. t can be made closer to x.

[0036] In formula (4), L shown in formula (6) obsis a term related to the positional relationship with the obstacle, and the corrected position p' t If L is on an obstacle, the loss will be large. obs In obs is p' t is a function that gives the shortest distance to an obstacle if the obstacle is inside it, and 0 if the obstacle is not collided with.

[0037] On the other hand, in formula (4), L shown in formula (7) bd d defined in Ω is the environment Ω=[0,1] 2 p' outside of t When L exists, the distance to the nearest boundary is calculated as the loss. bd is the corrected position p' t When the environment is outside, the loss is large. In equation (4), L shown in equation (8) reg is the regularization term, and the correction δ t By position p' t Suppresses the sudden change of λ vp ,λ obs ,λ bd ,λ reg are weights for each term, and can be appropriately set based on experience, etc.

[0038] In addition, L in formula (5) vp W in t is x and p' t This is the weight determined by the time of the event, and is defined as shown in equation (9) in “Mathematical Expression 3” below.

[0039]

number

[0040] In the formula (9) shown in "Math. 3", the first line is, for a visited position x, if the time t' is known, the position p' at the same time is known. tThe first line indicates that nothing is done if the visited position is not approached, and the third line indicates that, for a visited position x, if the time is unknown, a position p' close to the visited position x is brought closer to x. <Experiment and Evaluation> Next, in order to confirm the effect of correcting the movement trajectory by the above-mentioned correction network, the Inertial Localization Dataset (hereinafter sometimes referred to as the "dataset") described in the above-mentioned non-patent document 2 was used as experimental data. The dataset is composed of data from three different environments: University A (university campus A), University B (university campus B), and Office. University A and University B are outdoor environments, and Office is an indoor environment. The size of each environment is as follows: University A is 62.8 × 84.4 m 2 University B is 57.6 x 147.2 m 2 , and Office: 38.4 x 11.2 m 2 The data sets were acquired by users moving around in each environment while operating two smartphones equipped with IMUs 10 as user terminals 1.

[0041] In general, one of the two smartphones (hereinafter referred to as "smartphone A") is held in the hand and records the actual movement trajectory P using Visual Inertial SLAM. The other smartphone (hereinafter referred to as "smartphone B") is freely used by the participant, for example, by holding it in the hand, putting it in a pocket, or making a phone call. Note that in both smartphones, inertial information M is recorded in time synchronization with the movement trajectory P. The above dataset includes the movement trajectories and all sequences of inertial information M recorded by the two smartphones.

[0042] Then, the above data set was used to estimate a movement trajectory by the movement line estimation method according to the embodiment, and a movement line estimation method using a conventional inertial position estimation technology was used as a comparative example, and an experiment was conducted to compare the movement trajectories estimated by each of the embodiment and the comparative example. As the comparative examples, a method similar to Neural Inertial Navigation described in Non-Patent Document 1 (hereinafter, sometimes referred to as "Comparative Example 1") and a method similar to Neural Inertial Localization described in Patent Document 2 (hereinafter, sometimes referred to as "Comparative Example 2") were used. In addition, when estimating the movement trajectory P by the embodiment and comparative examples 1 and 2, 1 / 6 of all sequences in the above data set was used as test data. The remaining 5 / 6 was used as learning data for the neural network in comparative example 1 and comparative example 2.

[0043] In addition, Comparative Example 1 is a method of estimating the estimated value Q of the movement trajectory using a neural network, and P'=Q is treated as the movement trajectory. Then, the inertial information M of the smartphone B acquired in each of the three environments in the above data set is used to draw the movement trajectory using the neural network described in Comparative Example 1. In addition, the model of the neural network in Comparative Example 1 uses a part of the University A environment as training data.

[0044] Comparative Example 2 is the latest Neural Inertial Localization, and corrects the movement trajectory Q by the above Comparative Example 1 using a transformer network. As described above, Comparative Example 2 assumes a situation in which the environment is the same for learning and testing, and obtains a correction network model by learning different models independently for the above three environments.

[0045] Next, for each test sample used as test data, the floor plan information A is composed of a visited location X with unknown time and a visited location X with time. timedIn addition, we extracted the obstacles and set up an obstacle layout Ωobs. Here, we thinned the movable area in the environment and detected corners to calculate a set of places that the user can visit (maximum 12 points from University A, 10 points from University B, and 11 points from the office) as visited positions X with unknown times. For each test sample, we used a place that was a certain distance from the true movement trajectory as X. We also calculated the visited positions X with time information. timed The start and end points of the true trajectory of each test sample were used as Ωobs. The obstacle layout Ωobs was set by generating a two-dimensional occupancy map of the environment using the method described in Non-Patent Document 1, and then manually dividing the obstacle area into a set of rectangles.

[0046] In the method of the embodiment, the estimated value Q of the trajectory is corrected based on the floor plan information A from the results of the comparative example 1, as in the comparative example 2. In the above correction network in the embodiment, the output layer of the cross attention module is set to 32 dimensions, and the hidden layer of the MLP is set to 64 dimensions and 128 dimensions. The objective function is empirically set to λ vp = 1.0, λ obs = 0.01, λ bd = 1.0, λ reg = 0.01, T = 100.0, and optimization was performed for 2000 steps using Adam with a learning rate of 0.001.

[0047] 2, 3, and 4 respectively show the experimental results for the three different environments, University A, University B, and Office, and in each of Figs. 2 to 4, a movement space 40 is shown in a rectangular frame corresponding to each environment, and floor information 40b is shown in which rectangular blocks indicating obstacles 41 are arranged within the movement space 40. Within the movement space 40 of each environment, a true (actual) movement trajectory (Ground Truth: hereinafter sometimes referred to as "GT70") and movement trajectories (30, 80, and 60) estimated by the methods of Comparative Example 1, Comparative Example 2, and the embodiment are shown by thick solid lines.

[0048] 2 to 4, the movement trajectory 30 estimated by Comparative Example 1 is significantly different from GT70, for example, showing movement outside the movement space 40. In the method of Comparative Example 2 in which the movement trajectory 30 estimated by the method of Comparative Example 1 is corrected, a correction network is trained for each environment, and therefore a movement trajectory 80 closer to GT70 than in Comparative Example 1 is shown. However, it is easily conceivable that the movement trajectory 80 when the correction network of Comparative Example 2 is applied to an environment other than the environment at the time of learning will naturally deviate from GT70.

[0049] On the other hand, in the method of the embodiment, the network for trajectory estimation is not trained from an existing data set, but the movement trajectory 30 of the comparative example 1 is corrected based on the floor information 40b describing the movement space 40 and the obstacles 41 in the movement space 40, the visit location 42 with unknown time, and the floor plan 40a including the visit location 43 with time. In the experimental results corresponding to the embodiment in Figs. 2 to 4, a part of the visit location 42 with unknown time is shown by a white dot, and the visit location 43 with time is shown by a black dot. And the movement trajectory 60 by the method of the embodiment is very similar to the GT70. In this way, in the movement line estimation method according to the embodiment, it is possible to obtain a movement trajectory P (movement trajectory 60) closer to the GT70 by setting the floor plan 40a, without learning a movement trajectory obtained by some conventional movement line estimation method (for example, the comparative example 1) that obtains the movement trajectory Q (movement trajectory 30) from the inertial information M.

[0050] In the above experiment, in order to compare the movement trajectory by the flow line estimation method according to the embodiment with the movement trajectory by other methods, the data of the inertial information M obtained in the past experiment was used, and the floor plan information A was set from the above three environments in the past experiment and the true movement trajectory GT in those environments. However, in the real environment, the size of the real environment, the arrangement of obstacles, and the visit position (X, X) with unknown time and time that can be obtained in the real environment are different. timed ) based on which floor plan information A is set. Actual environments include the above-mentioned stores and facilities where traveling exhibitions and businesses are held.

[0051] Fig. 5 shows a procedure for estimating a user's movement trajectory P in a store using a traffic line estimation method according to an embodiment. As shown in Fig. 5, inertial information M is acquired by an IMU 10 mounted on a user terminal 1 carried by a user (s1), and an (inaccurate) movement trajectory Q is estimated by applying the inertial information M to an existing inertial position estimation technique (s3).

[0052] On the other hand, since the POS system 2 can obtain the items (purchase history) of the products purchased by the user at the store, by including in the floor information 40b the correspondence between the positions of the product shelves and the products, and the position of the checkout location (for example, the installation location of a POS terminal such as a cash register), it is possible to obtain a visited location X (42) with an unknown time based on the purchase history and the floor information 40b, and a visited location X (43) with a time attached based on the time of checkout and the position of the checkout location included in the floor information 40b. timed Also, the position Ω of an obstacle 41 in the store is obtained based on the floor information 40b (s4). obs (s5) and obtain the position Ω of the obstacle. obs and the visit location (X,X timed ) based on which floor plan information A (40a) is set (s6). Then, a calculation is performed to optimize a network that corrects an inaccurate movement trajectory Q estimated by an existing inertial position estimation technique based on the floor plan information A (40a) (s7). Then, as a result of the calculation, a more accurate movement trajectory P is output (s8).

[0053] In this way, the flow line estimation method according to the embodiment is different from the map matching described in the above Patent Document 2 in that the visiting position X with an unknown time and the visiting position X with an unknown time are estimated. timed The traffic line estimation method according to the embodiment is different in that it acquires the visit location X with unknown time and the visit location X with time. timed Based on this and floor plan information A, it is possible to estimate an extremely accurate flow line. ===Application Examples=== The flow of information processing when the traffic line estimation method according to the above embodiment is applied to a store will be specifically described below. Here, it is assumed that a dedicated application software (hereinafter, sometimes referred to as a "dedicated app") is installed in the user terminal 1 for collecting information required to estimate the movement trajectory P in the store and for data communication with a computer for a POS system or a computer that outputs the above inaccurate movement trajectory Q. Note that, although the process of finally estimating an accurate movement trajectory P based on the inertial information M may be performed by the dedicated app, in this application example, it is assumed that the process is performed by a computer (hereinafter, sometimes referred to as a "server device") managed by the store's operating company. It is assumed that the server device is configured to be able to communicate with the user terminal 1 and the POS terminal in the store via a communication network such as the Internet or a LAN.

[0054] FIG. 6 shows an example of a communication procedure in a computer system (hereinafter sometimes referred to as a "traffic line estimation system") consisting of a user terminal 1, a POS terminal 21, a server device 3, and a barcode reader 22. The server device 3 can be composed of a computer constituting the POS system 2, or a computer linked to the POS system 2. Of course, the server device 3 may be a dedicated computer that executes the estimation process of the movement trajectory P while linking with the POS system 2. Here, for ease of explanation, it is assumed that the function of the server device 3 is incorporated in the POS system 2.

[0055] The dedicated app has a function of recording the inertial information M measured by the IMU 10 of the user terminal 1 in the store, and a function of transmitting the inertial information M to the server device 3 in a timely manner. These functions are integrated into an app (for example, an app described in the above non-patent documents 5 and 6) for using the service described in the above non-patent document 4 (hereinafter, sometimes referred to as the "automatic settlement service").

[0056] The user terminal 1 running the dedicated app timely executes information processing and data communication required for the server device 3 to estimate the movement trajectory P while the user uses the above-mentioned automatic payment service in the store. The server device 3 executes processing to estimate the movement trajectory P of the user in the store while providing the automatic payment service to the user via the user terminal 1 based on data communication with the user terminal 1 and the POS terminals constituting the POS system 2. As described above, the server device 3 also serves as the computer constituting the POS system 2. Below, with reference to FIG. 6, information processing in the user terminal 1, POS terminal 21, and server device 3 in this application example, and the contents of data transmitted and received between the various information processing devices (1, 3, 21, 22) constituting the movement line estimation system will be described.

[0057] First, the user terminal 1 starts a dedicated app at a location that is the starting point of the movement trajectory, such as the entrance of a floor of a store (s11). When the dedicated app is started, the user terminal 1 starts recording the inertial information M (s12). In order to make the user terminal 1 start recording the inertial information M at the starting point, a user who visits the starting point must operate the user terminal 1 and start the dedicated app. Here, as with the service described in Non-Patent Document 4 above, the user uses a shopping cart equipped with a smartphone holder. The shopping cart is kept waiting at a predetermined position on each floor of the store, and the shopping cart is used only on that floor. In other words, on another floor, a shopping cart dedicated to that floor is also waiting.

[0058] When a user attaches their own smartphone or a dedicated smartphone always available at the store to the shopping cart as a user terminal 1 at the waiting position of the shopping cart, the user starts a dedicated app and performs an operation to start shopping. When the user uses their own user terminal 1, they can input information for designating the movement space, such as the store or floor, into the dedicated app before starting shopping. In the case of a store-specific smartphone, information such as the store and floor can be input in advance. The dedicated app then starts recording the inertial information M from the point in time when the operation to start shopping is accepted.

[0059] Next, the user moves around the store, picks up the product he or she wants to purchase from the shelves, and places the product in the shopping cart after having the user terminal 1 read the barcode displayed on the product. Each time the user terminal 1 reads the barcode of a product, it records information for identifying the product (hereinafter sometimes referred to as "product information") (s13).

[0060] After putting all the products the user wishes to purchase into the shopping cart, the user proceeds to a booth (cash register area) where multiple POS terminals 21 for well-known automatic checkout, known as "self-checkouts," are installed to check out, and has the user terminal 1 read the two-dimensional barcode 23 posted at the entrance to the cash register area (s14). The user terminal 1 then transmits the code written in the read barcode, terminal identification information for identifying the user terminal 1, such as the serial number of the dedicated app, and the product information recorded in the dedicated app, to the server device 3 via a communication network such as the Internet (s15).

[0061] When the server device 3 receives the various information transmitted from the user terminal 1, it identifies the floor information 40b of the store in which the specified cash register area is located based on the received code, and obtains the floor information from an external storage device attached to the server device 3 or a database installed on a communication network (s22). The server device 3 also identifies an available POS terminal 21 installed in the cash register area by using its function as the POS system 2, and transmits the number of the POS terminal 21 to the user terminal 1 (s23). The server device 3 also transmits payment information (product, quantity, unit price, total amount, etc.) based on the transmitted product information to the POS terminal 21 with the above number (s24). The POS terminal 21 displays the transmitted payment information, such as the product purchased by the user, its quantity, unit price, total amount, etc., on its own display (s31). Meanwhile, the number of the POS terminal 21 transmitted from the server device 3 is displayed on the user terminal 1 (s16), and the user performs payment operations at the POS terminal 21 with that number.

[0062] When the user operates the POS terminal 21 and pays for the purchase price of the product by credit card, electronic money, cash, or the like, the POS terminal 21 executes the settlement process (s32) and transmits a notice to that effect to the server device 3. The server device 3 generates a two-dimensional barcode for the user terminal 1 for which the settlement has been completed (s25) and transmits the barcode to the user terminal 1. The user terminal 1 displays the transmitted barcode (s17), and the user has the barcode read by the barcode reader 22 at the exit of the cash register area (s41). The barcode reader 22 immediately transmits the code written in the read barcode to the server device 3, and the server device 3 transfers a confirmation of the reading of the barcode to the user terminal 1. When the user terminal 1 receives this information, it stops recording the inertial information M (s18) and transmits its own terminal information and the inertial information M recorded up to that point to the server device 3 (s19).

[0063] The server device 3 acquires the time when the reading confirmation was received from the barcode reading device 22 as the time when the user terminal 1 visited the exit of the cash register area (s26), and sets a floor plan based on this acquired time, the previously acquired floor information 40b, and the payment information (s27). The server device 3 outputs a movement trajectory P which is corrected based on the floor plan which has been set and which is estimated based on the inertial information M from the user terminal 1 (s29, s30). That is, the server device 3 acquires two pieces of position information X with times added thereto based on the recording start time of the acquired inertial information M, the reception time of the code transmitted from the barcode reading device 22, the position X of the waiting place of the shopping cart in the identified floor information 40b, and the position X of the exit of the cash register area. timed Then, the calculation for estimating the movement trajectory P based on the flow line estimation method according to the embodiment described above is performed using the position information (X, X timed ), inertia information M, and floor plan information A to estimate a more accurate movement trajectory P of the user. The estimated movement trajectory P may be output as appropriate, for example, by storing it in an external storage device attached to the server device 3 or in a database arranged on the network, or by displaying and outputting it as a flow line as shown in Figs. 2 to 4. This allows the store operating company to use the output movement trajectory P as a reference for future business development and operation, and a reference for evaluating the effectiveness of advertising. ===Other Examples=== Although the embodiments and application examples of the present invention have been described above, the embodiments and application examples of the present invention are not limited to those described above, and include various modified examples and application examples. Furthermore, the above-mentioned embodiments and application examples are intended to easily explain the configuration and information processing content of the traffic line estimation method and traffic line estimation system according to the present invention, and are not necessarily limited to those having all of the configurations and information processing described. It is possible to add other configurations or other information processing, delete part of the configuration or part of the information processing, or replace it with other configurations or information processing. The order of one information processing and another information processing can also be reversed.

[0064] For example, in the above application example, the time to start recording the inertial information is not essential. It is sufficient to at least obtain the exit of the cash register area and the visit time to that position at the time of payment.

[0065] In the above application example, the location of the exit of the cash register area specified based on the floor information 40b and the location information X with time added based on the visit time are generated. timed However, the barcode displayed by the user terminal 1 may be read by the POS terminal 21 where the user has performed the operation for settlement. The location of each POS terminal 21 may be included in the floor information 40b. This provides location information X with time stamp based on the location of the POS terminal 21 where the user performed the operation for settlement and the time when the barcode was read by that POS terminal 21. timed In any case, the floor information 40b only needs to include the location of the place where the user performs a predetermined operation on the POS system 2 when making a payment, such as the exit of the cash register area and the location of the POS terminal 21.

[0066] In the above-mentioned application example, as described above, all or part of the information processing for estimating the movement trajectory P may be executed on the user terminal 1 side. The user terminal 1 and the POS terminal 21 may be configured to be able to communicate with each other. When the information processing for estimating the movement trajectory P is executed on the user terminal 1 side, the latest floor information 40b may be downloaded from the server device 3 at the time when the starting point or the end point of the movement trajectory P is visited. Of course, not limited to the above-mentioned application example, even in the above-mentioned traveling exhibition and business, the user terminal carried by the user may execute all or part of the information processing for estimating the movement trajectory. In any case, the computer system for estimating the movement trajectory P may be composed of at least one information processing device including the user terminal 1.

[0067] In the above application example, if a store has multiple entrances, or if a store has multiple floors and purchases made on one floor can be paid for on another floor, or if payment can be made on a different floor and each floor has multiple entrances, the user may be required to enter the entrance, floor number, or even a combination of a floor number and the entrance to that floor when launching the dedicated app.

[0068] A QR code (registered trademark) or the like including information on the location of the starting point may be posted at the entrance, and by reading the code with the user terminal 1, a dedicated app may be started and floor information 40b may be specified. If the dedicated app is set to be constantly running in the background, a beacon transmitter may be installed at the location of the starting point. Then, when the user terminal 1 receives a beacon, it stores the identifier of the beacon by the function of the running dedicated app, and starts recording the inertial information by the IMU 10. In any case, it is sufficient to be able to specify the start time of recording the inertial information, which is the time that is the starting point of the movement trajectory, and the location of the starting point.

[0069] The functions of the dedicated app may be integrated into an app for using a well-known point service or coupon service provided by a store management company, etc. In such a case, the user terminal displays a barcode on which the serial number of the dedicated app, etc. is described, and the barcode is read by the POS terminal 21 at the time of payment. The POS terminal 21 then transmits the read code, payment information, and its own identification information to the server device 3, and the user terminal 1 transmits terminal identification information and inertia information M to the server device 3 at the time of paying the product price to the POS terminal 21 through an electronic payment service, so that the server device 3 can obtain information required to estimate the movement trajectory P.

[0070] When the flow line estimation method according to the embodiment is applied to the patrol inspection work in the above-mentioned factory or other facility, the patrol inspection work can specify the inspection location as the visit location X with unknown time by including the location of the inspection location corresponding to the inspection list in the target floor information 40b. In addition, the start and end locations of the inspection are often specified. In this case, the inspector inputs the start and end times of the inspection into the server device 3 via the user terminal 1 after the inspection, and the visit location X with the time attached can be specified. timed The floor plan 40a can be set by specifying the time and place of the start and end of the patrol inspection. Alternatively, an inspector carries the user terminal 1 and performs the patrol inspection work, and inputs the inspection confirmation or the start or end of the inspection to the user terminal 1 at a specified inspection location or at the start and end of the inspection, thereby setting the visit location X with the time. timed It is also possible to identify

[0071] The position X with the time acquired in the position estimation method according to the above embodiment timed This is not limited to being based on floor information such as the waiting position of the cart or the position of the POS terminal in the above application example, but may also be based on time and position information obtained by sensing technologies other than IMU10, such as AI cameras or beacons, or a combination of multiple sensing technologies other than IMU10.

[0072] When combining an AI camera with the traffic line estimation method according to the embodiment, for example, the positions and movement directions of an unspecified number of users detected by the AI ​​camera from images captured by security cameras or the like can be collated with the traffic lines of users of the dedicated app estimated by the method of the above embodiment, thereby making it possible to identify users of the dedicated app included in the images captured by the camera. Then, a time-stamped position X is calculated based on the position and time when the identified user was photographed. timed can be obtained.

[0073] In the case of combining beacons with the traffic flow estimation method of the embodiment, in recent years, retail stores and the like have been utilizing BLE standard beacons and other such devices to obtain user location information for purchase promotion, etc., and by measuring the signal strength of a beacon emitted from a transmitter installed in the store with the user terminal 1, a combination of time and approximate location can be obtained.

[0074] In addition, information obtained using sensing technologies other than IMU includes the shooting range of the AI ​​camera, the reception range of the beacon, etc. k In some cases, the visited location may be ambiguous. In such cases, the visited location is treated as a set of coordinates with a certain extent, rather than as a point, so that the visited location X timed In any case, the flow line estimation method according to the embodiment may complement a conventional flow line estimation method using an AI camera, a beacon, or the like.

[0075] In the flow line estimation method according to the above embodiment, the movement trajectory Q to be corrected is obtained from the inertial information M based on the method of Comparative Example 1. However, as is well known, in the field of robotics and the like, there are various methods for estimating the movement trajectory Q from the inertial information M. The essence of the flow line estimation method according to the embodiment is to obtain the movement trajectory Q from the inertial information M by an existing inertial position estimation method, and then to calculate the movement trajectory Q by combining the floor information 40b and the visited position (X, X timed ) and the correction is performed based on floor plan information A (40a). [Explanation of symbols]

[0076] 1 User terminal, 2 POS system, 3 Server device, 10 IMU, 20 Neural network, 21 POS terminal, 22 Barcode reader, 23 Two-dimensional barcode, 30, 60, 70, 80 Movement trajectory, 40 Movement space, 40a Floor plan (floor plan information), 40b Floor information, 41 Obstacles, 42, 43 Visiting location, 50 Correction network

Claims

1. A method for estimating a user's movement line in a moving space using a computer system, comprising: The computer system comprises: a trajectory generating step of generating a first movement trajectory based on inertial information outputted by a user terminal equipped with an inertial measurement device and moving in the movement space together with the user; a floor information acquisition step of acquiring floor information describing a layout of the movement space; a visiting position acquiring step of acquiring visiting positions where the user has stopped in the moving space; a floor plan setting step of setting a floor plan including the floor information and the visiting location; a trajectory correction step of correcting the first movement trajectory based on the floor plan to output a second movement trajectory; Run In the visiting position acquisition step, the visiting position with an unknown time and the visiting position with a time are acquired, In the trajectory correction step, the first movement trajectory is corrected based on a movable area in the movement space included in the floor plan, a position of an obstacle that cannot be moved, and the visited position. Flow line estimation method.

2. The moving space is a store, The computer system is linked to a POS system of the store; A product information acquisition step of acquiring information on a product purchased by a user via the POS system, The floor information includes the locations of product shelves in the store, information on products displayed on each product shelf, and the location of a place where a predetermined operation for a POS system is performed when making a payment, In the visiting location acquisition step, the visiting location with unknown time based on the product information acquired in the product information acquisition step and the visiting location with time based on a predetermined operation time for the settlement are acquired. The method for estimating a traffic line according to claim 1 .

3. The floor information includes a recording start position at which recording of the inertial information is started by the user terminal, acquiring the visited location with a time stamp based on a time when an instruction to start recording the inertial information for the user terminal was received and the recording start location; The method for estimating a traffic line according to claim 2 .

4. The moving space is a facility where a patrol inspection is performed, The floor information includes an inspection location in the facility, In the user position acquisition step, the inspection position is acquired as the visit position, and the visit position includes the visit position accompanied with a time based on a user input to a predetermined device at the predetermined inspection position. The method for estimating a traffic line according to claim 1 .

5. The method for estimating a movement line according to claim 1 , wherein in the trajectory generating step, a trained neural network estimates the first movement trajectory.

6. The traffic line estimation method according to any one of claims 1 to 5, wherein the visited locations with time stamps include those obtained based on at least one of an image taken by an AI camera and a beacon received by the user terminal.

7. A computer system that is configured with one or more information processing devices and estimates a user's movement line in a moving space, The information processing device includes a user terminal equipped with an inertial measurement device and moving in the moving space together with the user; storing floor information describing the layout of the travel space; a trajectory generating step of generating a first movement trajectory based on inertial information output by the user terminal; a visiting position acquiring step of acquiring visiting positions where the user has stopped in the moving space; a floor plan setting step of setting a floor plan including the floor information and the visiting location; a trajectory correction step of correcting the first movement trajectory based on the floor plan to output a second movement trajectory; Run In the visiting position acquisition step, the visiting position with an unknown time and the visiting position with a time are acquired, In the trajectory correction step, the first movement trajectory is corrected based on a movable area in the movement space included in the floor plan, a position of an obstacle that cannot be moved, and the visited position. Traffic flow estimation system.

Images