Position estimation system, imaging system, position estimation method, and program

Abstract

To provide a position estimation system capable of easily estimating a position with high accuracy, an imaging system using the same, a position estimation method, and a program.SOLUTION: A position estimation system includes a gyro sensor, an acceleration sensor that measures acceleration of the gyro sensor in at least three or more axes, and a position estimation processing section that estimates a position of the gyro sensor. The position estimation processing section generates data of acceleration in a vertical direction of the gyro sensor and data of acceleration in a horizontal direction of the gyro sensor on the basis of measurement data of the gyro sensor and measurement data of the acceleration sensor, estimates a movement direction and a movement amount of the gyro sensor on the basis of periodic characteristics of the data of acceleration in the vertical direction and the data of acceleration in the horizontal direction, and estimates the position of the gyro sensor by integrating the movement amount in the movement direction.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a position estimation system, an imaging system, a position estimation method, and a program. [Background technology]

[0002] In inspections of road accessories such as signs and road lighting, as well as buildings, the objects to be inspected are sometimes photographed as inspection records, and then the photographs are attached to a ledger or similar along with the inspection results and location information. In this case, when photographing a large number of objects with very similar shapes, location information is recorded at the same time as the photograph is taken so that each subject can be identified. GPS (Global Positioning System) information is sometimes used to record this location information. However, photographs may also be taken in environments where GPS satellite signals cannot be received, such as indoors or underground.

[0003] Pedestrian Dead Reckoning (PDR) is a method for estimating the photographer's position based on gyro sensor information that reflects the photographer's walking, in order to obtain shooting location information even in environments where GPS information cannot be used.

[0004] As a technology related to PDR, Patent Document 1 discloses a technology for determining position by using a pedometer to calculate the distance traveled by multiplying the number of steps by the stride length, and combining this with the direction of travel measured by a compass. However, this technology has the problem that the estimation of the stride length and direction of travel is prone to be inaccurate.

[0005] As techniques for solving the problems of the technique of Patent Document 1, Patent Documents 2 and 3 disclose a method of determining the distance between two points using known radio wave sources such as GPS satellites, mobile phone base stations, and BLE (Bluetooth (registered trademark) Low Energy) beacons, and calculating the average stride length from the number of steps between them. Furthermore, Patent Document 4 discloses a method of correcting position by taking a photograph of a known facility.

[0006] Patent Document 5 discloses a technique for specifying the optimum timing for measuring the direction.

[0007] Patent Document 6 discloses a technique that enables estimation of a direction even when a terminal equipped with a direction sensor is changed while walking.

[0008] Patent Document 7 discloses a technology for estimating the amount and direction of movement of a pedestrian by a machine learning model using a DNN (Deep Neural Network). [Prior art documents] [Patent documents]

[0009] [Patent Document 1] Japanese Patent Application Publication No. 2-216011 [Patent Document 2] Japanese Patent Application Publication No. 9-89584 [Patent Document 3] Japanese Patent Application Laid-Open No. 2016-218026 [Patent Document 4] Patent Publication No. 2021-139701 [Patent Document 5] Japanese Patent Application Publication No. 11-194033 [Patent Document 6] Patent Publication No. 2021-121781 [Patent Document 7] International Publication No. 2020 / 008878 Summary of the Invention [Problem to be solved by the invention]

[0010] However, in Patent Documents 2 to 4, stride length varies greatly depending on the situation, so the average remains inaccurate. Patent Document 5 assumes that the orientation sensor is fixed to the pedestrian, and has the problem that orientation estimation is not possible if the relationship between the pedestrian, the orientation sensor, and the orientation changes. Patent Document 6 is considered difficult to handle when the pedestrian makes irregular movements such as walking sideways (moving left and right while facing forward) or moving backward (moving backward while facing forward).

[0011] Furthermore, with regard to Patent Document 7, recording acceleration waveforms during walking requires recording at a sufficient sampling rate (e.g., 20 Hz) and a time length sufficient to express the characteristic behavior of walking (e.g., one second for each foot to take one step), and this data is input together with three-axis acceleration and angular velocity. Since the DNN's hierarchy depth increases depending on the amount of data in the input layer, deep hierarchies are required to learn data with a high sampling rate, which increases the computational load during learning. Furthermore, the amount of training data required increases depending on the depth of the hierarchy, making it difficult to prepare this data.

[0012] The present invention has been made in view of the above, and aims to provide a position estimation system that can estimate a position easily and with high accuracy, an imaging system using the same, and a position estimation method and program. [Means for solving the problem]

[0013] In order to solve the above-mentioned problems and achieve the object, one aspect of the present invention is a position estimation system comprising a gyro sensor, an acceleration sensor that measures the acceleration of the gyro sensor in at least three axes, and a position estimation processing unit that estimates the position of the gyro sensor, wherein the position estimation processing unit generates vertical acceleration data of the gyro sensor and horizontal acceleration data of the gyro sensor based on the measurement data of the gyro sensor and the measurement data of the acceleration sensor, estimates the direction and amount of movement of the gyro sensor based on periodic characteristics of the vertical acceleration data and the horizontal acceleration data, and estimates the position of the gyro sensor by accumulating the amount of movement in the direction of movement.

[0014] In the position estimation system, the position estimation processing unit may perform a phase operation on at least one of the vertical acceleration data and the horizontal acceleration data, then evaluate the correlation, and estimate the direction and amount of movement of the gyro sensor based on the correlation evaluation result.

[0015] In the position estimation system, the horizontal acceleration data may include acceleration data in each of two axial directions that are approximately perpendicular to each other in the horizontal direction, and the system may further include a determination unit that performs a phase operation on at least one of the acceleration data in the two axial directions, evaluates a correlation, and determines whether or not to correct the result of estimation by the position estimation processing unit based on the correlation.

[0016] One aspect of the present invention is a position estimation system comprising a gyro sensor, an acceleration sensor that measures the acceleration of the gyro sensor in at least three or more axes, and a position estimation processing unit that estimates the position of the gyro sensor, wherein the position estimation processing unit generates vertical acceleration data of the gyro sensor and acceleration data in two horizontal axial directions that are approximately perpendicular to each other based on the measurement data of the gyro sensor and the measurement data of the acceleration sensor, estimates the direction and amount of movement of the gyro sensor using a trained model generated using teacher data including at least six pieces of data, including three pieces of data obtained by evaluating the correlation between two of the vertical acceleration data and the three acceleration data in the two horizontal axial directions, and three pieces of data obtained by performing a phase operation on at least one of two of the three acceleration data and then evaluating the correlation, and estimates the position of the gyro sensor by accumulating the amount of movement in the direction of movement.

[0017] In the position estimation system, the position estimation processing unit may perform the phase operation using a Hilbert filter.

[0018] One aspect of the present invention is an imaging system including an imaging device and the position estimation system, wherein the gyro sensor measures the angular velocity and imaging direction of the imaging device.

[0019] In the imaging system, the position estimation processing unit may generate orientation information data including information on the orientation of the imaging in a stationary coordinate system, and may include a recording unit that records imaging data taken by the imaging device at a certain time in association with estimated position data of the gyro sensor at a time corresponding to the time.

[0020] The imaging system may further include a correction unit that corrects the result of estimation by the position estimation processing unit based on imaging data obtained by imaging the same subject from different directions while moving.

[0021] One aspect of the present invention is a position estimation method comprising the steps of: generating vertical acceleration data of the gyro sensor and horizontal acceleration data of the gyro sensor based on measurement data of a gyro sensor and measurement data of an acceleration sensor that measures the acceleration of the gyro sensor in at least three axes; estimating the direction and amount of movement of the gyro sensor based on periodic characteristics of the vertical acceleration data and the horizontal acceleration data; and accumulating the amount of movement in the direction of movement to estimate the position of the gyro sensor.

[0022] One aspect of the present invention is a program that causes a computer to execute the following steps: generating vertical acceleration data of the gyro sensor and horizontal acceleration data of the gyro sensor based on measurement data from a gyro sensor and measurement data from an acceleration sensor that measures the acceleration of the gyro sensor in at least three axes; estimating the direction and amount of movement of the gyro sensor based on periodic characteristics of the vertical acceleration data and the horizontal acceleration data; and accumulating the amount of movement in the direction of movement to estimate the position of the gyro sensor. [Effects of the Invention]

[0023] According to the present invention, a position can be estimated easily and with high accuracy. [Brief explanation of the drawings]

[0024] [Figure 1] FIG. 1 is a configuration diagram of an inspection support system according to the first embodiment. [Figure 2] FIG. 2 is a diagram showing a worker performing an inspection work. [Figure 3] FIG. 3 is an explanatory diagram of position estimation in the position information acquisition system shown in FIG. [Figure 4] FIG. 4 is a diagram showing an example of the trajectory of the center of gravity marker of the torso of a pedestrian walking forward. [Figure 5]FIG. 5 is a diagram illustrating an example of time-series data of acceleration and acceleration correlation. [Figure 6] FIG. 6 is a diagram illustrating an example of a position estimated in the inspection support system according to the first embodiment. [Figure 7] FIG. 7 is a diagram showing another example of a position estimated in the inspection support system according to the first embodiment. [Figure 8] FIG. 8 is a configuration diagram of an inspection support system according to the second embodiment. [Figure 9] FIG. 9 is an explanatory diagram of information processing in the determination unit shown in FIG. [Figure 10] FIG. 10 is a diagram showing an example of determination data obtained in the determination unit shown in FIG. [Figure 11] FIG. 11 is a flow diagram of the inspection support system according to the second embodiment. [Figure 12] FIG. 12 is an explanatory diagram of the correction by the correction unit. [Figure 13] FIG. 13 is a diagram showing an image of a subject captured by the inspection support system according to the second embodiment. [Figure 14] FIG. 14 is a diagram illustrating an example of pre-correction position data obtained in the inspection support system according to the second embodiment. [Figure 15] FIG. 15 is a diagram illustrating an example of corrected position data obtained in the inspection support system according to the second embodiment. [Figure 16] FIG. 16 is a configuration diagram of a location information acquisition system according to the third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0025] Hereinafter, embodiments of the present invention will be described with reference to the drawings. Note that the present invention is not limited to the embodiments described below. Furthermore, in the description of the drawings, identical parts are appropriately designated by the same reference numerals, and duplicate explanations are appropriately omitted. It should be noted that the drawings are schematic, and the dimensional relationships and ratios of each element may differ from the actual ones. Even between the drawings, there may be parts with different dimensional relationships and ratios.

[0026] (Embodiment 1) 1 is a configuration diagram of an inspection support system according to embodiment 1. The inspection support system 1000 is an example of an imaging system. The inspection support system 1000 includes a position information acquisition system 100, an imaging device 200, and a recording device 300.

[0027] [Location information acquisition system] First, a description will be given of the position information acquisition system 100. The position information acquisition system 100 includes a sensor unit 110, a position estimation processing unit 120, a clock unit 130, and a transmission unit 140. The position information acquisition system 100 is an example of a position estimation system.

[0028] The sensor unit 110 includes a gyro sensor 111 and an acceleration sensor 112. The gyro sensor 111 measures the angular velocity of the gyro sensor 111 in three axial directions (roll, pitch, and yaw). The acceleration sensor 112 moves in conjunction with the movement of the gyro sensor 111 and measures the acceleration of the gyro sensor 111 in at least three axes. In this embodiment, the acceleration sensor 112 measures the acceleration of the gyro sensor 111 in three axial directions (x, y, and z) in a local coordinate system (a coordinate system fixed to the gyro sensor 111). In this embodiment, the sensor unit 110 can be realized by a known nine-axis gyro sensor. In this embodiment, the sensor unit 110 is attached to an imaging device 200 carried by a worker (pedestrian) walking for inspection. Therefore, the angular velocity measured by the gyro sensor 111 corresponds to the angular velocity of the imaging device 200 and the worker, and the acceleration measured by the acceleration sensor 112 corresponds to the acceleration of the imaging device 200 and the worker. The sensor unit 110 outputs the measured angular velocity and acceleration measurement data to the position estimation processing unit 120 .

[0029] The position estimation processing unit 120 includes a coordinate conversion unit 121, a movement amount estimation unit 122, and a position calculation unit 123, and estimates the position of the gyro sensor 111. The coordinate conversion unit 121 executes a step of generating vertical acceleration data and horizontal acceleration data of the gyro sensor 111 in stationary coordinates based on the input angular velocity measurement data and acceleration measurement data (an example of a step in a position estimation method and program). In this embodiment, the coordinate conversion unit 121 generates acceleration data in two axial directions that are substantially orthogonal to each other in the horizontal direction. The coordinate conversion unit 121 outputs the generated acceleration data to the movement amount estimation unit 122. The coordinate conversion unit 121 also generates orientation information data, which is information on the orientation of imaging in the imaging device 200. The orientation information data includes, for example, orientation vector data representing the orientation of imaging. The coordinate conversion unit 121 outputs the orientation information data to the transmission unit 140.

[0030] The movement amount estimation unit 122 executes a step of estimating the movement direction and movement amount of the gyro sensor 111 based on the periodic characteristics of the input vertical acceleration data and horizontal acceleration data (an example of a step in a position estimation method and program). In this embodiment, the movement direction is two axial directions that are substantially perpendicular to each other in the horizontal direction, and the movement amount is the movement amount per time period corresponding to the measurement cycle of the gyro sensor 111 and the acceleration sensor 112 in each movement direction. The movement amount estimation unit 122 outputs the estimated movement direction and movement amount data to the position calculation unit 123.

[0031] The position calculation unit 123 executes a step of accumulating the input movement amounts of the gyro sensor 111 in each movement direction to estimate the position of the gyro sensor 111 (an example of steps in a position estimation method and program). The position calculation unit 123 outputs data of the estimated position to the transmission unit 140.

[0032] The transmitting unit 140 transmits the estimated position data (position data) and direction information data of the gyro sensor 111 to the recording device 300 by wireless communication such as Bluetooth or wired communication, in association with the time data input from the clock unit 130.

[0033] The above-described location estimation processing unit 120, clock unit 130, and transmission unit 140 can be realized by a personal computer or a mobile information terminal such as a smartphone or tablet terminal. Functional units such as the location estimation processing unit 120 are realized by a combination of hardware and software, with a processor in the personal computer or the like executing a program read from memory. The processor is, for example, a CPU (Central Processing Unit) that performs arithmetic processing. The memory is, for example, a semiconductor memory. The semiconductor memory provides a workspace for the processor when it performs arithmetic processing and stores programs and data. The semiconductor memory is, for example, a RAM (Random Access Memory), a ROM (Read Only Memory), an EPROM (Erasable Programmable Read Only Memory), etc.

[0034] [Imaging device] Next, a description will be given of the imaging device 200. As described above, the imaging device 200 is an imaging device carried by a worker, and includes an imaging unit 210, a clock unit 220, and a transmission unit 230.

[0035] The imaging device 200 captures an image of, for example, an object to be inspected. The transmission unit 230 transmits the captured image data to the recording device 300 by wireless communication such as Bluetooth or by wired communication, in association with time data input from the clock unit 220. The imaging device 200 can be realized by a digital camera, a smartphone, or a tablet terminal.

[0036] [Recording Device] Next, the recording device 300 will be described. The recording device 300 includes a receiving unit 310, an associating unit 320, and a recording unit 330. The receiving unit 310 receives position data and orientation information data associated with time data transmitted from the position information acquisition system 100, and imaging data associated with time data transmitted from the imaging device 200. The associating unit 320 associates imaging data at a certain time with position data and orientation information data at a time corresponding to the certain time. The recording unit 330 records the associated imaging data, position data, and orientation information data.

[0037] The recording device 300 can be realized by a personal computer, a mobile information terminal, or a server device. Functional units such as the association unit 320 are realized by a combination of hardware and software, with a processor in the personal computer or the like reading a program from a memory and executing the program. The recording unit 330 can be realized by an auxiliary recording device such as an HDD (Hard Disk Drive) or SSD (Solid State Drive).

[0038] FIG. 2 illustrates workers performing an inspection. In FIG. 2, worker OP1 carries a digital camera C equipped with a sensor S and takes photographs, while worker OP2 accompanies worker OP1 with a tablet terminal T. Here, sensor S is an example of a means for implementing the sensor unit 110, and digital camera C is an example of a means for implementing the imaging device 200. Furthermore, tablet terminal T is an example of a means for implementing the position estimation processing unit 120, clock unit 130, and transmission unit 140 of the position information acquisition system 100, and the recording device 300. Note that when workers OP1 and OP2 walk together in this manner, the gyro sensor provided in sensor S may be used as the gyro sensor, and the acceleration sensor provided in tablet terminal T may be used as the acceleration sensor. Note that the inspection support system 1000 can also be implemented, for example, with a single mobile information terminal. In this case, a single worker may be used.

[0039] [Position estimation method] Next, position estimation in the position information acquisition system 100 will be specifically described with reference to the explanatory diagram of FIG. 3. First, the coordinate conversion unit 121 receives input of angular velocity measurement data in three axial directions (roll, pitch, yaw) of the gyro sensor 111 and acceleration measurement data (acc_x, acc_y, acc_z) in three axial directions (x, y, z) in the local coordinate system of the gyro sensor 111 from the sensor unit 110. The coordinate conversion unit 121 performs coordinate conversion on the input angular velocity and acceleration measurement data to generate vertical acceleration data (acc_w) and horizontal acceleration data (acc_u, acc_v) of the gyro sensor 111 in a stationary coordinate system (uvw coordinates), and outputs these to the movement amount estimation unit 122. Here, the positive u-axis direction is, for example, eastward, the positive v-axis direction is northward, and the positive w-axis direction is upward. Furthermore, acc_u, acc_v, and acc_w are all time-series data.

[0040] The movement amount estimation unit 122 includes multiplication units 122a and 122b, a Hilbert filter 122c, and low-pass filters (LPFs) 122d and 122e. The Hilbert filter 122c generates data (H(acc_w)) by performing a Hilbert transform on acc_w. The Hilbert transform is a transformation that shifts the phase by 90° and is an example of phase manipulation for acceleration data. The multiplication unit 122a outputs multiplied data obtained by multiplying acc_u by H(acc_w) to the LPF 122d. The LPF 122d performs a smoothing process on the input multiplied data. Similarly, the multiplication unit 122b outputs multiplied data obtained by multiplying acc_v by H(acc_w) to the LPF 122e. The LPF 122e performs a smoothing process on the input multiplied data. The steps of multiplication by the multiplication units 122a and 122b and smoothing processing by the LPFs 122d and 122e are an example of a step of evaluating correlation of data.

[0041] Figure 4 shows an example of the trajectory of the center of gravity marker on the torso of a pedestrian walking forward (see Saito et al., "Gait Analysis and Movement Analysis," p. 56, published by the Rehabilitation Department of Fujita Health University, 2015). Figure 4(a) shows the trajectory when the pedestrian is viewed from above, Figure 4(b) shows the trajectory when the pedestrian is viewed from the right, and Figure 4(c) shows the trajectory when the pedestrian is viewed from behind. The solid line indicates the stance phase of the right leg, the dashed line indicates the swing phase of the right leg, point P1 indicates the initial contact with the ground, and point P2 indicates toe-off.

[0042] As can be seen from Figure 4(b), the center of gravity of the torso of a walker walking forward traces a counterclockwise circular trajectory. This means that the period of the movement of the center of gravity in the forward direction (forward / backward movement) and the period of the movement of the center of gravity in the upward direction (up / down movement) are the same, with a phase difference of approximately 90 degrees. On the other hand, as can be seen from Figure 4(a), for forward / backward movement and left / right movement, the center of gravity traces a figure-eight trajectory, and the period of the forward / backward movement is about half that of the left / right movement. On the other hand, as can be seen from Figure 4(c), for up / down movement and left / right movement, the center of gravity traces a pendulum-like trajectory, and the period of the up / down movement is about half that of the left / right movement.

[0043] The inventors of the present invention have focused on the characteristics of walking from the results of FIG. 4 and found that, for example, when a pedestrian (gyro sensor 111) is walking eastward, the east direction is the walking direction (forward), so acc_u and H(acc_w) are in phase and highly correlated, and (acc_u)·H(acc_w) takes a relatively large positive value. Also, when a pedestrian is retreating westward, the movement of the circular trajectory is clockwise, so (acc_u)·H(acc_w) takes a relatively large negative value. Also, when a pedestrian is walking eastward, the north direction is the left-right direction, so acc_v and H(acc_w) act like pendulums with different periods, resulting in a low correlation, and therefore (acc_v)·H(acc_w) takes a relatively small value ((acc_v)·(acc_w) also takes a small value). Therefore, the moving direction of the pedestrian (gyro sensor 111) can be estimated from (acc_u)·H(acc_w) and (acc_v)·H(acc_w). (acc_u)·H(acc_w) and (acc_v)·H(acc_w) are examples of correlation evaluation results. The fact that the center of gravity traces a trajectory as shown in Figure 4 is an example of the vertical acceleration data and horizontal acceleration data having periodic characteristics with a predetermined relationship.

[0044] Furthermore, since the absolute value of acceleration (for example, |acc_u|) reflects the walking speed, the magnitude of (acc_u)·H(acc_w) reflects the pedestrian's stride (amount of movement). As a result, the movement amount estimation unit 122 can estimate the pedestrian's movement direction and movement amount. Specifically, the movement amount estimation unit 122 outputs Δu, which is the amount of movement eastward, and Δv, which is the amount of movement northward, as estimated data.

[0045] 4 shows a case where the pedestrian is walking forward, but even if the pedestrian is walking sideways, the center of gravity will trace a circular locus in the direction of movement and in the up and down directions as shown in FIG. 3(b). Therefore, the movement amount estimation unit 122 can make suitable estimations even when the pedestrian is walking irregularly, such as walking sideways.

[0046] The Hilbert filter 122c and LPFs 122d and 122e may be realized by a convolution operation such as an FIR (Finite Impulse Response) filter, or may be realized by converting the signal onto the frequency axis using an FFT or the like and then performing a phase operation. The same applies to the LPFs 122d and 122e.

[0047] Next, position calculation unit 123 includes accumulators 123a and 123b. Accumulator 123a accumulates the amount of movement in the u direction and outputs u as estimated data of the pedestrian's position in the u direction. Accumulator 123b accumulates the amount of movement in the v direction and outputs v as estimated data of the pedestrian's position in the v direction. In this way, estimated data of the pedestrian's position is obtained.

[0048] 5A and 5B are diagrams showing an example of time-series data of acceleration and acceleration correlation obtained when a pedestrian is walking in embodiment 1. Fig. 5A shows time-series data of acceleration (acc_u, acc_v, acc_w), and Fig. 5B shows time-series data of acceleration correlation. The acceleration correlation is, for example, (acc_u)·H(acc_w) smoothed using an LPF, and is shown in the figure as, for example, LPF((acc_u)·H(acc_w)).

[0049] As can be seen from Figure 5(a), the acceleration value fluctuates in a short cycle due to the body swaying slightly up and down and side to side when walking. Therefore, a sampling rate of about 10 Hz to 20 Hz is required to adequately capture the characteristics of walking using acceleration. In contrast, Figure 5(b) shows the results of extracting walking characteristics by evaluating acceleration correlation, which results in slower changes than acceleration. As a result, the sampling rate for position estimation can be significantly reduced, which simplifies the data processing involved and reduces the processing load.

[0050] In the inspection support system 1000 configured as described above, suitable support for inspection work can be realized by data in which the position data estimated simply and highly accurately by the position information acquisition system 100 and the associated image data and orientation information data of the subject of the inspection target are associated with the position data. For example, when there are multiple inspection targets, data that appropriately reflects their positional relationships can be recorded in the recording device 300.

[0051] [Example of location estimation result] Next, an example of position estimation by the inspection support system 1000 according to the first embodiment will be described. The inspection support system 1000 according to the first embodiment was constructed using a digital camera, a 9-axis gyro sensor, and a laptop PC. An operator held the digital camera equipped with the 9-axis gyro sensor in both hands in front of his chest and performed measurements while walking, as shown in FIG. 6(a). That is, the operator first started the measurement from a reference position (0), then (1) walked 1.5 m west-northwest from the reference position, (2) walked 6 m north-northeast, (3) made a U-turn on the spot, (4) walked 6 m south-southwest, and (5) returned to the reference position to finish the measurement.

[0052] Figure 6(b) shows the position estimation results when measurements similar to those in Figure 6(a) are performed. In Figure 6(b), the horizontal axis indicates the eastward position, and the vertical axis indicates the northward position. The solid line indicates the estimated pedestrian's trajectory, and the arrow indicates the orientation vector corresponding to the digital camera's imaging direction at that position; for example, [16:09:37] indicates the time. Note that the discrepancy between the measurement start position and the measurement end position in Figure 6(b) is likely due to some measurement noise and can be ignored. As can be seen from Figures 6(a) and (b), highly accurate position estimation corresponding to the actual walking of a worker was achieved.

[0053] (Embodiment 2) The results shown in Figure 6(b) are from a case where the worker moved while holding the digital camera in front of his chest with both hands. In contrast, when the worker moved while holding the digital camera in one hand, the results shown in Figure 7 were obtained for the measurements shown in Figure 6(a). In Figure 7, the estimated trajectory is shifted in the negative direction of the u-axis compared to Figure 6(b). The results in Figure 7 are thought to be due to the fact that the movement of the hand hanging from the shoulder is added to the movement of the torso when the digital camera is equipped with a 9-axis gyro sensor.

[0054] In contrast to this, the second embodiment relates to an inspection support system that can correct the result of position estimation even when the result shown in FIG. 7 is obtained for the measurement shown in FIG. 6(a).

[0055] 8 is a configuration diagram of an inspection support system according to embodiment 2. The inspection support system 1000A is an example of an imaging system. The inspection support system 1000A has a configuration in which the position information acquisition system 100 in the inspection support system 1000 according to embodiment 1 is replaced with a position information acquisition system 100A, and the recording device 300 is replaced with a recording device 300A.

[0056] The position information acquisition system 100A has a configuration in which a determination unit 150 is added to the position information acquisition system 100. The recording device 300A has a configuration in which a correction unit 340 is added to the recording device 300.

[0057] 9 is an explanatory diagram of information processing in the determination unit 150 shown in FIG. 8. The determination unit 150 includes a multiplication unit 151, a Hilbert filter 152, an LPF 153, and a comparison / determination unit 154. The Hilbert filter 152 generates data (H(acc_w)) by performing a Hilbert transform on acc_w input from the coordinate transformation unit 121. The multiplication unit 151 multiplies acc_u input from the coordinate transformation unit 121 by H(acc_v) to output multiplied data to the LPF 153. The LPF 153 performs a smoothing process on the input multiplied data to generate determination data and outputs the determination data to the comparison / determination unit 154. The comparison / determination unit 154 compares the value of the determination data with a threshold, and outputs determination information, which is a determination result as to whether or not to correct the estimation result, to the transmission unit 140.

[0058] FIG. 10 is a diagram showing an example of LPF((acc_v)·H(acc_u)), which is determination data obtained by determination unit 150. As shown in FIG. 10, when a digital camera serving as imaging device 200 is held with both hands, LPF((acc_v)·H(acc_u)) is a value close to zero in all cases of moving forward, backward, and sideways. In contrast, when imaging device 200 is held with one hand (right hand), LPF((acc_v)·H(acc_u)) sometimes becomes a negative value with a relatively large absolute value. This is thought to be because, when imaging device 200 is held with one hand (right hand), imaging device 200 and sensor unit 110 attached thereto make a circular motion in the horizontal direction with the shoulder as a fulcrum.

[0059] Therefore, the comparison / determination unit 154 compares the absolute value of the determination data with a threshold value, and outputs a determination result data indicating that the estimation result needs to be corrected (correction required) if the absolute value of the determination data is equal to or greater than the threshold value (positive value).On the other hand, if the absolute value of the determination data is less than the threshold value, the comparison / determination unit 154 outputs a determination result data indicating that the estimation result does not need to be corrected (correction not required).

[0060] The transmitting unit 140 associates the position data and direction information data of the sensor unit 110 with the time data and determination result data input from the clock unit 130, and transmits them to the recording device 300A.

[0061] In recording device 300A, as in recording device 300, receiving unit 310 receives position data, orientation information data, and determination result data associated with time data transmitted from position information acquisition system 100A, and imaging data associated with time data transmitted from imaging device 200. Associating unit 320 associates imaging data at a certain time with position data, orientation information data, and determination result data at a time corresponding to the certain time. Recording unit 330 records the associated imaging data, determination result data, position data, and orientation information data.

[0062] Furthermore, the correction unit 340 executes the processing flow shown in FIG. 11, and corrects or does not correct the position data according to the determination result data.

[0063] First, in step S101, the correction unit 340 reads out from the recording unit 330 a data set including associated imaging data, determination result data, position data, and orientation information data.

[0064] Next, in step S102, the correction unit 340 determines whether the read data sets include a data set whose position data needs to be corrected, based on the determination result data. If it is determined that a data set that needs correction is included (step S102, No), the correction unit 340 ends the processing flow. If it is determined that a data set that needs correction is not included (step S102, Yes), the processing proceeds to step S103.

[0065] Next, in step S103, the correction unit 340 selects imaging data of the same subject and data associated therewith. There are various methods by which the correction unit 340 can identify imaging data of the same subject from multiple imaging data. For example, the correction unit 340 may identify imaging data of the same subject based on time data associated with the imaging data. In this case, the correction unit 340 may identify imaging data captured within a certain period as imaging data of the same subject. Furthermore, the correction unit 340 may perform the identification based on information (e.g., file name) added to the imaging data as a record of the inspection work, or may perform the identification using AI processing or image processing.

[0066] Next, in step S104, the correction unit 340 corrects the position data that needs to be corrected based on the orientation vector data included in the orientation information data. Next, in step S105, the correction unit 340 outputs the corrected position data to the recording unit 330.

[0067] An example of correcting position data based on orientation vector data will be described with reference to FIG. 12. For example, FIG. 12(a) shows position data and an orientation vector obtained by position information acquisition system 100A. Positions A, B, and C are locations corresponding to position data expressed in the uv coordinate system, and it is assumed that the same subject was captured at positions A, B, and C using imaging device 200. Furthermore, it is assumed that the position data corresponding to position A is determined to be position data that does not require correction, and the position data corresponding to positions B and C is position data that is determined to be position data that requires correction. A worker carrying imaging device 200 walks from position A to position B to position C, and the dashed lines in the figure represent the worker's walking trajectory. The arrows in the figure indicate the orientation vectors of imaging device 200 at positions A, B, and C. Because the same subject was captured, the three dashed-dotted lines extending from the three arrows should converge to a single point. However, due to estimation errors in the position data, the intersections of the dashed-dotted lines do not coincide, as shown at intersections AB, AC, and BC.

[0068] In this case, as shown in FIG. 12(b), the correction unit 340 rotates the line segment AB around position A as the central axis without changing its length. After the rotation, position B becomes position B'. Furthermore, the correction unit 340 rotates the line segment BC around position B' as the central axis without changing its length. After the rotation, position C becomes position C'. As a result of this rotation, the intersections AB, AC, and BC become intersections AB', AC', and B'C', but the distance between these intersections is shorter than the distance between points AB, AC, and BC. This means that by correcting position B to position B' and position C to position C', the three dashed dotted lines come closer to converging to a single point, and therefore positions B and C have been corrected to approach more accurate positions.

[0069] Therefore, the correction unit 340 evaluates the distances between multiple intersections formed when the arrows of the orientation vectors are extended at positions where the same subject is imaged, and repeats the correction shown in Fig. 12 until the distances become equal to or less than a predetermined value or the intersections coincide. Fig. 12(c) shows a state where position B' is further corrected to position B'', and position C' is further corrected to position C'', where the three intersections AB'', AC'', and B''C'' coincide.

[0070] In the inspection support system 1000A configured as above, corrected position data can be obtained, thereby realizing more suitable support for inspection work.

[0071] FIG. 13 is a diagram showing a schematic diagram of an image of a subject captured by the inspection support system 1000A. As shown in FIG. 13, the inspection worker captured images of a road sign O, which is attached to a support pole, from various directions as the subject to be inspected at each time point after 13:31:10. Note that FIGS. 13(c) and 13(d) are images of the base of the support pole. In contrast, FIG. 14 is a diagram showing an example of pre-correction position data obtained by the inspection support system 1000A. At the points indicated by the arrows in the diagram, there was data in which the position did not match, even though images of the road sign O were captured.

[0072] In contrast, Fig. 15 is a diagram showing an example of corrected position data obtained in the inspection support system 1000A. In Fig. 15, the directions of the arrows converge to approximately the same point, which indicates that suitable correction has been achieved.

[0073] (Embodiment 3) 16 is a configuration diagram of a position information acquisition system according to embodiment 3. The position information acquisition system 100B includes a sensor unit 110 and a position estimation processing unit 120B. The position estimation processing unit 120B includes a coordinate conversion unit 121, a preprocessing unit 122B, and an estimation unit 123B.

[0074] The sensor unit 110 and the coordinate conversion unit 121 have the same configuration as the corresponding units in the first and second embodiments, and therefore a description thereof will be omitted.

[0075] The preprocessing unit 122B includes multiplication units 122Ba and 122Bb, a Hilbert filter 122Bc, and LPFs 122Bd and 122Be. The preprocessing unit 122B receives input of the vertical acceleration data (acc_w) and horizontal acceleration data (acc_u, acc_v) of the gyro sensor 111 from the coordinate conversion unit 121, and generates three types of IQ data from these data.

[0076] Specifically, the Hilbert filter 122Bc performs a Hilbert transform on acc_0, which is one of acc_u, acc_v, and acc_w, to generate H(acc_0). The multiplication unit 122Ba integrates acc_0 with acc_1, which is another one of acc_u, acc_v, and acc_w, to generate acc_0·acc_1. The LPF 122Bd smooths acc_0·acc_1 to generate LPF(acc_0·acc_1) ("I" in the figure). The multiplication unit 122Bb integrates acc_1 with H(acc_0) to generate acc_1·H(acc_0). The LPF 122Be smooths acc_1·H(acc_0) to generate LPF(acc_1·H(acc_0)) ("Q" in the figure). The IQ data is an example of at least six data including three data in which the correlation between two of acc_u, acc_v, and acc_w is evaluated, and three data in which the correlation is evaluated after performing a phase operation on at least one of two of acc_u, acc_v, and acc_w.

[0077] The estimation unit 123B receives input of these six types of IQ data as correlation data. The estimation unit 123B is equipped with a trained model that has been machine-learned using the IQ data and the position of the gyro sensor of the sensor unit 110 as training data and the estimation results (i.e., u, v) of the movement direction and movement amount of the gyro sensor of the sensor unit 110 as output parameters. Such a trained model can be realized using a DNN, for example, as in Patent Document 7. Therefore, the estimation unit 123B can output estimation data (i.e., u, v) of the movement direction and movement amount of the gyro sensor of the sensor unit 110.

[0078] Here, if there is no pre-processing unit and the estimation unit uses a trained model that has been machine-learned using training data including acceleration data (acc_u, acc_v, and acc_w), a deep hierarchy is required for learning as described above, and the computational load during learning increases. In contrast, in this embodiment, the training data includes data (i.e., IQ data) resulting from correlation evaluation by the pre-processing unit 122B. Since such data changes slowly as shown in FIG. 6(b), it is possible to simplify the hierarchy and reduce the computational load. Furthermore, the computational load during machine learning and the amount of training data required are also reduced.

[0079] In the inspection support system 1000B configured as above, it is possible to realize position estimation with reduced computation load and the like.

[0080] In the above embodiment, the Hilbert transform is used as the phase manipulation when evaluating the correlation, but the phase manipulation is not limited to a transformation that shifts the phase by 90°, and any transformation that shifts the phase can be used so that the correlation that reflects the periodic characteristics of the acceleration data can be evaluated.

[0081] Furthermore, the present invention is not limited to the above-described embodiments. The present invention also includes configurations in which the above-described components are appropriately combined. Furthermore, further effects and modifications can be easily derived by those skilled in the art. Therefore, the broader aspects of the present invention are not limited to the above-described embodiments, and various modifications are possible. [Explanation of symbols]

[0082] 100, 100A, 100B: Location information acquisition system 110: Sensor unit 111: Gyro sensor 112: Acceleration sensor 120, 120B: Position estimation processing unit 121: Coordinate conversion section 122: Movement amount estimation section 122B: Pre-processing section 122a, 122b, 122Ba, 122Bb, 151: Multiplication section 122c, 122Bc, 152: Hilbert filter 122d, 122e, 122Bd, 122Be, 153:LPF 123:Position calculation unit 123a, 123b: accumulator 123B: Estimation section 130, 220: Clock section 140, 230: Transmitter 150: Judgment section 154: Comparison judge 200: Imaging device 300, 300A: Recording device 310: Receiving unit 320: Association section 330: Recording section 340: Correction unit 1000, 1000A, 1000B: Inspection support system C: Digital camera O: Road sign OP1, OP2: Workers S: Sensor T: Tablet device

Claims

1. A gyro sensor, an acceleration sensor for measuring the acceleration of the gyro sensor in at least three axes; a position estimation processing unit that estimates a position of the gyro sensor; Equipped with The position estimation processing unit generating vertical acceleration data of the gyro sensor and horizontal acceleration data of the gyro sensor based on the measurement data of the gyro sensor and the measurement data of the acceleration sensor; estimating a direction and amount of movement of the gyro sensor based on periodic characteristics of the vertical acceleration data and the horizontal acceleration data; The position of the gyro sensor is estimated by accumulating the amount of movement in the movement direction. Location estimation system.

2. The position estimation processing unit A phase operation is performed on at least one of the vertical acceleration data and the horizontal acceleration data, and then a correlation is evaluated, and a moving direction and a moving amount of the gyro sensor are estimated based on the correlation evaluation result. The location estimation system of claim 1 .

3. the horizontal acceleration data includes acceleration data in each of two axial directions that are substantially perpendicular to each other in the horizontal direction, The position estimation processing unit further includes a determination unit that performs a phase operation on at least one of the two-axis acceleration data, evaluates a correlation, and determines whether or not to correct the result of estimation by the position estimation processing unit based on the correlation. The location estimation system of claim 1 .

4. A gyro sensor, an acceleration sensor for measuring the acceleration of the gyro sensor in at least three axes; a position estimation processing unit that estimates a position of the gyro sensor; Equipped with The position estimation processing unit generating acceleration data in a vertical direction of the gyro sensor and acceleration data in two axial directions that are substantially perpendicular to each other in a horizontal direction of the gyro sensor based on the measurement data of the gyro sensor and the measurement data of the acceleration sensor; a trained model is generated using training data including at least six pieces of data, including three pieces of data obtained by evaluating the correlation between two of the vertical acceleration data and the three horizontal acceleration data, and three pieces of data obtained by performing a phase operation on at least one of two of the three acceleration data and then evaluating the correlation; and a trained model is generated using training data including at least six pieces of data, including the vertical acceleration data and the three horizontal acceleration data, and The position of the gyro sensor is estimated by accumulating the amount of movement in the movement direction. Location estimation system.

5. The position estimation processing unit performs the phase operation using a Hilbert filter.

5. The position estimation system according to claim 2.

6. An imaging device; A position estimation system according to any one of claims 1 to 4; Equipped with The gyro sensor measures the angular velocity and imaging direction of the imaging device. Imaging system.

7. the position estimation processing unit generates orientation information data including information on the orientation of the image capture in a stationary coordinate system; a recording unit that records image data captured by the imaging device at a certain time and data on the estimated position of the gyro sensor at a time corresponding to the time in association with each other; The imaging system according to claim 6 .

8. A correction unit is provided that corrects the result of estimation by the position estimation processing unit based on image data obtained by capturing images of the same object from different directions while moving. The imaging system according to claim 7 .

9. generating vertical acceleration data of the gyro sensor and horizontal acceleration data of the gyro sensor based on measurement data of the gyro sensor and measurement data of an acceleration sensor that measures acceleration of the gyro sensor in at least three axes; estimating a direction and amount of movement of the gyro sensor based on periodic characteristics of the vertical acceleration data and the horizontal acceleration data; estimating a position of the gyro sensor by integrating the amount of movement in the movement direction; A location estimation method comprising:

10. generating vertical acceleration data of the gyro sensor and horizontal acceleration data of the gyro sensor based on measurement data of the gyro sensor and measurement data of an acceleration sensor that measures acceleration of the gyro sensor in at least three axes; estimating a direction and amount of movement of the gyro sensor based on periodic characteristics of the vertical acceleration data and the horizontal acceleration data; estimating a position of the gyro sensor by integrating the amount of movement in the movement direction; A program that causes a computer to execute the following.

Images