Information processing device, information processing method, and program

Abstract

One example of a purpose of the present disclosure is to improve the accuracy of location information when adding the location information of a capture location to moving-image data.　An information processing device (10) comprises: a first trajectory generation unit (11) that calculates the location of a moving body (20), which captures moving-image data, following a time series and using positioning data received by the moving body (20) from a positioning satellite (40), and that connects locations in the time-series order to generate a trajectory of the moving body (20); a feature point extraction unit (12) that extracts a feature point in each frame of the moving-image data; an environment map generation unit (13) that identifies a set of corresponding feature points between frames and calculates three-dimensional coordinates of the feature points, so as to generate an environment map constituted by a collection of the feature points; a second trajectory generation unit (14) that generates a trajectory of a frame using the environment map; a capture location identification unit (15) that superimposes the trajectory of the moving body and the trajectory of the frame to identify a capture location for each frame; and a location information addition unit (16) that adds information indicating a location to each frame.

Description

[Technical Field]

[0001] This disclosure relates to an information processing device and an information processing method for processing video data, and further to a program for realizing these. Mu To relate to. [Background technology]

[0002] Conventionally, techniques have been proposed to extract corresponding pairs of feature points from multiple still images obtained by photographing an object from different angles, and then use these extracted pairs of feature points to create 3D point cloud data of the object. Furthermore, techniques have also been proposed to generate 3D point cloud data of an object using video instead of multiple still images.

[0003] Unlike the former technique which uses still images, the latter method using video does not require multiple shots; it only requires recording the entire object in video. For this reason, the latter method using video is useful when generating 3D point cloud data of large and complex structures such as plants and bridges.

[0004] Incidentally, when generating 3D point cloud data from video, the shooting position changes from frame to frame because the video is shot while moving. Therefore, in order to extract corresponding pairs of feature points between frames, it is necessary to specify the shooting position for each frame.

[0005] For example, Patent Document 1 discloses a device for determining the position of a moving object that is taking video. The device disclosed in Patent Document 1 determines the position of a moving object equipped with a camera using a GPS (Global Positioning System) receiver, and then records the shooting time and positioning time of each frame. Compare the frames and, based on the comparison results, determine the position of each frame. [Prior art documents] [Patent Documents]

[0006]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0007] However, in the device disclosed in the above Patent Document 1, since the position of the moving object is specified by general GPS positioning, there is a problem that the accuracy of the specified position is low. This will be specifically described below.

[0008] First, in general GPS positioning, a GPS receiver receives radio waves from three positioning satellites alone. And since the transmission time is added to the received radio waves by the positioning satellites, the GPS receiver calculates the difference between the reception time, which is the time when the GPS receiver itself received the radio waves, and the transmission time for each positioning satellite, and calculates the distance to the positioning satellite based on the difference. Then, the GPS receiver calculates its own position from the positions of each positioning satellite and the distances to each positioning satellite.

[0009] At this time, the accuracy of time measurement in the GPS receiver is lower than the accuracy of the time added to the radio waves from the positioning satellites, and an error occurs in the reception time. Also, when the GPS receiver receives radio waves reflected by obstacles, an error also occurs in the reception time due to this. In such a situation, an error also occurs in the distances to each positioning satellite calculated by the GPS receiver, so as a result, the device disclosed in the above Patent Document 1 has a problem that the accuracy of the specified position is low.

[0010] An example of the object of the present disclosure is to improve the accuracy of position information when adding position information of a shooting position to video data.

Means for Solving the Problems

[0011] To achieve the above object, an information processing device according to one aspect of the present disclosure is A first trajectory generation unit calculates the position of a moving object that captures video data using positioning data received from multiple positioning satellites, in a time-series order, and further generates the trajectory of the moving object by connecting the calculated positions in chronological order. A feature point extraction unit extracts feature points from each frame that makes up the aforementioned video data, An environment map generation unit identifies corresponding sets of feature points between the aforementioned frames, calculates the 3D coordinates of the identified set of feature points, and uses the calculation results to generate an environment map composed of the set of identified feature points. A second trajectory generation unit generates the trajectory of the frame using the aforementioned environment map, A shooting position identification unit superimposes the trajectory of the moving object generated by the first trajectory generation unit and the trajectory of the frame generated by the second trajectory generation unit, thereby identifying the position corresponding to the shooting position of each frame. A position information adding unit adds information indicating the specified position to each frame, It is characterized by having the following features.

[0012] Furthermore, in order to achieve the above objectives, the information processing method in one aspect of this disclosure is: A first trajectory generation step involves a moving object capturing video data using positioning data received from multiple positioning satellites to calculate the position of the moving object in chronological order, and further generating the trajectory of the moving object by connecting the calculated positions in chronological order. A feature point extraction step is performed to extract feature points for each frame that makes up the aforementioned video data. An environment map generation step involves identifying corresponding pairs of feature points between the aforementioned frames, calculating the 3D coordinates of the identified feature points, and using the calculation results to generate an environment map composed of the set of identified feature points. A second trajectory generation step involves generating the trajectory of the frame using the aforementioned environment map, A shooting position identification step involves superimposing the trajectory of the moving object generated by the first trajectory generation step and the trajectory of the frame generated by the second trajectory generation step, thereby identifying the position corresponding to the shooting position of each frame. For each frame, a position information addition step is performed, in which information indicating the specified position is added to the frame. It is characterized by having the following:

[0013] Furthermore, in order to achieve the above objectives, one aspect of this disclosure program teeth, On the computer, A first trajectory generation step involves a moving object capturing video data using positioning data received from multiple positioning satellites to calculate the position of the moving object in chronological order, and further generating the trajectory of the moving object by connecting the calculated positions in chronological order. A feature point extraction step is performed to extract feature points for each frame that makes up the aforementioned video data. An environment map generation step involves identifying corresponding pairs of feature points between the aforementioned frames, calculating the 3D coordinates of the identified feature points, and using the calculation results to generate an environment map composed of the set of identified feature points. A second trajectory generation step involves generating the trajectory of the frame using the aforementioned environment map, A shooting position identification step involves superimposing the trajectory of the moving object generated by the first trajectory generation step and the trajectory of the frame generated by the second trajectory generation step, thereby identifying the position corresponding to the shooting position of each frame. For each frame, a position information addition step is performed, in which information indicating the specified position is added to the frame. Let's execute it ru, It is characterized by the following: [Effects of the Invention]

[0014] As described above, this disclosure makes it possible to improve the accuracy of location information when adding location information of the shooting location to video data. [Brief explanation of the drawing]

[0015] [Figure 1] Figure 1 is a schematic diagram showing an example of an information processing device. [Figure 2] Figure 2 is a configuration diagram showing an example of an information processing device and its surrounding equipment. [Figure 3] Figure 3 shows an example of the process for generating the trajectory of a moving object. [Figure 4] Figure 4 shows an example of the process of overlaying trajectories. [Figure 5] Figure 5 shows an example of the process for adding location information. [Figure 6] Figure 6 is a flowchart illustrating an example of the operation of an information processing device. [Figure 7] Figure 7 is a block diagram showing an example of a computer that implements an information processing device. [Modes for carrying out the invention]

[0016] (Embodiment) The information processing apparatus, information processing method, and program in the embodiment will be described below with reference to Figures 1 to 7.

[0017] [Device configuration] First, we will explain the schematic configuration of an example of an information processing device using Figure 1. Figure 1 is a configuration diagram showing the schematic configuration of an example of an information processing device.

[0018] As shown in Figure 1, the information processing device 10 is a data generation support device for assisting in the generation of 3D point cloud data from video data. As shown in Figure 1, the information processing device 10 includes a first trajectory generation unit 11, a feature point extraction unit 12, an environment map generation unit 13, a second trajectory generation unit 14, a shooting location identification unit 15, and a location information addition unit 16.

[0019] First, the first trajectory generation unit 11 calculates the position of the moving object, which is capturing video data, in chronological order using positioning data (hereinafter also referred to as "first positioning data") received from multiple positioning satellites. Next, the first trajectory generation unit 11 generates the trajectory of the moving object by connecting the calculated positions in chronological order.

[0020] The feature point extraction unit 12 extracts feature points from image data for each frame that makes up the video data captured by the moving object.

[0021] The environment map generation unit 13 identifies corresponding pairs of feature points between frames and calculates the 3D coordinates of the identified feature points. Furthermore, the environment map generation unit 13 uses the calculation results to generate an environment map composed of the set of identified feature points.

[0022] The second trajectory generation unit 14 generates the frame trajectory using the environment map generated by the environment map generation unit 13.

[0023] The shooting position identification unit 15 superimposes the trajectory of the moving object, generated by the first trajectory generation unit 11, and the trajectory of the frame, generated by the second trajectory generation unit 14. Then, by superimposing these trajectories, the shooting position identification unit 15 identifies the position corresponding to the shooting position of each frame.

[0024] The location information addition unit 16 adds location information (hereinafter referred to as "location information") to each frame, indicating the location identified by the shooting location identification unit 15.

[0025] In this embodiment, the information processing device 10 determines the position on a frame-by-frame basis by matching the frame trajectory obtained from the environmental map with the trajectory of the moving object obtained from positioning data from positioning satellites. Therefore, the accuracy of the position information can be improved when adding position information of the shooting location to video data.

[0026] Next, we will specifically explain the configuration and functions of an example of an information processing device using Figures 2 to 5. Figure 2 is a configuration diagram showing an example of an information processing device and its surrounding equipment.

[0027] As shown in Figure 2, in this embodiment, the mobile object that captures video data is an unmanned aerial vehicle (UAV) 20 equipped with a camera. Hereafter, it will also be referred to as the mobile object 20. The UAV 20 is equipped with a GNSS (Global Navigation Satellite System) receiver and receives positioning data (first positioning data) from each of the positioning satellites 40. The video data captured by the UAV 20 is stored in the mobile object database 21. The first positioning data received by the GNSS receiver of the UAV 20 is also stored in the mobile object database 21.

[0028] Base station 30 is installed at a fixed point. Base station 30 is also equipped with a GNSS receiver, which receives positioning data (second positioning data) from each of the positioning satellites 40. The second positioning data received by base station 30 is stored in base station database 31. Base station 30 also receives satellite orbit data from each of the positioning satellites, which indicates the position of the positioning satellite at each time point. Base station database 31 also stores the satellite orbit data.

[0029] Furthermore, each positioning satellite 40 transmits positioning data at a fixed frequency, and the first and second positioning data are received at set intervals. Therefore, the mobile database 21 stores the first positioning data for each positioning satellite in chronological order. The base station database 31 stores the second positioning data (carrier phase data and route data) and satellite orbit data for each positioning satellite in chronological order.

[0030] In this embodiment, the first trajectory generation unit 11 uses, in addition to the first positioning data, second positioning data received by the base station 30 from each positioning satellite 40. The first trajectory generation unit 11 then calculates the position of the unmanned aerial vehicle 20 in chronological order based on the relative relationship between the first positioning data and the second positioning data for each positioning satellite 40, and the position of the fixed point where the base station 30 is installed.

[0031] More specifically, the first trajectory generation unit 11 first acquires the stored first positioning data from the mobile object database 21. The first trajectory generation unit 11 also acquires the stored second positioning data and satellite orbit data from the base station database 31.

[0032] Furthermore, the first trajectory generation unit 11 generates first observation data including the carrier phase of the first positioning data, and second observation data including the carrier phase of the second positioning data. The first and second observation data each include the pseudo-distance from the GNSS receiver to the positioning satellite 40 and the carrier phase. The first trajectory generation unit 11 then uses the first observation data, the second observation data, and the satellite orbit data to determine the relative relationship between the first positioning data and the second positioning data for each positioning satellite in a time series. Furthermore, the first trajectory generation unit 11 calculates the position of the unmanned aerial vehicle 20 in a time series based on the determined relative relationship and the position of the fixed point.

[0033] Here, the position calculation process of the unmanned aerial vehicle 20 by the first trajectory generation unit 11 will be described in detail below. Furthermore, the following references will be made as appropriate in the following explanation. Reference: Tomoji Takasu (Tokyo University of Marine Science and Technology), "A.4 RTK-GPS and Network-Type RTK-GPS Positioning" "Technology"<https: / / gpspp.sakura.ne.jp / paper2005 / gpssymp_2007a.pdf>

[0034] First, as mentioned above, the observational data includes pseudodistance and carrier phase. Here, as shown in the above-mentioned literature, pseudodistance is defined as the signal propagation time measured by the positioning code (RPN code) multiplied by the speed of light.

[0035] The carrier phase is determined from the difference between the phase of the received carrier wave and the phase of the GNSS receiver's reference oscillator. The carrier phase allows us to determine the wavenumber at which the received radio waves arrived, based on the phase at which they were transmitted. By multiplying this wavenumber by the wavelength, the distance from the positioning satellite 40 can be calculated.

[0036] The pseudo-distance (observed value) P and carrier phase (observed value) φ of the positioning satellite s received by the GNSS receiver r are expressed by the observation equation shown in Equation 1 below. In Equation 1 below, ρ is the geometric distance between the positioning satellite and the receiver [m], and c is the speed of light [m / s]. The geometric distance ρ is obtained from satellite orbit data. dt is the time error of the GNSS receiver [s], and dT is the clock error of the positioning satellite [s]. I is the ionospheric delay [m], T is the tropospheric delay [m], and λ is the carrier wave. Wavelength "m", ε is the observation error "m", and N is the carrier phase bias [number of cycles].

[0037]

number

[0038] Furthermore, in the observation equation for relative positioning in Equation 1 above, once the double difference of the carrier phase (double phase difference) is determined, the carrier phase bias N becomes an integer as the initial phase term is eliminated. Hereafter, the carrier phase bias N will also be referred to as "integer ambiguous N".

[0039] Here, the carrier phases of positioning satellites a and b, measured almost simultaneously by two GNSS receivers, u (base station 30) and r (unmanned aerial vehicle 20), are defined as φ. u a , φ u b , φ r a , φ r b is set as, and the pseudo distance is p u a , p u b , p r a , p r b is set as. Here, the superscript indicates the positioning satellite 40, and the subscript indicates the GNN receiver (observation point). The carrier phase double difference φ ur ab and the pseudo distance double difference p ur ab are defined by the following Equation 2.

[0040] [Equation]

[0041] Also, the above Equation 2 can be transformed as the following Equation 3.

[0042] [Equation]

[0043] Here, the carrier phase double difference φ ur ab and the pseudo distance double difference p ur ab are as shown in the following Equation 4 using the notation of the above Equation 1.

[0044] [Equation]

[0045] Also, considering that the positioning data of each positioning satellite 40 is received simultaneously in each GNSS receiver, the relationship shown in the following Equation 5 holds.

[0046] [Equation]

[0047] Furthermore, considering that the transmission times of positioning data are approximately simultaneous for each positioning satellite 40, and that the positioning satellite clocks are sufficiently stable within a short period of time, the following equation 6 also holds true.

[0048]

number

[0049] By applying equations 5 and 6 above to equation 4 above, the observation equations for carrier phase and pseudo-distance double difference are as shown in equation 7 below. In equation 7 below, the terms for positioning satellite clock error and receiver clock error are eliminated from the carrier phase double difference.

[0050]

number

[0051] Here, we consider the case where the distance (baseline length ur) between the GNSS receiver u (base station 30) and the GNSS receiver r (unmanned aerial vehicle 20) is sufficiently close. When positioning data from the same positioning satellite 40 is received at the same time at two sufficiently close locations, the atmospheric propagation path of the positioning signal (signal of positioning data), the ionosphere at each location, and the tropospheric delay at each location will be approximately the same. Therefore, the approximation shown in Equation 8 below becomes possible.

[0052]

number

[0053] Then, by applying equation 8 above to equation 7 above, the observation equations for carrier phase and pseudo-distance double difference can be approximated as shown in equation 9 below.

[0054]

number

[0055] Here, the position of the GNSS receiver r (unmanned aerial vehicle 20) is defined as r.u The received carrier wave L i The single-difference integer ambiguity between receivers is N i Let's assume that the estimated parameter x is represented by the following equation 10, and the double difference observable vector y is represented by the following equation 11.

[0056]

number

[0057]

number

[0058] Apply equations 10 and 11 above to equation 9 above. This allows the observation equation and partial derivative matrix of the double difference observable vector y to be expressed as follows.

[0059]

number

[0060]

number

[0061] The observation error covariance matrix R of the double difference observation is as shown in Equation 14 below.

[0062]

number

[0063] The first trajectory generation unit 11 uses the observation equation described above and an extended Kalman filter to estimate the unknown parameter x of each GNSS receiver x k We find the hat. The equations for correction and updating by the extended Kalman filter in each GNSS receiver k are expressed as follows using equations 12 to 14 above.

[0064]

number

[0065] In the above number 15, x k - The hat is the pre-estimated vector of the unknown parameter x in the GNSS receiver k. k + The hat is the posterior estimate vector of the unknown parameter x at the GNSS receiver k. k - This is the prior error covariance matrix in the GNSS receiver k. Yes. k + This is the posterior error covariance matrix at the GNSS receiver k. k is the Kalman gain matrix at the GNSS receiver k. I is the identity matrix.

[0066] Furthermore, using the extended Kalman filter, we can determine the position of each GNSS receiver, its integer ambiguous estimate x-hat, and its covariance matrix P. The integer ambiguous values ​​obtained here are real-valued estimates that do not take into account the constraint of integer conditions, and the simultaneously obtained estimate of the GNSS receiver positions is called a FLOAT solution. Since the GNSS receiver u (base station 30) is fixed, its position is known in advance. Therefore, only the position of the GNSS receiver r (unmanned aerial vehicle 20) is estimated by the FLOAT solution.

[0067] The first trajectory generation unit 11 calculates integer solutions to integer ambiguity using the integer least squares method with the FLOAT solution and covariance matrix P as input, and then calculates the final FIX solution. The FIX solution becomes the final position of the unmanned aerial vehicle 20.

[0068] Specifically, a single-difference integer ambiguous is transformed into a double-difference integer bias estimate x' hat, as shown in Equation 16 below. In Equation 16 below, D is the transformation matrix for transforming from single-difference to double-difference.

[0069]

number

[0070] The above number 16 can be rewritten as the following number 17. This separates the double-difference integer ambiguous estimate x' hat and its covariance matrix Q into real and integer variables.

[0071]

number

[0072] Here, the least squares method for integers is applied. The first trajectory generation unit 11 then defines the optimal integer solution N caron as the integer solution N that satisfies the following condition 18 and is closest to the real solution N hat calculated using the distance r defined by the covariance matrix Q.

[0073]

number

[0074] Furthermore, in this embodiment, LAMBDA (Least Square Ambiguity Decorrelation Adjustment) is used for integer ambiguity search in the integer least squares method. In this case, the integer ambiguity candidate with the minimum residual value, the second integer ambiguity candidate, and their respective residual values, which were searched using the LAMBDA method, are used to perform a Ratio test as shown in Equation 19 below. Then, the test result is determined to be the threshold (generally 3~ 5) If the result is 5 or higher, the test is considered successful and an integer solution to the integer ambiguity problem is obtained.

[0075]

number

[0076] Furthermore, if an integer solution to the integer ambiguity that passed the test is found, the fixed solution r of the position of the GNSS receiver r of the unmanned aerial vehicle 20 is determined based on the following equation 20 obtained by rearranging equations 16 and 17 above. u Caron is needed.

[0077]

number

[0078] In this embodiment, as described above, the first trajectory generation unit 11 calculates the position of the unmanned aerial vehicle 20 using positioning data stored in the database. Therefore, the first trajectory generation unit 11 can also calculate the position of the unmanned aerial vehicle 20 in the forward direction from the past to the future, the reverse direction from the future to the past, or in both the forward and reverse directions. In particular, when the position of the unmanned aerial vehicle 20 is calculated in both the forward and reverse directions, the position calculation accuracy is further improved.

[0079] Next, the first trajectory generation unit 11 calculates the position of the unmanned aerial vehicle 20 from the positioning data in a time series, and then, as shown in Figure 3, connects the coordinates of the calculated position in a time series to generate the trajectory of the unmanned aerial vehicle 20 (see Figure 2). In Figure 3, "○" indicates the position of the unmanned aerial vehicle 20 calculated from the positioning data.

[0080] Furthermore, the first trajectory generation unit 11 receives data from the unmanned aerial vehicle 20 via the IMU (Inertial Measurement Unit). If data from the Measurement Unit can be acquired, as shown in Figure 3, the acquired IMU The data can also be used to interpolate the position of the unmanned aerial vehicle. Figure 3 shows an example of the process for generating the trajectory of a moving object. In Figure 3, "□" indicates the interpolated position.

[0081] Specifically, the IMU is a device equipped with an acceleration sensor and an angular velocity sensor, and outputs data that identifies the attitude of the unmanned aerial vehicle 20, such as its orientation and tilt. Therefore, the first trajectory generation unit 11 uses the orientation and velocity of the unmanned aerial vehicle 20 identified from the IMU data to complete the position. The trajectory obtained in this way is composed of a point cloud.

[0082] In this embodiment, the feature point extraction unit 12 acquires frames of video data sequentially from the mobile object database 21 in chronological order, and for each acquired frame, it extracts feature points from the image using, for example, a general FAST algorithm.

[0083] The environment map generation unit 13 selects one frame, and for each feature point extracted from it, it determines whether it is identical to each feature point extracted from previous frames, associates the feature point determined to be identical with the selected feature point, and identifies a pair of feature points. The environment map generation unit 13 performs this feature point association for all acquired frames.

[0084] Next, the environment map generation unit 13 calculates the camera matrix for each frame using the set of feature points identified in that frame. Then, the environment map generation unit 13 calculates the 3D coordinates of the feature points using the calculated camera matrix and the 2D coordinates of the feature points in the frame.

[0085] Next, the environment map generation unit 13 calculates a camera matrix and, for each frame, calculates the camera position (i.e., the position of the unmanned aerial vehicle 20) from the camera matrix and performs self-localization. Subsequently, the environment map generation unit 13 generates an environment map composed of a 3D point cloud using the feature points for which 3D coordinates have been calculated. In this specification, the generation of an environment map also includes updating an already generated environment map.

[0086] In this embodiment, the second trajectory generation unit 14 generates a frame trajectory by sequentially connecting the camera position coordinates calculated during the generation of the environmental map in chronological order. The frame trajectory obtained in this way is also composed of a point cloud.

[0087] By the way, the trajectory of the moving object generated by the first trajectory generation unit 11 and the trajectory of the frame generated by the second trajectory generation unit 14 are in different coordinate systems. For this reason, in this embodiment, the shooting position identification unit 15 performs a transformation process (rotation, translation, scaling) on ​​the frame trajectory, as shown in Figure 4, and superimposes the transformed frame trajectory onto the trajectory of the moving object. Figure 4 shows an example of the trajectory superposition process.

[0088] Specifically, the shooting position identification unit 15 performs superposition by calculating a transformation matrix that minimizes the error between the point cloud constituting the frame's trajectory and the point cloud constituting the moving object's trajectory. The transformation matrix is ​​as shown in Equation 21 below. In Equation 21, A is the rotation matrix, B is the translation matrix, and C is the scaling matrix, and their product "A·B·C" is the transformation matrix. (x g ,y g ,z g ) indicates the coordinates of a point on the trajectory of the moving object, and (x c ,y c ,z c ) indicates the coordinates of a point on the frame's trajectory.

[0089]

number

[0090] Next, the shooting position identification unit 15 applies the calculated transformation matrix to the above number 21, thereby determining the coordinates (x) of each frame in the coordinate system of the moving object's trajectory (world coordinate system). c ,y c ,z c ) corresponds to coordinates (x g ,y g ,z g The shooting position identification unit 15 identifies the coordinates (x) of each frame after superimposing them. c,y c ,z c The coordinates (points) of the nearest moving object to ) can be identified, and these identified coordinates may be used as the corresponding coordinates.

[0091] Then, as shown in Figure 5, the position information addition unit 16 adds position information for each frame, indicating the position (coordinates) of the moving object corresponding to the coordinates of that frame. Figure 5 is a diagram showing an example of the position information addition process.

[0092] [Device operation] Next, an example of the operation of the information processing device 10 will be explained using Figure 6. Figure 6 is a flowchart showing an example of the operation of the information processing device. In the following explanation, Figures 1 to 5 will be referred to as appropriate. In Embodiment 1, the information processing method is implemented by operating the information processing device. Therefore, the explanation of the information processing method in this embodiment will be replaced by the following explanation of the operation of the information processing device 10.

[0093] As shown in Figure 6, first, the first trajectory generation unit 11 acquires first positioning data from the mobile database 21 and second positioning data and satellite orbit data from the base station database 31 (step A1).

[0094] Next, the first trajectory generation unit 11 uses the first positioning data, the second positioning data, and satellite orbit data to calculate the position of the unmanned aerial vehicle 20 in chronological order (step A2).

[0095] Specifically, in step A2, the first trajectory generation unit 11 generates first observation data including the carrier phase of the first positioning data and second observation data including the carrier phase of the second positioning data. Using these and satellite orbit data, it determines the relative relationship between the first positioning data and the second positioning data for each positioning satellite in a time series. Then, based on the determined relative relationship and the position of the base station 30, which is a fixed point, the first trajectory generation unit 11 calculates the position of the unmanned aerial vehicle 20 in a time series.

[0096] Next, the first trajectory generation unit 11 sequentially connects the position coordinates calculated in step A2 in a time series to generate the trajectory of the unmanned aerial vehicle 20 (see Figure 2) (step A3). In step A3, the first trajectory generation unit 11 can also interpolate the position using the orientation and velocity of the unmanned aerial vehicle 20 identified from the IMU data of the unmanned aerial vehicle 20.

[0097] Next, the feature point extraction unit 12 retrieves frames of video data from the mobile object database 21 in chronological order, and for each retrieved frame, extracts feature points from the image using, for example, a general FAST algorithm (step A4).

[0098] Next, the environment map generation unit 13 identifies corresponding pairs of feature points between frames, calculates the 3D coordinates of the identified feature points, and uses the calculation results to generate an environment map composed of the set of feature points (step A5).

[0099] Specifically, in step A5, the environment map generation unit 13 calculates the camera matrix for each frame using the set of feature points identified in that frame. Then, the environment map generation unit 13 calculates the 3D coordinates of the feature points using the calculated camera matrix and the 2D coordinates of the feature points in the frame. Furthermore, for each frame, the environment map generation unit 13 calculates the camera position from the camera matrix and performs self-localization. After that, it generates an environment map composed of a 3D point cloud using the feature points for which the 3D coordinates have been calculated.

[0100] Next, the second trajectory generation unit 14 connects the camera position coordinates calculated during the generation of the environment map in step A5 in chronological order to generate the frame trajectory (step A6).

[0101] Next, the shooting position identification unit 15 superimposes the trajectory of the unmanned aerial vehicle 20 generated in step A3 with the trajectory of the frame generated in step A6, and by this superimposition, identifies the position corresponding to the shooting position of each frame (step A7).

[0102] Subsequently, the location information addition unit 16 adds location information to each frame, indicating the location identified in step A7 (step A8).

[0103] As described above, in this embodiment, the information processing device 10 determines the position on a frame-by-frame basis by matching the frame trajectory obtained from the environmental map with the trajectory of the moving object obtained from positioning data from positioning satellites. Furthermore, since the trajectory of the moving object is generated using not only positioning data received by the moving object alone, but also positioning data received by both the moving object and the base station, the accuracy of the trajectory of the moving object is improved compared to when positioning is performed by the moving object alone. For this reason, according to this embodiment, it is possible to reduce positioning errors in the GPS receiver and to improve the accuracy of position information when adding location information of the shooting location to video data.

[0104] [program] The program in this embodiment can be any program that causes a computer to execute steps A1 to A8 shown in Figure 6. By installing and running this program on a computer, the information processing device and information processing method in this embodiment can be realized. In this case, the computer's processor functions as a first trajectory generation unit 11, a feature point extraction unit 12, an environment map generation unit 13, a second trajectory generation unit 14, a shooting location identification unit 15, and a location information addition unit 16, and performs the processing. In addition to a general-purpose PC, the computer can also be a smartphone or a tablet terminal device.

[0105] Furthermore, the program in the embodiment may be executed by a computer system constructed by multiple computers. In this case, for example, each computer may function as one of the following: a first trajectory generation unit 11, a feature point extraction unit 12, an environment map generation unit 13, a second trajectory generation unit 14, a shooting location identification unit 15, and a location information addition unit 16.

[0106] [Physical configuration] Here, a computer that implements an information processing device by executing the program in the embodiment will be described using Figure 7. Figure 7 is a block diagram showing an example of a computer that implements an information processing device.

[0107] As shown in Figure 7, the computer 110 has a CPU (Central Processing Unit) 11 The system comprises a main memory 112, a storage device 113, an input interface 114, a display controller 115, a data reader / writer 116, and a communication interface 117. Each of these components is connected to the others via a bus 121, enabling data communication between them.

[0108] Furthermore, the computer 110 may, in addition to or instead of the CPU 111, have a GPU (Graphics Processing Unit) or an FPGA (Field-Programmable Gate Array). It may also include the following. In this embodiment, the GPU or FPGA can execute the program in the embodiment.

[0109] The CPU 111 loads the program in the embodiment, which consists of a set of codes stored in the storage device 113, into the main memory 112, and performs various calculations by executing each code in a predetermined order. The main memory 112 is typically a volatile storage device such as DRAM (Dynamic Random Access Memory).

[0110] Furthermore, the program in this embodiment is provided stored on a computer-readable recording medium 120. The program in this embodiment may also be distributed over the internet via a communication interface 117.

[0111] Specific examples of the storage device 113 include hard disk drives and semiconductor storage devices such as flash memory. The input interface 114 mediates data transmission between the CPU 111 and input devices 118 such as a keyboard and mouse. The display controller 115 is connected to the display device 119 and controls the display on the display device 119.

[0112] The data reader / writer 116 mediates data transmission between the CPU 111 and the recording medium 120, reads programs from the recording medium 120, and writes processing results from the computer 110 to the recording medium 120. The communication interface 117 mediates data transmission between the CPU 111 and other computers.

[0113] Furthermore, specific examples of recording media 120 include CF (Compact Flash®) and S Examples include general-purpose semiconductor memory devices such as D (Secure Digital), magnetic recording media such as Flexible Disks, and optical recording media such as CD-ROMs (Compact Disk Read Only Memory).

[0114] Furthermore, the information processing device 10 can be implemented not only by a computer on which a program is installed, but also by using hardware corresponding to each part, such as electronic circuits. Moreover, the information processing device 10 may be partially implemented by a program and the remaining part by hardware. In this embodiment, the computer is not limited to the computer shown in Figure 7.

[0115] Some or all of the embodiments described above can be expressed by (Appendix 1) to (Appendix 21) described below, but are not limited to the following descriptions.

[0116] (Note 1) A first trajectory generation unit calculates the position of a moving object that captures video data using positioning data received from multiple positioning satellites, in a time-series order, and further generates the trajectory of the moving object by connecting the calculated positions in chronological order. A feature point extraction unit extracts feature points from each frame that makes up the aforementioned video data, An environment map generation unit identifies corresponding sets of feature points between the aforementioned frames, calculates the 3D coordinates of the identified set of feature points, and uses the calculation results to generate an environment map composed of the set of identified feature points. A second trajectory generation unit generates the trajectory of the frame using the aforementioned environment map, A shooting position identification unit superimposes the trajectory of the moving object generated by the first trajectory generation unit and the trajectory of the frame generated by the second trajectory generation unit, thereby identifying the position corresponding to the shooting position of each frame. A position information adding unit adds information indicating the specified position to each frame, An information processing device characterized by having the following features.

[0117] (Note 2) The information processing device according to Appendix 1, wherein the shooting position identification unit determines a transformation matrix for superimposing the trajectory of the frame onto the trajectory of the position of the moving object, and for each frame, applies the coordinates of the frame on the environmental map to the transformation matrix to identify the position corresponding to the shooting position of the frame.

[0118] (Note 3) The shooting position identification unit determines the transformation matrix by performing at least one of rotation, translation, scaling, and reduction on the frame's trajectory so as to minimize the error between the frame's trajectory and the trajectory of the moving object's position. The information processing device described in Appendix 2.

[0119] (Note 4) The first trajectory generation unit, in addition to the positioning data, further uses second positioning data received by a base station installed at a fixed point from each of the multiple positioning satellites, and calculates the position of the moving object in a time series based on the relative relationship between the positioning data for each positioning satellite and the second positioning data, and the position of the fixed point where the base station is installed. The information processing device described in Appendix 1.

[0120] (Note 5) The first trajectory generation unit further uses satellite orbit data indicating the position of each positioning satellite at each time interval, which is received by the base station from each of the plurality of positioning satellites. First observation data including the carrier phase of the positioning data is generated, and second observation data including the carrier phase of the second positioning data is generated. Using the first observation data, the second observation data, and the satellite orbit data, the relative relationship between the positioning data and the second positioning data for each positioning satellite is determined in a time series, and the position of the moving object is calculated based on the determined relative relationship and the position of the fixed point. The first observation data and the second observation data each include the pseudo-distance to the positioning satellite and the carrier phase, The information processing apparatus described in Appendix 4, characterized in that it is a processing apparatus.

[0121] (Note 6) The positioning data for each positioning satellite and the second positioning data are accumulated in chronological order. The first trajectory generation unit uses the stored positioning data for each positioning satellite and the second positioning data to determine the relative relationship for each positioning satellite in a time series, and calculates the position of the moving object based on the determined relative relationship and the position of the fixed point. The information processing device described in Appendix 4.

[0122] (Note 7) The first trajectory generation unit calculates the position of the moving object in a forward direction from the past to the future, a reverse direction from the future to the past, or in two directions, the forward direction and the reverse direction. The information processing device described in Appendix 6.

[0123] (Note 8) A first trajectory generation step involves a moving object capturing video data using positioning data received from multiple positioning satellites to calculate the position of the moving object in chronological order, and further generating the trajectory of the moving object by connecting the calculated positions in chronological order. A feature point extraction step is performed to extract feature points for each frame that makes up the aforementioned video data. An environment map generation step involves identifying corresponding pairs of feature points between the aforementioned frames, calculating the 3D coordinates of the identified feature points, and using the calculation results to generate an environment map composed of the set of identified feature points. A second trajectory generation step involves generating the trajectory of the frame using the aforementioned environment map, A shooting position identification step involves superimposing the trajectory of the moving object generated by the first trajectory generation step and the trajectory of the frame generated by the second trajectory generation step, thereby identifying the position corresponding to the shooting position of each frame. For each frame, a position information addition step is performed, in which information indicating the specified position is added to the frame. An information processing method characterized by having the following:

[0124] (Note 9) In the aforementioned step of identifying the shooting position, a transformation matrix is ​​obtained to superimpose the trajectory of the frame onto the trajectory of the position of the moving object, and for each frame, the coordinates of the frame on the environmental map are applied to the transformation matrix to identify the position corresponding to the shooting position of that frame. The information processing method described in Appendix 8.

[0125] (Note 10) In the aforementioned shooting position determination step, the transformation matrix is ​​determined by performing at least one of rotation, translation, scaling, and reduction on the frame's trajectory so as to minimize the error between the frame's trajectory and the trajectory of the moving object's position. The information processing method described in Appendix 9.

[0126] (Note 11) In the first trajectory generation step, in addition to the positioning data, a base station installed at a fixed point further uses second positioning data received from each of the multiple positioning satellites, and calculates the position of the moving object in chronological order based on the relative relationship between the positioning data for each positioning satellite and the second positioning data, and the position of the fixed point where the base station is installed. The information processing method described in Appendix 8.

[0127] (Note 12) In the first trajectory generation step, the base station further uses satellite orbit data indicating the position of each positioning satellite at each time interval, First observation data including the carrier phase of the positioning data is generated, and second observation data including the carrier phase of the second positioning data is generated. Using the first observation data, the second observation data, and the satellite orbit data, the relative relationship between the positioning data and the second positioning data for each positioning satellite is determined in a time series, and the position of the moving object is calculated based on the determined relative relationship and the position of the fixed point. The first observation data and the second observation data each include the pseudo-distance to the positioning satellite and the carrier phase, The information processing method described in Appendix 11, characterized by the features described herein.

[0128] (Note 13) The positioning data for each positioning satellite and the second positioning data are accumulated in chronological order. In the first trajectory generation step, the relative relationship for each positioning satellite is determined in chronological order using the stored positioning data for each positioning satellite and the second positioning data, and the position of the moving object is calculated based on the determined relative relationship and the position of the fixed point. The information processing method described in Appendix 11.

[0129] (Note 14) In the first trajectory generation step, the position of the moving object is calculated in the forward direction from the past to the future, the reverse direction from the future to the past, or in both the forward and reverse directions. The information processing method described in Appendix 13.

[0130] (Note 15) On the computer, A first trajectory generation step involves a moving object capturing video data using positioning data received from multiple positioning satellites to calculate the position of the moving object in chronological order, and further generating the trajectory of the moving object by connecting the calculated positions in chronological order. A feature point extraction step is performed to extract feature points for each frame that makes up the aforementioned video data. An environment map generation step involves identifying corresponding pairs of feature points between the aforementioned frames, calculating the 3D coordinates of the identified feature points, and using the calculation results to generate an environment map composed of the set of identified feature points. A second trajectory generation step involves generating the trajectory of the frame using the aforementioned environment map, A shooting position identification step involves superimposing the trajectory of the moving object generated by the first trajectory generation step and the trajectory of the frame generated by the second trajectory generation step, thereby identifying the position corresponding to the shooting position of each frame. For each frame, a position information addition step is performed, in which information indicating the specified position is added to the frame. Let's execute it ru, Professional Hmm.

[0131] (Note 16) In the aforementioned step of identifying the shooting position, a transformation matrix is ​​obtained to superimpose the trajectory of the frame onto the trajectory of the position of the moving object, and for each frame, the coordinates of the frame on the environmental map are applied to the transformation matrix to identify the position corresponding to the shooting position of that frame. As described in Appendix 15 program .

[0132] (Note 17) In the aforementioned shooting position determination step, the transformation matrix is ​​determined by performing at least one of rotation, translation, scaling, and reduction on the frame's trajectory so as to minimize the error between the frame's trajectory and the trajectory of the moving object's position. As described in Appendix 16 program .

[0133] (Note 18) In the first trajectory generation step, in addition to the positioning data, a base station installed at a fixed point further uses second positioning data received from each of the multiple positioning satellites, and calculates the position of the moving object in chronological order based on the relative relationship between the positioning data for each positioning satellite and the second positioning data, and the position of the fixed point where the base station is installed. As described in Appendix 15 program .

[0134] (Note 19) In the first trajectory generation step, the base station further uses satellite orbit data indicating the position of each positioning satellite at each time interval, First observation data including the carrier phase of the positioning data is generated, and second observation data including the carrier phase of the second positioning data is generated. Using the first observation data, the second observation data, and the satellite orbit data, the relative relationship between the positioning data and the second positioning data for each positioning satellite is determined in a time series, and the position of the moving object is calculated based on the determined relative relationship and the position of the fixed point. The first observation data and the second observation data each include the pseudo-distance to the positioning satellite and the carrier phase, The features described in Appendix 18 program .

[0135] (Note 20) The positioning data for each positioning satellite and the second positioning data are accumulated in chronological order. In the first trajectory generation step, the relative relationship for each positioning satellite is determined in chronological order using the stored positioning data for each positioning satellite and the second positioning data, and the position of the moving object is calculated based on the determined relative relationship and the position of the fixed point. As described in Appendix 18 program .

[0136] (Note 21) In the first trajectory generation step, the position of the moving object is calculated in the forward direction from the past to the future, the reverse direction from the future to the past, or in both the forward and reverse directions. As described in Appendix 20 program .

[0137] Although the present invention has been described above with reference to embodiments, the present invention is not limited to the above embodiments. Various modifications to the structure and details of the present invention can be made, as can be understood by those skilled in the art within the scope of the present invention.

[0138] This application claims priority based on Japanese Patent Application No. 2023-031948, filed on 2 March 2023, and incorporates all of its disclosures herein. [Industrial applicability]

[0139] As described above, this disclosure makes it possible to improve the accuracy of location information when adding location information of the shooting location to video data. This disclosure is useful for systems that generate 3D point cloud data of an object from video data. [Explanation of symbols]

[0140] 10 Information Processing Devices 11. First Trajectory Generation Unit 12 Feature point extraction unit 13. Environmental Map Generation Unit 14. Second Trajectory Generation Unit 15. Shooting location identification unit 16 Location information addition unit 20 Unmanned aerial vehicle 21 Mobile Database 21 30 base station 31 Base Station Database 40 positioning satellites 110 Computer 111 CPU 112 Main Memory 113 Storage device 114 Input Interface 115 Display Controller 116 Data Readers / Writers 117 Communication Interface 118 Input devices 119 Display device 120 recording media 121 Bus

Claims

1. A first trajectory generation unit calculates the position of a moving object that captures video data using positioning data received from multiple positioning satellites, in a time-series order, and further generates the trajectory of the moving object by connecting the calculated positions in chronological order. A feature point extraction unit extracts feature points from each frame that makes up the aforementioned video data, An environment map generation unit identifies corresponding sets of feature points between the aforementioned frames, calculates the three-dimensional coordinates of the identified set of feature points, and uses the calculation results to generate an environment map composed of the set of identified feature points. A second trajectory generation unit generates the trajectory of the frame using the aforementioned environmental map, A shooting position identification unit superimposes the trajectory of the moving object generated by the first trajectory generation unit and the trajectory of the frame generated by the second trajectory generation unit, thereby identifying the position corresponding to the shooting position of each frame. A position information adding unit adds information indicating the specified position to each frame, An information processing device characterized by having the following features.

2. The shooting position identification unit determines a transformation matrix for superimposing the trajectory of the frame onto the trajectory of the moving object's position, and for each frame, it identifies the position corresponding to the shooting position of that frame by applying the coordinates of that frame on the environmental map to the transformation matrix. The information processing apparatus according to claim 1.

3. The shooting position identification unit determines the transformation matrix by performing at least one of rotation, translation, scaling, and reduction on the frame's trajectory so as to minimize the error between the frame's trajectory and the trajectory of the moving object's position. The information processing apparatus according to claim 2.

4. The first trajectory generation unit, in addition to the positioning data, further uses second positioning data received by a base station installed at a fixed point from each of the multiple positioning satellites, and calculates the position of the moving object in a time series based on the relative relationship between the positioning data for each positioning satellite and the second positioning data, and the position of the fixed point where the base station is installed. The information processing apparatus according to claim 1.

5. The first trajectory generation unit further uses satellite orbit data indicating the position of each positioning satellite at each time interval, which is received by the base station from each of the plurality of positioning satellites. First observation data including the carrier phase of the positioning data is generated, and second observation data including the carrier phase of the second positioning data is generated. Using the first observation data, the second observation data, and the satellite orbit data, the relative relationship between the positioning data and the second positioning data for each positioning satellite is determined in chronological order, and the position of the moving object is calculated based on the determined relative relationship and the position of the fixed point. The first observation data and the second observation data each include the pseudo-distance to the positioning satellite and the carrier phase, The information processing apparatus according to feature 4.

6. The positioning data for each positioning satellite and the second positioning data are accumulated in chronological order. The first trajectory generation unit uses the stored positioning data for each positioning satellite and the second positioning data to determine the relative relationship for each positioning satellite in a time series, and calculates the position of the moving object based on the determined relative relationship and the position of the fixed point. The information processing apparatus according to claim 4.

7. The first trajectory generation unit calculates the position of the moving object in a forward direction from the past to the future, a reverse direction from the future to the past, or in two directions, the forward direction and the reverse direction. The information processing apparatus according to claim 6.

8. A moving object capturing video data uses positioning data received from multiple positioning satellites to calculate the object's position in chronological order, and then connects the calculated positions in chronological order to generate the object's trajectory. For each frame that makes up the aforementioned video data, feature points are extracted. The system identifies corresponding pairs of feature points between the aforementioned frames, calculates the three-dimensional coordinates of the identified feature points, and uses the calculation results to generate an environment map composed of the set of identified feature points. Using the aforementioned environment map, the trajectory of the frame is generated. The trajectory of the moving object and the trajectory of the frame are superimposed, thereby identifying the position corresponding to the shooting location of each frame. For each frame, information indicating the specified location is added to that frame. An information processing method characterized by the following:

9. In identifying the shooting location, a transformation matrix is ​​obtained to superimpose the trajectory of the frame onto the trajectory of the moving object's position, and for each frame, the coordinates of that frame on the environmental map are applied to the transformation matrix to identify the location corresponding to the shooting location of that frame. The information processing method according to claim 8.

10. In determining the shooting position, the transformation matrix is ​​obtained by performing at least one of rotation, translation, scaling, and reduction on the frame's trajectory so as to minimize the error between the frame's trajectory and the trajectory of the moving object's position. The information processing method according to claim 9.

11. In generating the trajectory of the moving object, in addition to the positioning data, a base station installed at a fixed point uses second positioning data received from each of the multiple positioning satellites, and calculates the position of the moving object in chronological order based on the relative relationship between the positioning data for each positioning satellite and the second positioning data, and the position of the fixed point where the base station is installed. The information processing method according to claim 8.

12. In generating the trajectory of the moving object, the base station further uses satellite orbit data indicating the position of each positioning satellite at each time interval, which is received from each of the multiple positioning satellites. First observation data including the carrier phase of the positioning data is generated, and second observation data including the carrier phase of the second positioning data is generated. Using the first observation data, the second observation data, and the satellite orbit data, the relative relationship between the positioning data and the second positioning data for each positioning satellite is determined in chronological order, and the position of the moving object is calculated based on the determined relative relationship and the position of the fixed point. The first observation data and the second observation data each include the pseudo-distance to the positioning satellite and the carrier phase, The information processing method according to feature 11.

13. The positioning data for each positioning satellite and the second positioning data are accumulated in chronological order. In generating the trajectory of the moving object, the relative relationship for each positioning satellite is determined in chronological order using the stored positioning data for each positioning satellite and the second positioning data, and the position of the moving object is calculated based on the determined relative relationship and the position of the fixed point. The information processing method according to claim 11.

14. In generating the trajectory of the moving object, the position of the moving object is calculated in the forward direction from the past to the future, the reverse direction from the future to the past, or in both the forward and reverse directions. The information processing method according to claim 13.

15. On the computer, A moving object capturing video data uses positioning data received from multiple positioning satellites to calculate the object's position in chronological order, and then connects the calculated positions in chronological order to generate the object's trajectory. For each frame that makes up the aforementioned video data, feature points are extracted. The system identifies corresponding pairs of feature points between the aforementioned frames, calculates the three-dimensional coordinates of the identified feature points, and uses the calculation results to generate an environment map composed of the set of identified feature points. Using the aforementioned environment map, the trajectory of the frame is generated. The trajectory of the moving object and the trajectory of the frame are superimposed, thereby identifying the position corresponding to the shooting location of each frame. For each frame, information indicating the specified location is added to that frame. program.

16. In identifying the shooting location, a transformation matrix is ​​obtained to superimpose the trajectory of the frame onto the trajectory of the moving object's position, and for each frame, the coordinates of that frame on the environmental map are applied to the transformation matrix to identify the location corresponding to the shooting location of that frame. The program according to claim 15.

17. In determining the shooting position, the transformation matrix is ​​obtained by performing at least one of rotation, translation, scaling, and reduction on the frame's trajectory so as to minimize the error between the frame's trajectory and the trajectory of the moving object's position. L as described in claim 16.

18. In generating the trajectory of the moving object, in addition to the positioning data, a base station installed at a fixed point uses second positioning data received from each of the multiple positioning satellites, and calculates the position of the moving object in chronological order based on the relative relationship between the positioning data for each positioning satellite and the second positioning data, and the position of the fixed point where the base station is installed. The program according to claim 15.

19. In generating the trajectory of the moving object, the base station further uses satellite orbit data indicating the position of each positioning satellite at each time interval, which is received from each of the multiple positioning satellites. First observation data including the carrier phase of the positioning data is generated, and second observation data including the carrier phase of the second positioning data is generated. Using the first observation data, the second observation data, and the satellite orbit data, the relative relationship between the positioning data and the second positioning data for each positioning satellite is determined in chronological order, and the position of the moving object is calculated based on the determined relative relationship and the position of the fixed point. The first observation data and the second observation data each include the pseudo-distance to the positioning satellite and the carrier phase, The program according to feature 18.

20. The positioning data for each positioning satellite and the second positioning data are accumulated in chronological order. In generating the trajectory of the moving object, the relative relationship for each positioning satellite is determined in chronological order using the stored positioning data for each positioning satellite and the second positioning data, and the position of the moving object is calculated based on the determined relative relationship and the position of the fixed point. The program according to claim 18.

21. In generating the trajectory of the moving object, the position of the moving object is calculated in the forward direction from the past to the future, the reverse direction from the future to the past, or in both the forward and reverse directions. The program according to claim 20.

Images