Information processing device, information processing method, and program

Abstract

An information processing device 10 comprises: a time information addition unit 11 that adds time information indicating the time of image capture to each of frames of moving image data captured by a mobile body; a data acquisition unit 12 that acquires, in chronological order, first positioning data received by the mobile body from each of positioning satellites and second positioning data received by a base station from each of the positioning satellites; a position calculation unit 13 that calculates the position of the mobile body on the basis of the position of the base station, and the relative relationship between the first positioning data and the second positioning data for each of the positioning satellites; an image capture position identification unit 14 that identifies a position corresponding to the image capture position of each of the frames from among the calculated positions of the mobile body by matching the time of image capture indicated by the time information added to each of the frames with the time associated with the first positioning data; and a position information addition unit 15 that adds information indicating the position to each of the frames.

Description

Information processing device, information processing method, and computer-readable recording medium

[0001] The present disclosure relates to an information processing device and information processing method for processing video data, and further to a computer-readable recording medium on which a program for realizing these is recorded.

[0002] A technique has been proposed in the past to extract sets of corresponding feature points from multiple still images of an object taken from different angles, and to create 3D point cloud data of the object using the extracted sets of feature points. A technique has also been proposed to generate 3D point cloud data of an object using a video instead of multiple still images.

[0003] Unlike the former technique using still images, the latter technique using video does not require multiple shots, but simply requires video of the entire target, making it useful for generating 3D point cloud data for large, complex-shaped structures such as plants and bridges.

[0004] However, when generating 3D point cloud data from a video, the video is captured while moving, so the capturing position changes for each frame. Therefore, in order to extract sets of corresponding feature points between frames, it is necessary to identify the capturing position for each frame.

[0005] For example, Patent Document 1 discloses an apparatus for identifying the position of a moving object that is capturing video. The apparatus disclosed in Patent Document 1 measures the position of the moving object equipped with a camera using a GPS (Global Positioning System) receiver, compares the shooting time of each frame with the positioning time, and identifies the position of each frame based on the comparison result.

[0006] Japanese Patent Application Laid-Open No. 2006-250917

[0007] However, in the device disclosed in Patent Document 1, the position of the mobile object is determined by general GPS positioning, which causes a problem of low accuracy in the determined position. This will be explained in detail below.

[0008] First, in typical GPS positioning, a GPS receiver receives radio waves from three positioning satellites by itself. Since the received radio waves are marked with the transmission time of the positioning satellites, the GPS receiver calculates the difference between the reception time, which is the time the GPS receiver itself received the radio waves, and the transmission time for each positioning satellite, and calculates the distance to the positioning satellite based on the difference. The GPS receiver then calculates its own position from the position and distance of each positioning satellite.

[0009] In this case, the accuracy of the time measurement by the GPS receiver is lower than the accuracy of the time added to the radio waves from the positioning satellites, resulting in an error in the reception time. Furthermore, if the GPS receiver receives radio waves reflected by an obstacle, this also causes an error in the reception time. In such a situation, an error also occurs in the distance to each positioning satellite calculated by the GPS receiver, resulting in the problem of low accuracy in the determined position of the device disclosed in Patent Document 1.

[0010] An example of an object of the present disclosure is to improve the accuracy of location information when location information on a shooting location is added to video data.

[0011] In order to achieve the above object, an information processing device according to one aspect of the present disclosure comprises: a time information adding unit that adds time information indicating the shooting time to each frame constituting video data; a data acquiring unit that acquires, in chronological order, first positioning data received from each of a plurality of positioning satellites by a mobile body that shoots the video data, and second positioning data received from each of the plurality of positioning satellites by a base station installed at a fixed point; a position calculating unit that calculates, in chronological order, the position of the mobile body based on the relative relationship between the first positioning data and the second positioning data for each of the positioning satellites and the position of the fixed point where the base station is installed; a shooting position identifying unit that, for each frame, compares the shooting time indicated by the time information added to the frame with the time associated with the first positioning data, and identifies a position corresponding to the shooting position of the frame from among the positions of the mobile body calculated in chronological order; and a position information adding unit that adds, for each frame, information indicating the identified position to the frame.

[0012] In order to achieve the above object, an information processing method according to one aspect of the present disclosure comprises: a time information adding step of adding time information indicating a shooting time to each frame constituting video data; a data acquiring step of acquiring, in chronological order, first positioning data received from each of a plurality of positioning satellites by a mobile body that shoots the video data, and second positioning data received from each of the plurality of positioning satellites by a base station installed at a fixed point; a position calculating step of calculating, in chronological order, a position of the mobile body based on a relative relationship between the first positioning data and the second positioning data for each of the positioning satellites and the position of the fixed point where the base station is installed; a shooting position identifying step of, for each frame, comparing the shooting time indicated by the time information added to the frame with a time associated with the first positioning data, and identifying a position corresponding to the shooting position of the frame from among the positions of the mobile body calculated in chronological order; and a position information adding step of adding, for each frame, information indicating the identified position to the frame.

[0013] Furthermore, in order to achieve the above object, a computer-readable recording medium according to one aspect of the present disclosure includes: a time information adding step of adding time information indicating a shooting time to each frame constituting video data; a data acquiring step of acquiring, in chronological order, first positioning data received from each of a plurality of positioning satellites by a mobile body that shoots the video data, and second positioning data received from each of the plurality of positioning satellites by a base station installed at a fixed point; a position calculating step of calculating, in chronological order, a position of the mobile body based on a relative relationship between the first positioning data and the second positioning data for each of the positioning satellites and the position of the fixed point where the base station is installed; a shooting position identifying step of, for each frame, comparing the shooting time indicated by the time information added to the frame with the time associated with the first positioning data, and identifying a position corresponding to the shooting position of the frame from among the positions of the mobile body calculated in chronological order; and a position information adding step of adding, for each frame, information indicating the identified position to the frame. The present invention is characterized in that it records a program including instructions for executing the above.

[0014] As described above, according to the present disclosure, when location information on the shooting location is added to video data, the accuracy of the location information can be improved.

[0015] FIG. 1 is a configuration diagram showing a schematic configuration of an information processing device in an embodiment. FIG. 2 is a configuration diagram showing an information processing device in an embodiment and its peripheral devices. FIG. 3 is a diagram explaining an example of processing in a time information adding unit in an embodiment. FIG. 4 is a diagram explaining an example of processing in a shooting position identifying unit in an embodiment. FIG. 5 is a flow diagram showing the operation of an information processing device in an embodiment. FIG. 6 is a block diagram showing an example of a computer that realizes an information processing device in an embodiment.

[0016] (Embodiments) Hereinafter, an information processing device, an information processing method, and a program according to an embodiment will be described with reference to FIGS.

[0017] [Device Configuration] First, the schematic configuration of an information processing device according to an embodiment will be described with reference to Fig. 1. Fig. 1 is a diagram showing the schematic configuration of an information processing device according to an embodiment.

[0018] 1, an information processing device 10 according to an embodiment is a data generation support device for supporting the generation of three-dimensional point cloud data from video data. As shown in FIG. 1, the information processing device 10 includes a time information adding unit 11, a data acquiring unit 12, a position calculating unit 13, a shooting position identifying unit 14, and a position information adding unit 15.

[0019] The time information adding unit 11 adds time information indicating the shooting time to each frame constituting the video data. Here, the time information is not limited to information indicating the absolute time at the time of shooting, but may be information indicating a relative time in which the start time of the first frame of the video is set to zero.

[0020] The data acquisition unit 12 acquires first positioning data and second positioning data in chronological order. The first positioning data is positioning data received from a plurality of positioning satellites by a mobile object capturing video data. The second positioning data is positioning data received from a plurality of positioning satellites by a base station installed at a fixed point.

[0021] The position calculation unit 13 calculates the position of the mobile body based on the relative relationship between the first positioning data and the second positioning data for each positioning satellite in time series, and the position of the fixed point where the base station is installed.

[0022] The shooting position identifying unit 14 compares the shooting time indicated by the time information added to each frame of the video data with the time associated with the first positioning data for each frame. Then, based on the comparison result, the shooting position identifying unit 14 identifies a position corresponding to the shooting position of each frame from among the positions of the moving object calculated in chronological order. The position information adding unit 15 adds information indicating the identified position to each frame.

[0023] However, since the frame interval of video data is generally shorter than the transmission interval when a satellite transmits positioning data, the shooting position identification unit 14 does not need to compare the transmission times of all frames. In this case, the shooting position identification unit 14 may perform comparison only for the frame corresponding to the transmission time of the first positioning data, for example, for one frame out of a specific number of frames.

[0024] In this manner, in the embodiment, the information processing device 10 determines the location not only by using the positioning data received by the mobile unit alone but also by using the positioning data received by both the mobile unit and the base station. Therefore, the information processing device 10 can improve the accuracy of the location information when adding the location information of the shooting location to video data.

[0025] Next, the configuration and functions of the information processing device in the embodiment will be specifically described with reference to Figures 2 to 4. Figure 2 is a configuration diagram showing the information processing device in the embodiment and its peripheral devices.

[0026] 2 , in this embodiment, the moving object that captures the video data is an unmanned aerial vehicle 20 equipped with a camera. The unmanned aerial vehicle 20 is equipped with a Global Navigation Satellite System (GNSS) receiver and receives positioning data (first positioning data) from each of the positioning satellites 40. The video data captured by the unmanned aerial vehicle 20 is stored in a moving object database 21. The first positioning data received by the GNSS receiver of the unmanned aerial vehicle 20 is also stored in the moving object database 21.

[0027] The base station 30 also includes a GNSS receiver, and receives positioning data (second positioning data) from each positioning satellite 40 using the GNSS receiver. The second positioning data received by the base station 30 is stored in the base station database 31. The base station 30 also receives satellite orbit data from each positioning satellite, which indicates the position of the positioning satellite at each time. The base station database 31 also stores the satellite orbit data.

[0028] Furthermore, each positioning satellite 40 transmits positioning data at a constant frequency, and the first positioning data and second positioning data are received at set intervals. Therefore, the mobile object database 21 stores the first positioning data for each positioning satellite in chronological order. The base station database 31 stores the second positioning data (carrier phase data and route data) and satellite orbit data for each positioning satellite in chronological order.

[0029] In this embodiment, the data acquisition unit 12 acquires the accumulated first positioning data and video data from the mobile object database 21. The data acquisition unit 12 also acquires the accumulated second positioning data and satellite orbit data from the base station database 31. The data acquisition unit 12 also passes the acquired video data to the time information addition unit 11, and passes the acquired first positioning data and second positioning data to the position calculation unit 13.

[0030] In this embodiment, the time information adding unit 11 receives video data captured by the unmanned aerial vehicle 20 from the data acquiring unit 12 and encodes the acquired video data. When encoding is performed, time information indicating the time of capture is embedded as a time code in each frame constituting the video data, as shown in Fig. 3. Fig. 3 is a diagram illustrating an example of processing by the time information adding unit in this embodiment.

[0031] Furthermore, if it is difficult to transmit video data from unmanned aerial vehicle 20 to time information adding unit 11 due to a problem such as a transmission bandwidth problem, time information adding unit 11 may be mounted on unmanned aerial vehicle 20. In this case, time information adding unit 11 performs encoding and embedding of time codes within unmanned aerial vehicle 20.

[0032] In this embodiment, the position calculation unit 13 generates first observation data from the first positioning data and generates second observation data from the second positioning data. The first observation data and second observation data described above are data including the pseudorange and carrier phase from the GNSS receiver to the positioning satellite 40. The position calculation unit 13 then uses the first observation data, the second observation data, and the satellite orbit data to determine the relative relationship between the first positioning data and the second positioning data for each positioning satellite along a time series, and calculates the position of the moving body (unmanned aerial vehicle 20) based on the determined relative relationship and the position of the fixed point.

[0033] The process of calculating the position of the unmanned aerial vehicle 20 by the position calculation unit 13 will now be described in detail. The following description will refer to the following document as appropriate: "A.4 RTK-GPS and Network-Type RTK-GPS Positioning Technology" by Tomoji Takasu (Tokyo University of Marine Science and Technology) <https: / / gpspp.sakura.ne.jp / paper2005 / gpssymp_2007a.pdf>

[0034] First, as described above, the observation data is data including a pseudorange and a carrier phase. Here, as shown in the above document, the pseudorange is defined as the signal propagation time measured by the positioning code (RPN code) multiplied by the speed of light.

[0035] The carrier wave phase is determined from the difference between the phase of the received carrier wave and the phase of the reference oscillator of the GNSS receiver. The carrier wave phase indicates the number of waves from the phase when the radio wave received from the positioning satellite was transmitted, and the distance from the positioning satellite 40 can be calculated by multiplying this number of waves by the wavelength.

[0036] The pseudorange (observed value) P and carrier phase (observed value) φ of a positioning satellite s received by a GNSS receiver r are expressed by the observation equation shown in the following equation 1. In the following equation 1, ρ is the geometric distance [m] between the positioning satellite and the receiver, and c is the speed of light [m / s]. The geometric distance ρ is calculated from satellite orbit data. dt is the time error [s] of the GNSS receiver, and dT is the clock error [s] of the positioning satellite. I is the ionospheric delay [m], T is the tropospheric delay [m], λ is the carrier wavelength "m", ε is the observation error "m", and N is the carrier phase bias [number of cycles].

[0037]

[0038] Furthermore, when the double difference (double phase difference) of the carrier phase is calculated in the observation equation for relative positioning in the above equation 1, the carrier phase bias N becomes an integer with the initial phase term eliminated. Hereinafter, the carrier phase bias N is also specifically referred to as "integer ambiguous N."

[0039] Here, the carrier phases of the positioning satellites a and b measured almost simultaneously by the two GNSS receivers u (base station 30) and r (unmanned aerial vehicle 20) are defined as φ ｕ ａ , φ ｕ ｂ , φ ｒ ａ , φ ｒ ｂ and the pseudorange is p ｕ ａ , p ｕ ｂ , p ｒ ａ , p ｒ ｂ Here, the superscript indicates the positioning satellite 40, and the subscript indicates the GNN receiver (observation point). Carrier phase double difference φ ｕｒ ａｂ and pseudorange double difference p ｕｒ ａｂ is defined by the following equation 2.

[0040]

[0041] Moreover, the above equation 2 can be transformed into the following equation 3.

[0042]

[0043] Here, the carrier phase double difference φ ｕｒ ａｂ and pseudorange double difference p ｕｒ ａｂ Using the notation of the above-mentioned formula 1, it becomes as shown in the following formula 4.

[0044]

[0045] Furthermore, taking into consideration that the positioning data from each positioning satellite 40 is received simultaneously in each GNSS receiver, the relationship shown in the following equation 5 holds.

[0046]

[0047] Furthermore, taking into consideration that the positioning data is transmitted at approximately the same time in each positioning satellite 40 and that the positioning satellite clock is sufficiently stable within a short period of time, the following equation 6 also holds.

[0048]

[0049] By applying the above-mentioned equations 5 and 6 to the above-mentioned equation 4, the observation equations of the carrier phase and pseudorange double difference are obtained as shown in the following equation 7. In the following equation 7, the terms of the positioning satellite clock error and the receiver clock error are eliminated from the carrier phase double difference.

[0050]

[0051] Here, consider the case where the distance (baseline length u-r) between the GNSS receiver u (base station 30) and the GNSS receiver r (unmanned aerial vehicle 20) is sufficiently close. When positioning data is received from the same positioning satellite 40 at the same time at two points that are sufficiently close to each other, the propagation path of the positioning signal (positioning data signal) through the atmosphere, the ionosphere at each point, and the tropospheric delay at each point will be approximately the same. For this reason, the approximation shown in the following equation 8 is possible.

[0052]

[0053] Then, by applying the above-mentioned equation 8 to the above-mentioned equation 7, the observation equations of the carrier phase and the pseudorange double difference can be approximated as shown in the following equation 9.

[0054]

[0055] Here, the position of the GNSS receiver r (unmanned aerial vehicle 20) is expressed as r ｕ and the received carrier L ｉ The receiver-to-receiver single difference integer ambiguity is N ｉ Then, the estimation parameter x is expressed by the following Expression 10, and the double-difference observation vector y is expressed by the following Expression 11.

[0056]

[0057]

[0058] The above-mentioned expressions (10) and (11) are applied to the above-mentioned expression (9). As a result, the observation equation and the partial differential coefficient matrix of the double-difference observable vector y are expressed as follows:

[0059]

[0060]

[0061] The observation error covariance matrix R of the double-differenced observation is as shown in the following equation 14.

[0062]

[0063] The position calculation unit 13 uses the above-mentioned observation equation to calculate an estimated value x of the unknown parameter x of each GNSS receiver by an extended Kalman filter. ｋ The equations for correction and update by the extended Kalman filter in each GNSS receiver k are expressed as follows using the above equations 12 to 14:

[0064]

[0065] In the above formula 15, x ｋ － is the a priori estimate vector of unknown parameter x at GNSS receiver k. ｋ ＋ is the a posteriori estimate vector of the unknown parameters x at GNSS receiver k. ｋ - is the a priori error covariance matrix at GNSS receiver k. ｋ ＋is the a posteriori error covariance matrix at GNSS receiver k. ｋ is the Kalman gain matrix at GNSS receiver k. I is the identity matrix.

[0066] Then, using the extended Kalman filter, the position of each GNSS receiver, the integer ambiguity estimate x, and its covariance matrix P can be obtained. The integer ambiguities obtained here are real-valued estimates that do not take into account the constraints of integer conditions, and the estimated values ​​of the GNSS receiver positions obtained at the same time are called FLOAT solutions. Because GNSS receiver u (base station 30) is fixed, its position is known in advance. Therefore, only the position of GNSS receiver r (unmanned aerial vehicle 20) is estimated using the FLOAT solution.

[0067] The position calculation unit 13 calculates an integer solution of the integer ambiguity and calculates a final FIX solution using the integer least squares method with the FLOAT solution and the covariance matrix P as input. The FIX solution becomes the final position of the unmanned aerial vehicle 20.

[0068] Specifically, the single-difference integer ambiguities are converted to double-difference integer bias estimates x′ as shown in Equation 16 below: where D is a transformation matrix for converting from single differences to double differences.

[0069]

[0070] The above equation (16) can be rewritten as the following equation (17): As a result, the double-difference integer ambiguous estimate x' and its covariance matrix Q are separated into real variables and integer variables.

[0071]

[0072] Here, the integer least squares method is applied. Then, the position calculation unit 13 determines the integer solution N that satisfies the condition of the following equation 18 and is closest to the real solution N calculated at the distance r defined by the covariance matrix Q as the optimal integer solution N Caron.

[0073]

[0074] In the embodiment, LAMBDA (Least Square Ambiguity Decorrelation Adjustment) is used for integer ambiguity search in the integer least squares method. In this case, a test is performed by a ratio test shown in the following equation 19 using the integer ambiguity candidate with the smallest residual value searched by the LAMBDA method, the second integer ambiguity candidate, and their respective residual values. If the test result is equal to or greater than a threshold value (generally 3 to 5), the test is deemed to be successful and an integer solution for the integer ambiguity is found.

[0075]

[0076] In addition, when an integer solution of the integer ambiguity that passes the test is obtained, the FIX solution r of the position of the GNSS receiver r of the unmanned aerial vehicle 20 is obtained based on the following equation 20 obtained by transforming the above equations 16 and 17. ｕ Caron is required.

[0077]

[0078] In the embodiment, as described above, the position calculation unit 13 calculates the position of the unmanned aerial vehicle 20 using positioning data stored in the database. Therefore, the position calculation unit 13 can also calculate the position of the unmanned aerial vehicle 20 in the forward direction from the past to the future, the backward direction from the future to the past, or in two directions, the forward direction and the backward direction. In particular, when the position of the unmanned aerial vehicle 20 is calculated in two directions, the forward direction and the backward direction, the accuracy of the position calculation is further improved.

[0079] In the embodiment, the shooting position identification unit 14 first compares, for each frame constituting the video data, the shooting time indicated by the time information added to each frame with the time associated with the first positioning data. Here, in the embodiment, the time associated with the first positioning data is the time when the positioning satellite transmitted the first positioning data, i.e., the transmission time of the first positioning data.

[0080] The photographing position identifying unit 14 then links each frame to the first observation data generated from the first positioning data so that the photographing time and transmission time match, as shown in Figure 4. Furthermore, since the position calculation unit 13 has calculated the position of the unmanned aerial vehicle 20 for each piece of first observation data whose photographing time and transmission time match, the linking identifies a position corresponding to the photographing position of each frame. Figure 4 is a diagram illustrating an example of processing by the photographing position identifying unit in this embodiment.

[0081] In addition, in the embodiment, it is preferable that the GNSS receiver of the unmanned aerial vehicle 20 and the GNSS receiver of the base station 30 receive the positioning data so that the capture timing of each frame of the video data and the transmission time of the positioning data coincide with each other. In this case, the capture time of each frame coincides with the transmission time (reception time) of the positioning data.

[0082] However, as described above, the frame interval of video data is generally shorter than the transmission interval when a satellite transmits positioning data. Therefore, the shooting position identification unit 14 does not need to check the transmission time for all frames. Specifically, in this case, the shooting position identification unit 14 skips a certain number of frames of video data and adjusts the frame interval when encoding the video data so that the shooting timing of each specific frame matches the transmission time of the positioning data.

[0083] In the above case, the calculation of the position by the position calculation unit 13 and the identification of the position by the photographing position identification unit 14 are performed using only frames that match the transmission timing of the positioning data. In this case, it is preferable that the photographing time and transmission time of the matching frame (specific frame) match, but they do not need to match completely as long as the difference between them is within a set range.

[0084] Then, as shown in Figure 4, for each specific frame, the location information addition unit 15 identifies the location calculated by the location calculation unit 13 for the first observation data linked to each frame, and adds information indicating the identified location.

[0085] [Device Operation] Next, the operation of the information processing device in the embodiment will be described with reference to FIG. 5. FIG. 5 is a flow diagram showing the operation of the information processing device in the embodiment. In the following description, reference will be made to FIGS. 1 to 4 as appropriate. In the embodiment, an information processing method is implemented by operating the information processing device 10. Therefore, the description of the information processing method in the embodiment will be replaced by the following description of the operation of the information processing device 10.

[0086] As shown in Figure 5, first, the data acquisition unit 12 acquires the accumulated first positioning data and video data from the mobile object database 21, and acquires the accumulated second positioning data and satellite orbit data from the base station database 31 (step A1).

[0087] Next, the time information adding unit 11 adds time information indicating the shooting time to each frame constituting the video data acquired in step A1 (step A2).

[0088] Specifically, in step A2, the time information adding unit 11 encodes the video data acquired in step A1, thereby embedding time information indicating the shooting time as a time code into each frame constituting the video data (see FIG. 3).

[0089] Next, the position calculation unit 13 calculates the position of the unmanned aerial vehicle (mobile body) 20 based on the relative relationship between the first positioning data and the second positioning data for each positioning satellite, and the position of the fixed point where the base station is installed, in chronological order (step A3).

[0090] Specifically, in step A3, the position calculation unit 13 generates first observation data from the first positioning data and generates second observation data from the second positioning data. Then, using the first observation data, the second observation data, and the satellite orbit data acquired in step A1, the position calculation unit 13 calculates the relative relationship between the first positioning data and the second positioning data for each positioning satellite in time series. Furthermore, the position calculation unit 13 calculates the position of the unmanned aerial vehicle 20 based on the determined relative relationship and the position of the base station 30 (fixed point).

[0091] Next, for each specific frame constituting the video data, the shooting position identifying unit 14 compares the shooting time indicated by the time information added to each frame with the transmission time of the first positioning data, and then, based on the comparison result, identifies a position corresponding to the shooting position of each frame from among the positions of the moving object calculated in chronological order (step A4).

[0092] Specifically, in step A4, the photographing position identifying unit 14 associates each frame with the first observation data so that the photographing time and the transmission time match or so that the difference between the photographing time and the transmission time falls within a set range (see FIG. 4). By this association, a position corresponding to the photographing position of each frame is identified.

[0093] Next, for each specific frame, the position information adding unit 15 identifies the position calculated in step A3 for the first observation data linked to each frame, and adds information indicating the identified position (step A5).

[0094] By executing steps A1 to A5, the position information of the photographing position is added to the specific frame. As a result, it becomes possible to create 3D point cloud data of the target using the image data of the frame to which the position information is linked and the linked position information.

[0095] As described above, the information processing device 10 determines the location not only by using the positioning data received by the mobile unit alone but also by using the positioning data received by both the mobile unit and the base station. Therefore, the information processing device 10 can improve the accuracy of the location information when adding the location information of the shooting location to video data.

[0096] In the above example, the information processing device 10 executes processing using data that has been accumulated in advance in chronological order as the video data, the first positioning data, the second positioning data, and the satellite orbit data, but the embodiment is not limited to this example. The information processing device 10 can also execute processing in real time, sequentially, when the video data, the first positioning data, the second positioning data, and the satellite orbit data are transmitted.

[0097] [Program] The program in the embodiment may be a program that causes a computer to execute steps A1 to A5 shown in Figure 5. By installing and executing this program on a computer, the information processing device 10 and information processing method in the embodiment can be realized. In this case, the processor of the computer functions and performs processing as a time information adding unit 11, a data acquiring unit 12, a position calculating unit 13, a shooting position identifying unit 14, and a position information adding unit 15. In addition to a general-purpose PC, examples of the computer include a smartphone and a tablet terminal device.

[0098] The program in the embodiment may be executed by a computer system constructed by a plurality of computers, in which case, for example, each computer may function as any one of the time information adding unit 11, data acquiring unit 12, position calculating unit 13, shooting position identifying unit 14, and position information adding unit 15.

[0099] [Modification] Here, a modification of the embodiment will be described. In the modification, the information processing device 10 also has the configuration shown in Figures 1 and 2. As in the example of Figures 1 and 2, in the modification, the information processing device 10 also has a time information addition unit 11, a data acquisition unit 12, a position calculation unit 13, a shooting position identification unit 14, and a position information addition unit 15.

[0100] However, in the modified example, the time associated with the first positioning data is the reception time of the first positioning data, which is different from the above-described example. Therefore, in the modified example, the shooting position identification unit 14 compares the shooting time indicated by the time information added to each frame of the video data with the reception time of the first positioning data for each frame.

[0101] Furthermore, the reception time is assigned to the first positioning data by, for example, the GNSS receiver of the unmanned aerial vehicle 20, which is a moving body. The method of assigning the reception time is not particularly limited. In a modified example, in step A4 shown in Figure 5, the reception time is used instead of the transmission time. However, as in the above example, steps A1 to A5 shown in Figure 5 are also executed in the modified example.

[0102] [Physical Configuration] A computer that implements the information processing device 10 by executing a program according to the embodiment will now be described with reference to Fig. 6. Fig. 6 is a block diagram showing an example of a computer that implements the information processing device according to the embodiment.

[0103] 6, the computer 110 includes a CPU (Central Processing Unit) 111, a main memory 112, a storage device 113, an input interface 114, a display controller 115, a data reader / writer 116, and a communication interface 117. These components are connected to each other via a bus 121 so as to be able to communicate data with each other.

[0104] Furthermore, the computer 110 may include a GPU (Graphics Processing Unit) or an FPGA (Field-Programmable Gate Array) in addition to or instead of the CPU 111. In this aspect, the GPU or FPGA can execute the programs in the embodiments.

[0105] The CPU 111 loads a program in the embodiment, which is composed of a group of codes and stored in the storage device 113, into the main memory 112 and executes each code in a predetermined order to perform various calculations. The main memory 112 is typically a volatile storage device such as a DRAM (Dynamic Random Access Memory).

[0106] The program in the embodiment is provided in a state stored in a computer-readable recording medium 120. The program in the embodiment may be distributed over the Internet connected via the communication interface 117.

[0107] Specific examples of the storage device 113 include a hard disk drive and a semiconductor storage device such as a flash memory. The input interface 114 mediates data transmission between the CPU 111 and input devices 118 such as a keyboard and a mouse. The display controller 115 is connected to a display device 119 and controls the display on the display device 119.

[0108] The data reader / writer 116 mediates data transmission between the CPU 111 and the recording medium 120, reads programs from the recording medium 120, and writes processing results from the computer 110 to the recording medium 120. The communication interface 117 mediates data transmission between the CPU 111 and other computers.

[0109] Specific examples of the recording medium 120 include general-purpose semiconductor storage devices such as CF (Compact Flash (registered trademark)) and SD (Secure Digital), magnetic recording media such as flexible disks, or optical recording media such as CD-ROMs (Compact Disk Read Only Memory).

[0110] The information processing device 10 in the embodiment can be realized not by a computer on which a program is installed, but by hardware corresponding to each unit, such as an electronic circuit. Furthermore, the information processing device 10 may be partially realized by a program and the remaining unit by hardware. In the embodiment, the computer is not limited to the computer shown in FIG. 6.

[0111] Some or all of the above-described embodiments can be expressed by (Supplementary Note 1) to (Supplementary Note 18) described below, but are not limited to the following descriptions.

[0112] (Supplementary Note 1) An information processing device comprising: a time information adding unit that adds time information indicating a shooting time to each frame constituting video data; a data acquiring unit that acquires, in chronological order, first positioning data received from each of a plurality of positioning satellites by a mobile body that shoots the video data, and second positioning data received from each of the plurality of positioning satellites by a base station installed at a fixed point; a position calculating unit that calculates, in chronological order, a position of the mobile body based on a relative relationship between the first positioning data and the second positioning data for each of the positioning satellites and the position of the fixed point where the base station is installed; a shooting position identifying unit that, for each of the frames, compares the shooting time indicated by the time information added to the frame with a time associated with the first positioning data, and identifies a position corresponding to the shooting position of the frame from among the positions of the mobile body calculated in chronological order; and a position information adding unit that adds, for each of the frames, information indicating the identified position to the frame.

[0113] (Supplementary Note 2) The information processing device according to Supplementary Note 1, characterized in that the data acquisition unit further acquires satellite orbit data indicating the position of the positioning satellite at each time, which is received by the base station from each of the plurality of positioning satellites; the position calculation unit generates first observation data including a carrier phase of the first positioning data, and generates second observation data including a carrier phase of the second positioning data; uses the first observation data, the second observation data, and the satellite orbit data to determine a relative relationship between the first positioning data and the second positioning data for each of the positioning satellites along a time series, and calculates the position of the moving body based on the determined relative relationship and the position of the fixed point; and the first observation data and the second observation data each include a pseudorange to the positioning satellite and a carrier phase.

[0114] (Supplementary Note 3) The information processing device described in Supplementary Note 1 or 2, wherein the first positioning data and the second positioning data for each of the positioning satellites are accumulated in chronological order, and the position calculation unit uses the accumulated first positioning data and the second positioning data for each of the positioning satellites to determine the relative relationship for each of the positioning satellites in chronological order, and calculates the position of the moving body based on the determined relative relationship and the position of the fixed point.

[0115] (Supplementary Note 4) The information processing device according to Supplementary Note 3, wherein the position calculation unit calculates the position of the moving object in a forward direction from the past to the future, a backward direction from the future to the past, or in two directions of the forward direction and the backward direction.

[0116] (Supplementary Note 5) The information processing device described in Supplementary Note 1, wherein the time associated with the first positioning data is the transmission time of the first positioning data, and the shooting location identification unit, based on the result of the comparison, identifies a location corresponding to the shooting location of the frame by linking the corresponding first positioning data to the frame so that the shooting time and the transmission time of the first positioning data match for each frame or so that the difference between the shooting time and the transmission time of the first positioning data is within a set range.

[0117] (Supplementary Note 6) The information processing device described in Supplementary Note 1, wherein the time associated with the first positioning data is the reception time of the first positioning data, and the shooting position identification unit, based on the result of the comparison, identifies a position corresponding to the shooting position of each frame by linking the corresponding first positioning data to the frame so that the shooting time and the reception time of the first positioning data match or so that the difference between the shooting time and the reception time of the first positioning data is within a set range.

[0118] (Supplementary Note 7) An information processing method comprising: a time information adding step of adding time information indicating a shooting time to each frame constituting video data; a data acquiring step of acquiring, in chronological order, first positioning data received from each of a plurality of positioning satellites by a mobile body that shoots the video data, and second positioning data received from each of the plurality of positioning satellites by a base station installed at a fixed point; a position calculating step of calculating, in chronological order, a position of the mobile body based on a relative relationship between the first positioning data and the second positioning data for each of the positioning satellites and the position of the fixed point where the base station is installed; a shooting position identifying step of, for each of the frames, comparing the shooting time indicated by the time information added to the frame with a time associated with the first positioning data, and identifying a position corresponding to the shooting position of the frame from among the positions of the mobile body calculated in chronological order; and a position information adding step of adding, for each of the frames, information indicating the identified position to the frame.

[0119] (Supplementary Note 8) The information processing method according to Supplementary Note 7, wherein the data acquisition step further acquires satellite orbit data indicating the position of the positioning satellite at each time, which is received by the base station from each of the plurality of positioning satellites; the position calculation step generates first observation data including a carrier phase of the first positioning data, and generates second observation data including a carrier phase of the second positioning data; using the first observation data, the second observation data, and the satellite orbit data, determines a relative relationship between the first positioning data and the second positioning data for each positioning satellite along a time series, and calculates the position of the moving body based on the determined relative relationship and the position of the fixed point; and the first observation data and the second observation data each include a pseudorange to the positioning satellite and a carrier phase.

[0120] (Supplementary Note 9) The information processing method described in Supplementary Note 7, wherein the first positioning data and the second positioning data for each of the positioning satellites are accumulated in chronological order, and in the position calculation step, the relative relationship for each of the positioning satellites is determined in chronological order using the accumulated first positioning data and the second positioning data for each of the positioning satellites, and the position of the moving body is calculated based on the determined relative relationship and the position of the fixed point.

[0121] (Supplementary Note 10) The information processing method according to Supplementary Note 9, wherein in the position calculation step, the position of the moving object is calculated in a forward direction from the past to the future, a backward direction from the future to the past, or in two directions of the forward direction and the backward direction.

[0122] (Supplementary Note 11) An information processing method as described in Supplementary Note 7, wherein the time associated with the first positioning data is the transmission time of the first positioning data, and in the photographing position identification step, based on the result of the comparison, the photographing time and the transmission time of the first positioning data are matched for each frame, or the difference between the photographing time and the transmission time of the first positioning data is within a set range, and the corresponding first positioning data is linked to the frame to identify a position corresponding to the photographing position of the frame.

[0123] (Supplementary Note 12) An information processing method as described in Supplementary Note 7, wherein the time associated with the first positioning data is the reception time of the first positioning data, and in the photographing position identification step, based on the result of the comparison, the photographing time and the reception time of the first positioning data are matched for each frame, or the difference between the photographing time and the reception time of the first positioning data is within a set range, and the corresponding first positioning data is linked to the frame to identify a position corresponding to the photographing position of the frame.

[0124] (Supplementary Note 13) A computer-readable recording medium having recorded thereon a program including instructions to cause a computer to execute the following steps: a time information adding step of adding time information indicating a shooting time to each frame constituting video data; a data acquiring step of acquiring, in chronological order, first positioning data received from each of a plurality of positioning satellites by a mobile body that shoots the video data, and second positioning data received from each of the plurality of positioning satellites by a base station installed at a fixed point; a position calculating step of calculating, in chronological order, a position of the mobile body based on a relative relationship between the first positioning data and the second positioning data for each of the positioning satellites and the position of the fixed point where the base station is installed; a shooting position identifying step of, for each of the frames, comparing the shooting time indicated by the time information added to the frame with a time associated with the first positioning data, and identifying a position corresponding to the shooting position of the frame from among the positions of the mobile body calculated in chronological order; and a position information adding step of adding, for each of the frames, information indicating the identified position.

[0125] (Supplementary Note 14) The computer-readable recording medium according to Supplementary Note 13, wherein the data acquisition step further acquires satellite orbit data indicating the position of the positioning satellite at each time, which is received by the base station from each of the plurality of positioning satellites; the position calculation step generates first observation data including a carrier phase of the first positioning data, and generates second observation data including a carrier phase of the second positioning data; using the first observation data, the second observation data, and the satellite orbit data, determines a relative relationship between the first positioning data and the second positioning data for each of the positioning satellites along a time series, and calculates the position of the moving body based on the determined relative relationship and the position of the fixed point; and the first observation data and the second observation data each include a pseudorange to the positioning satellite and a carrier phase.

[0126] (Supplementary Note 15) A computer-readable recording medium as described in Supplementary Note 13, wherein the first positioning data and the second positioning data for each of the positioning satellites are accumulated in chronological order, and in the position calculation step, the relative relationship for each of the positioning satellites is determined in chronological order using the accumulated first positioning data and the second positioning data for each of the positioning satellites, and the position of the moving body is calculated based on the determined relative relationship and the position of the fixed point.

[0127] (Supplementary Note 16) The computer-readable recording medium according to Supplementary Note 15, wherein in the position calculation step, the position of the moving object is calculated in a forward direction from the past to the future, a backward direction from the future to the past, or in two directions of the forward direction and the backward direction.

[0128] (Supplementary Note 17) A computer-readable recording medium as described in Supplementary Note 13, wherein the time associated with the first positioning data is the transmission time of the first positioning data, and in the photographing position identification step, based on the result of the comparison, the photographing time and the transmission time of the first positioning data are matched for each frame, or the difference between the photographing time and the transmission time of the first positioning data is within a set range, and the corresponding first positioning data is linked to the frame to identify a position corresponding to the photographing position of the frame.

[0129] (Supplementary Note 18) A computer-readable recording medium as described in Supplementary Note 13, wherein the time associated with the first positioning data is the reception time of the first positioning data, and in the photographing position identification step, based on the result of the comparison, the photographing time and the reception time of the first positioning data are matched for each frame, or the difference between the photographing time and the reception time of the first positioning data is within a set range, and the corresponding first positioning data is linked to the frame to identify a position corresponding to the photographing position of the frame.

[0130] Although the present invention has been described above with reference to the embodiments, the present invention is not limited to the above-described embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention.

[0131] This application claims priority based on Japanese Patent Application No. 2022-198640, filed December 13, 2022, the disclosure of which is incorporated herein in its entirety.

[0132] As described above, according to the present disclosure, when position information of a shooting location is added to video data, the accuracy of the position information can be improved. The present disclosure is useful for a system for generating three-dimensional point cloud data of an object from video data.

[0133] 10 Information processing device 11 Time information addition unit 12 Data acquisition unit 13 Position calculation unit 14 Shooting position identification unit 15 Position information addition unit 20 Unmanned aerial vehicle (mobile body) 21 Mobile body database 30 Base station 31 Base station database 40 Positioning satellite 110 Computer 111 CPU 112 Main memory 113 Storage device 114 Input interface 115 Display controller 116 Data reader / writer 117 Communication interface 118 Input device 119 Display device 120 Recording medium 121 Bus

Claims

1. a time information adding unit that adds time information indicating a shooting time to each frame that constitutes the video data; a data acquisition unit that acquires, in time series, first positioning data received from each of a plurality of positioning satellites by a mobile object that captures the video data, and second positioning data received from each of the plurality of positioning satellites by a base station installed at a fixed point; a position calculation unit that calculates a position of the moving object based on a relative relationship between the first positioning data and the second positioning data for each positioning satellite along a time series and a position of the fixed point where the base station is installed; a photographing position specifying unit that, for each of the frames, compares the photographing time indicated by the time information added to the frame with a time associated with the first positioning data, and specifies a position corresponding to the photographing position of the frame from among positions of the moving object calculated along a time series; a position information adding unit that adds, for each of the frames, information indicating the identified position to the frame; An information processing device comprising:

2. the data acquisition unit further acquires satellite orbit data indicating the position of the positioning satellite at each time, the satellite orbit data being received by the base station from each of the plurality of positioning satellites; The position calculation unit generating first observation data including a carrier phase of the first positioning data, and generating second observation data including a carrier phase of the second positioning data; using the first observation data, the second observation data, and the satellite orbit data, a relative relationship between the first positioning data and the second positioning data for each of the positioning satellites is obtained along a time series, and a position of the moving body is calculated based on the obtained relative relationship and the position of the fixed point; the first observation data and the second observation data each include a pseudorange to the positioning satellite and a carrier phase; 2. The information processing apparatus according to claim 1, wherein:

3. the first positioning data and the second positioning data for each of the positioning satellites are accumulated in chronological order, the position calculation unit uses the first positioning data and the second positioning data stored for each of the positioning satellites to determine the relative relationship for each of the positioning satellites along a time series, and calculates the position of the moving body based on the determined relative relationship and the position of the fixed point. The information processing device according to claim 1 .

4. the position calculation unit calculates the position of the moving object in a forward direction from the past to the future, a backward direction from the future to the past, or in two directions, the forward direction and the backward direction; The information processing device according to claim 3 .

5. the time associated with the first positioning data is a transmission time of the first positioning data, the photographing position specifying unit, based on the result of the comparison, adjusts, for each of the frames, the photographing time and the transmission time of the first positioning data to match, or adjusts the difference between the photographing time and the transmission time of the first positioning data to be within a set range, by associating the frame with the corresponding first positioning data, a position corresponding to the photographing position of the frame is identified; The information processing device according to claim 1 .

6. the time associated with the first positioning data is a reception time of the first positioning data, the photographing position specifying unit, based on the result of the comparison, adjusts, for each of the frames, the photographing time and the reception time of the first positioning data to match, or adjusts the difference between the photographing time and the reception time of the first positioning data to be within a set range, by associating the frame with the corresponding first positioning data, a position corresponding to the photographing position of the frame is identified; The information processing device according to claim 1 .

7. Adding time information indicating the shooting time to each frame that constitutes the video data, acquiring, in time series, first positioning data received from each of a plurality of positioning satellites by a mobile body that captures the video data, and second positioning data received from each of the plurality of positioning satellites by a base station installed at a fixed point; calculating a position of the moving body based on a relative relationship between the first positioning data and the second positioning data for each of the positioning satellites along a time series and a position of the fixed point where the base station is installed; for each of the frames, comparing the shooting time indicated by the time information added to the frame with a time associated with the first positioning data, and identifying a position corresponding to the shooting position of the frame from among the positions of the moving object calculated in chronological order; adding information indicating the identified position to each of the frames; An information processing method comprising:

8. In acquiring the first positioning data and the second positioning data, satellite orbit data indicating a position of the positioning satellite at each time point is further acquired, the satellite orbit data being received by the base station from each of the plurality of positioning satellites; In calculating the position, generating first observation data including a carrier phase of the first positioning data, and generating second observation data including a carrier phase of the second positioning data; using the first observation data, the second observation data, and the satellite orbit data, a relative relationship between the first positioning data and the second positioning data for each of the positioning satellites is obtained along a time series, and a position of the moving body is calculated based on the obtained relative relationship and the position of the fixed point; the first observation data and the second observation data each include a pseudorange to the positioning satellite and a carrier phase; 8. The information processing method according to claim 7,

9. the first positioning data and the second positioning data for each of the positioning satellites are accumulated in chronological order, In calculating the position, the relative relationship for each of the positioning satellites is determined along a time series using the first positioning data and the second positioning data stored for each of the positioning satellites, and the position of the moving body is calculated based on the determined relative relationship and the position of the fixed point. The information processing method according to claim 7.

10. In calculating the position, the position of the moving body is calculated in a forward direction from the past to the future, a backward direction from the future to the past, or in two directions, the forward direction and the backward direction. The information processing method according to claim 9.

11. the time associated with the first positioning data is a transmission time of the first positioning data, In identifying the photographing position, based on the result of the comparison, the photographing time and the transmission time of the first positioning data are made to coincide with each other, or the difference between the photographing time and the transmission time of the first positioning data is made to fall within a set range, for each of the frames; by associating the frame with the corresponding first positioning data, a position corresponding to the photographing position of the frame is identified; The information processing method according to claim 7.

12. the time associated with the first positioning data is a reception time of the first positioning data, In identifying the photographing position, based on the result of the comparison, the photographing time and the reception time of the first positioning data are made to coincide with each other, or the difference between the photographing time and the reception time of the first positioning data is made to fall within a set range, for each of the frames; by associating the frame with the corresponding first positioning data, a position corresponding to the photographing position of the frame is identified; The information processing method according to claim 7.

13. On the computer, Adding time information indicating the shooting time to each frame constituting the video data, acquiring, in time series, first positioning data received from each of a plurality of positioning satellites by a mobile body that captures the video data, and second positioning data received from each of the plurality of positioning satellites by a base station installed at a fixed point; calculating a position of the moving body based on a relative relationship between the first positioning data and the second positioning data for each of the positioning satellites and a position of the fixed point where the base station is installed, along a time series; for each of the frames, comparing the shooting time indicated by the time information added to the frame with the time associated with the first positioning data, and identifying a position corresponding to the shooting position of the frame from among the positions of the moving object calculated in chronological order; adding information indicating the identified position to each of the frames; program.

14. In acquiring the first positioning data and the second positioning data, satellite orbit data indicating a position of the positioning satellite at each time point, which is received by the base station from each of the plurality of positioning satellites, is further acquired; In calculating the position, generating first observation data including a carrier phase of the first positioning data, and generating second observation data including a carrier phase of the second positioning data; using the first observation data, the second observation data, and the satellite orbit data, a relative relationship between the first positioning data and the second positioning data for each of the positioning satellites is obtained along a time series, and a position of the moving body is calculated based on the obtained relative relationship and the position of the fixed point; the first observation data and the second observation data each include a pseudorange to the positioning satellite and a carrier phase; 14. The program according to claim 13 .

15. the first positioning data and the second positioning data for each of the positioning satellites are accumulated in chronological order, In calculating the position, the relative relationship for each of the positioning satellites is obtained along a time series using the first positioning data and the second positioning data stored for each of the positioning satellites, and the position of the moving body is calculated based on the obtained relative relationship and the position of the fixed point. The program according to claim 13.

16. In the calculation of the position, the position of the moving body is calculated in a forward direction from the past to the future, a backward direction from the future to the past, or in two directions, the forward direction and the backward direction. The program according to claim 15.

17. the time associated with the first positioning data is a transmission time of the first positioning data, In identifying the photographing position, based on the result of the comparison, the photographing time and the transmission time of the first positioning data are made to coincide with each other, or the difference between the photographing time and the transmission time of the first positioning data is made to fall within a set range, for each of the frames; by associating the frame with the corresponding first positioning data, a position corresponding to the photographing position of the frame is identified; The program according to claim 13.

18. the time associated with the first positioning data is a reception time of the first positioning data, In identifying the photographing position, based on the result of the comparison, the photographing time and the reception time of the first positioning data are made to coincide with each other, or the difference between the photographing time and the reception time of the first positioning data is made to fall within a set range, for each of the frames; by associating the frame with the corresponding first positioning data, a position corresponding to the photographing position of the frame is identified; The program according to claim 13.