Image processing device, image processing method, and program

Abstract

The present invention makes it possible to accurately display the flight path of a drone on an actual image of the drone. A display unit which is provided to the present invention and displays an image captured by a camera which depicts a moving device such as a drone has a data processing unit for displaying the movement path of the moving device. The data processing unit generates a coordinate transformation matrix for performing coordinate transformation processing for transforming position information which corresponds to a first coordinate system expressing the movement path of the moving device, such as a NED system, for example, into a second coordinate system in which it is possible to identify pixel positions in the display image on the display unit, such as a camera coordinate system, for example. The data processing unit outputs, to the display unit, a movement path which has position information which corresponds to the camera coordinate system and is generated by the coordinate transformation processing which uses the generated coordinate transformation matrix.

Description

[Technical field]

[0001] The present disclosure relates to an image processing device, an image processing method, and a program. More specifically, the present disclosure relates to an image processing device, an image processing method, and a program that enable accurate display of a movement trajectory of a moving device, such as a drone, on a real-life image. [Background technology]

[0002] In recent years, the use of drones, which are small flying objects, has increased dramatically. For example, drones are equipped with cameras and are used to take pictures of the ground from the sky. There are also plans to use drones for parcel delivery, and various experiments are being conducted.

[0003] Currently, many countries require drones to be controlled under human supervision, i.e., by operating a controller within the range of human line of sight. However, in the future, it is expected that autonomous drones that do not require human visual supervision, i.e., drones that fly autonomously from a departure point to a destination, will be widely used.

[0004] Such autonomous drones fly from their starting point to their destination using, for example, communication information with a control center and GPS location information.

[0005] One specific use of autonomous drones is parcel delivery by drone. When parcel delivery by drone is performed, it is expected that a user who has requested parcel delivery will want to look up at the sky to see the drone carrying his / her parcel and check its flight path and planned flight path when the scheduled arrival time of the drone carrying his / her parcel approaches.

[0006] In addition, even for drones other than those used to deliver packages, it is expected that there will be a demand to check the flight path of drones flying overhead, for example, to ensure safety.

[0007] One possible process for realizing such a user request would be to take an image of a drone in the sky using the camera of a camera-equipped user device such as a smartphone, display this captured image on a display unit, and also superimpose the drone's flight path and planned flight path on the image, on top of the actual image of the drone. In other words, it generates and displays an AR (Augumented Reality) image in which a line indicating the flight path is superimposed on the actual image of the drone.

[0008] Information regarding the drone's flight path and planned flight path can be transmitted from the drone or the control center that manages the drone's flight to a user terminal such as a smartphone via communication over a communications network. The drone or the control center holds information about the drone's flight path and planned flight path, i.e., flight path information, and can provide this flight path information to a user terminal such as a smartphone.

[0009] However, most drones use GPS satellite communication information for position control. The position information obtained from GPS satellites consists of latitude, longitude, and altitude information. Most drones use this information to fly, so they confirm their position and control their flight according to the NED coordinate system. The NED coordinate system is a coordinate system that has three axes: North, East, and Down.

[0010] The drone or the control center stores flight route information, which is information about the drone's flight path and planned flight path, as route information (N,E,D) using this NED coordinate system, and the route information according to this NED coordinate system is provided to user devices such as smartphones.

[0011] On the other hand, camera-captured images displayed on user terminals such as smartphones are image data based on a camera coordinate system that is set according to the shooting direction of the camera. The image position of a drone captured by a camera on a user device such as a smartphone can be specified as an image position on the camera coordinate system. However, it is difficult to calculate which position on the NED coordinate system the drone's actual image position corresponds to.

[0012] In this way, the position of the drone, which is the actual image displayed on a user device such as a smartphone, is a position that can be identified in the camera coordinate system, but the drone's flight path information received from the drone or the control center is route position information specified in the NED coordinate system, and there is a problem in that it is difficult to confirm which position in the camera coordinates this route position corresponds to.

[0013] As a result, when a user terminal such as a smartphone receives drone flight route information from the drone or a control center and attempts to display the flight route on a display unit based on this received information, an accurate route cannot be displayed.

[0014] Incidentally, for example, Patent Document 1 (Japanese Patent No. 5192598) is a conventional technique that discloses a configuration for AR display of the position and trajectory of a self-propelled robot on an image captured by a fixed camera such as a surveillance camera.

[0015] This configuration discloses a configuration in which an AR tag is placed on a self-propelled robot or within the working area of ​​the self-propelled robot, the AR tag is recognized from an image captured by a camera, and a single piece of reference coordinate information is generated, and the position and route of the self-propelled robot can be identified using the generated reference coordinate information.

[0016] However, this configuration requires that the AR tag be attached within the work area, and is applicable only within a limited work area. In a configuration that flies in the air, such as a drone, it is impossible to attach an AR tag in the air. [Prior art documents] [Patent documents]

[0017] [Patent Document 1] Patent No. 5192598 Summary of the Invention [Problem to be solved by the invention]

[0018] The present disclosure has been made in consideration of the above-mentioned problems, for example, and aims to provide an image processing device, an image processing method, and a program that enable an actual image of a drone, which is an image captured by a camera, to be displayed on a user terminal such as a smartphone, and that enables the drone's flight path and planned flight path to be accurately superimposed and displayed on the actual image. [Means for solving the problem]

[0019] A first aspect of the present disclosure is a data processing unit that displays a moving route of the mobile device on a display unit that displays an image captured by a camera of the mobile device; The data processing unit includes: execute a coordinate conversion process for converting position information according to a first coordinate system that indicates a moving path of the moving device into a second coordinate system that can identify a pixel position of a display image on the display unit; The image processing device outputs to a display unit a movement path having position information according to the second coordinate system generated by the coordinate conversion process.

[0020] Furthermore, a second aspect of the present disclosure is An image processing method executed in an image processing device, the image processing device has a data processing unit that displays a moving route of the mobile device on a display unit that displays an image captured by a camera of the mobile device; The data processing unit: execute a coordinate conversion process for converting position information according to a first coordinate system that indicates a moving path of the moving device into a second coordinate system that can identify a pixel position of a display image on the display unit; The image processing method includes outputting to a display unit a movement path having position information according to the second coordinate system generated by the coordinate conversion process.

[0021] Furthermore, a third aspect of the present disclosure is A program for causing an image processing device to execute image processing, the image processing device has a data processing unit that displays a moving route of the mobile device on a display unit that displays an image captured by a camera of the mobile device; The program causes the data processing unit to a coordinate conversion process for converting position information according to a first coordinate system that indicates a moving path of the moving device into a second coordinate system that can identify a pixel position of a display image on the display unit; The program executes a process of outputting to a display unit a travel path having position information according to the second coordinate system generated by the coordinate conversion process.

[0022] The program of the present disclosure is, for example, a program that can be provided by a storage medium or a communication medium in a computer-readable format to an information processing device or a computer system capable of executing various program codes. By providing such a program in a computer-readable format, processing according to the program is realized on the information processing device or the computer system.

[0023] Further objects, features, and advantages of the present disclosure will become apparent from the following detailed description of the embodiments of the present disclosure and the accompanying drawings. In this specification, a system refers to a logical collective configuration of multiple devices, and each device is not necessarily located in the same housing.

[0024] According to the configuration of one embodiment of the present disclosure, a configuration is realized that makes it possible to accurately display the flight path of a drone on a real-life image of the drone. Specifically, for example, the display unit displays an image captured by a camera of a mobile device such as a drone, and the data processing unit displays the moving route of the mobile device. The data processing unit generates a coordinate transformation matrix that performs coordinate transformation processing to convert position information according to a first coordinate system, for example, the NED coordinate system, which indicates the moving route of the mobile device, into a second coordinate system, for example, the camera coordinate system, which can identify the pixel position of the display image on the display unit, and outputs the moving route having position information according to the camera coordinate system generated by the coordinate transformation processing to which the generated coordinate transformation matrix is ​​applied to the display unit. This configuration makes it possible to accurately display the drone's flight path on a live image of the drone. It should be noted that the effects described in this specification are merely examples and are not limiting, and additional effects may also be provided. [Brief description of the drawings]

[0025] [Figure 1] 11 is a diagram illustrating an example of drone photographing processing using a user terminal and route information display processing for the photographed image. [Diagram 2] A figure explaining a specific example of a drone route information display process on a display unit of a user terminal. [Diagram 3] 11 is a diagram illustrating an example of transmission and reception of flight route information, which is information regarding the flight route and planned flight route of a drone. [Figure 4] FIG. 13 is a diagram illustrating a problem in the drone route information display process. [Diagram 5] FIG. 13 is a diagram explaining why the drone route information display may be inaccurate. [Figure 6] FIG. 13 is a diagram explaining why the drone route information display may be inaccurate. [Figure 7] 1 is a diagram illustrating a plurality of coordinate systems used in processing executed by an image processing device of the present disclosure. [Figure 8] FIG. 13 is a diagram illustrating an example of a process for converting position information in one coordinate system into position information in another coordinate system. [Figure 9]1A to 1C are diagrams illustrating three examples of coordinate transformation matrices. [Figure 10] 11 is a diagram illustrating a specific example of a calculation process of a coordinate transformation matrix (CTNED) executed by a user terminal which is an image processing device of the present disclosure. FIG. [Figure 11] 11 is a diagram illustrating a specific example of a calculation process of a coordinate transformation matrix (CTNED) executed by a user terminal which is an image processing device of the present disclosure. FIG. [Figure 12] FIG. 1 is a diagram illustrating a pinhole camera model. [Figure 13] FIG. 1 is a diagram illustrating a pinhole camera model. [Figure 14] 11 is a diagram illustrating a specific example of a calculation process of a coordinate transformation matrix (CTNED) executed by a user terminal which is an image processing device of the present disclosure. FIG. [Figure 15] 11 is a diagram illustrating a specific example of a calculation process of a coordinate transformation matrix (CTNED) executed by a user terminal which is an image processing device of the present disclosure. FIG. [Figure 16] This is an example of an AR image display executed by a user terminal, which is an image processing device of the present disclosure, and is a figure that explains an example in which a drone's flight path (flight path or planned flight path) without deviation is output and displayed for a drone image as a real-life image. [Figure 17] FIG. 13 is a diagram illustrating a process that takes into account changes in the position and orientation of the camera during shooting periods of drones in three different positions. [Figure 18] FIG. 13 is a diagram illustrating a process that takes into account changes in the position and orientation of the camera during shooting periods of drones in three different positions. [Figure 19] FIG. 2 is a diagram showing a flowchart illustrating a sequence of processing executed by an image processing device of the present disclosure. [Figure 20] 4 is a diagram illustrating an example of data recorded in a storage unit (memory) of the image processing device of the present disclosure. FIG. [Figure 21] FIG. 2 is a diagram showing a flowchart illustrating a sequence of processing executed by an image processing device of the present disclosure. [Figure 22]4 is a diagram illustrating an example of data recorded in a storage unit (memory) of the image processing device of the present disclosure. FIG. [Diagram 23] FIG. 2 is a diagram showing a flowchart illustrating a sequence of processing executed by an image processing device of the present disclosure. [Figure 24] FIG. 1 is a diagram illustrating an example configuration of a user terminal, which is an image processing device of the present disclosure, and a drone. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0026] The image processing device, image processing method, and program of the present disclosure will be described in detail below with reference to the drawings. The description will be made according to the following items. 1. Issues with drone route information display processing 2. Processing performed by the image processing device of the present disclosure 3. Sequence of processes executed by the image processing device of the present disclosure 4. Configuration examples of the image processing device and drone of the present disclosure 5. Summary of the Disclosure

[0027] [1. Problems with drone route information display processing] First, we will explain the problems with the display processing of drone route information, with reference to Figure 1 and subsequent figures.

[0028] As explained above, in many countries, drones today are required to be under human supervision, i.e., flight control is performed by operating a controller within the range of human line of sight. However, in the future, it is expected that autonomous drones that do not require human visual supervision, i.e., drones that fly autonomously from a departure point to a destination, will be used. Such autonomous drones fly from a departure point to a destination using, for example, communication information with a control center and GPS location information.

[0029] One specific use case for autonomous drones is the delivery of parcels by drone. When a parcel is delivered by drone, it is expected that a user who has requested the parcel delivery will want to look up at the sky to see the drone carrying his / her parcel and check its flight path and planned flight path when the scheduled arrival time of the drone carrying his / her parcel approaches.

[0030] One possible process for realizing this user request is for the user to take an image of a drone in the sky using the camera on a camera-equipped user device such as a smartphone, and then display this captured image on a display unit while also superimposing the drone's flight path and planned flight path on the image.

[0031] A specific example of this process will be described with reference to FIG. FIG. 1 shows a drone 20 flying in the sky and a user 1 having a user terminal 10 which is a smartphone with a camera. User 1 points the camera of user terminal 10 at drone 20 in the sky to capture an image of drone 20.

[0032] FIG. 2 shows an example of a display image of the user terminal 10. As shown in FIG. 2 is an image captured by the camera of the user terminal 10, and shows a drone 20. The image of the drone 20 is a real object that is actually flying in the sky. Lines indicating the flight path and planned flight path of the drone 20 are displayed on this actual image as a virtual image generated by the data processing unit of the user terminal 10. In other words, an AR (Augumented Reality) image is generated and displayed in which a virtual line indicating the flight path is superimposed on the actual image of the drone.

[0033] The "flight route" shown in Fig. 2 is the route flown up to the current position of the drone 20. The "planned flight route" shown in the figure is the planned flight route from the current position of the drone 20. In the example shown in the figure, the end of the "flight route" coincides with the current position of drone 20, and the start of the "planned flight route" also coincides with the current position of drone 20, and this route information is displayed accurately. It can be seen that drone 20 will fly from its current position according to the “planned flight route” and land in front of the house to deliver a package addressed to the user.

[0034] Information regarding the drone's flight path and planned flight path can be received by the user terminal from the drone or the control center. As shown in FIG. 3, the user terminal 10 can receive flight route information, which is information regarding the flight route and planned flight route of the drone, from the drone 20 or the drone management server 30, which is a control center.

[0035] However, as mentioned above, most drones use communication information from GPS satellites for position control. The position information obtained from GPS satellites consists of latitude, longitude, and altitude information. Most drones use this information to fly, so they often use the NED coordinate system to confirm their position and control their flight route.

[0036] NED coordinates are set on three axes: North, East, and Down.

[0037] The drone 20 shown in Figure 3, or the drone management server 30, which is the control center, holds flight route information, which is information regarding the drone's flight route and planned flight route, as route information (N, E, D) to which this NED coordinate system is applied, and the route information according to this NED coordinate system is provided to a user terminal 10, such as a smartphone.

[0038] On the other hand, a camera-captured image displayed on a user terminal 10 such as a smartphone is image data according to a camera coordinate system that is set according to the shooting direction of the camera. The data processing unit of the user terminal 10 such as a smartphone can identify the image position of the actual image of the drone 20 captured by the camera as the image position on the camera coordinates. However, it cannot calculate which position on the NED coordinates the actual image position of the drone 20 corresponds to.

[0039] In this way, the position of the drone 20, which is the actual image displayed on a user terminal 10 such as a smartphone, is a position that can be identified in the camera coordinate system, but the flight path information of the drone 20 received from the drone 20 or the drone management server 30 is route position information identified in the NED coordinate system, and it is not possible to perform an accurate analysis of which position in the camera coordinates this route position corresponds to.

[0040] As a result, even if a user terminal 10 such as a smartphone receives flight route information of the drone 20 from the drone 20 or the drone management server 30 and attempts to display the flight route on the display unit based on this received information, it is unable to display an accurate route. For example, as shown in Figure 4, a problem may occur in which the "flight route" or "planned flight route" is displayed as a route that deviates from the current position of the drone 20.

[0041] The reason for this is that, as shown in Figure 5, the camera-captured image displayed on a user terminal 10 such as a smartphone is image data based on a camera coordinate system that is set according to the camera's shooting direction, whereas the flight path information of the drone 20 received from the drone 20 or the drone management server 30 is position information in the NED coordinate system.

[0042] As a result, as shown in Figure 6, even if one tries to display route information according to the NED coordinate system (N, E, D) on a camera-captured image according to the camera coordinate system (Xc, Yc, Zc), the accurate route position cannot be displayed because the correspondence between the camera coordinate system and the NED coordinate system is unclear.

[0043] [2. Processing performed by the image processing device of the present disclosure] Next, the process executed by the image processing device of the present disclosure will be described.

[0044] The image processing device disclosed herein is, for example, a user terminal 10 such as a smartphone owned by a user, and performs processing to accurately display route information according to the drone's NED coordinate system (N, E, D) on a camera-captured image according to the camera coordinate system (Xc, Yc, Zc), for example, an image captured by a drone.

[0045] The user terminal 10, which is an image processing device of the present disclosure, converts the flight path information of the drone 20 received from the drone 20 and the drone management server 30, i.e., the flight path position in the NED coordinate system, into position information in the camera coordinate system, which is the coordinate system of the camera-captured image. After that, a line indicating the flight path converted into the position information in the camera coordinate system is displayed on the camera-captured image. By performing these processes, it is possible to accurately display the drone's route information on the images captured by the drone.

[0046] First, a plurality of coordinate systems used in the processing executed by the image processing device of the present disclosure will be described with reference to FIG. Figure 7 shows the following three coordinate systems: (1) Camera coordinate system (2) NED coordinate system (3) World coordinate system (SLAM coordinate system)

[0047] (1) The camera coordinate system is a coordinate system capable of defining the image position (pixel position) of an image captured by the camera of the user terminal 10. The camera coordinate system is a coordinate system in which the focal point of the camera is the origin C, the image plane is a two-dimensional plane of Xc, Yc, and the optical axis direction (depth) is Zc, and the origin C moves with the movement of the camera. For example, it is a coordinate system in which the horizontal axis of an imaging element such as a C-MOS is the Xc axis, the vertical axis is the Yc axis, and the optical axis direction is the Zc axis. The two-dimensional plane (UV plane) of the display unit corresponds to the XcYc plane of the camera coordinate system.

[0048] (2) The NED coordinate system is a coordinate system that indicates the position of drone 20 and the flight path, and is used for flight control of drone 20. Many of the drones 20 perform position control using communication information from GPS satellites. The position information obtained from GPS satellites is latitude information, longitude information, and altitude information, and many drones use the NED coordinate system to fly using this information.

[0049] The NED coordinate system is a coordinate system that has three axes: North, East, and Down. The drone 20, or a drone management server 30 such as a control center, holds flight route information, which is information regarding the flight route and planned flight route of the drone 20, as route information (N, E, D) in the NED coordinate system, and the route information according to this NED coordinate is provided to a user terminal 10 such as a smartphone.

[0050] (3) The world coordinate system (SLAM coordinate system) is primarily used as a coordinate system to define the entire space used in 3D graphics. In the processing of the present disclosure, the world coordinate system is a coordinate system (SLAM coordinate system) applied to the SLAM processing performed by the user terminal 10, i.e., SLAM (simultaneous localization and mapping) processing that performs camera position identification (localization) and environment map creation (mapping) in parallel.

[0051] As illustrated in the lower center of FIG. 7, each pixel position (u, v) of the display image has a one-to-one correspondence with the XY coordinates (Xc, Yc) of the camera coordinate system. Therefore, if the position of the flight path shown in the NED coordinate system can be converted to a position shown in the camera coordinate system, the path can be accurately output on the display image shown in the camera coordinate system.

[0052] FIG. 8 is a diagram illustrating an example of a process for converting position information in one coordinate system into position information in another coordinate system. The example shown in FIG. 8 is a coordinate transformation matrix required to transform position information (Xw, Yw, Zw) in the world coordinate system into position information (Xc, Yc, Zc) in the camera coordinate system. C T Ws This shows that.

[0053] The position of a point (X) in the three-dimensional space shown in the upper center of Figure 8 in the world coordinate system (SLAM coordinate system) and the position of the camera coordinate system are expressed as follows: Position in world coordinate system (SLAM coordinate system): Ws P X Camera coordinate system position: C P X

[0054] Here, the position ( Ws P X ) to the position in the camera coordinate system ( C P X ) is the coordinate transformation matrix C T Ws Let us assume that.

[0055] As shown in the lower part of Figure 8, the position ( Ws P X ) to the position in the camera coordinate system ( C P X ) can be expressed as follows (Equation 1): C P X = C T Ws × Ws P X ...(Formula 1)

[0056] Here, the position in the world coordinate system (SLAM coordinate system) ( Ws P X ) to the position in the camera coordinate system ( C P X ), the coordinate transformation matrix that transforms C T Ws This coordinate transformation matrix can be expressed as the following determinant (Equation 2).

[0057]

number

[0058] In the above formula (2), C R Ws is the rotation matrix that transforms the pose defined in the world coordinate system (SLAM coordinate system) to the pose defined in the camera coordinate system. Ws P C is the camera position in the world coordinate system (SLAM coordinate system), It is. In this example, the camera position corresponds to the position of the camera of the user terminal 10.

[0059] Coordinate transformation matrix shown in (Equation 2) above C T Ws is the position in the world coordinate system (SLAM coordinate system) ( Ws P X ) to the position in the camera coordinate system ( C P X ) is the coordinate transformation matrix. As previously described with reference to FIG. 7, the process of the present disclosure uses three different coordinate systems: (1) Camera coordinate system (2) NED coordinate system (3) World coordinate system (SLAM coordinate system) Processing is carried out using these coordinate systems.

[0060] Examples of coordinate transformation matrices for these three coordinate systems are shown in FIG. FIG. 9 shows the following three coordinate transformation matrices: C T Ws : World coordinate system (SLAM coordinate system) position ( Ws P X ) to the position in the camera coordinate system ( C P X ), the coordinate transformation matrix that transforms C T NED: NED coordinate system position ( NED P X ) to the position in the camera coordinate system ( C P X ), the coordinate transformation matrix that transforms Ws T NED : NED coordinate system position ( NED P X ) to the world coordinate system (SLAM coordinate system) position ( Ws P X ), the coordinate transformation matrix that transforms

[0061] Each of these three coordinate transformation matrices can be calculated from the other two coordinate transformation matrices. For example, Position in the NED coordinate system ( NED P X ) to the world coordinate system (SLAM coordinate system) position ( Ws P X ) coordinate transformation matrix: Ws T NED is the other two coordinate transformation matrices ( C T Ws , C T NED ) can be calculated according to the following formula: Ws T NED = C T Ws -1 × C T NED

[0062] In addition, C T Ws -1 teeth, C T Ws is the inverse matrix of C T Ws It is a matrix that can be calculated from As above, World coordinate system (SLAM coordinate system) position ( Ws P X ) to the position in the camera coordinate system ( C P X ) coordinate transformation matrix C T Ws is the other two coordinate transformation matrices (Ws T NED , C T NED ) can be calculated according to the following formula: C T Ws = C T NED × Ws T NED -1

[0063] Also, the position of the NED coordinate system ( NED P X ) to the position in the camera coordinate system ( C P X ) coordinate transformation matrix C T NED is the other two coordinate transformation matrices ( Ws T NED , C T Ws ) can be calculated according to the following formula: C T NED = C T Ws × Ws T NED

[0064] As explained above with reference to Figure 7, each pixel position (u, v) of the display image displayed on the display unit of the user terminal 10 has a one-to-one correspondence with the XY coordinates (Xc, Yc) of the camera coordinate system, and if the position of the flight path shown in the NED coordinate system can be converted to a position in the camera coordinate system, the path can be accurately output on the display image shown in the camera coordinate system.

[0065] That is, the position in the NED coordinate system ( NED P X ) to the position in the camera coordinate system ( C P X ), the coordinate transformation matrix that transforms C T NED If this coordinate transformation matrix can be calculated, it becomes possible to convert the flight path position shown in the NED coordinate system to a position in the camera coordinate system, and the drone's flight path can be accurately output on the display image shown in the camera coordinate system. The user terminal 10, which is an image processing device of the present disclosure, calculates the coordinate transformation matrix: C T NED Calculate.

[0066] With reference to FIG. 10 and subsequent figures, the coordinate transformation matrix executed by the user terminal 10, which is the image processing device of the present disclosure, is: C T NED A specific example of the calculation process will be described.

[0067] As shown in Fig. 10, the user photographs a drone 20 flying in the sky for a predetermined period of time using the user terminal 10. This drone 20 does not have to be a drone carrying a package to be delivered to the user. However, it is necessary that the drone is capable of acquiring the drone's flight path position information (position information on the NED coordinate system). In the example shown in FIG. 10, the drone 20 flies from right to left as times (t1), (t2), and (t3) elapse.

[0068] The data processing unit of the user terminal 10 records in a memory the shooting image positions of the drone in at least three different positions. As shown in FIG. 11, drone imaging positions 52-1 to 52-3 corresponding to drones at three different positions on a camera imaging surface 51 such as a C-MOS are recorded in a memory. The drone imaging position corresponds to the display image position displayed on the display unit, and here, a processing example is shown that uses the coordinate positions (u1, v1) to (u3, v3) on the display image.

[0069] Using these three different drone positions and the three drone image positions corresponding to these three drone positions, the coordinate transformation matrix: C T NED can be calculated.

[0070] Specific coordinate transformation matrix ( C T NEDBefore describing the calculation process of M, we will describe the pinhole camera model, which is a relational equation that defines the relationship between the three-dimensional position M of an object in three-dimensional space when the object is photographed with a general camera (pinhole camera) and the imaging position (imaging pixel position) m of the object on the imaging plane of the camera, with reference to Figs. 12 and 13.

[0071] In the pinhole camera model, the relation between the three-dimensional position M of an object that is a photographed subject and the imaging position (imaging pixel position) m of the camera of that object is expressed by the following (Equation 3).

[0072]

number

[0073] The meaning of the above formula (3) will be explained with reference to FIG. 12 and FIG. As shown in FIG. 12, when an object 61, which is a subject to be photographed, is photographed by a camera, an object image 62 is photographed on a camera imaging surface (C-MOS or the like) 51 of the camera. The above (Equation 3) is an equation that shows the correspondence between the pixel position on the camera-captured image plane of point (m) of object image 62 contained in the camera-captured image, i.e., the position expressed by the camera coordinate system, and the three-dimensional position (M) of object 61 in the world coordinate system.

[0074] The position (pixel position) of a point (m) of an object image 62 included in an image captured by a camera is expressed by a camera coordinate system. The camera coordinate system is a coordinate system in which the focal point of the camera is the origin C, the image plane is a two-dimensional plane of Xc, Yc, and the optical axis direction (depth) is Zc, and the origin C moves with the movement of the camera.

[0075] On the other hand, the three-dimensional position (M) of object 61, which is the subject of the photograph, is indicated by a world coordinate system consisting of three axes, XwYwZw, with an origin O that does not move with the movement of the camera. The equation indicating the correspondence relationship between the object positions in these different coordinate systems is defined as the pinhole camera model shown above in (Equation 3).

[0076] The value of this (Equation 3) includes the following parameters, as shown in FIG. λ: Regularization parameter A: Camera internal parameters, Cw: camera position, Rw: camera rotation matrix, moreover,

[0077]

number

[0078] is the position on the imaging plane of the camera expressed in a homogeneous coordinate system. λ is a regularization parameter,

[0079]

number

[0080] This is the value that satisfies the third term of the above and can be calculated using a formula.

[0081] As shown in FIG. 13, the camera internal parameter A is expressed by the following determinant:

[0082]

number

[0083] The camera internal parameters A include the following values: f: focal length θ: Orthogonality of image axes (ideal value is 90°) k u : Vertical axis scale (conversion from 3D position scale to 2D image scale) k v : Scale of the horizontal axis (conversion from the scale of the 3D position to the scale of the 2D image) (u0,v0): Image center position

[0084] The above-described (Equation 3), that is, the relational expression between the three-dimensional position M of the object that is the photographed subject and the imaging position (imaging pixel position) m of the camera of the object, is included in (Equation 3). Cw: camera position, Rw: camera rotation matrix, These parameters can be acquired in a SLAM process executed by the user terminal 10, that is, a SLAM (simultaneous localization and mapping) process that executes camera position identification (localization) and environment map creation (mapping) in parallel.

[0085] SLAM processing is a process in which a camera captures images (video) and analyzes the trajectories of feature points contained in multiple captured images to estimate the 3D positions of feature points as well as the position and orientation of the camera (self) (localization), and the 3D position information of the feature points can be used to create (mapping) a map of the surroundings (environmental map). In this way, the process of identifying the position of the camera (self) (localization) and creating (mapping) a map of the surroundings (environmental map) in parallel is called SLAM.

[0086] One of the SLAM techniques is EKF-based SLAM, which uses an Extended Kalman Filter (EKF). EKF-based SLAM is a method in which, for example, images are continuously captured while moving a camera, the trajectory (tracking information) of feature points contained in each image is obtained, and the amount of camera movement and the three-dimensional positions of the feature points are simultaneously estimated using the moving stereo method.

[0087] In this EKF-based SLAM processing, for example, Camera position, orientation, velocity, angular velocity, Position information of each feature point, We use "state data" consisting of multidimensional normal distribution data as a probability distribution model that includes each of these pieces of information. We use a Kalman filter or an extended Kalman filter to update the "state data" and estimate the feature point positions, camera position, etc.

[0088] "State data" is composed of multidimensional normal distribution data including the camera's position, orientation, speed, and angular velocity, as well as the mean vector and variance-covariance matrix indicating the position information of each feature point. The variance-covariance matrix is ​​a matrix that includes the camera's position, orientation, speed, and angular velocity, the position information of each feature point, the [variance] of these inherent state values, and the [covariance] corresponding to the correlation information of combinations of different state values ​​of the above state values.

[0089] The parameters included in the above (Equation 3), namely, λ: Regularization parameter A: Camera internal parameters, Cw: camera position, Rw: camera rotation matrix, Of these parameters, λ and A are known, and Cw and Rw can be calculated by SLAM.

[0090] Using these parameters, it is possible to generate the relational equation between the three-dimensional position M of the object being photographed and the camera's imaging position (imaging pixel position) m of that object, i.e., the above (Equation 3), and analyze the correspondence between the three-dimensional position M of the object being photographed, shown in the world coordinate system, and the object imaging position, shown in the camera coordinate system.

[0091] The above formula (3) is (1) Object position (M) in the world coordinate system, (2) Object imaging position (m) in the camera coordinate system; These are equations that show the positional relationship of each point (Mm) in two different coordinate systems, the world coordinate system and the camera coordinate system. However, the relational equation shown in (Equation 3) is not limited to the combination of the world coordinate system and the camera coordinate system, and can also be expanded as a relational equation that shows the positional relationship of two points (Mm) in any other combination of two different coordinate systems.

[0092] Specifically, for example, (1) Object position (M) in the NED coordinate system, (2) Object imaging position (m) in the camera coordinate system; It is also possible to develop an equation showing the positional relationship of each point (Mm) in these two different coordinate systems, the NED coordinate system and the camera coordinate system.

[0093] In this case, the relation is: (1) Object position (M) in the NED coordinate system, (2) Object imaging position (m) in the camera coordinate system; The relational expression showing the positional relationship between each point (Mm) in these two different coordinate systems, the NED coordinate system and the camera coordinate system, can be expressed as the following (Equation 4).

[0094]

number

[0095] The above (Equation 4) is the parameter corresponding to the world coordinate system of (Equation 3) explained above, that is, Rw: camera rotation matrix, M: object position, Cw: camera position, These parameters correspond to the equations converted into parameters of the NED coordinate system. That is, this is an equation in which the above parameters are changed to the following parameters of the NED coordinate system: R NED : camera rotation matrix, M NED : object position, CNED : Camera position,

[0096] The relational expression shown in (Equation 4) defines the correspondence between the object position in the NED coordinate system and the object imaging position in the camera coordinate system, which is the object imaging position in the imaging element when the object is imaged by the camera. By using this relational expression, the position of the NED coordinate system ( NED P X ) to the position in the camera coordinate system ( C P X ), the coordinate transformation matrix that transforms C T NED This coordinate transformation matrix can be calculated.

[0097] As explained above with reference to Figure 7, each pixel position (u, v) of the display image displayed on the display unit of the user terminal 10 has a one-to-one correspondence with the XY coordinates (Xc, Yc) of the camera coordinate system, and if the position of the flight path shown in the NED coordinate system can be converted to a position in the camera coordinate system, the path can be accurately output on the display image shown in the camera coordinate system.

[0098] Position in the NED coordinate system ( NED P X ) to the position in the camera coordinate system ( C P X ) coordinate transformation matrix C T NED A specific example of the calculation process will be described.

[0099] FIG. 14(1) is a diagram similar to that described above with reference to FIG. 10 and FIG. 11. The user photographs a drone 20 flying in the sky for a predetermined period of time using the user terminal 10. This drone 20 does not have to be a drone carrying a package to be delivered to the user. However, it is necessary that the drone is capable of acquiring the drone's flight path position information (position information on the NED coordinate system). In the example shown in FIG. 14(1), the drone 20 flies from right to left as times (t1), (t2), and (t3) pass.

[0100] The data processing unit of the user terminal 10 records in a memory the shooting image positions of the drone in at least three different positions. As shown in FIG. 14, drone imaging positions 52-1 to 52-3 corresponding to drones at three different positions on a camera imaging surface 51 such as a C-MOS are recorded in a memory.

[0101] The drone positions in the NED coordinate system at times (t1), (t2), and (t3) are shown as follows: Drone position at time (t1) = NED P Dronet1 , Drone position at time (t2) = NED P Dronet2 , Drone position at time (t3) = NED P Dronet3 ,

[0102] Moreover, the imaging positions in the camera coordinate system at times (t1), (t2), and (t3) are expressed as follows: Drone imaging position at time (t1) = m Dronet1 , Drone imaging position at time (t2) = m Dronet2 , Drone imaging position at time (t3) = m Dronet3 , In the above, the (~) at the top of m is omitted. These drone image capture positions are the position information of the camera coordinate system shown in a three-dimensional homogeneous coordinate system.

[0103] The previously explained (Equation 4), that is, the (Equation 4) that defines the correspondence relationship between the object position in the NED coordinate system and the object imaging position in the camera coordinate system, which is the object imaging position in the imaging element when the object is photographed by the camera, is expressed as follows: Drone position in NED coordinate system NED P Drone , Camera position in NED coordinate system NED P C , Drone imaging position in camera coordinate system = m Drone , Using these parameters, it can be expressed as the following (Equation 5).

[0104]

number

[0105] Furthermore, based on the above (Equation 5), the following (Equation 6) is derived.

[0106]

number

[0107] In addition, C R T NED is the rotation matrix to transform the NED coordinate system into the camera coordinate system: C R NED is the transpose of A -1 is the inverse matrix of the camera internal parameter A described above with reference to FIG.

[0108] By substituting the drone positions in the three different NED coordinate systems at times (t1) to (t3) shown in Figure 14 and the drone imaging positions in the camera coordinate system corresponding to these drone positions into the above (Equation 6), we obtain the three simultaneous equations shown in the following (Equation 7).

[0109]

number

[0110] In the simultaneous equations of (Equation 7) above, the following parameters are known: Drone position in NED coordinate system: NED P Drone can be obtained from the drone or the drone management server. Inverse matrix A of camera internal parameters A -1 is known. Drone imaging position from time (t1) to (t3) = m Dronet1 ~m Dronet3 These are coordinate position information of the camera imaging system, and can be obtained by analyzing the image captured by the camera.

[0111] Therefore, the unknown parameters in the simultaneous equations shown above (Equation 7) are Camera position in NED coordinate system: NED P C , Rotation matrix to transform NED coordinate system to camera coordinate system: C R NED Transpose of matrix: C R T NED , Normalization factor: λ t1 ,λ t2 ,λ t3 , These are the parameters.

[0112] Here, the unknown parameters in the simultaneous equations shown above (Equation 7) are Camera position in NED coordinate system: NED P C , Rotation matrix to transform NED coordinate system to camera coordinate system: C R NED Transpose of matrix: C R T NED , Normalization factor: λ t1 ,λ t2 ,λ t3 , These nine parameters (three position elements, three orientation elements, and three normalization coefficients) can be calculated by solving a system of three simultaneous equations (with a volume of information of 9).

[0113] By using the values ​​of these calculation parameters, the position ( NED P X ) to the position in the camera coordinate system ( C P X ) to convert the coordinate transformation matrix ( C TNED ) can be calculated.

[0114] The coordinate transformation matrix ( C T NED ), i.e., the position in the NED coordinate system ( NED P X ) to the position in the camera coordinate system ( C P X ) to convert the coordinate transformation matrix ( C T NED ) is the position ( NED P X ) to the position in the camera coordinate system ( C P X ) to convert the coordinate transformation matrix ( C T NED ) is a matrix in which the world coordinate system elements of the matrix elements are replaced with NED coordinate system elements.

[0115] That is, the position in the NED coordinate system ( NED P X ) to the position in the camera coordinate system ( C P X ) to convert the coordinate transformation matrix ( C T NED ) can be expressed as the following (Equation 8).

[0116]

number

[0117] The coordinate transformation matrix ( C T NED The matrix elements of (x,y) are composed of parameters obtained by solving the simultaneous equations shown in (Equation 7) above. Therefore, by solving the simultaneous equations shown above (Equation 7), the position ( NED P X ) to the position in the camera coordinate system ( C P X ) to convert the coordinate transformation matrix ( C T NED ) can be calculated.

[0118] In this way, the user terminal 10, which is an image processing device of the present disclosure, first acquires drone positions in three different NED coordinate systems at times (t1) to (t3) shown in Figures 14(1) and 15(1) and drone imaging positions in the camera coordinate system corresponding to these drone positions.

[0119] Next, the simultaneous equations shown in (Equation 7) above are solved to obtain the following unknown parameters. Camera position in NED coordinate system: NED P C , Rotation matrix to transform NED coordinate system to camera coordinate system: C R NED Transpose of matrix: C R T NED ,

[0120] Next, the calculated parameters are used to transform the coordinates C T NED ), i.e., the position in the NED coordinate system ( NED P X ) to the position in the camera coordinate system ( C P X ) to convert the coordinate transformation matrix ( C T NED )

[0121] This coordinate transformation matrix ( C T NED ) can be used to convert the flight path position shown in the NED coordinate system to a position shown in the camera coordinate system.

[0122] As previously explained with reference to Figures 3 and 5, the user terminal 10, which is an image processing device of the present disclosure, can obtain flight path information shown in the NED coordinate system from the drone 20 or the drone management server 30, which serves as a control center that controls the drone 20.

[0123] The user terminal 10 converts the acquired flight route information shown in the NED coordinate system into the coordinate transformation matrix shown in the above (Equation 8). C TNED By applying this, it becomes possible to obtain the flight path position indicated in the camera coordinate system, and the flight path position (flight path or planned flight path) indicated in the obtained camera coordinate system is output on an actual image, i.e., an image including an actual drone image.

[0124] As a result, as shown in Figure 16, it is possible to output and display the drone's flight path (flight path or planned flight path) without any deviation from the drone image as a real-life image. The image display example shown in Figure 16 is an AR image display example executed by the user terminal 10, which is an image processing device of the present disclosure, and is a figure explaining an example in which a drone's flight path (flight path or planned flight path) without deviation is output and displayed for a drone image as a real-life image.

[0125] Note that the above-mentioned processing is based on the assumption that the position and orientation of the camera remains unchanged during the shooting periods of the drones in the three different positions, i.e., during the drone shooting periods from time (t1) to (t3) shown in Figure 14(1) and Figure 15(1).

[0126] If the camera position and orientation change during the shooting period of the drones in three different positions, processing must be performed that takes into account the change in the camera position and orientation. Hereinafter, this processing example will be described with reference to FIG. In this processing example, the drone positions in three different NED coordinate systems and the drone imaging positions in the camera coordinate system corresponding to these drone positions are obtained, and the position ( NED P X ) to the position in the camera coordinate system ( C P X ) to convert the coordinate transformation matrix ( C T NED )

[0127] If the position and attitude of the camera changes during the drone's shooting period, the position and attitude of the camera shooting drone 20 in different positions will be different, as shown in Figure 17 (1). In the example shown in Figure 17 (1), the imaging plane of the camera photographing the drone at time (t1) is camera imaging plane 51 (t1), and the imaging plane of the camera photographing the drone at time (t2) is camera imaging plane 51 (t2), which have different positions and attitudes.

[0128] Here, the coordinate transformation matrix that transforms the world coordinate system (SLAM coordinate system) into the camera coordinate system at time (t1) is ( Ct1 T Ws ) In addition, the coordinate transformation matrix that transforms the world coordinate system (SLAM coordinate system) into the camera coordinate system at time (t2) is ( Ct2 T Ws )

[0129] The coordinate transformation matrix ( Ctn T Ws ) is the position ( Ws P X ) to the position in the camera coordinate system ( C P X ) to convert the coordinate transformation matrix ( C T Ws ) and time(n) corresponding matrix.

[0130] A coordinate transformation matrix ( Ctn T Ws The matrix elements constituting (1) can be obtained in the SLAM process executed by the user terminal 10, that is, in the SLAM (simultaneous localization and mapping) process that performs camera position identification (localization) and environment map creation (mapping) in parallel.

[0131] Therefore, the coordinate transformation matrix for transforming the world coordinate system (SLAM coordinate system) shown in FIG. 17 into the camera coordinate system at time (t1) is Ct1 T Ws or, A coordinate transformation matrix ( Ct2 T Ws ), These coordinate transformation matrices for time (tn) ( Ctn T Ws ) can be calculated using SLAM processing.

[0132] In addition, the coordinate transformation matrix for transforming the camera coordinate system at time (t1) to the camera coordinate system at time (t2) is ( Ct2 T Ct1 ) and Ct2 T Ct1 = Ct2 T Ws × Ct1 T Ws -1 It can be calculated as follows:

[0133] The user terminal 10, which is an image processing device of the present disclosure, calculates the above coordinate transformation matrix: Ct2 T Ct1 By this coordinate transformation, the drone imaging position in the camera coordinate system at time (t1) is transformed into the drone imaging position in the camera coordinate system at time (t2). Also, the position of the drone converted into the imaging plane of the camera at time (t1), NED P' Dronet1 can be expressed by the following formula. NED P' Dronet1 = NED P Dronet1 +( Ct2 R T NED )·( Ct2 R Ct1 )·( Ct1 P Ct2 ) As a result, drone imaging positions on two different camera coordinate systems can be transformed into drone imaging positions according to one common camera coordinate system.

[0134] By performing the above processing for drone imaging positions corresponding to three different drone positions, it is possible to set drone imaging positions corresponding to the three different drone positions on one common camera coordinate system.

[0135] Figure 18(1) shows an example where the positions and orientations of the cameras photographing drones in three different positions are different. As shown in FIG. 18(1), the position and orientation of the camera (user terminal 10) are different at times (t1), (t2), and (t3), and images of the drone 20 are taken at times (t1), (t2), and (t3) using cameras with different positions and orientations.

[0136] It is assumed that the flight path and planned flight path of the drone 20 at the latest time (t3) are output. In this case, the data processing unit of the user terminal 10 executes the following process. (1) For the drone imaging position at time (t1), the coordinate transformation matrix ( Ct3 T Ct1 ) to set up an equation to perform the coordinate transformation. (2) For the drone imaging position at time (t2), the coordinate transformation matrix ( Ct3 T Ct2 ) to set up an equation to perform the coordinate transformation. Through these coordinate conversion processes, the drone imaging positions in the camera coordinate system at times (t1) and (t2) are converted to the drone imaging positions in the camera coordinate system at time (t3). By formulating these equations, it is possible to formulate simultaneous equations that set drone imaging positions corresponding to three different drone positions on one common camera coordinate system (camera coordinate system at time (t3)).

[0137] That is, a simultaneous equation can be established that sets the following three draw imaging positions on one common camera coordinate system (camera coordinate system at time (t3)). Drone imaging position at time (t1) = m Dronet1 , Drone imaging position at time (t2) = m Dronet2, Drone imaging position at time (t3) = m Dronet3 , In the above, the (~) at the top of m is omitted. These drone image capture positions are the position information of the camera coordinate system shown in a three-dimensional homogeneous coordinate system.

[0138] Thereafter, the same processing as that previously described with reference to FIGS. 14 and 15 is executed. First, as shown in Figure 18(2), the drone imaging position = m Dronet1 ~m Dronet3 and the corresponding equation with the drone position in the NED coordinate system, that is, the simultaneous equation (7) described above is generated.

[0139] Next, using the parameters obtained by solving the simultaneous equations, the coordinate transformation matrix ( C T NED ), i.e., the position in the NED coordinate system ( NED P X ) to the position in the camera coordinate system ( C P X ) to convert the coordinate transformation matrix ( C T NED ) is calculated.

[0140] This coordinate transformation matrix ( C T NED ), the position of the flight path shown in the NED coordinate system is converted to a position shown in the camera coordinate system, and the flight path position (flight path and planned flight path) shown in the acquired camera coordinate system is output on an actual image, i.e., an image including an actual drone image. In the example of FIG. 18, the flight path position (flight path or planned flight path) is output on the image captured by the camera at time (t3).

[0141] It should be noted that thereafter, in the case where the flight path position (flight path or planned flight path) is to be output continuously at times (t4), (t5), and so on, the above-mentioned processing may be continuously executed.

[0142] When outputting the flight path position (flight path or planned flight path) on the camera image at time (t4), the drone imaging positions at times (t2), (t3), and (t4) = m Dronet2 ~m Dronet4 and the corresponding equation with the drone position in the NED coordinate system, that is, the simultaneous equation (7) described above is generated.

[0143] When outputting the flight path position (flight path or planned flight path) on the camera image at time (t5), the drone imaging positions at times (t3), (t4), and (t5) = m Dronet3 ~m Dronet5 and the corresponding equation with the drone position in the NED coordinate system, that is, the simultaneous equation (7) described above is generated.

[0144] If the same process is continued, the flight path position (flight path or planned flight path) on the image captured by the camera will move in accordance with the movement of the camera, and the correct flight path will be continuously output.

[0145] [3. Sequence of processing executed by the image processing device of the present disclosure] Next, a processing sequence executed by the image processing device of the present disclosure will be described.

[0146] The flowcharts shown in FIG. 19 and subsequent figures are diagrams illustrating a processing sequence executed by an image processing device of the present disclosure, for example, a user terminal 10 such as a smartphone.

[0147] The processing according to the flow shown in FIG. 19 and subsequent figures can be executed under the control of a control unit (data processing unit) composed of a CPU and other components having the program execution function of the image processing device, in accordance with a program stored in a memory inside the image processing device. The process of each step in the flow shown in FIG. 19 and subsequent figures will be described below in order.

[0148] Note that the process of steps S111 to S114 and the process of steps S121 to S123 shown in FIG. 19 can be executed in parallel. First, the processes in steps S111 to S114 will be described.

[0149] (Step S111) The process of step S111 is a process of photographing the sky using a drone by the user terminal 10. At time t(n), a drone 20 in flight is photographed using a camera of a user terminal 10 such as a smartphone.

[0150] That is, as described above with reference to Figures 10 and 11, an image of the drone 20 in flight is taken. As shown in Fig. 10, the user photographs a drone 20 flying in the sky for a predetermined period of time using the user terminal 10. This drone 20 does not have to be a drone carrying a package to be delivered to the user. However, it is necessary that the drone is capable of acquiring the drone's flight path position information (position information on the NED coordinate system).

[0151] (Step S112) Next, in step S112, the user terminal 10 calculates drone imaging position information (imaging position information (m Dronet(n) In this document, the (~) at the top of m will be omitted. This drone imaging position is an imaging position indicated in the camera coordinate system (homogeneous coordinate system) at time t(n).

[0152] (Step S113) Next, in step S113, the user terminal 10 receives the drone's position information (position information in the NED coordinate system ( NED P Dronet(n) )) to get the

[0153] As previously explained with reference to Figures 3 and 5, the user terminal 10, which is an image processing device of the present disclosure, can obtain flight path information shown in the NED coordinate system from the drone 20 or a drone management server 30, such as a control center that controls the drone 20.

[0154] (Step S114) Next, in step S114, the user terminal 10 acquires the imaging position information of the drone at time t(n) (imaging position information in the camera coordinate system (m Dronet(n) )) and location information (NED coordinate system location information ( NED P Dronet(n) ) is recorded in memory in correspondence with time t(n).

[0155] Next, the processing of steps S121 to S123 executed in parallel with the processing of steps S111 to S114 will be described.

[0156] (Step S121) In step S121, the user terminal 10 executes the following process. SLAM processing is executed in accordance with time t(n), that is, the timing of the drone photographing in step S111.

[0157] As mentioned above, SLAM processing is a process that performs camera position identification (localization) and environment mapping in parallel.

[0158] (Step S122) Next, in step S122, the user terminal 10 calculates a coordinate transformation matrix ( Ct(n) T Ws ) is calculated.

[0159] The process of step S122 corresponds to the process described above with reference to FIG. 17(1).

[0160] (Step S123) Next, in step S123, the user terminal 10 calculates the coordinate transformation matrix ( Ct(n) T Ws ), that is, the coordinate transformation matrix ( Ct(n) T Ws ) is recorded in memory.

[0161] When the processes of steps S111 to S114 and steps S121 to S123 are completed, the process of step S124 is executed.

[0162] (Step S124) In step S124, the user terminal 10 determines whether or not the number of entries recorded in the memory has reached three or more. That is, it is determined whether data based on images taken at three different drone positions is recorded in memory.

[0163] A specific example of record data recorded in the memory will be described with reference to FIG. Figure 20 shows an example of recorded data based on images captured from three different drone positions.

[0164] As shown in Fig. 20, the memory records entries corresponding to the time (t(n) etc.) taken by a drone in the sky. Each entry records the following data corresponding to the time taken by a drone: (1) Time (t) (= drone shooting time) (2) Drone imaging position in camera coordinate system (3) Drone position in NED coordinate system (4) A coordinate transformation matrix that transforms the world coordinate system (SLAM coordinate system) into the camera coordinate system.

[0165] "(2) Drone imaging position in camera coordinate system" is drone imaging position information (imaging position information in camera coordinate system (m Dronet(n) ) etc.

[0166] "(3) Drone position in NED coordinate system" is the position information of the drone in the sky at the image capture timing acquired in step S113 of the flow shown in FIG. 19 (position information in the NED coordinate system ( NED P Dronet(n) ) etc.

[0167] "(4) Coordinate transformation matrix for transforming the world coordinate system (SLAM coordinate system) into the camera coordinate system" is a coordinate transformation matrix ( Ct(n) T Ws etc.).

[0168] Each entry in the memory corresponding to a shooting time records data corresponding to these drone shooting times. In step S124 of the flowchart shown in FIG. 19, it is determined whether or not the number of entries recorded in the memory has reached three or more. That is, it is determined whether data based on images taken at three different drone positions is recorded in memory.

[0169] As shown in FIG. 20, when data based on images captured at three different drone positions is recorded in memory, the process proceeds to the next step S131. On the other hand, if data based on images captured at three different drone positions is not recorded in the memory, the determination in step S124 is No, and the process proceeds to step S125.

[0170] (Step S125) In step S125, the time setting parameter n is set to the next time n+1, and the processes of steps S111 to S114 and steps S121 to S123 are executed at the next time (n+1). In other words, at time (n+1), an image of a drone in a different position than at time (n) is captured and processed.

[0171] (Step S131) In step S124, if it is determined that data based on images captured at three different drone positions as shown in FIG. 20 is recorded in the memory, the process proceeds to step S131.

[0172] As shown in FIG. 21, the user terminal 10 executes the following process in step S131. A coordinate transformation matrix ( Ct(out) T Ct(n) etc.) and record them in memory. This process corresponds to the process previously described with reference to FIG.

[0173] Referring to FIG. 22, the coordinate transformation matrix ( Ct(out) T Ct(n) Examples of such cases will be explained below. The data recorded in the memory in step S131 is the data (5) shown in FIG. (5) A coordinate transformation matrix that converts the camera coordinate system during drone photography into the camera coordinate system when outputting the flight path This is the data.

[0174] In the example shown in FIG. 22, among the three shooting time corresponding entries shown in FIG. 22, time=t(n+2) is set as the drone flight path output time t(out). That is, t(n+2)=t(out) Let us assume that.

[0175] In this case, the coordinate transformation matrix that is additionally recorded as "(5) coordinate transformation matrix that transforms the camera coordinate system at the time of drone shooting into the camera coordinate system at the time of flight path output" is as shown in Figure 22. There are only shooting time = t(n) and shooting time = t(n+1). For the data at shooting time = t(n + 2), the camera coordinate system at the time of drone shooting matches the camera coordinate system at the time of flight path output, so there is no need to additionally record the coordinate transformation matrix.

[0176] For the entry for the shooting time = t(n), A coordinate transformation matrix that converts the camera coordinate system (Ct(n)) at the time of drone shooting (t=(n)) into the camera coordinate system (Ct(out)) at the time of flight path output, Ct(out) T Ct(n) Add the following:

[0177] Also, for the entry with shooting time = t(n+1), A coordinate transformation matrix that converts the camera coordinate system (Ct(n+1)) at the time of drone shooting (t=(n+1)) into the camera coordinate system (Ct(out)) at the time of flight path output, Ct(out) T Ct(n+1) Add the following:

[0178] (Step S132) As shown in the flow of FIG. 21, next, the user terminal 10 executes the following process in step S132.

[0179] The coordinate transformation matrix ( Ctc T Ctn ) is applied to perform coordinate conversion processing, and the drone imaging position corresponding to the camera coordinate system at the drone flight path output time t(out) is calculated and recorded in memory.

[0180] This process corresponds to the process previously described with reference to FIG. 17 and FIG. 18(1).

[0181] With reference to FIG. 22, an example of the drone imaging position corresponding to the camera coordinate system at the drone flight path output time t(out) recorded in the memory will be described. The data recorded in the memory in step S132 is data (6) shown in FIG. (6) Drone imaging position corresponding to the camera coordinate system at drone flight path output time t(out) This is the data.

[0182] In the example shown in FIG. 22, among the three shooting time corresponding entries shown in FIG. 22, time=t(n+2) is set as the time t(out) at which the drone flight path is output. That is, t(n+2)=t(out) Let us assume that.

[0183] In this case, the user terminal 10 calculates the following data and records them in memory in step S132. The processing for the data of drone shooting time = t(n) is as follows. The camera coordinate system (C t(n) ) drone imaging position (m Dront(n) ), the coordinate transformation matrix ( Ct(out) T Ct(n) That is, the following coordinate transformation process is performed. λ(m Dronetn )=A·( Ctout T NED )·( NED P Dronetn ) The coordinates obtained by the above calculation formula are the drone imaging position corresponding to the camera coordinate system at the drone flight path output time t(out). This coordinate position is recorded in memory.

[0184] In addition, the processing for the data of drone shooting time = t(n+1) is as follows. The camera coordinate system (C t(n+1) ) drone imaging position (m Dront(n+1) ), the coordinate transformation matrix ( Ct(out) T Ct(n+1) That is, the following coordinate transformation process is performed. λ(m Dronetn+1 )=A·( Ctout T NED )·( NED P Dronetn+1 ) The coordinates obtained by the above calculation formula are the drone imaging position corresponding to the camera coordinate system at the drone flight path output time t(out). This coordinate position is recorded in memory.

[0185] In addition, the processing for the data of drone shooting time = t(n+2) is as follows. The camera coordinate system (C t(n+2) ) is the camera coordinate system (C t(out) ). Therefore, coordinate transformation is not necessary, and the camera coordinate system (C t(n+2) ) drone imaging position (m Dront(n+2) ) is recorded in memory as is. These recorded data are the data recorded in item (6) shown in FIG.

[0186] Next, the process from step S133 onward in the flow shown in FIG. 23 will be described. (Step S133) In step S133, the user terminal 10 executes the following process.

[0187] A simultaneous equation (Equation 7) is generated consisting of correspondence equations between the drone positions in the NED coordinate system of three different positions recorded in the memory and the drone imaging positions (imaging positions in the camera coordinate system at time t(out)) corresponding to each drone position.

[0188] The simultaneous equations to be generated are the simultaneous equations previously described with reference to FIG. 14(2), FIG. 15(2), and FIG. 18(2), and are the simultaneous equations previously described as (Equation 7). In addition, the drone imaging position (m Dronetn ), the position calculated in step S132, i.e., the drone imaging position corresponding to the camera coordinate system at the drone flight path output time t(out), is used. That is, the converted coordinate position recorded in item (6) of the memory recording data described with reference to FIG. 22 is used.

[0189] (Step S134) Next, in step S134, the user terminal 10 calculates a coordinate transformation matrix ( Ct(out) T NED ), i.e., the position in the NED coordinate system ( NED P X ) to the position in the camera coordinate system ( Ct(out) P X ) to convert the coordinate transformation matrix ( Ct(out) T NED ) (Equation 8) is calculated.

[0190] This coordinate transformation matrix ( Ct(out) T NED ) is the coordinate transformation matrix ( C T NED ) which corresponds to the coordinate transformation matrix ( C T NED )

[0191] (Step S135) Next, in step S135, the user terminal 10 calculates the coordinate transformation matrix ( Ct(out) T NED ) to transform the drone flight path position in the NED coordinate system to a position in the camera coordinate system.

[0192] The drone flight path (already flown path and planned flight path) in the NED coordinate system is obtained from the drone 20 or the drone management server 30.

[0193] (Step S136) Next, in step S136, the user terminal 10 outputs the flight path converted into position information in the camera coordinate system obtained by the coordinate conversion in step S135 to the display unit of the user terminal.

[0194] Through this processing, it becomes possible to accurately display the drone's route information on the captured image. That is, as shown in FIG. 16 described above, it is possible to output and display the drone's flight path (flight path or planned flight path) without deviation from the drone image as a real-life image.

[0195] [4. Configuration examples of the image processing device and drone of the present disclosure] Next, configuration examples of the image processing device and drone of the present disclosure will be described.

[0196] FIG. 24 is a diagram showing an example configuration of a user terminal 100 as an image processing device of the present disclosure, and a drone 200.

[0197] The user terminal 100 as an image processing device of the present disclosure is, for example, a communication terminal with a camera such as a smartphone, etc. It is not limited to a smartphone, and may be a device such as a PC or a camera device.

[0198] The user terminal 100 is configured to be able to communicate with the drone 200 and the drone management server 300. The drone 200 flies according to a predetermined flight path using, for example, communication information with the drone management server 300 and communication information with a GPS satellite 400.

[0199] As shown in the figure, a user terminal 100 includes a camera 101, a data processing unit 102, a storage unit (memory) 103, a communication unit 104, a display unit 105, an input unit 106, and an output unit 107.

[0200] The camera 101 is used, for example, for taking images for drone photography and SLAM processing.

[0201] The data processing unit 102 performs output control of the flight path of the drone described above. That is, the data processing unit 102 performs processing such as generating an AR image in which the flight path is superimposed on an actual image of the drone or the like and displaying the image on the display unit 105. Furthermore, it controls the processes executed in the user terminal 100, such as SLAM processing and image capture control. The data processing unit 102 includes a processor such as a CPU having a program execution function, and executes processing according to a program stored in the storage unit 103 .

[0202] The storage unit (memory) 103 is used as a storage area and a work area for the programs executed by the data processing unit 102. It is also used as a storage area for various parameters applied to the processing. The storage unit (memory) 103 is composed of a RAM, a ROM, etc.

[0203] The communication unit 104 executes communication with the drone 200 and the drone management server 300. For example, the communication unit 104 performs a process of receiving flight path information of the drone 200 from the drone 200 or the drone management server 300.

[0204] The display unit 105 displays the image captured by the camera, and further outputs the flight path information of the drone generated by the data processing unit 102. That is, it displays an AR image in which the flight path is superimposed on the actual image of the drone or the like.

[0205] The input unit 106 is a unit operated by the user, and is used for various processes, such as inputting user requests for image capture, starting and ending route display, and the like. The output unit 107 includes an audio output unit, an image output unit, and the like.

[0206] Next, the configuration of the drone 200 will be described. The drone 200 has a route planning unit 201, a route control unit 202, a positioning sensor (GPS information receiving and analyzing unit) 203, and a communication unit 204.

[0207] The route planning unit 201 plans and determines a flight route of the drone 200. For example, based on information received from the drone management server 300, the route planning unit 201 plans and determines a specific flight route.

[0208] The route control unit 202 executes flight control to fly the drone 200 according to the flight route determined by the route planning unit 201.

[0209] The positioning sensor (GPS information receiving and analyzing unit) 203 communicates with GPS satellites 400, analyzes the current position (latitude, longitude, altitude) of the drone 200 based on the communication information with the GPS satellites 400, and outputs the analysis information to the route control unit 202. The route control unit 202 refers to input information from a positioning sensor (GPS information receiving and analyzing unit) 203 and executes flight control for flying the drone 200 according to the flight route determined by the route planning unit 201.

[0210] The communication unit 204 performs communication with the drone management server 300 and the user terminal 100.

[0211] In the above-described embodiment, an example of the display processing for a drone's flight path was described. However, the processing of the present disclosure is not limited to the display of a drone's flight path, and can also be applied to the display processing of route information for other moving bodies, such as robots and autonomous vehicles. Similar processing can be performed by replacing the drones in the above-described embodiments with robots or autonomous vehicles.

[0212] [5. Summary of the configuration of this disclosure] The embodiments of the present disclosure have been described in detail above with reference to specific embodiments. However, it is obvious that a person skilled in the art can modify or substitute the embodiments without departing from the gist of the present disclosure. In other words, the present invention has been disclosed in the form of an example, and should not be interpreted as being limited. In order to judge the gist of the present disclosure, the claims should be taken into consideration.

[0213] The technology disclosed in this specification can have the following configurations. (1) A data processing unit is provided that displays a moving route of a mobile device on a display unit that displays an image captured by a camera of the mobile device; The data processing unit includes: execute a coordinate conversion process for converting position information according to a first coordinate system that indicates a moving path of the moving device into a second coordinate system that can identify a pixel position of a display image on the display unit; an image processing device that outputs to a display unit a travel path having position information according to the second coordinate system generated by the coordinate conversion process.

[0214] (2) An image processing method executed in an image processing device, the image processing device has a data processing unit that displays a moving route of the mobile device on a display unit that displays an image captured by a camera of the mobile device; The data processing unit: execute a coordinate conversion process for converting position information according to a first coordinate system that indicates a moving path of the moving device into a second coordinate system that can identify a pixel position of a display image on the display unit; an image processing method for outputting, to a display unit, a travel path having position information according to the second coordinate system generated by the coordinate conversion process;

[0215] (3) The mobile device is a drone, the first coordinate system is a NED coordinate system; The image processing method according to (2), wherein the second coordinate system is a camera coordinate system corresponding to a camera that photographs the mobile device.

[0216] (4) The data processing unit generating a relational expression indicating a correspondence relationship between a position of the mobile device corresponding to the first coordinate system and an imaging position in an image captured by the mobile device; The image processing method according to (2) or (3), further comprising generating a coordinate transformation matrix to be applied to the coordinate transformation process using parameters calculated from the generated relational expression.

[0217] (5) The data processing unit generating simultaneous equations including relational expressions that indicate a correspondence relationship between a position of the mobile device corresponding to the first coordinate system and an imaging position in an image captured by the mobile device; The image processing method according to any one of (2) to (4), further comprising generating a coordinate transformation matrix to be applied to the coordinate transformation process using parameters calculated from the generated simultaneous equations.

[0218] (6) The data processing unit includes: The image processing method according to (5), further comprising generating the simultaneous equations using images captured by a camera of the mobile device at a plurality of different positions.

[0219] (7) The data processing unit includes: The image processing method according to (5) or (6), further comprising generating a coordinate transformation matrix in which parameters obtained from the simultaneous equations are set as matrix elements of the coordinate transformation matrix.

[0220] (8) The data processing unit includes: The image processing method according to any one of (5) to (7), further comprising generating the simultaneous equations consisting of three relational expressions using images captured by a camera of the mobile device at three different positions.

[0221] (9) The data processing unit includes: When the positions or attitudes of the cameras of the mobile device in the multiple different positions are different from each other in the images captured by the cameras, An image processing method according to any one of (6) to (8), which calculates a coordinate transformation matrix for transforming a plurality of different camera coordinates during image capture into a single camera coordinate, and executes a process of setting the imaging positions of mobile devices at a plurality of different positions onto the single camera coordinate by applying the calculated camera coordinate transformation matrix.

[0222] (10) The data processing unit includes: The image processing method according to (9), further comprising: executing a SLAM (simultaneous localization and mapping) process to calculate a position and orientation of a camera that captures the moving device; and calculating the camera coordinate transformation matrix using a result of the SLAM process.

[0223] (11) The data processing unit includes: Based on the result of the SLAM process, Calculating a world coordinate camera coordinate transformation matrix for performing coordinate transformation between a world coordinate system, which is a coordinate system applied to the SLAM processing, and a camera coordinate system corresponding to the camera; The image processing method according to (10), further comprising calculating a camera coordinate transformation matrix for transforming a plurality of different camera coordinates at the time of capturing an image into a single camera coordinate by applying the calculated world coordinate camera coordinate transformation matrix.

[0224] (12) The data processing unit includes: Obtaining flight path information according to the NED coordinate system, which is the first coordinate system, from a drone, which is the mobile device, or a drone management server that manages the drone; The image processing method according to any one of (2) to (11), further comprising converting the acquired flight path information into a camera coordinate system, which is the second coordinate system.

[0225] (13) The data processing unit includes: The image processing method according to any one of (2) to (12), further comprising displaying a route already traveled and a planned route of travel of the mobile device on the display unit that displays an image captured by a camera of the mobile device.

[0226] (14) A program for causing an image processing device to execute image processing, the image processing device has a data processing unit that displays a moving route of the mobile device on a display unit that displays an image captured by a camera of the mobile device; The program causes the data processing unit to a coordinate conversion process for converting position information according to a first coordinate system that indicates a moving path of the moving device into a second coordinate system that can identify a pixel position of a display image on the display unit; A program that causes a process of outputting, to a display unit, a travel path having position information according to the second coordinate system generated by the coordinate conversion process.

[0227] In addition, the series of processes described in the specification can be executed by hardware, software, or a combination of both. When executing processes by software, a program recording the processing sequence can be installed and executed in a memory in a computer built into dedicated hardware, or the program can be installed and executed in a general-purpose computer capable of executing various processes. For example, the program can be pre-recorded on a recording medium. In addition to installing the program from the recording medium to the computer, the program can be received via a network such as a LAN (Local Area Network) or the Internet, and installed on a recording medium such as a built-in hard disk.

[0228] In addition, the various processes described in the specification may not only be executed in chronological order as described, but may also be executed in parallel or individually depending on the processing capacity of the device executing the processes or as necessary. Furthermore, in this specification, a system refers to a logical collective configuration of multiple devices, and is not limited to devices in the same housing. [Industrial Applicability]

[0229] As described above, according to one embodiment of the present disclosure, a configuration is realized that makes it possible to accurately display the flight path of a drone on an actual image of the drone. Specifically, for example, the display unit displays an image captured by a camera of a mobile device such as a drone, and the data processing unit displays the moving route of the mobile device. The data processing unit generates a coordinate transformation matrix that performs coordinate transformation processing to convert position information according to a first coordinate system, for example, the NED coordinate system, which indicates the moving route of the mobile device, into a second coordinate system, for example, the camera coordinate system, which can identify the pixel position of the display image on the display unit, and outputs the moving route having position information according to the camera coordinate system generated by the coordinate transformation processing to which the generated coordinate transformation matrix is ​​applied to the display unit. This configuration makes it possible to accurately display the drone's flight path on a live image of the drone. [Explanation of symbols]

[0230] 10 User terminal 20. Drone 30 Drone Management Server 40 GPS satellites 51 Camera imaging surface 61 Objects 62 Object Images 100 user terminals 101 Camera 102 Data processing section 103 Memory unit (memory) 104 Communications Department 105 Display section 106 Input section 107 Output section 200 Drone 201 Route Planning Department 202 Route control unit 203 Positioning sensor (GPS information reception and analysis section) 204 Communications Department 300 Drone Management Server 400 GPS satellites

Claims

1. a data processing unit that displays a moving route of the mobile device on a display unit that displays an image of the mobile device captured by a camera; The data processing unit includes: execute a coordinate conversion process for converting position information according to a first coordinate system that indicates a moving path of the moving device into a second coordinate system that can specify a pixel position of a display image on the display unit; A process of outputting a moving path having position information according to the second coordinate system generated by the coordinate conversion process to a display unit is executed; The data processing unit includes: In the process of generating a coordinate transformation matrix to be applied to the coordinate transformation process, generating simultaneous equations including relational expressions that indicate a correspondence relationship between a position of the mobile device corresponding to the first coordinate system and an imaging position in a captured image of the mobile device captured by a camera; Executing a process of generating the coordinate transformation matrix using parameters calculated from the generated simultaneous equations; an image processing device that performs processing using images captured by a camera of the mobile device at a plurality of different positions in the process of generating the simultaneous equations;

2. An image processing method executed in an image processing device, the image processing device has a data processing unit that displays a moving route of the moving device on a display unit that displays a captured image of the moving device captured by a camera; The data processing unit: a coordinate conversion processing step of converting position information according to a first coordinate system that indicates a moving path of the moving device into a second coordinate system that can specify a pixel position of a display image on the display unit; a movement path display step of outputting a movement path having position information according to the second coordinate system generated in the coordinate conversion processing step to a display unit; The data processing unit includes: In the coordinate conversion processing step, a simultaneous equation generating step of generating simultaneous equations including relational expressions indicating a correspondence relationship between a position of the mobile device corresponding to the first coordinate system and an imaging position in an image captured by a camera of the mobile device; A coordinate transformation matrix generating step is performed to generate a coordinate transformation matrix to be applied to the coordinate transformation processing performed in the coordinate transformation processing step, using parameters calculated from the generated simultaneous equations; The data processing unit includes: In the simultaneous equation generating step, An image processing method for generating the simultaneous equations by using images captured by a camera of the mobile device at a plurality of different positions.

3. the mobile device is a drone; the first coordinate system is a NED coordinate system; 3. The image processing method according to claim 2, wherein the second coordinate system is a camera coordinate system corresponding to a camera that photographs the mobile device.

4. The data processing unit includes:

3. The image processing method according to claim 2, further comprising the step of generating a coordinate transformation matrix in which parameters obtained from the simultaneous equations are set as matrix elements of the coordinate transformation matrix.

5. The data processing unit includes:

3. The image processing method according to claim 2, wherein the simultaneous equations consisting of three relational expressions are generated using images captured by a camera of the mobile device at three different positions.

6. The data processing unit includes: When the position or posture of the camera in the captured images of the mobile device at a plurality of different positions is different, The image processing method according to claim 2, further comprising the steps of: calculating a coordinate transformation matrix for transforming a plurality of different camera coordinates during image capture into a single camera coordinate; and applying the calculated camera coordinate transformation matrix to set the imaging positions of a mobile device at a plurality of different positions onto the single camera coordinate.

7. The data processing unit includes:

7. The image processing method according to claim 6, further comprising the steps of: executing a simultaneous localization and mapping (SLAM) process to calculate a position and orientation of a camera that captures the image of the moving device; and calculating the camera coordinate transformation matrix by using a result of the SLAM process.

8. The data processing unit includes: Based on the results of the SLAM process, Calculating a world coordinate camera coordinate transformation matrix for performing coordinate transformation between a world coordinate system, which is a coordinate system applied to the SLAM processing, and a camera coordinate system corresponding to the camera; The image processing method according to claim 7 , wherein the calculated world coordinate camera coordinate transformation matrix is ​​applied to calculate the camera coordinate transformation matrix for transforming a plurality of different camera coordinates at the time of capturing an image into a single camera coordinate.

9. The data processing unit includes: Acquire flight path information according to the NED coordinate system, which is the first coordinate system, from a drone, which is the mobile device, or a drone management server that manages the drone; The image processing method according to claim 2 , wherein the acquired flight path information is transformed into the camera coordinate system, which is the second coordinate system.

10. The data processing unit includes:

3. The image processing method according to claim 2, further comprising the step of displaying a route already traveled and a planned route of travel of the mobile device on the display unit which displays an image of the mobile device photographed by a camera.

11. A program for causing an image processing device to execute image processing, the image processing device has a data processing unit that displays a moving route of the moving device on a display unit that displays a captured image of the moving device captured by a camera; The program causes the data processing unit to a coordinate conversion processing step of converting position information according to a first coordinate system that indicates a moving path of the moving device into a second coordinate system that can specify a pixel position of a display image on the display unit; a movement path display step of outputting a movement path having position information according to the second coordinate system generated in the coordinate conversion processing step to a display unit; In the coordinate conversion processing step, a simultaneous equation generating step of generating simultaneous equations including relational expressions indicating a correspondence relationship between a position of the mobile device corresponding to the first coordinate system and an imaging position in an image captured by a camera of the mobile device; executing a coordinate transformation matrix generating step of generating a coordinate transformation matrix to be applied to the coordinate transformation processing executed in the coordinate transformation processing step, using parameters calculated from the generated simultaneous equations; In the simultaneous equation generating step, A program that generates the simultaneous equations by using images captured by a camera of the mobile device at multiple different positions.

Images