Image processing device, image processing method, and program

Abstract

To provide an image processing device capable of appropriately displaying images necessary for operation.SOLUTION: An image processing device 1 includes an altitude information acquisition unit 11 that acquires the altitude of a mobile body, a photographic image acquisition unit 12 that acquires multiple images taken by multiple photographing units provided included in the mobile body, and a peripheral image generation unit 13 that composes a plurality of images on the basis of altitudes to generate an image of the periphery around a landing place. The peripheral image generation unit 13 generates a peripheral image as viewed from a virtual viewpoint located at a predetermined distance from the altitude of the mobile body when the altitude of the mobile body is less than or equal to a first altitude and greater than or equal to a second altitude.SELECTED DRAWING: Figure 7

Description

[Technical field]

[0001] The present disclosure relates to an image processing device, an image processing method, and a program. [Background technology]

[0002] In recent years, a technology for supporting the landing of a moving object such as a drone has been known. As a related technology, Patent Document 1 discloses a descent system for landing an unmanned aerial vehicle at a target position. The descent system disclosed in Patent Document 1 has a target sign arranged in an area including the target position. The target sign is composed of a plurality of levels of floor signs having different external sizes in the horizontal direction, and floor signs having relatively small external sizes are arranged inside floor signs having relatively large external sizes. When a floor sign having a relatively large external size can be recognized within the shooting range of an image acquisition means (e.g., a camera), the unmanned aerial vehicle descends while controlling the floor sign having a relatively large external size to be positioned within the shooting range. Furthermore, when a floor sign having a relatively large external size becomes unrecognizable within the shooting range as a result of the descent of the unmanned aerial vehicle, the unmanned aerial vehicle descends while controlling the floor sign having a relatively small external size to be positioned within the shooting range. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2021-172318 A Summary of the Invention [Problem to be solved by the invention]

[0004] In the case of a moving body that takes off and lands by remote control or automatic piloting, the moving body is aligned using information from a global positioning system (GPS) or the like, so there is a possibility of misalignment. If misalignment occurs, the camera equipped on the moving body cannot properly photograph the surroundings of the landing site. In this case, the user operating the moving body cannot visually confirm images of the surroundings of the landing site, which may make it difficult to perform accurate takeoff and landing of the moving body. Patent Document 1 does not fully consider such problems.

[0005] In view of the above-mentioned problems, an object of the present disclosure is to provide an image processing device, an image processing method, and a program capable of appropriately displaying images necessary for operations. [Means for solving the problem]

[0006] The image processing device according to the present disclosure includes: an altitude information acquisition unit that acquires the altitude of a moving object; a captured image acquisition unit that acquires a plurality of images captured by a plurality of image capture units included in the moving object; a surrounding image generating unit that synthesizes the plurality of images based on the altitude and generates a surrounding image of the surroundings of the landing site; Equipped with The peripheral image generating unit is When the altitude of the moving body is equal to or lower than a first altitude and equal to or higher than a second altitude, the surrounding image viewed from a virtual viewpoint located at a predetermined distance from the altitude of the moving body is generated.

[0007] The image processing method according to the present disclosure includes: The computer an altitude information acquisition step of acquiring the altitude of a moving object; a photographed image acquisition step of acquiring a plurality of images photographed by a plurality of photographing units included in the moving object; a surrounding image generating step of synthesizing the plurality of images based on the altitude to generate a surrounding image of the surroundings of the landing site; Run In the peripheral image generating step, When the altitude of the moving body is equal to or lower than a first altitude and equal to or higher than a second altitude, the surrounding image viewed from a virtual viewpoint located at a predetermined distance from the altitude of the moving body is generated.

[0008] The program according to the present disclosure is an altitude information acquisition step of acquiring the altitude of a moving object; a photographed image acquisition step of acquiring a plurality of images photographed by a plurality of photographing units included in the moving object; a surrounding image generating step of synthesizing the plurality of images based on the altitude to generate a surrounding image of the surroundings of the landing site; Run the following on your computer: In the peripheral image generating step, When the altitude of the moving body is equal to or lower than a first altitude and equal to or higher than a second altitude, the surrounding image viewed from a virtual viewpoint located at a predetermined distance from the altitude of the moving body is generated. Effect of the Invention

[0009] The image processing device, image processing method, and program according to the present disclosure make it possible to appropriately display images required for an operation. [Brief description of the drawings]

[0010] [Figure 1] 1 is a block diagram showing an overall configuration of an image processing system according to the present disclosure. [Diagram 2] 1 is a block diagram showing a functional configuration of a moving body according to the present disclosure. [Diagram 3] FIG. 2 is a diagram showing camera information relating to a camera according to the present disclosure. [Figure 4A] FIG. 2 is a diagram illustrating the mounting position of a camera according to the present disclosure. [Figure 4B] FIG. 2 is a diagram illustrating the mounting position of a camera according to the present disclosure. [Diagram 5] 1A and 1B are diagrams illustrating the mounting angle of a camera according to the present disclosure. [Figure 6]FIG. 13 is a diagram showing a composite image based on photographed images acquired by a camera in a peripheral image according to the present disclosure. [Figure 7] FIG. 1 is a block diagram showing a functional configuration of an image processing device according to the present disclosure. [Figure 8] FIG. 1 illustrates an example of a predetermined pattern provided on the ground surface according to the present disclosure. [Figure 9] 1 is a diagram showing the relationship between the altitude of a moving body and the height of a virtual viewpoint according to the present disclosure. [Figure 10] 10A to 10C are diagrams showing examples of surrounding images corresponding to each position of the moving object in FIG. [Figure 11] 4 is a flowchart showing a process performed by an image processing device according to the present disclosure. [Figure 12] FIG. 13 is a diagram illustrating an example of parameter information according to the present disclosure. [Figure 13] FIG. 2 is a diagram for explaining the angle of a camera with respect to the Z axis according to the present disclosure. [Figure 14] FIG. 13 is a diagram showing an example of an angle of a camera relative to the Z axis according to the present disclosure. [Figure 15] FIG. 13 is a diagram showing an example of a target area image according to the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In each drawing, the same or corresponding elements are denoted by the same reference numerals. For clarity of explanation, duplicate explanations will be omitted as necessary.

[0012] (Image processing system 100) An image processing system 100 according to this embodiment will be described with reference to Fig. 1. Fig. 1 is a block diagram showing the overall configuration of the image processing system 100. The image processing system 100 includes an image processing device 1 and a moving object 2. Although one image processing device 1 and one moving object 2 are shown in Fig. 1, the number of image processing devices 1 and the number of moving objects 2 are not limited to one. The number of image processing devices 1 may be any number equal to or greater than one. The number of moving objects 2 may also be any number equal to or greater than one.

[0013] The image processing device 1 and the mobile object 2 are connected via a network N. The network N is a wired or wireless communication line. The network N may be configured using, for example, a personal area network, a mesh network, a mobile communication network such as 3G, 4G, or 5G, the Internet, a public telephone line network, or a satellite communication network.

[0014] The image processing system 100 is a system capable of supporting the landing of a moving object 2 by acquiring images captured by a plurality of cameras equipped on the moving object 2 with an image processing device 1 and performing predetermined processing in the image processing device 1. The image processing device 1 is constituted by an information processing device such as a PC (Personal Computer).

[0015] The moving body 2 is, for example, a drone capable of flying autonomously in a space. Instead of a drone, another moving body may be used as the moving body 2. For example, an air vehicle such as an airplane may be used as the moving body 2. The space through which the moving body 2 moves may be outdoors or indoors.

[0016] (Mobile unit 2) The moving object 2 will be described with reference to Fig. 2. Fig. 2 is a block diagram showing the functional configuration of the moving object 2. The moving object 2 includes an imaging control unit 21, an altitude detection unit 22, a position detection unit 23, a movement control unit 24, a communication unit 27, and a camera unit 20. The camera unit 20 includes five cameras C0 to C4.

[0017] The moving object 2 includes a processor, a memory, and a storage device, which are not shown in the figure. The storage device stores a computer program that implements the processing according to this embodiment. The processor can load the computer program from the storage device into the memory and execute the computer program. In this way, the processor realizes the functions of an imaging control unit 21, an altitude detection unit 22, a position detection unit 23, and a movement control unit 24.

[0018] Alternatively, the imaging control unit 21, the altitude detection unit 22, the position detection unit 23, and the movement control unit 24 may each be realized by dedicated hardware. Also, some or all of the components of each device may be realized by a general-purpose or dedicated circuit, processor, etc., or a combination of these. These may be configured by a single chip, or may be configured by multiple chips connected via a bus. Some or all of the components of each device may be realized by a combination of the above-mentioned circuits, etc., and a program.

[0019] The photographing control unit 21 controls photographing in the camera unit 20. The photographing control unit 21 transmits photographed images acquired by photographing in the camera unit 20 to the image processing device 1 via the communication unit 27. The photographing control unit 21 may transmit the photographed images to the image processing device 1 at a predetermined time interval.

[0020] The altitude detection unit 22 detects the altitude of the moving body 2. The altitude of the moving body 2 may be a relative altitude from the ground surface or an absolute altitude such as above sea level. In this embodiment, the altitude of the moving body 2 is described using the relative altitude from the ground surface where the moving body 2 lands. The altitude detection unit 22 detects the vertical distance from the center of the landing place to the moving body 2 as the altitude. The altitude detection unit 22 includes, for example, an altitude sensor, and detects the altitude of the moving body 2 using the altitude sensor. The altitude sensor performs altitude positioning using, for example, a transmitting antenna (not shown) and a receiving antenna (not shown) installed at the bottom of the moving body, a radio altimeter, or the like. The altitude detection unit 22 transmits altitude information indicating the detected altitude to the image processing device 1 via the communication unit 27. The altitude detection unit 22 may transmit the altitude information to the image processing device 1 at a predetermined time interval.

[0021] The position detection unit 23 detects the position of the moving object 2. The position of the moving object 2 indicates the position of the moving object 2 in the horizontal direction. The position detection unit 23 includes, for example, a position sensor, and detects the position of the moving object 2 using the position sensor. The position sensor performs positioning using, for example, a technology such as a Global Navigation Satellite System (GNSS) including a Global Positioning System (GPS). The position detection unit 23 transmits position information indicating the detected position to the image processing device 1 via the communication unit 27. The position detection unit 23 may transmit the position information to the image processing device 1 at a predetermined time interval.

[0022] The movement control unit 24 controls the movement of the moving body 2. The movement control unit 24 controls, for example, the rotation of a propeller unit (not shown) equipped on the moving body 2. The propeller unit is configured to enable the moving body 2 to ascend, descend, turn left and right, and move forward and backward. The movement control unit 24 moves the moving body 2 in the atmosphere by controlling the rotation of the propeller unit.

[0023] The communication unit 27 communicates with the image processing device 1. The communication unit 27 may be, for example, a communication interface for wireless communication. The type of communication is not limited.

[0024] The camera unit 20 includes a plurality of cameras C0 to C4. Each of the cameras C0 to C4 functions as an imaging unit that captures images in a predetermined imaging direction. The cameras C0 to C4 are attached at different positions on the main body 2a of the moving object 2 shown in Figs. 4A and 4B described below. The cameras C0 to C4 capture images from their respective attachment positions. Each of the cameras C0 to C4 outputs captured images acquired by imaging to the imaging control unit 21. This allows the imaging control unit 21 to acquire captured images corresponding to the five imaging ranges.

[0025] Camera C0 is attached to the lower center of the bottom surface of main body 2a, and captures the area directly below moving body 2. Hereinafter, camera C0 may be referred to as center camera C0. Cameras C1 to C4 are attached to the front, rear, left side, and right side of main body 2a, respectively, and capture images in a predetermined direction from their respective attachment positions. Camera C1 is a front camera that captures images in front of moving body 2. Camera C2 is a rear camera that captures images behind moving body 2. Camera C3 is a left camera that captures images on the left side of moving body 2. Camera C4 is a right camera that captures images on the right side of moving body 2.

[0026] The cameras C0 to C4 are imaging devices such as color cameras that are configured with, for example, a lens, an image sensor such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor), a driver that digitizes a video signal output by the image sensor, a DRAM (Dynamic Random Access Memory) that temporarily stores image data for image processing, and a DSP (Digital Signal Processor), etc. Here, the cameras C0 to C4 are described as fisheye cameras that are equipped with a fisheye lens or the like and are capable of wide-angle imaging.

[0027] The cameras C0 to C4 capture images in their respective shooting directions to obtain fisheye images that are significantly distorted. Note that cameras other than fisheye cameras may be used as the cameras C0 to C4. Furthermore, the cameras C0 to C4 may have the same configuration, or may have different configurations. Here, the cameras C0 to C4 are described as all having the same configuration. Note that, although the description will be given using five cameras C0 to C4, the number of cameras included in the camera unit 20 is not limited to five. The camera unit 20 may have any number of cameras equal to or greater than two.

[0028] The camera unit 20 will be described in detail with reference to Figs. 3 to 6. Fig. 3 is a diagram showing camera information 91 relating to cameras C1 to C4. Figs. 4A and 4B are diagrams explaining the mounting positions of cameras C0 to C4. Fig. 5 is a diagram explaining the mounting angles of cameras C1 to C4. Fig. 6 is a diagram showing a composite image based on the captured images acquired by cameras C1 to C4 in the peripheral image.

[0029] 3, the camera information 91 includes various parameters related to the cameras C1 to C4. The camera information 91 may be set in advance by a user or the like. Here, it is assumed that the preset camera information 91 is stored in a storage unit 19 of the image processing device 1 shown in FIG. 7, which will be described later. The camera information 91 includes, as examples of parameters, information related to the mounting positions (X, Y, H) of the cameras C1 to C4 and the mounting angles (A, B, C) of the cameras C1 to C4.

[0030] FIG. 4A shows the mounting positions of cameras C1 to C4 when the moving body 2 is viewed from the top. FIG. 4B shows the mounting positions of cameras C0 and C2 to C4 when the moving body 2 is viewed from the side. P1 shown in FIG. 4A indicates the center of the main body 2a of the moving body 2. Camera mounting position X indicates the left-right distance between the center P1 and each camera. Furthermore, camera mounting position Y indicates the front-rear distance between the center P1 and each camera. Furthermore, camera mounting position H indicates the up-down distance between the center P1 and each camera. The center camera C0 is attached to the bottom surface at the back side of the main body 2a of the moving body 2 shown in FIG. 4A.

[0031] FIG. 5 shows the mounting angle of camera C1 when the moving body 2 is viewed obliquely. Here, the mounting angle will be explained using camera C1, but it is similar for other cameras. The Z axis shown in the figure indicates the shooting direction of camera C1. Mounting angle A indicates the angle between the vertical direction of the mounting surface of camera C1 and the Z axis. The mounting surface of camera C1 is approximately parallel to the vertical direction. Mounting angle B indicates the angle between the vertical direction of the mounting surface of camera C1 and the left-right direction of camera C1. Mounting angle C indicates the rotation angle around the Z axis, which is the shooting direction of camera C1.

[0032] FIG. 6 shows the shooting ranges of the cameras C1 to C4 in the peripheral image. FIG. 6 is also a diagram showing a composite image generated by image synthesis. I1 to I4 shown in FIG. 6 indicate areas of a composite image based on the captured images acquired by the cameras C1 to C4. As will be described later, the image processing device 1 uses a predetermined pattern M to convert the captured images acquired by the cameras C1 to C4 into images of the composite image areas I1 to I4, respectively, and synthesizes them. In this way, the image processing device 1 generates one composite image from the four captured images.

[0033] (Image processing device 1) Next, the image processing device 1 will be described with reference to Fig. 7. Fig. 7 is a block diagram showing the functional configuration of the image processing device 1. The image processing device 1 includes an altitude information acquisition unit 11, a captured image acquisition unit 12, a surrounding image generation unit 13, a projection image generation unit 14, a display control unit 15, a display unit 16, a communication unit 17, and a storage unit 19.

[0034] The image processing device 1 includes a processor and a memory as components not shown. A computer program in which the processing according to this embodiment is implemented is stored in a storage unit 19. The processor can load the computer program from the storage unit 19 into the memory and execute the computer program. In this way, the processor realizes the functions of an altitude information acquisition unit 11, a captured image acquisition unit 12, a surrounding image generation unit 13, a projection image generation unit 14, and a display control unit 15.

[0035] Alternatively, the altitude information acquisition unit 11, the photographed image acquisition unit 12, the peripheral image generation unit 13, the projection image generation unit 14, and the display control unit 15 may each be realized by dedicated hardware. Also, some or all of the components of each device may be realized by a general-purpose or dedicated circuit, processor, etc., or a combination of these. These may be configured by a single chip, or may be configured by multiple chips connected via a bus. Some or all of the components of each device may be realized by a combination of the above-mentioned circuits, etc., and a program.

[0036] The altitude information acquiring unit 11 acquires altitude information indicating the altitude of the moving object 2. The altitude information acquiring unit 11 may acquire the altitude information from the moving object 2 at predetermined time intervals.

[0037] The captured image acquisition unit 12 acquires a plurality of images captured by a plurality of imaging units included in the moving object 2. In this embodiment, the captured image acquisition unit 12 acquires captured images captured by each of the five cameras C0 to C4. The captured image acquisition unit 12 may acquire these captured images from the moving object 2 at a predetermined time interval.

[0038] The surrounding image generating unit 13 synthesizes the multiple images acquired by the photographed image acquiring unit 12 based on the altitude information acquired by the altitude information acquiring unit 11. In this way, the surrounding image generating unit 13 generates a surrounding image around the landing site of the moving body 2. When the altitude of the moving body 2 is between the first altitude and a second altitude lower than the first altitude (when the altitude of the moving body 2 is equal to or lower than the first altitude and equal to or higher than the second altitude), the surrounding image generating unit 13 generates a surrounding image viewed from a virtual viewpoint located at a predetermined distance from the altitude of the moving body 2. For example, the surrounding image generating unit 13 generates a surrounding image viewed from a virtual viewpoint at a height corresponding to the altitude of the moving body 2. Also, when the altitude of the moving body 2 is less than the second altitude, the surrounding image generating unit 13 generates a surrounding image viewed from a virtual viewpoint at a height corresponding to the second altitude. Note that the second altitude is an altitude lower than the first altitude by a predetermined altitude.

[0039] The virtual viewpoint is a virtual viewpoint for viewing the moving body 2 from above. The position of the virtual viewpoint is set at a position a predetermined distance away from the center position of the moving body 2 in a planar view in a straight upward direction. The viewing direction of the virtual viewpoint is set in a straight downward direction from the position of the virtual viewpoint toward the moving body 2. In this way, the peripheral image generating unit 13 generates an image of the moving body 2 viewed from a virtual viewpoint located directly above the moving body 2. The peripheral image is an image of the moving body 2 viewed from a virtual viewpoint, and includes the surroundings of the landing site. Specifically, the peripheral image generating unit 13 synthesizes the images captured by the cameras C1 to C4 using a predetermined pattern M provided on the ground surface of the landing target.

[0040] FIG. 8 is a diagram showing an example of a predetermined pattern M provided on the ground surface. The ground surface is, for example, the ground or floor surface. The pattern M includes a marker for identifying the position of the pattern M from the sky. The pattern M and the marker are provided in a size that can be photographed by the cameras C1 to C4. The marker is, for example, a combination of a rectangle, a dot, etc. The marker is not limited to this, and may include various figures, characters, etc. In order to detect a change in the size of the pattern M in the photographed image, the marker is provided, for example, at regular intervals. The surrounding image generating unit 13 generates a surrounding image by synthesizing images using the markers included in the pattern M. Note that the shape of the pattern M is not limited to the rectangular shape shown in the figure, and may be any shape. A specific method for generating a surrounding image will be described later.

[0041] Returning to FIG. 7, the explanation will be continued. The projection image generating unit 14 generates a projection image of the moving body 2 based on the altitude information acquired by the altitude information acquiring unit 11. The projection image is an image to be superimposed on a surrounding image to assist the landing of the moving body 2. The projection image is, for example, an image imitating the appearance of the moving body 2. By using the projection image, the user can intuitively grasp the current position of the moving body 2. Note that the projection image may be any figure, icon, or the like.

[0042] When the altitude of the moving body 2 is between the first altitude and the second altitude, the projection image generating unit 14 generates a projection image of a predetermined size. When the altitude of the moving body 2 is less than the second altitude, the projection image generating unit 14 generates a projection image of a size according to the altitude of the moving body 2. A specific method for generating the projection image will be described later.

[0043] The display control unit 15 causes the display unit 16 to display the peripheral image generated by the peripheral image generation unit 13. The display control unit 15 also causes the display unit 16 to display a superimposed image in which the peripheral image generated by the peripheral image generation unit 13 and the projection image generated by the projection image generation unit 14 are superimposed. The display control unit 15 may cause the display unit 16 to display an image in which a target area image indicating a target area for landing is further superimposed on the superimposed image. The display control unit 15 may also cause the display unit 16 to display an image in which only the target area image is superimposed on the peripheral image. In other words, the display control unit 15 may cause at least one of the projection image and the target area image to be superimposed on the peripheral image.

[0044] The display unit 16 outputs the images generated by the peripheral image generation unit 13 and the projection image generation unit 14. The display unit 16 is, for example, a display device such as a liquid crystal display. The display unit 16 may be a touch panel or the like having the function of an input unit. Here, the image processing device 1 is described using a configuration in which it is equipped with the display unit 16, but the display unit 16 may be provided outside the image processing device 1. For example, the display unit 16 may be a display unit such as a liquid crystal display equipped in a terminal used by a user.

[0045] The communication unit 17 communicates with the mobile object 2. The communication unit 17 may be, for example, a communication interface for wireless communication. The type of communication is not limited.

[0046] The storage unit 19 is composed of, for example, a semiconductor memory, and stores various data and programs. At least a part of the storage unit 19 is composed of a non-volatile memory so that data is retained even when the power of the image processing device 1 is turned off. The storage unit 19 stores camera information 91 and parameter information 92 in addition to various data and programs.

[0047] The configuration of the image processing system 100 has been described above. The configuration of the image processing system 100 described above is merely an example and may be modified as appropriate. For example, when some or all of the components of the image processing device 1 are realized by a plurality of information processing devices, circuits, etc., the plurality of information processing devices, circuits, etc. may be centrally arranged or distributed. For example, the information processing devices, circuits, etc. may be realized as a client-server system, a cloud computing system, etc., in a form in which each is connected via a communication network. In addition, the functions of the image processing device 1 may be provided in a SaaS (Software as a Service) format.

[0048] In addition, in the above description, the image processing device 1 and the moving body 2 are provided separately, but the image processing device 1 may be provided in the moving body 2. Alternatively, a part of the configuration of the image processing device 1 may be provided in the moving body 2. For example, a peripheral image and a projected image may be generated in the moving body 2, and the generated peripheral image and projected image may be transmitted from the moving body 2 to the image processing device 1.

[0049] (overview) Next, an overview of the processing performed by the image processing system 100 will be described. The image processing device 1 generates one image from a plurality of captured images by synthesizing (stitching processing) the plurality of captured images acquired by the cameras C1 to C4. Specifically, the image processing device 1 synthesizes the captured images of the cameras C1 to C4 according to a pattern M of the captured image of the camera C0 captured in advance in the initial calibration.

[0050] The image processing device 1 calculates parameters for correcting the images captured by each camera using coordinates indicating the mounting positions of the cameras C1 to C4 and their positional relationship with the pattern M installed around the landing site, and performs synthesis processing. The parameters include, for example, correction values ​​for rotation, enlargement, reduction, and distortion of the captured images.

[0051] Here, the initial calibration performed by the image processing device 1 will be described. The initial calibration is a process that is required before performing image synthesis processing. The image processing device 1 performs the initial calibration at a stage before the moving body 2 starts landing. Here, the image processing device 1 performs the initial calibration at a timing when the moving body 2 is flying at a predetermined altitude at which the cameras C0 to C4 can photograph the landing site pattern M from directly above. This is to avoid a situation in which, when the moving body 2 lowers its altitude to land, the center camera C0 cannot photograph the surroundings of the landing site due to the influence of a narrow angle of view, etc., and the initial calibration described below cannot be performed.

[0052] At the start of the initial calibration, the moving body 2 photographs the pattern M with the cameras C0 to C4 and transmits the photographed images to the image processing device 1. The image processing device 1 stores the transmitted photographed images in the storage unit 19 or the like. The image processing device 1 analyzes the images of the pattern M photographed by each of the cameras C1 to C4 in order to correctly combine the images photographed by the cameras C1 to C4 at the altitude at which the pattern M was photographed. The image processing device 1 performs the initial calibration to calculate the initial values ​​of the image correction values ​​(rotation angle, enlargement / reduction ratio, distortion correction value, etc.) of the cameras C1 to C4 using the coordinates indicating the mounting positions of the cameras C1 to C4 so that the pattern M in the images photographed by the cameras C1 to C4, the pattern M in the image photographed by the camera C0, and the pattern M in the images photographed by the cameras C1 to C4 correspond to each other. This completes the initial calibration based on the ground surface of the landing site at the altitude at which the pattern M was photographed. The image processing device 1 may perform the initial calibration using a known method.

[0053] After performing the initial calibration, the moving body 2 lowers its altitude for landing. The image processing device 1 acquires images taken while the moving body 2 descends from the altitude at which the initial calibration was performed to the target landing location. The image processing device 1 also acquires altitude information at the timing when the images were captured. The image processing device 1 adjusts the image correction values ​​of the cameras C1 to C4 based on the acquired altitude information, and generates a peripheral image for assisting the landing based on an image corrected using the adjusted image correction value. The target landing location is an area where a pattern M is provided. The target landing location may be a predetermined area within the pattern M, or may be a predetermined point. The peripheral image includes the pattern M and a peripheral area of ​​the pattern M.

[0054] Furthermore, the image processing device 1 generates a projected image of the moving object 2 based on the altitude information. The image processing device 1 generates a superimposed image in which the surrounding image and the projected image are superimposed, and displays the superimposed image on the display unit 16. This allows the user to perform landing operations for the moving object 2 while viewing the superimposed image displayed on the display unit 16.

[0055] In addition, as the moving object 2 descends in altitude, the center camera C0 may not be able to capture the pattern M because the moving object 2 is close to the landing site. In such a case, the user cannot visually recognize the pattern M in the peripheral image, making it difficult to land the moving object 2 correctly. To prevent this, the image processing device 1 may switch the superimposed image to be displayed when the moving object 2 is too close to the landing site and the camera C0 cannot capture an image, or when the camera C0 cannot capture an image of the area below the moving object 2. Specifically, the image processing device 1 switches the superimposed image to be displayed from an image of only the center camera C0 to a composite image of the cameras C1 to C4. This allows the user to visually recognize the pattern M.

[0056] When the moving body 2 descends toward the landing site, the image processing device 1 changes parameters related to altitude according to changes in altitude information of the moving body 2. Specifically, the image processing device 1 adjusts the image correction values ​​and composition positions of the cameras C1 to C4 according to changes in the altitude information of the moving body 2. The image processing device 1 corrects the images of the cameras C1 to C4 using the image correction values ​​of the cameras C1 to C4 adjusted according to changes in the altitude information, and adjusts the composition positions of the corrected images of the cameras C1 to C4 according to the positions and intervals of the corresponding markers in the pattern M of each image of the cameras C1 to C4 acquired in the initial calibration.

[0057] In the present disclosure, the landing place of the moving body 2 is a horizontal ground surface. In addition, it is assumed that the image processing device 1 performs the initial calibration in a state in which the horizontality of the moving body 2 is automatically maintained by the control of the movement control unit 24. Therefore, the image processing device 1 can perform adjustments so that an image identical to the surrounding image including the pattern M can be obtained by the synthesis process simply by changing the parameters related to the altitude. As a result, the image processing device 1 can appropriately generate the surrounding image even at an altitude different from that at the time of the initial calibration by changing the parameters related to the altitude.

[0058] When adjusting parameters related to altitude, the image processing device 1 calculates the distance between the markers of pattern M from the captured image by converting the image into a central projection image while taking into account the distortion characteristics of the camera lens, and then calculates the distance between the markers. In this way, the image processing device 1 can obtain a surrounding image based on the ground surface of the landing site, thereby supporting a safe landing.

[0059] It is also possible to change parameters without performing the above-mentioned initial calibration. In that case, the parameters to be changed will be diverse, and it will require an increased convergence time and a huge amount of computational power until the adjustment is complete. For this reason, the above-mentioned initial calibration and calibration adaptation method based on altitude changes are effective.

[0060] (Specific examples) Next, a specific example of the processing performed by the image processing system 100 will be described with reference to Fig. 9 and Fig. 10. Fig. 9 is a diagram showing the relationship between the altitude h of the moving body 2 and the height of the virtual viewpoint. The dashed line shown in Fig. 9 indicates the field of view from the virtual viewpoint. Fig. 10 is a diagram showing an example of a surrounding image corresponding to each position of the moving body 2 in Fig. 9.

[0061] FIG. 9 shows an example in which the moving object 2 gradually descends from the leftmost position (a) to the rightmost position (d) at an altitude h. Virtual viewpoints V1 to V4 are virtual viewpoints at the positions (a) to (d), respectively. FIG. 10 shows peripheral images Q1 to Q4 corresponding to the positions (a) to (d) shown in FIG. 9. Pattern M shown in FIG. 10 is a simplification of pattern M shown in FIG. 8. It is assumed that the above-mentioned initial calibration was performed at a timing when the moving object 2 flies higher in the sky than the altitude h at (a).

[0062] The peripheral image generating unit 13 of the image processing device 1 generates peripheral images Q1 to Q4, which are images of the moving object 2 viewed from the virtual viewpoints V1 to V4, respectively. Here, an example of four peripheral images Q1 to Q4 is shown, but the peripheral image generating unit 13 may generate any number of peripheral images. The virtual viewpoints V1 to V4 are set to be located directly above the moving object 2.

[0063] In Fig. 9, a height of 10 m above the ground surface is set as the first altitude H1. The first altitude H1 indicates a height corresponding to a position where a predetermined pattern M provided at a landing site on the ground can be photographed from the cameras C0 to C4. Also in the same figure, a height of 3 m above the ground surface is set as the second altitude H2. The second altitude H2 is set to an altitude lower than the first altitude H1.

[0064] In (a) and (b), the moving body 2 flies between a first altitude H1 and a second altitude H2, that is, between 3m and 10m above the ground. In (c) and (d), the moving body 2 flies at an altitude less than the second altitude H2, that is, at a position less than 3m above the ground. The distances between the virtual viewpoints V1 to V4 and the moving body 2 are respectively defined as virtual viewpoint heights d1 to d4.

[0065] Here, the heights d1 and d2 of the virtual viewpoints are set to 3 m. In this case, the distance between each virtual viewpoint and the moving body 2 is constant at 3 m. Therefore, in (a) and (b), the virtual viewpoints V1 and V2 are set to higher positions as the altitude h of the moving body 2 increases, and are set to lower positions as the altitude h of the moving body 2 decreases. Note that the distance between each virtual viewpoint and the moving body 2 does not have to be constant.

[0066] Also, in (c) and (d), the heights d3 and d4 of the virtual viewpoints are set according to the second altitude H2. The heights d3 and d4 of the virtual viewpoints are, for example, 3 m higher than the second altitude H2. By setting the heights of the virtual viewpoints in this way, the position of the virtual viewpoints does not change even if the moving body 2 further descends from the second altitude H2. As shown by the virtual viewpoints V3 and V4, the position of the virtual viewpoints is fixed even if the moving body 2 continues to descend from the second altitude H2. In this case, the distance between the virtual viewpoints and the moving body 2 increases as it descends.

[0067] 10, in the cases of (a) and (b), the peripheral image generation unit 13 generates peripheral images Q1 and Q2 viewed from virtual viewpoints V1 and V2, respectively, at heights corresponding to the altitude h of the moving object 2. In addition, in the cases of (c) and (d), the peripheral image generation unit 13 generates peripheral images Q3 and Q4 viewed from virtual viewpoints V3 and V4, respectively, at heights corresponding to a second altitude H2. In the cases of (c) and (d), the peripheral image generation unit 13 generates peripheral images viewed from fixed virtual viewpoints.

[0068] The projection image generating unit 14 of the image processing device 1 generates projection images T1 to T4 to be displayed by being superimposed on the peripheral images Q1 to Q4. In (a) and (b), the projection image generating unit 14 generates projection images T1 and T2 of a predetermined size. The predetermined size may be appropriately set by a user or the like. In this way, the projection image generating unit 14 generates projection images T1 and T2 of a fixed size. In (c) and (d), the projection image generating unit 14 generates projection images T3 and T4 of a size according to the altitude h of the moving object 2. For example, when the size of the projection images T1 and T2 is 100%, the projection image generating unit 14 generates the projection images T3 and T4 so that the size is less than 100%. In addition, the projection image generating unit 14 may generate the projection images T3 and T4 so that the image becomes smaller as the altitude h of the moving object 2 becomes lower (the height of the virtual viewpoint becomes higher). This allows the projection image generating unit 14 to generate projection images T3 and T4 whose sizes correspond to the change in the height of the virtual viewpoint.

[0069] In this way, the image processing device 1 can generate a peripheral image and a projected image according to the altitude h of the moving object 2. The image processing device 1 generates a superimposed image by superimposing the generated projected image on the peripheral image, and displays the superimposed image on the display unit 16. As a result, for example, in the case of a moving object 2 flying between the first altitude H1 and the second altitude H2, the virtual viewpoint is fixed, so that the user can easily align the moving object 2 with respect to the pattern M. Also, in the case of a moving object 2 descending and flying to an altitude lower than the second altitude H2, the projected image is generated with a variable size according to the height of the virtual viewpoint, so that the user can confirm the exact position of the moving object 2 with respect to the pattern M in an overhead image, and can more easily align the moving object 2.

[0070] (Processing of image processing device 1) The process performed by the image processing device 1 will be described in detail with reference to Fig. 11. Fig. 11 is a flowchart showing the process performed by the image processing device 1. Note that the description will be given here assuming that the above-mentioned initial calibration has been completed.

[0071] In step S10, the altitude information acquisition unit 11 acquires altitude information indicating the altitude of the moving body 2. For example, the altitude information acquisition unit 11 acquires altitude information transmitted from the moving body 2 at a predetermined time interval. The altitude information may be acquired in advance and stored in the storage unit 19.

[0072] In step S11, the captured image acquisition unit 12 acquires captured images from the cameras C0 to C4. For example, the captured image acquisition unit 12 acquires captured images transmitted from the moving object 2 at a predetermined time interval. The captured images may be acquired in advance and stored in the storage unit 19. The order of steps S11 and S12 may be reversed.

[0073] In step S12, the surrounding image generating unit 13 acquires parameter information. The parameter information indicates information on various parameters used in the processing of the image processing device 1. The parameter information may be stored in advance in the storage unit 19, or may be acquired by the surrounding image generating unit 13 performing a predetermined calculation. Here, it is assumed that the surrounding image generating unit 13 acquires parameter information 92 stored in advance in the storage unit 19.

[0074] 12 is a diagram showing an example of parameter information 92. The parameter information 92 includes, for example, information on the mounting angle A of the camera, the pattern width w, the height d of the virtual viewpoint, the rotation angle of the captured image, and the size of the projected image. Although not shown, the parameter information 92 may also include information on a first altitude and a second altitude. Here, it is assumed that the first altitude is set to 10 m and the second altitude is set to 3 m.

[0075] The mounting angle A of the camera corresponds to the mounting angle A of the cameras C1 to C4 described with reference to Figures 3 and 5. Here, the mounting angle A is 30 degrees. The pattern width w indicates the width of the pattern M provided at the landing site. Here, the pattern width w is 5 m.

[0076] The height d of the virtual viewpoint indicates the distance between the virtual viewpoint and the moving body 2, as described with reference to FIG. 9. For example, when the altitude h of the moving body 2 is 10 m, the height d of the virtual viewpoint is 3 m directly above the moving body 2. As shown in the figure, when the altitude h of the moving body 2 is 3 m to 10 m, the height d of the virtual viewpoint is fixed at a position 3 m directly above the moving body 2. Therefore, the height of the virtual viewpoint based on the ground surface in this case is set to a height according to the altitude h of the moving body 2. On the other hand, when the altitude h of the moving body 2 is less than 3 m, the height d of the virtual viewpoint is set to fluctuate to a value larger than when the altitude h of the moving body 2 is 3 m to 10 m. The height of the virtual viewpoint based on the ground surface in this case is set to a height according to the second altitude.

[0077] The rotation angle of a captured image is information indicating the angle by which the captured images of cameras C1 to C4 are rotated when combining images. Since cameras C1 to C4 capture images in different directions from their respective mounting positions on moving object 2, different rotation angles are set for each. As shown in the figure, the rotation angles of the captured images of cameras C1 to C4 are 0 degrees, 180 degrees, 270 degrees, and 90 degrees, respectively.

[0078] The size of the projected image indicates the size of the projected image corresponding to the altitude h of the moving object 2. As shown in the figure, the size of the projected image is set to be different when the altitude h is between 3m and 10m and when it is less than 3m. The above-mentioned parameter information 92 is an example, and the parameter information 92 may include information other than the above-mentioned information. Note that although the altitude h is expressed in units of meters in the figure, the unit of the altitude h may be a finer unit.

[0079] Returning to Fig. 11, the description will continue. In step S13, the peripheral image generating unit 13 calculates the angle θ of the camera with respect to the Z axis.

[0080] FIG. 13 is a diagram for explaining the angle θ of the camera C1 with respect to the Z axis. Here, the explanation will be given using the camera C1, but the same applies to other cameras. The Z axis of the camera C1 is the shooting direction of the camera C1. The angle θ indicates the angle between the straight line connecting the camera C1 and the end of the pattern M and the Z axis indicating the shooting direction of the camera C1 when the moving body 2 is viewed from the side. The angle θ changes according to the altitude h of the moving body 2. It is assumed that the moving body 2 descends from the sky in a straight posture parallel to the ground surface G and lands in the center of the landing site in accordance with the position of the pattern M installed at the landing site.

[0081] The peripheral image generating unit 13 calculates the angle θ of the camera C1 with respect to the Z axis using the following formula (1). Angle θ = Mounting angle A of camera C1 - Angle δ according to altitude h (1)

[0082] In equation (1), the angle δ according to the altitude h is expressed by the following equation (2). Angle δ = atan(((pattern width w / 2)-x) / height h) (2)

[0083] In formula (2), atan represents the arc tangent. Also, x represents the amount of deviation from the center of pattern M to the mounting position of camera C1. For example, if camera C1 is located vertically above the center of pattern M, x=0.

[0084] 14 is a diagram showing an example of the angle θ of the camera C1 with respect to the Z axis calculated by the above-mentioned calculation. As shown in the figure, the angle θ is calculated according to the altitude h of the moving object 2. The surrounding image generating unit 13 may store the altitude h and the angle θ in the storage unit 19 in association with each other.

[0085] Returning to Fig. 11, the description will be continued. In step S14, the peripheral image generating unit 13 performs coordinate conversion on the captured image of the camera C1 using the calculated angle θ. As a result, the peripheral image generating unit 13 develops the captured image from a fisheye image to a planar image. The peripheral image generating unit 13 may perform the coordinate conversion using, for example, a pinhole model.

[0086] In step S15, the peripheral image generating unit 13 further performs coordinate conversion of the planar image converted in step S14 into an overhead image viewed from the virtual viewpoint height d, using the virtual viewpoint height d. The peripheral image generating unit 13 may perform, for example, perspective projection conversion.

[0087] In step S16, the peripheral image generating unit 13 rotates the overhead image whose coordinates have been converted in step S15 in accordance with the mounting position of the camera. The peripheral image generating unit 13 acquires a rotation angle of the overhead image by referring to the parameter information 92, and rotates the overhead image according to the acquired rotation angle.

[0088] In step S17, the peripheral image generation unit 13 determines whether the processing of the cameras C1 to C4 is complete. If it is determined that the processing of the cameras C1 to C4 is not complete (NO in step S17), the peripheral image generation unit 13 returns to step S12 and repeats the subsequent processes. If it is determined that the processing of the cameras C1 to C4 is complete (YES in step S17), the peripheral image generation unit 13 proceeds to step S18.

[0089] In step S18, the peripheral image generating unit 13 generates a peripheral image. Specifically, the peripheral image generating unit 13 synthesizes the overhead images of the cameras C1 to C4 that have been subjected to the processes of steps S12 to S16. The peripheral image generating unit 13 may synthesize the overhead images using a known method. For example, the peripheral image generating unit 13 generates the peripheral image using a method of cutting out overlapping image areas and pasting together the remaining image areas, a method of synthesizing by alpha blending, or the like.

[0090] In step S19, the projection image generating unit 14 generates a projection image. Specifically, the projection image generating unit 14 acquires the size of the projection image according to the altitude h by referring to the parameter information 92. The projection image generating unit 14 generates the projection image based on the acquired size of the projection image.

[0091] For example, assume that the altitude h of the moving object 2 is 2 m from the ground surface. In the example of parameter information 92 shown in FIG. 12, a size R2 is associated with an altitude h of 2 m. The size R2 indicates 75% of the size R1 associated with an altitude h of 3 m to 10 m. Therefore, the projection image generating unit 14 generates a projection image at 75% of the size R1. For example, the projection image generating unit 14 may generate a projection image of a corresponding size by linearly decreasing the size of the projection image (for example, the ratio of the vertical and horizontal widths) with respect to an increment in the distance between the virtual viewpoint and the moving object 2.

[0092] In step S20, the peripheral image generating unit 13 generates a superimposed image by synthesizing the projected image with the peripheral image. The superimposed image is an image in which the projected image is superimposed on the peripheral image, as shown in Fig. 10. Note that the peripheral image generating unit 13 may generate an image in which a target area image indicating a target area for landing is further superimposed on the superimposed image.

[0093] FIG. 15 is a diagram showing an example of a target area image. In the diagram, an example is shown in which a target area image U is further superimposed on an image in which a projection image T is superimposed on a peripheral image Q. The target area image U is, for example, a predetermined area or a predetermined point provided within a pattern M. The peripheral image generating unit 13 may generate the target area image U so as to be distinguishable from other areas, for example, by shading or color display. In this way, for example, as shown in the diagram, when the projection image T (the current position of the moving body 2) is deviated from the position of the target area image U, the user can adjust the position of the moving body 2. Note that the image processing device 1 may further support the landing by generating character information or audio information for supporting the landing and outputting the generated information.

[0094] Returning to Fig. 11, the description will continue. In step S21, the display control unit 15 causes the superimposed image to be displayed on the display unit 16. This allows the user to visually recognize the superimposed image.

[0095] In the above description, an example in which only the angle θ of the camera C1 with respect to the Z axis is calculated in step S13 has been described, but this is not limiting. The peripheral image generating unit 13 may calculate the angles of the camera C1 with respect to the X axis and the Y axis. The X axis and the Y axis of the camera C1 are mutually orthogonal, and each is orthogonal to the Z axis.

[0096] As described above, in the image processing system 100 according to this embodiment, in the image processing device 1, the altitude information acquisition unit 11 acquires altitude information indicating the altitude of the moving body 2. The captured image acquisition unit 12 acquires a plurality of images captured by a plurality of capture units provided in the moving body 2. The surrounding image generation unit 13 synthesizes the plurality of images based on the altitude information to generate a surrounding image around the landing site of the moving body 2. When the altitude of the moving body 2 is between the first altitude and the second altitude, the surrounding image generation unit 13 generates a surrounding image viewed from a virtual viewpoint at a height corresponding to the altitude of the moving body 2, and when the altitude of the moving body 2 is less than the second altitude, the surrounding image generation unit 13 generates a surrounding image viewed from a virtual viewpoint at a height corresponding to the second altitude.

[0097] Furthermore, the projection image generating unit 14 generates a projection image of the moving object 2 based on the altitude information. When the altitude of the moving object 2 is between the first altitude and the second altitude, the projection image generating unit 14 generates a projection image of a predetermined size, and when the altitude of the moving object 2 is less than the second altitude, the projection image of a size according to the altitude of the moving object 2. The display control unit 15 causes the display unit to display a superimposed image in which the generated peripheral image and the projection image are superimposed.

[0098] With this configuration, the image processing device 1 can appropriately display images necessary for operation, thereby supporting the user in accurate landing of the moving body 2. By visually checking the displayed images, the user can ensure the safety of operating the moving body 2.

[0099] Each functional component of the image processing device 1 and the mobile object 2 described above may be realized by hardware that realizes each functional component (e.g., hardwired electronic circuitry, etc.), or may be realized by a combination of hardware and software (e.g., a combination of an electronic circuitry and a program that controls it, etc.). For example, any process in the present disclosure can be realized by having a CPU (Central Processing Unit) execute a computer program.

[0100] The program includes instructions (or software code) that, when loaded into a computer, cause the computer to perform one or more functions described in the embodiments. The program may be stored on various types of non-transitory computer readable medium or tangible storage medium. By way of example and not limitation, non-transitory computer readable medium or tangible storage medium includes random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drive (SSD) or other memory technology, CD-ROM, digital versatile disc (DVD), Blu-ray® disk or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage device. The program may also be transmitted on various types of transitory computer readable medium or communication medium. By way of example and not limitation, transitory computer readable medium or communication medium includes electrical, optical, acoustic, or other form of propagated signal.

[0101] The present disclosure is not limited to the above-described embodiment, and can be modified as appropriate without departing from the spirit and scope of the present disclosure. [Explanation of symbols]

[0102] 1 Image processing device 2. Mobile 2a Main body 11 Advanced information acquisition section 12 Image capture unit 13 Surrounding image generation unit 14 Projection image generation unit 15 Display control section 16 Display 17 Communications Department 19 Memory section 20 Camera Section 21 Shooting control section 22 Altitude detection unit 23 Position detection unit 24 Movement control section 27 Communications Department 91 Camera Information 92 Parameter Information 100 Image Processing System A~C Mounting angle C0~C4 Camera d, d1~d4 Virtual viewpoint height G ground surface h altitude H1 1st altitude H2 2nd altitude I1~I4 Composite image area M-Pattern N Network Q, Q1~Q4 surrounding images T, T1~T4 projection images U Target Area Image V1~V4 Virtual viewpoint w Pattern width δ Angle according to altitude θ Angle relative to the camera's Z axis

Claims

1. an altitude information acquisition unit that acquires the altitude of a moving object; a captured image acquisition unit that acquires a plurality of images captured by a plurality of image capture units included in the moving object; a surrounding image generating unit that synthesizes the plurality of images based on the altitude and generates a surrounding image of the surroundings of the landing site; Equipped with The peripheral image generating unit is When the altitude of the moving object is equal to or lower than a first altitude and equal to or higher than a second altitude, the surrounding image viewed from a virtual viewpoint located at a predetermined distance from the altitude of the moving object is generated. Image processing device.

2. a projection image generating unit that generates a projection image of the moving object based on the altitude, The projection image generating unit includes: When the altitude of the moving object is equal to or lower than the first altitude and equal to or higher than the second altitude, the projection image having a predetermined size is generated. The image processing device according to claim 1 .

3. a display control unit that displays the peripheral image on a display unit; The display control unit displays at least one of the projection image and a target area image indicating a target area for landing in a superimposed manner on the peripheral image. The image processing device according to claim 2 .

4. The computer an altitude information acquisition step of acquiring the altitude of a moving object; a photographed image acquisition step of acquiring a plurality of images photographed by a plurality of photographing units included in the moving object; a surrounding image generating step of synthesizing the plurality of images based on the altitude to generate a surrounding image of the surroundings of the landing site; Run In the peripheral image generating step, When the altitude of the moving object is equal to or lower than a first altitude and equal to or higher than a second altitude, the surrounding image viewed from a virtual viewpoint located at a predetermined distance from the altitude of the moving object is generated. Image processing methods.

5. an altitude information acquisition step of acquiring the altitude of a moving object; a photographed image acquisition step of acquiring a plurality of images photographed by a plurality of photographing units included in the moving object; a surrounding image generating step of synthesizing the plurality of images based on the altitude to generate a surrounding image of the surroundings of the landing site; Run the following on your computer: In the peripheral image generating step, When the altitude of the moving object is equal to or lower than a first altitude and equal to or higher than a second altitude, the surrounding image viewed from a virtual viewpoint located at a predetermined distance from the altitude of the moving object is generated. program.

Images