Ground support equipment control apparatus, ground support equipment and control method based on aircraft loading dock position

Abstract

To grasp, as accurately as possible, a loading dock of an aircraft for automatic traveling of ground support equipment.SOLUTION: A control apparatus disclosed is for controlling automatic traveling of ground support equipment of an aircraft to reach the aircraft, and is configured to perform first control comprising: detecting, on the basis of point cloud data of the aircraft measured by a first sensor mounted on the ground support equipment, a position of an edge of a loading dock in a state where a door for closing the loading dock formed on an airframe of the aircraft is open; and using the detected position of the edge to control the traveling of the ground support equipment to the loading dock.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to a control device, ground support equipment, and a control method for ground support equipment based on an aircraft loading dock position. [Background technology]

[0002] Patent Document 1 discloses a travel route setting system that sets travel routes for multiple ground support devices that perform ground support operations for aircraft. This travel route setting system determines the work positions of each ground support device based on aircraft model information, etc., and sets travel routes from the current positions of the ground support devices to the work positions. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2021-178533 A Summary of the Invention

[0004] In Patent Document 1, the working position of the ground support equipment is determined based on aircraft model information and the like. When the aircraft model information is used, the general position of the aircraft can be known, but the exact position cannot be known. In order to load and unload cargo, etc., the ground support equipment of the aircraft needs to arrive at the aircraft entrance in a state of being accurately aligned with the entrance.

[0005] Therefore, in order to enable automated navigation of ground support equipment, it is desirable to determine the aircraft entrance as accurately as possible.

[0006] One aspect of the present disclosure is a control device. The disclosed control device is a control device for controlling automatic travel of ground support equipment of an aircraft to reach the aircraft. The disclosed control device is configured to execute a first control including detecting a position of an end of a loading entrance formed on a body of the aircraft when a cargo door for closing the loading entrance is open based on point cloud data obtained by measuring the aircraft by a first sensor mounted on the ground support equipment, and controlling travel of the ground support equipment to the loading entrance using the detected position of the end.

[0007] Another aspect of the present disclosure is a ground support equipment comprising a control device.

[0008] Yet another aspect of the present disclosure is a control method. The disclosed control method is a control method for automatic travel of ground support equipment of an aircraft to reach the aircraft. The disclosed method includes detecting a position of an end of a service entrance formed on a body of the aircraft when a door for closing the service entrance is open based on point cloud data obtained by measuring the aircraft by a first sensor mounted on the ground support equipment, and controlling travel of the ground support equipment to the service entrance using the detected position of the end.

[0009] Further details will be described in the following embodiments. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a configuration diagram of a sensor device and a control device. [Diagram 2] FIG. 2 is a side view of the ground support equipment. [Diagram 3] FIG. 3 is a flowchart of the automatic driving control. [Figure 4] FIG. 4 is a flowchart of the waypoint determination process. [Diagram 5] FIG. 5 is an explanatory diagram of waypoints and guiding lines. [Figure 6] FIG. 6 is an explanatory diagram of the detection range of the sensor device. [Figure 7] FIG. 7 is an explanatory diagram of an automatic driving route. [Figure 8] FIG. 8 is an explanatory diagram of an automatic driving route. [Figure 9] FIG. 9 is a flowchart of control based on sensor data. [Figure 10] FIG. 10 is a flowchart of a process for recognizing the vehicle body position and angle using the first logic. [Figure 11] FIG. 11 is a diagram showing the relationship between the camera field of view and the LiDAR reference coordinates. [Figure 12] FIG. 12 shows a camera image in which a cargo entrance is identified. [Figure 13] FIG. 13 is an explanatory diagram of how to determine the right end of the carry-in entrance. [Figure 14] FIG. 14 is an explanatory diagram of how to find the normal line. [Figure 15] FIG. 15 is an explanatory diagram of the coordinate system. [Figure 16] FIG. 16 is an explanatory diagram of lock identification. [Figure 17] FIG. 17 is a flowchart for recognizing the cargo entrance height. [Figure 18] FIG. 18 is a diagram showing the relationship between the camera image coordinate system and the camera coordinates. [Figure 19] FIG. 19 is a diagram showing the relationship between the camera coordinate system and the HL coordinate system. [Figure 20] FIG. 20 is a flowchart of a process for recognizing the vehicle body position and angle using the second logic. [Figure 21] FIG. 21 is a diagram showing a camera image in which the entrance area is identified. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] <1. Aircraft ground support equipment control device, ground support equipment, and control method>

[0012] (1) A control device according to an embodiment is a control device for controlling automatic travel of ground support equipment of an aircraft to reach the aircraft, and is configured to execute a first control including detecting a position of an end of a service entrance formed on a body of the aircraft when a door for closing the service entrance is open based on point cloud data obtained by measuring the aircraft by a first sensor mounted on the ground support equipment, and controlling travel of the ground support equipment to the service entrance using the detected end position. The end position of a service entrance with an open door has clearer shape characteristics than when the door is closed, so that the position can be detected with high accuracy. As a result, the service entrance can be accurately grasped, and the accuracy of travel of the ground support equipment can be improved. The service entrance is, for example, a cargo entrance. The cargo entrance can also be used for unloading cargo. The service entrance includes a boarding gate for passengers and other people to enter and exit.

[0013] (2) It is preferable that the position of the end portion is detected based on partial point cloud data in a partial region including the end portion, among the point cloud data. In this case, the end portion is efficiently detected based on the partial point cloud data in the partial region.

[0014] (3) It is preferable that the partial area is an area that includes one of both widthwise ends of the entrance, but does not include the other end.

[0015] (4) The partial region is preferably determined by detecting the entire range of the entrance and shifting the range in the width direction of the entrance. Note that the entire entrance may be approximately the entire entrance, and does not have to be completely the entire entrance. The shifting method may be approximately the width direction, and does not have to be strictly the width direction. For example, shifting in the horizontal direction of the image will result in a shift approximately in the width direction.

[0016] (5) The range of the entire service entrance is preferably detected by determining the service entrance in image data obtained by capturing an image of the aircraft with a second sensor whose positional relationship with the first sensor is known. The service entrance may be determined, for example, by a machine learning model described below, or by image recognition processing such as pattern matching.

[0017] (6) It is preferable that the entrance in the image data is determined using a machine learning model that has been trained to identify the entrance from the image data.

[0018] (7) It is preferable that the first control further includes detecting a direction facing the entrance based on the point cloud data. It is preferable that the travel of the ground support equipment to the entrance is controlled further using the direction.

[0019] (8) It is preferable that controlling the movement of the ground support equipment to the loading entrance includes controlling the movement of the ground support equipment so that the ground support equipment arrives at the loading entrance while facing the loading entrance directly.

[0020] (9) Detecting the direction directly facing the loading entrance includes at least one of detecting a first direction perpendicular to a first plane based on the point cloud data and detecting a second direction parallel to a second plane based on the point cloud data, wherein the first plane is at least one of an interior wall of the cargo hold at the back of the cargo hold as viewed from the loading entrance and a first side wall of a container loaded in the cargo hold that is parallel to the interior wall of the cargo hold, and it is preferable that the second plane is a surface of the side wall of the container loaded in the cargo hold that is perpendicular to the interior wall of the cargo hold.

[0021] (10) It is preferable that the ground support equipment includes a height-adjustable lifting unit, and the first control further includes detecting a height of the entrance and controlling a height of the lifting unit according to the detected height of the entrance. It is preferable that the height control is performed after the ground support equipment arrives at the entrance.

[0022] (11) The step of detecting the height of the loading entrance preferably includes detecting a plurality of parts located near the loading entrance, the relative positional relationship of which is known, and calculating the height of the loading entrance based on the positions of the detected plurality of parts and the relative positional relationship. The plurality of parts are preferably located on the floor of the cargo hold. The plurality of parts are preferably located in the cargo hold.

[0023] (12) It is preferable that the relative positional relationship is determined based on model information of the aircraft.

[0024] (13) When the door is closed, a second control different from the first control is executed, and the second control includes detecting a position of the loading entrance based on point cloud data measured of the aircraft by the first sensor, and controlling the travel of the ground support equipment to the loading entrance using the detected position of the loading entrance, and detecting the position of the loading entrance is performed based on partial point cloud data within a partial area of ​​the point cloud data that includes the loading entrance with its door closed, and it is preferable that the partial area includes the range of the loading entrance detected in image data captured of the aircraft with its door closed by a second sensor whose positional relationship with the first sensor is known.

[0025] (14) It is preferable that the ground support equipment according to the embodiment includes the control device.

[0026] (15) A control method according to an embodiment is a control method for automatic navigation of ground support equipment of an aircraft to reach the aircraft, comprising: detecting a position of an end of a service entrance formed on a body of the aircraft when a door for closing the service entrance is open based on point cloud data of the aircraft measured by a first sensor mounted on the ground support equipment; and controlling the navigation of the ground support equipment to the cargo entrance using the detected position of the end.

[0027] <2. Examples of control devices, ground support equipment, and control methods for aircraft ground support equipment>

[0028] Hereinafter, examples of a control device for aircraft ground support equipment, the ground support equipment, and a control method will be described with reference to the drawings.

[0029] 1 shows a control device 60 of the ground support equipment 10 of the aircraft 100. The control device 60 is, for example, mounted on the ground support equipment 10 and controls the running of the ground support equipment 10. The control device 60 may be a device provided outside the ground support equipment 10 and remotely controls the ground support equipment 10. The control device 60 may include both a device provided on the ground support equipment 10 and a device provided outside the ground support equipment 10.

[0030] The control device 60 may be configured, for example, by a computer that executes a computer program. The computer may include a processor and a storage device connected to the processor. The storage device may include, for example, a primary storage device and a secondary storage device. The primary storage device is, for example, a RAM. The secondary storage device is, for example, a hard disk drive (HDD) or a solid state drive (SSD). The memory includes a computer program executed by the processor. The processor reads and executes the computer program stored in the memory. The computer program has program code for causing the computer to execute processes for controlling 61, 62 the ground support equipment 10. Some or all of the functions of the control device 60 may be configured by hardware logic circuits.

[0031] The control device 60 may, for example, execute an initial travel control 61. The control device 60 may, for example, execute an approach and height adjustment control 62. The control 62 includes a door open state control 62A (first control) and a door closed state control 62B (second control). These controls will be described later.

[0032] The ground support equipment 10 controlled by the control device 60 is equipment for ground handling work of the aircraft 100. There are various types of ground support equipment 10 depending on the work content. Examples of the ground support equipment 10 include a cargo loader, a catering car, a stairs car, and a cargo truck conveyor car. The cargo loader is used to load and unload cargo into the cargo hold of the aircraft. The catering car is a car that loads drinks, food, and in-flight items onto the aircraft. The catering car approaches the passenger boarding gate. The catering car has a lifting section that is adjusted to the height of the boarding gate. The boarding gate that the catering car approaches is not the boarding gate on the left side of the aircraft that is usually used for passenger entry and exit, but the boarding gate on the right side of the aircraft. The boarding gate is closed by a passenger door. The stairs car is a car that passengers use to get on and off the aircraft. The stairs car approaches the boarding gate. The height of the ladder car is adjusted to the boarding gate. The boarding gate to which the ladder car approaches is the boarding gate on the left side of the aircraft. The cargo truck conveyor can be used to approach the bulk cargo hold of the aircraft. The cargo truck conveyor vehicle is equipped with a lift for workers to stand on, and the height of the lift is adjusted. The height is adjusted to a height that makes it easy for workers to load cargo into the cargo hold. In the following, a cargo loader for loading and unloading cargo into the cargo hold of an aircraft will be described as an example of the ground support equipment 10 that approaches the loading entrance and has its height adjusted. Examples of cargo loaders include a high lift loader, a main deck loader, a belt loader, and a transporter. The high lift loader is a vehicle used to load and unload containers into the lower deck cargo hold or the main deck cargo hold of the aircraft 100.

[0033] The containers loaded and unloaded by the high lift loader are also called Unit Load Devices (ULDs). ULDs are storage equipment exclusively for aircraft. The cargo hold in which the containers are loaded is also called a container room. The cargo hold of the aircraft 100 also includes a bulk room in which baggage is loaded loose. Cargo is loaded and unloaded into the bulk room by a belt loader.

[0034] The aircraft 100 may, for example, include a forward cargo compartment and an aft cargo compartment. The aft cargo compartment is located aft of the forward cargo compartment. The cargo compartment is provided inside the fuselage of the aircraft. An entrance for carrying cargo into the forward cargo compartment may be called a forward cargo entrance. The forward cargo entrance may be located, for example, forward of the main wing. An entrance for carrying cargo into the aft cargo compartment may be called an aft cargo entrance. The aft cargo entrance may be located, for example, aft of the main wing. The entrances may also be used to carry out cargo. The aircraft 100 may be provided with one or more cargo entrances.

[0035] The front cargo hatch and the rear cargo hatch may be provided on the underside of the fuselage of the aircraft 100. The front cargo hatch and the rear cargo hatch are generally provided on the underside of the right side of the fuselage. Note that the right side of the fuselage refers to the side on the right side of the fuselage when viewed from the rear of the aircraft 100 facing forward.

[0036] As shown in FIG. 2, a door called a cargo door 103 is provided at a cargo entrance 102 of the fuselage 101 of the aircraft 100. The cargo entrance 102 is opened and closed by the cargo door 103. The cargo door 103 is driven to open and close by, for example, an electric motor. The cargo door 103 provided on the side of the fuselage 101 is called a side cargo door. As shown in FIG. 2, the side cargo door 103 opens upward on the outside of the fuselage 101. Note that, since FIG. 2 is a view of the aircraft 100 viewed from the front toward the rear, the side cargo door 103 provided on the right side of the fuselage 101 is located on the left side of the fuselage 101 in FIG. 2. Note that the cargo door may be a nose cargo door formed by the nose portion at the front of the fuselage 101. The nose cargo door opens when the nose portion is pushed upward. The cargo entrance opened and closed by the nose cargo door can be called a nose cargo entrance.

[0037] The high lift loader 10 is used to load or unload a container C with the side cargo door 103 open as shown in Fig. 2. The high lift loader 10 approaches the cargo entrance while facing the cargo entrance directly in order to load or unload cargo.

[0038] The high lift loader 10 includes a front vehicle section 20 and a rear vehicle section 30. The front vehicle section 20 includes a lifting section 21 connected to a cargo entrance 102. The lifting section 31 is height adjustable. The lifting section 21 is also called a lifting platform 21. The height of the lifting platform 21 is adjusted to match the floor surface 106 of the cargo hold 105.

[0039] The front part 20 of the vehicle is equipped with a cab 22 that rises and falls together with the lifting part 21. The cab 22 is used by an operator to manually operate the high lift loader 10. Operations for manual operation include, for example, steering and forward / reverse switching. The operator in the cab 22 can also manually open and close the cargo door 103 by operating an operation panel (not shown) provided near the cargo door 103.

[0040] The vehicle front portion 20 includes a sensor device 50. As an example, the sensor device 50 may be disposed in front of the cab 22. The sensor device 50 may rise and fall in accordance with the rise and fall of the lifting section 21. The location of the sensor device 50 in the ground support equipment 10 is not particularly limited.

[0041] The rear vehicle section 30 includes a lift-up deck 31 provided behind the front vehicle section 20. The lift-up deck 31 is used for loading and unloading cargo such as containers C. As shown in FIG. 2, the height of the lift-up deck 31 can be adjusted to the same height as the lift-up platform 21 of the front vehicle section 20. The rear vehicle section 30 includes a work platform 32 provided behind the lift-up deck 31. The work platform 32 is used, for example, for loading and unloading cargo with a dolly (not shown).

[0042] Returning to FIG. 1, the sensor device 50 will be described. The sensor device 50 acquires sensor data related to the aircraft 100 to which the ground support equipment 10 is to arrive. The sensor data acquired by the sensor device 50 is used, for example, to control the automatic travel of the ground support equipment 10. The sensor device 50 shown in FIG. 1 includes, as an example, cameras 51, 52, 53, and 54 for capturing images, and a LiDAR sensor 55. The sensor device 50 includes, as an example, a plurality of cameras 51, 52, 53, and 54. The plurality of cameras includes, as an example, a first camera 51, a second camera, a third camera, and a fourth camera.

[0043] The LiDAR sensor 55 irradiates a laser beam onto a measurement target to measure the distance to the measurement target, etc. The LiDAR sensor 55 acquires point cloud data indicating the three-dimensional coordinates of a large number of points on the surface of the measurement target. Note that the type of sensor for acquiring the point cloud data is not particularly limited.

[0044] The images and point cloud data acquired by the sensor device 50 are provided to the control device 60 as sensor data. The control device 60 uses the acquired sensor data to control, for example, the traveling device 80 and the lifting unit 21 of the ground support equipment 10. The traveling device 80 is a device that is responsible for the traveling of the ground support equipment 10, and is equipped with a drive unit, a steering mechanism, wheels, etc. for traveling. The automatic traveling by the traveling device 80 is controlled by the control device 60. The control by the control device 60 includes steering and forward / reverse switching of the ground support equipment 10. The control by the control device 60 may also include speed adjustment.

[0045] For example, the control device 60 uses the sensor data to detect a target (target location) that the ground support equipment 10 should reach, and automatically drives the ground support equipment 10 to reach the target location. Here, the target location of the aircraft 101 is, for example, a cargo entrance provided on the side of the aircraft 101. In this case, the control device 60 determines the position of the cargo entrance that the ground support equipment 10 should reach, and controls the travel of the ground support equipment 10 to reach the cargo entrance. The control device 60 also detects the height of the cargo entrance, and adjusts the height of the lifting section 21 according to the height of the cargo entrance. The control device 60 can also adjust the height of the lifting deck 31 according to the height of the lifting section 21.

[0046] The control device 60 can also acquire data or signals other than the sensor data as necessary for controlling the ground support equipment 10. As shown in Fig. 2, other data that the control device 60 can acquire is, for example, at least one data selected from the current position of the ground support equipment 10, data stored in the database 70, data detected by the guide wire detector 91, data acquired by the aircraft marking reading camera 92, the type of cargo entrance where the ground support equipment 10 should arrive, and flight schedule data. These data will be described later.

[0047] Fig. 3 shows the procedure of automatic travel control of the ground support equipment 10 by the control device 60. The procedure shown in Fig. 3 shows the procedure from when the ground support equipment 10 is parked at a waiting position at the airport until it reaches the aircraft 100. Here, as an example, the ground support equipment 10 passes through a waypoint before reaching the aircraft 100. For this reason, in the procedure of Fig. 3, the waypoint is determined (step S33). In order to determine the waypoint, necessary data is acquired prior to determining the waypoint (steps S31, S32).

[0048] Data required to determine waypoints is, for example, aircraft model information. Since the location of the cargo entrance of aircraft 100 may vary depending on the aircraft model, the aircraft model information is useful for appropriate automatic navigation in response to various aircraft models. For this reason, in step S31, aircraft model information (aircraft model information) is acquired. Note that, if the location of the cargo entrance can be identified even without aircraft model information, step S31 may be omitted.

[0049] The aircraft type information is obtained, for example, by the control device 60 accessing a database having flight schedule data (see FIG. 1). The flight schedule data is stored, for example, in a database server for managing flight schedules. The flight schedule data includes aircraft type information of the aircraft 100 that the ground support equipment 10 is to reach. The aircraft type information may be obtained by reading the aircraft registration attached to the aircraft 100 that the ground support equipment 10 is to reach, using a camera 92 provided on the ground support equipment 10 or a camera 92 provided outside the ground support equipment 10 (see FIG. 1).

[0050] The aircraft symbol is a symbol for identifying the aircraft 100 as an individual. When the aircraft symbol includes a number or symbol that directly indicates the model, it is possible to identify the model from the aircraft symbol. When the aircraft symbol does not directly indicate the model, it is possible to identify the model from the aircraft symbol by referring to a table that associates the aircraft symbol with the model. Note that the aircraft symbol may be affixed to the aircraft with some of the symbols or numbers omitted.

[0051] The aircraft symbol may be displayed on various parts of the aircraft 101, such as the aircraft's main wing, the aircraft fuselage, the nose gear cover, etc. The aircraft symbol read by the camera 92 may be on any part. For example, the aircraft model information can be obtained by using the camera 92 to recognize the aircraft symbol on the nose gear cover.

[0052] In addition, in order to determine the waypoint, the control device 60 may acquire the type of the cargo entrance that the ground support equipment 10 (high lift loader 10) should reach (step S32). Here, the type of the cargo entrance is data indicating which of the multiple cargo entrances equipped in the aircraft 100 the cargo entrance is. The types of the cargo entrances are, for example, the forward cargo bay (FWD) and the aft cargo bay (AFT). For example, the high lift loader 10 that should reach the forward cargo bay acquires data indicating "forward cargo bay (FWD)" as the type of the cargo entrance. Also, the high lift loader 10 that should reach the aft cargo bay acquires data indicating "aft cargo bay (AFT)" as the type of the cargo entrance. The data regarding which high lift loader 10 should reach which cargo entrance is set in advance in a management computer (not shown), or may be appropriately set in the management computer by the work manager. The control device 60 may acquire the type of the cargo entrance that the ground support equipment 10 should reach by accessing the management computer. In addition, if the type of cargo entrance to be reached is already known to the control device 60, the step may be omitted.

[0053] The control device 60 determines the waypoint in step S33. FIG. 4 shows the procedure of the waypoint determination process corresponding to step S33. First, the control device 60 acquires the current position of the ground support equipment 10 (step S41). The current position here is a waiting position before the ground support equipment 10 runs toward the aircraft 100. The current position is, for example, a position acquired by a GNSS (Global Navigation Satellite System) receiver mounted on the ground support equipment 10. When the ground support equipment 10 is parked at a predetermined waiting position, data indicating the waiting position may be acquired as the current position. Note that, when the waiting position is known to the control device 60 or the waypoint can be determined even without the waiting position (current position), step S41 may be omitted.

[0054] The control device 60, for example, refers to a model information table 71 stored in the database 70 to determine the waypoints (step S42). The table 71, for example, stores the waypoints P FWD1 ,P FWD2 ,P AFT As shown in FIG. 5, the waypoints can be set according to the type of cargo entrance (the position of the cargo entrance). For example, the waypoint P FWD1 ,P FWD2 and waypoint P for rear cargo entrance 102B AFT By selecting a waypoint according to the location of the cargo entrance, the control device 60 can determine an appropriate waypoint for reaching the cargo entrance.

[0055] The waypoint may be set according to the current position (standby position) of the ground support equipment 10. For example, the waypoint for the forward cargo entrance 102A may be the first waypoint P FWD1 and the second waypoint P FWD2 By selecting a waypoint according to the current position (standby position) of the ground support equipment 10, the control device 60 can determine an appropriate waypoint according to the current position (standby position).

[0056] As shown in FIG. 5, possible waiting positions for the ground support equipment 10 include, for example, positions 10C, 10D, 10E, and 10F shown in FIG. 5. Here, the waiting positions for the ground support equipment 10 are within equipment staging areas (ESA) 302, 302 at the airport. The equipment waiting areas are set outside an equipment restraint area (ERA) 301, and are used as waiting areas for equipment used in ground handling operations of the aircraft 100 parked at the spots. Note that the spots are positions where the aircraft 100 are parked. In the equipment waiting area 302, other equipment such as dollies, catering vehicles, and fuel trucks may be parked in addition to the high-lift loader 10. The equipment restricted area 301 is set inside the equipment waiting area, and when the aircraft 100 enters or leaves the spot, the equipment is prohibited from entering.

[0057] 5, the ground support equipment 10C, 10D, 10E, and 10F in a standby state are in an equipment standby area 302, and are located forward of the cargo entrances 102A and 102B of the parked aircraft 100. The ground support equipment 10C, 10D, 10E, and 10F in a standby state are also facing in the same direction as the parked aircraft 100. Therefore, as viewed from the ground support equipment 10 in a standby state, the cargo entrances 102A and 102B of the parked aircraft 100 are located behind the ground support equipment 10. For this reason, it may be difficult for the sensor device 50 provided at the front of the ground support equipment 10 to detect the cargo entrances 102A and 102B. In addition, since the distance from the ground support equipment 10C, 10D, 10E, 10F in standby to the cargo entrances 102A, 102B is relatively large (several tens of meters or more), it may be difficult for the sensor device 50 installed on the ground support equipment 10 in standby to accurately detect the position of the cargo entrances 102A, 102B, which are far away and not very large.

[0058] Therefore, as an example, the control device 60 starts the automatic travel based on the sensor data acquired by the sensor device 50 from a waypoint once, rather than starting from the waiting position of the ground support equipment 10. The waypoint is set at a position where the cargo entrances 102A, 102B to be reached by the ground support equipment 10 are within the detection range of the sensor device 50 mounted on the ground support equipment 10. The ground support equipment 10 moves to the waypoint so that the cargo entrances 102A, 102B are within the detection range of the sensor device 50.

[0059] For example, as shown in FIG. 6, when the detection range D of the sensor device 50 is set to be directed from the front to the left of the ground support equipment 10, as in the case of the ground support equipment 10A in FIG. 6, the cargo entrance 102 is located within the detection range D of the sensor device 50 by facing the cargo entrance 102 almost directly at a position on the right side of the aircraft 101. Also, as in the case of the ground support equipment 10B in FIG. 6, the cargo entrance 102 is located within the detection range of the sensor device 50 by facing the front of the aircraft 101 in a position diagonally rearward to the right of the cargo entrance 102. In this embodiment, the waypoint is set to be the position of the ground support equipment 10A or 10B shown in FIG. 6 with respect to the cargo entrance 102 to be reached. The waypoint may be set within the equipment waiting area 302 or within the equipment restricted area 301.

[0060] Returning to FIG. 5, the first waypoint P for the forward cargo entrance 102A FWD1 6. The second waypoint P for the front cargo entrance 102A FWD2 and waypoint P for rear cargo entrance 102B AFT is a position corresponding to 10B in FIG. 6. As an example, the first waypoint P FWD1 is provided as a waypoint for ground support equipment waiting at position 10C in Figure 5, and the second waypoint P FWD2 is provided as a waypoint for ground support equipment waiting at position 10E in FIG. 5. Waypoint P for rear cargo entrance 102B AFTare provided as waypoints for ground support equipment waiting at positions 10D and 10E in Fig. 5. The waypoints may be set within the equipment waiting area or within the equipment restricted area.

[0061] The control device 60 refers to the model information table 71 based on the acquired model information (step S31) to determine the candidate route point P FWD1 ,P FWD2 ,P AFT If the acquired "Type of Cargo Entrance" is "Forward Cargo Entrance (FWD)", the candidate is point P FWD1 ,P FWD2 In addition, if the current position of the ground support equipment in standby (step S41) is the position of 10C, the route point P FWD1 If the position is 10E, the waypoint P FWD2 If the acquired "type of cargo entrance" is "rear cargo entrance (AFT)", the waypoint P AFT (Step S42). In this embodiment, the "waypoint" is also referred to as the "turnaround position." As an example, the ground support equipment 10 moves backward from the waiting position to the turnaround position, and then moves forward from the turnaround position to the cargo entrance 102. By moving backward and then forward to approach the aircraft 100, the number of turns required from the waiting position to approach the aircraft 100 can be reduced.

[0062] When the waypoint determination process S33 in Fig. 4 is completed, the control device 60 executes automatic driving (initial driving; first driving) to move the ground support equipment 10 to the determined waypoint (step S34 in Fig. 3). The driving to the waypoint is, for example, reverse driving. When the control device 60 reaches the waypoint, the control device 60 controls the ground support equipment 10 based on the sensor data acquired by the sensor device 50 so that the ground support equipment 10 reaches the cargo entrance (step S35).

[0063] As shown in Fig. 5, in the equipment restricted area 301 and the equipment waiting area 302, induction lines 200 are provided to guide the ground support equipment 10 to the waypoint. The induction lines 200 are markers painted on the ground of the airport, electromagnetic induction lines buried in the ground, or other route guidance markers. The ground support equipment 10 can move along the induction lines 200. The induction lines 200 are detected by an induction line detector 91 provided on the ground support equipment 10. The induction line detector 91 is, for example, a camera that captures an image of the induction line 200, or a sensor that detects an electromagnetic induction line.

[0064] 5 includes, as an example, a first guiding line 201, a second guiding line 202 and a fourth guiding line 204 branching off from the first guiding line, a third guiding line 203 branching off from the second guiding line 202, and a fifth guiding line 205 branching off from the fourth guiding line 204. The first guiding line 201 is provided within an equipment restricted area 301, and includes a waypoint P AFT The second guiding wire 202 extends from the equipment restricted area 301 to the equipment waiting area 302, and is provided so as to reach the position of 10C from the first guiding wire 201. The third guiding wire 203 extends from the equipment restricted area 301 to the equipment waiting area 302, and is provided so as to reach the position of 10D from the second guiding wire 202. The fourth guiding wire 204 extends from the equipment restricted area 301 to the equipment waiting area 302, and is provided so as to reach the position of 10E from the first guiding wire 201. The fifth guiding wire 205 is provided in the equipment waiting area 302, and is provided so as to reach the position of 10F from the fourth guiding wire 204.

[0065] FIG. 7 shows an example of a travel route when the ground support equipment 10 is waiting at the positions 10C and 10D in FIG. 5. When the ground support equipment 10C in FIG. 7 is heading to the forward cargo entrance 102A, the route point is P FWD1 In this case, the ground support equipment 10C is determined to be the waypoint P FWD1 Then, as shown by the ground support equipment 10C-1, the ground support equipment 10C-1 faces the cargo entrance 102A.FWD1 In the first run 15 of the ground support equipment 10C, for example, a first part of the run is a run guided by the second guide line 202, and thereafter, the ground support equipment 10C is guided to the waypoint P by the GNSS signal received by the ground support equipment 10C. FWD1 The ride will be guided to the designated spot.

[0066] When the ground support equipment 10D in FIG. 7 heads for the rear cargo entrance 102B, the waypoint is P AFT In this case, the ground support equipment 10D is determined to be the waypoint P AFT Then, as shown by the ground support equipment 10D-1, the ground support equipment 10D-1 is oriented approximately parallel to the aircraft 101 and facing the front of the aircraft so that the cargo entrance 102B is located diagonally forward and to the left. AFT The ground support equipment 10D performs a second run 16 in which the ground support equipment 10D advances from the rear cargo entrance 102B and approaches the rear cargo entrance 102B. The first run 15 of the ground support equipment 10D is, for example, a run guided by the third guiding wire 203, the second guiding wire 202, and the first guiding wire 201.

[0067] FIG. 8 shows an example of a travel route when the ground support equipment 10 is waiting at the positions 10E and 10F in FIG. 5. When the ground support equipment 10E in FIG. 8 is heading toward the forward cargo entrance 102A, the model information table route point is P FWD2 In this case, the ground support equipment 10E is determined to be the waypoint P FWD2 Then, as shown by the ground support equipment 10E-1, the ground support equipment 10E-1 is oriented approximately parallel to the aircraft 101 and facing the front of the aircraft so that the cargo entrance 102A is located diagonally forward to the left. FWD2 The ground support equipment 10E advances from the forward cargo entrance 102A and approaches the forward cargo entrance 102A to perform a second run 16. The first run 15 of the ground support equipment 10E is, for example, a run guided by the fourth guiding wire and the first guiding wire 201.

[0068] When the ground support equipment 10F in FIG. 8 heads for the rear cargo entrance 102B, the waypoint is P AFTIn this case, the ground support equipment 10F is determined to be the waypoint P AFT Then, as shown by the ground support equipment 10F-1, the equipment is oriented approximately parallel to the aircraft 101 and facing the front of the aircraft, with the cargo entrance 102B located diagonally forward and to the left. AFT The ground support equipment 10F performs a second run 16 in which the ground support equipment 10F advances from the rear cargo entrance 102B and approaches the rear cargo entrance 102B. The first run 15 of the ground support equipment 10F is, for example, a run guided by the fifth guiding wire 205 and the first guiding wire 201.

[0069] When the ground support equipment reaches the positions 10C-1, 10D-1, 10E-1, and 10F-1 shown in Figs. 7 and 8, even if the cargo entrance is outside the detection range D of the sensor device 60 at the waiting position, the cargo entrances 102A and 102B can be positioned within the detection range D. Therefore, after the waypoint, the control device 60 controls the second run, etc. based on the sensor data measuring the aircraft 100 (step S35 in Fig. 3). Note that the sensor data may also be utilized in the first run 15 as necessary.

[0070] FIG. 9 shows a procedure of control (step S35) by the control device 60 based on sensor data. In the procedure shown in FIG. 9, as an example, first, it is determined whether the cargo doors of the cargo entrances 102A, 102B to which the ground support equipment 10 is going to arrive are open or closed (step S91). When the ground support equipment 10 is going to approach the cargo entrances 102A, 102B, whether the cargo doors are open or closed in advance varies from case to case. Whether the cargo doors are open or closed varies depending on, for example, the type of aircraft 100. For example, in some types of aircraft, the cargo doors may be opened in advance before the ground support equipment 10 is going to approach the cargo entrances 102A, 102B after parking. In other types of aircraft, the cargo doors remain closed after parking, and an operator on board the ground support equipment 10 must operate a cargo door operation panel (not shown) provided on the aircraft near the cargo doors to open the cargo doors. Therefore, in this embodiment, as an example, the control device 60 executes different controls depending on whether the cargo door is open or closed.

[0071] In the door open / close determination in step S91, for example, it is determined whether the cargo door is open or closed based on the aircraft model information acquired in step S31 of FIG. 3. In order to determine whether the door is open or closed, the aircraft model information table 71 preferably has door open / close information (not shown) associated with the aircraft model information. The door open / close information associated with the aircraft model information indicates whether the cargo door is open or closed for each aircraft model. The control device 60 refers to the aircraft model information table 71 having the door open / close information and determines whether the cargo entrance of the aircraft 100 to which the ground support equipment 10 should arrive is closed or open. If it is determined to be open, a first control (door open state control) is executed, and if it is determined to be closed, a second control (door closed state control) is executed. Note that if the second control is executed, the cargo door is open after the second control, so the first control may be executed.

[0072] In the first control, in step S911, a process is executed to recognize the position and angle of the ground support equipment 10 relative to the cargo entrance. Note that the same recognition is performed in the second control, but in the first control, the recognition is performed by a first logic, and in the second control, the recognition is performed by a second logic different from the first logic. The difference between the first logic and the second logic is due to whether the cargo door is open or closed.

[0073] 10 shows the recognition process (step S911) by the first logic. Here, camera images acquired by three cameras (second sensors) of the first camera 51, the second camera 52, and the third camera 53 of the sensor device 50 and point cloud data acquired by the LiDAR sensor 55 (first sensor) are used.

[0074] In step S101 shown in FIG. 10, a cargo entrance is detected from a camera image. For example, an entrance identifier built in the control device 60 is used to detect the cargo entrance from the camera image. The entrance identifier is, for example, a machine learning model that is trained to identify a cargo entrance in an image when an image of an aircraft is input. Note that, as an example, the entrance identifier here is trained by machine learning using both images with the cargo door open and images with the cargo door closed for various types of aircraft, and can detect a cargo entrance whether the cargo door is open or closed. Therefore, the entrance identifier according to the embodiment can be advantageously used for the second control as well.

[0075] Note that the entrance identifier may be provided separately for the first control and for the second control. In this case, the entrance identifier for the first control may be machine-trained to identify a cargo entrance whose cargo door is open, and the entrance identifier for the second control may be machine-trained to identify a cargo entrance whose cargo door is closed.

[0076] 11 and 12 are diagrams for explaining a method of obtaining the coordinates of the cargo entrance from the camera image. FIG. 11 shows the positional relationship between the camera coordinates and the LiDAR central polar coordinates. The three cameras 51, 52, and 53 are provided on the sensor device 50 at the same height and with different directions of their fields of view. In FIG. 11, the x direction (upward in FIG. 11) corresponds to the forward direction of the ground support equipment 10, and the y direction corresponds to the left direction of the ground support equipment 10. The first camera 51 is provided so as to face the x direction (forward), and the third camera 53 is provided so as to face the y direction (leftward). The second camera 52 is provided so as to face the direction between the first camera 51 and the third camera 53 (diagonally forward to the left). The fields of view of the three cameras partially overlap each other. The range of the combined fields of view of these three cameras 51, 52, and 53 is the entire camera field of view. The LiDAR sensor 55 is also configured to measure the distance to the object in a range including the entire camera field of view. The distance to the object can be appropriately determined in a range where the entire camera field of view and the detection range of the LiDAR sensor 55 overlap. Thus, in this case, the entire camera field of view of the three cameras corresponds to the detection range D of the sensor device 50.

[0077] The entrance identifier detects the cargo entrance from each of the three images. The cargo entrance is detected, for example, by determining the coordinates of a rectangular area that includes the cargo entrance in each image. The rectangular area that includes the cargo entrance is, for example, a bounding box that indicates the cargo entrance identified by the entrance identifier. The bounding box is obtained by bounding box annotation based on the result of identification by the entrance identifier.

[0078] In FIG. 12, as an example, a cargo entrance with an open cargo door is shown in a first camera image and a second camera image. As shown in FIG. 12, the entrance identifier extracts a rectangular area including the cargo entrance in each of the first camera image and the second camera image. Since the coordinates of the rectangular area are obtained as camera image coordinates (u, v), the control device 60 converts the camera image coordinates of the rectangular area into the LiDAR center polar coordinates (θ, φ). The coefficients for this conversion are determined from the relative positional relationship between the camera and the LiDAR. The positional relationship between the camera and the LiDAR is known.

[0079] As shown in FIG. 12, when the cargo entrance straddles two camera images, the control device 60 calculates the range including the cargo entrance by adding up the polar coordinate ranges indicating the rectangular ranges of the two images. For example, if the range of the polar coordinate θ of the rectangular range in the first camera image is 20° to 35° and the range of the polar coordinate θ of the rectangular range in the second camera image is 30° to 40°, the range of the polar coordinate θ of the range including the entire cargo entrance is 20° to 40°. The polar coordinate φ is also calculated by adding up. Through these processes, the range including almost the entire cargo entrance is extracted as coordinate values ​​based on the LiDAR sensor 55. That is, the azimuth angle range (θ range) and the elevation / depression angle range (φ range) of the entire cargo entrance based on the LiDAR sensor are extracted (step S101).

[0080] If a cargo entrance is detected in each of the two images and the bounding boxes of the cargo entrances do not overlap, it may be determined that another cargo entrance has been erroneously detected. If an erroneous detection occurs, a more likely cargo entrance position can be adopted based on its positional relationship with the waypoint. If an erroneous detection occurs, the ground support equipment 10 can be moved tentatively closer to the aircraft 100 from the waypoint so that the other cargo entrance does not appear in the image, and the cargo entrance can be redetected at that point.

[0081] Once the azimuth angle range (θ range) and elevation / depression angle range (φ range) of the cargo entrance are extracted, a predetermined reference position at the cargo entrance is estimated in steps S112 and S113. The reference position here is the right end (right edge) of the cargo entrance with the cargo door open. The right end is the end of the periphery of the cargo entrance that is on the right side when viewed directly at the cargo entrance from outside the aircraft. The reference position may also be the left end of the cargo entrance. In other words, the reference position is preferably one of the two widthwise ends of the cargo entrance.

[0082] Since the position of the cargo entrance detected in step S101 is a rough one represented by the vertex coordinates of a bounding box, in order to accurately connect the ground support equipment 10 to the cargo entrance 102, it is desirable to detect the position of the cargo entrance 102 more accurately. In addition, since a cargo entrance with an open door is an opening, it is not possible to directly measure the distance to the cargo entrance, which is an opening, so it is preferable to use the periphery of the cargo entrance as the reference position. Therefore, in steps S112 and S113, the position of the right end, which is the reference position at the cargo entrance, is estimated using the position of the cargo entrance detected in step S101 and the distance information measured by the LiDAR sensor 55.

[0083] To estimate the reference position (right end) of the cargo entrance, the control device 60 acquires point cloud data (first partial point cloud data) within an area (first partial area) required to estimate the reference position (right end) of the cargo entrance, from among all point cloud data acquired by the LiDAR sensor 55. The first partial area is an area that includes the right end, which is the reference position. The first partial area is an area that does not include the left end. Since the first partial area includes only one of both ends in the width direction of the cargo entrance, there is no need to perform processing to prevent erroneous detection of the other end, making processing easier.

[0084] The first partial area is obtained by using the entire range of the cargo entrance detected in step S101 in Fig. 10 (hereinafter, simply referred to as "entrance range"). In the camera image shown in Fig. 13, the first partial area is indicated by a solid line as the "SIDE point cloud data area." In the camera image shown in Fig. 13, the entrance range detected in step S101 is indicated by a dotted rectangle.

[0085] The control device 60 obtains the first partial region (SIDE point cloud data region), for example, by shifting the entrance range to the right in the width direction of the entrance range by 0.5 width of the entrance range. In the entrance range that captures the entire entrance, the right end, which serves as the reference position, is near the boundary of the entrance range, so it is difficult to utilize information around the right end when detecting the right end. In contrast, in the first partial region, the right end is located in the center of the first partial region in the width direction, so information around the right end can be utilized. Also, in the first partial region, the left end is not included due to the shift in the width direction. In this way, the first partial region is a region that includes almost the entire right end from its top to its bottom.

[0086] When the control device 60 extracts point cloud data (first partial point cloud data) in the first partial region from the point cloud data (step S112), it detects the right end of the cargo entrance based on the first partial point cloud data (step S113). By detecting the right end based on the first partial point cloud data instead of the entire point cloud data, the processing load can be reduced. In addition, it is not necessarily easy to identify the cargo entrance from the entire point cloud data including information on other parts other than the cargo entrance without using the first partial region. However, by determining the first partial region including the right end (reference position) of the cargo entrance from image data that can easily identify the cargo entrance, and using distance information included in the point cloud data in the first partial region, the position of the right end can be determined with high accuracy. Note that image data is suitable for identifying the cargo entrance, but does not include distance information. Therefore, in order to determine the position of the right end with high accuracy, it is suitable to use point cloud data including distance information.

[0087] The right end, which is the reference of the cargo entrance, can be detected by utilizing the fact that a relatively large difference in distance may occur between the distance from the LiDAR sensor 55 to the aircraft surface around the cargo entrance (particularly, the aircraft surface to the right of the right end) and the distance measured in the inner area (particularly, the area to the left of the right end) of the periphery of the cargo entrance where the cargo door is open. When the cargo door is open, the LiDAR sensor 55 measures the distance to the inside of the cargo room, and a relatively large difference in distance is likely to occur between the distance to the inside of the cargo room and the distance to the aircraft surface. Therefore, the reference position (right end) can be easily detected. In contrast, when the cargo door is closed, even if the LiDAR sensor 55 away from the aircraft measures the distance to the cargo entrance and its vicinity, the difference between the distance to the cargo entrance and its vicinity is small, making it difficult to detect the reference position (right end). For this reason, in the second logic, the position of the cargo entrance is detected by a different method.

[0088] In addition, detecting the right end of the cargo entrance within the first partial area can also be said to be detecting the left end of the aircraft surface within the first partial area.

[0089] An example of a method for determining the right end position from the first partial point cloud data in the first partial region will be described below. As shown in the "camera image with point cloud data superimposed" in FIG. 13, the point cloud data has multiple layers at a predetermined interval in the height direction (vertical direction). For example, the first partial region (SIDE point cloud data region) in FIG. 13 includes four layers. In each layer, multiple measurement points (hereinafter simply referred to as "points") are arranged at a predetermined interval in the width direction (horizontal direction). Note that the vertical interval of the layers and the horizontal interval of the points are determined by the measurement performance of the LiDAR sensor 55.

[0090] FIG. 13 shows a method for detecting the position of the right end using the first partial point cloud data. In FIG. 13, the first partial point cloud data is shown as "SIDE point cloud data". The control device 60 arranges the point cloud data of each layer in the first partial region from the left, and calculates the distance to the left and right points of the same layer for each point of each layer. The control device 60 extracts a plurality of points in each layer, from a point whose distance to the right neighboring point is significantly longer than the distance to the left neighboring point (for example, a point whose distance difference to the left and right neighboring points is equal to or greater than a threshold) to a point whose distance to the right neighboring point is significantly shorter than the distance to the left neighboring point (for example, a point whose distance difference to the left and right neighboring points is less than a threshold). Then, the control device 60 detects a point immediately to the right of the plurality of points as a right end candidate in that layer. The average position of the two rightmost points among the detected right end candidate points is estimated as the right end position in that layer. In addition, the right end positions of each layer may be averaged and used as the final right end of the cargo entrance, or the right end position of any one layer may be used as the final right end of the cargo entrance.

[0091] By calculating the right end position in this way, the right end can be calculated according to various situations in the cargo hold. A typical situation in the cargo hold is when there are no containers (cargo) in the cargo hold, as in "CASE.1" in Figure 13. Also, there are cases where containers C1, C2, C3, C4, and C5 are present in the cargo hold, as in "CASE.2" and "CASE.3."

[0092] In "CASE.1" in Figure 13, the first partial point cloud data in the first partial region is data measuring the distance to the aircraft surface B1 to the right of the right end, and the distance to the cargo hold inner wall B2 at the back of the cargo hold when viewed from the cargo entrance on the left of the right end.

[0093] In "CASE.2" in Fig. 13, the first partial point cloud data in the first partial area is obtained by measuring the distance to the aircraft surface B1 on the right side of the right end, and the distance to the side wall of container C2 facing the cargo entrance on the left side of the right end. This side wall is parallel to the inner wall B2 at the back of the cargo hold, but is located in front of the inner wall B2.

[0094] In "CASE.3" in Figure 13, the first partial point cloud data in the first partial region measures the distance to the aircraft surface B1 to the right of the right end, and the distance to the side wall of container C2 and the inner wall B2 at the back of the cargo hold to the left of the right end.

[0095] In the cases of CASE.1 and CASE.2, for example, for point P2, the distance to the point P1 on the left side and the distance to the point P3 on the right side are almost the same. The same is true for points P3, P4, and P5. In contrast, for point P6, the distance to the point P7 on the right side is significantly longer than the distance to the point P5 on the left side. Also, for point P7, the distance to the point P8 on the right side is significantly shorter than the distance to the point P6 on the left side. Therefore, points P6 and P7 are extracted, and points P7 and P8, which are immediately to the right of points P6 and P7, respectively, are detected as right end candidates. Here, the detected points are two, P7 and P8, so these points P7 and P8 are determined as the right end positions. The average position of points P7 and P8 is estimated as the right end position in this layer.

[0096] In the case of CASE.3, four points P12, P13, P17, and P18 are detected as candidates for the right end. Among the four points P12, P13, P17, and P18, the two rightmost points P17 and P18 are determined as the right end positions. The average position of points P17 and P18 is estimated as the right end position in this layer.

[0097] By taking the average of the two rightmost points among the multiple detected points, it is expected that one of the points will be the target point, and the actual right end position can be found with high accuracy. Also, by taking the average of the right end positions of each layer, the right end position can be found with even higher accuracy. If only one point is detected, that one point can be determined as the right end position.

[0098] Returning to Fig. 10, in steps S122 and S123, the angle (θ) of the cargo entrance is estimated. Here, in an xy coordinate system (see Fig. 11) based on the LiDAR, the angle of the direction directly facing the cargo entrance with respect to the x direction is estimated as the angle (θ) of the cargo entrance.

[0099] Since the cargo entrance with its door open is an opening, it is difficult to detect the surface of the cargo entrance itself. Therefore, the control device 60 detects some wall inside the cargo room measured through the open cargo entrance. The control device 60 calculates the normal of the wall inside the cargo room, and if the calculated normal (first normal) is within a predetermined angle (for example, within 30°) from the normal (second normal) of the aircraft surface near the cargo entrance, the control device 60 selects the first normal as the angle of the cargo entrance (direction facing the cargo entrance) (step S123). Note that the normal here is a line obtained by projecting the normal to the surface consisting of the point cloud data onto a horizontal plane (xy plane in FIG. 11). In other words, the normal here indicates a direction on the horizontal plane. Therefore, the angle of the cargo entrance is estimated as an angle on the horizontal plane.

[0100] In addition, since the surface of the aircraft is curved, the accuracy decreases if the second normal line itself is calculated as the angle of the cargo entrance. Since the cargo bay has a relatively large number of planes that are parallel or perpendicular to the fore-aft direction of the aircraft, it is advantageous to calculate the direction directly facing the cargo bay as the lateral direction of the aircraft, which is perpendicular to the fore-aft direction of the aircraft on a horizontal plane.

[0101] In order to obtain the angle of the cargo entrance, the control device 60 acquires second partial point cloud data and third partial point cloud data necessary for estimating the angle of the cargo entrance from among all point cloud data acquired by the LiDAR sensor 55 (step S122). The second partial point cloud data is point cloud data within the second partial region from among all point cloud data acquired by the LiDAR sensor 55. The third partial point cloud data is point cloud data within the third partial region from among all point cloud data acquired by the LiDAR sensor 55.

[0102] The second partial area is an area including the entire cargo entrance, and the entrance range detected in step S101 in Fig. 10 is used as is. In the camera image shown in Fig. 14, the second partial area is shown as an "in point cloud data area."

[0103] The third partial area is an area including the surface of the aircraft near the cargo entrance, but does not include the cargo entrance. It is determined based on the entrance range detected in step S101 of FIG. 10. In the camera image shown in FIG. 14, the third partial area is shown as the "out point cloud data area." The third partial area is set at a position away from the entrance range detected in step S101 by a predetermined distance (e.g., 0.2 times the width of the entrance) in the entrance width direction (e.g., to the right). The width of the third partial area is set to, for example, 0.4 times the entrance range. By making the width of the third partial area smaller than the width of the entrance range, it is possible to prevent a decrease in the accuracy of the estimated angle.

[0104] The control device 60 detects the walls of the cargo hold measured through the open cargo entrance using the second partial point cloud data in the second partial area indicating the range of the entrance, and detects the aircraft surface using the third partial point cloud data in the third partial area indicating the aircraft surface (step S123). The detection of planes such as the walls of the cargo hold and the aircraft surface is performed using an algorithm such as RANSAC (Random Sample Consensus) based on the point cloud data.

[0105] Variations of the walls inside the cargo hold detected using the second partial point cloud data include "CASE.1", [CASE.2], and [CASE.3] shown in FIG. 14. In CASE.1, the front of the container C2 is detected as the wall inside the cargo hold, and the first normal N11 is calculated. In CASE.2, the inner wall B2 at the back of the cargo hold is detected as the wall inside the cargo hold, and the first normal N12 is calculated. In CASE.3, the side wall of the container C4 and the inner wall B2 at the back of the cargo hold are detected, and the first normal N13 of the side wall of the container C4 and the first normal N14 of the inner wall B2 are calculated. Depending on the orientation of the ground support equipment 10, the side wall of the container C4 may be captured as in CASE.3. In all of "CASE.1", [CASE.2], and [CASE.3], the second normal N2 of the aircraft surface is calculated using the third partial point cloud data.

[0106] In CASE.1, the first normal N11 is approximately parallel to the second normal N2 (within 30°), so the first normal N11 is selected as the cargo bay angle (step S123). In CASE.2, the first normal N12 is approximately parallel to the second normal N2, so the first normal N12 is selected as the cargo bay angle (step S123).

[0107] Regarding CASE.3, depending on the orientation of the ground support equipment 10, the first normal N14 may not be detected, and only the first normal N13 may be detected. In preparation for such a case, the first normal N13 perpendicular to the second normal N2 (for example, a normal at 90°±30° from the second normal N2) may also be selected. When the first normal N13 perpendicular to the second normal N2 is selected, the angle minus 90° is estimated as the cargo hold angle.

[0108] As in CASE.3, when both normals parallel to and perpendicular to the second normal N2 are obtained, the normal that contains the most points on the surface corresponding to the normal may be selected as the cargo hold angle (step S123).

[0109] As described above, the cargo hold angle (direction facing the cargo entrance) may be detected as the first normal N11, N12, N14 perpendicular to the first plane that is roughly parallel to the cargo entrance opening, or may be detected as a direction parallel to the second plane that is roughly perpendicular to the cargo entrance opening (a direction 90° minus the direction of the first normal N13). The first plane here is at least one of the cargo hold inner wall B2 located at the back of the cargo hold as seen from the cargo entrance, and the first side wall of the side wall of the container C2 loaded in the cargo hold that is parallel to the inner wall B2. The second plane here is the side wall of the container C4 loaded in the cargo hold that is perpendicular to the inner wall B2.

[0110] 10, the LiDAR-based entrance position (x, y) obtained in step S113 and the LiDAR-based entrance angle (θ) obtained in step S123 are coordinate-converted (step S104). By the coordinate conversion, the LiDAR-based position (x, y) and angle (θ) are converted into a vehicle position (X, Y) and angle (yaw) based on the cargo entrance (the right end of the cargo entrance). The angles θ and yaw here indicate the attitude of the ground support equipment 10.

[0111] FIG. 15 shows xy coordinates based on the LiDAR, i.e., xy coordinates based on the ground support equipment 10 and XY coordinates based on the right end of the cargo entrance. In the coordinate conversion in step S104, the position (x, y) and angle (θ) based on the LiDAR are converted into the vehicle position (X, Y) and angle (yaw) in the world coordinate system based on the cargo entrance (right end of the cargo entrance) according to equations (1) to (4) shown in FIG. 15. The X-axis direction in the world coordinate system is the forward direction of the aircraft, and the Y-axis direction is to the left of the aircraft. The angle (yaw) is the angle of the x-axis as viewed from the x-axis.

[0112] Returning to FIG. 9, once the position and angle of the ground support equipment 10 relative to the cargo entrance are determined as described above (step S911), the control device 60 generates a route for approaching the cargo entrance by automatic driving (step S912). The control device 60 outputs an instruction to the traveling device 80 to move the ground support equipment 10 along the generated route (step S913). The control device 60 controls the traveling device 80 so that the ground support equipment 10 reaches the cargo entrance while facing the cargo entrance. Therefore, the ground support equipment 10 can reach the cargo entrance in an attitude suitable for loading and unloading cargo. Note that the ground support equipment 10 can reach the cargo entrance with high accuracy by reacquiring sensor data during movement and repeatedly executing steps S911 to S913.

[0113] When the ground support equipment 10 reaches the cargo entrance, the control device 60 detects the height of the cargo entrance (step S914). Here, the height of the bottom surface of the cargo hold is detected as the height of the cargo entrance. For example, the fourth camera 54 in the sensor device 50 is used as a sensor for detecting the height of the cargo entrance. The control device 60 controls the height of the lifting section 21 (and the lifting deck 31, if necessary) according to the detected height of the cargo entrance (step S915). The height of the cargo entrance may be detected while the ground support equipment 10 is moving to reach the cargo entrance. Also, a worker responsible for loading and unloading the cargo may get on the ground support equipment 10 after the ground support equipment 10 approaches the cargo entrance. The worker may get on before or after the height of the lifting section 21 is adjusted.

[0114] In this embodiment, as an example, a plurality of parts (reference parts) whose relative positional relationships are known are used in the cargo hold to detect the height of the cargo entrance. The control device 60 determines the height of the cargo entrance by using the positions of the plurality of parts whose relative positional relationships are known in an image captured by the camera 54 and the relative positional relationships of the plurality of parts.

[0115] Here, an example of the reference portion is a portion 107 provided on a floor surface 106 (see FIG. 1) inside the cargo hold 105 and located near the bottom end of the cargo entrance. An example of the portion 107 is a cargo entrance lock provided near the cargo entrance, as shown in the entrance image in FIG. 17. Such a lock is, for example, a lock for closing a cargo door or a lock for securing a container. Multiple such locks (generally four or two) are lined up in a straight line in the width direction of the cargo entrance in the cargo hold near the cargo entrance, and the spacing between them is known, although it differs depending on the model. Using these, the height of the cargo entrance is detected.

[0116] More specifically, the control device 60 uses the lock identifier 65 (reference portion identifier) ​​to identify a lock (reference portion) from an image captured by the camera 54 near the cargo entrance. The lock identifier 65 is a machine learning model that has been trained to identify a lock (reference portion) in an image when an image of the vicinity of the cargo entrance is input. The lock identifier 65 has been trained to identify various types of locks. The lock identifier 65 determines the position coordinates of each of the multiple locks in the image. Since the relative positional relationship of each of the multiple locks is known, the control device 60 can determine the height of the lock with respect to the camera 54 as the height of the cargo entrance.

[0117] FIG. 17 shows an example of the procedure for the process (step S914) of detecting the height of the cargo entrance. First, the control device 60 acquires lock data (reference part data) based on the model information acquired in step S31 of FIG. 3 (step S171). The above-mentioned model information table 71 preferably has lock data (not shown) associated with the model information. The lock data associated with the model information has data indicating the relative positional relationship between a plurality of locks, such as the lock interval for each model. The control device 60 refers to the model information table 71 based on the model information to acquire the lock data.

[0118] Then, the control device 60 detects the positions of the locks from the camera image of the locks using the lock identifier 65 (step S172: see FIG. 16). In the image of the lock identification result in FIG. 16, four white rectangular bounding boxes indicate the positions of the four identified cargo entrance locks.

[0119] The control device 60 calculates the center point coordinates (center points of rectangular bounding boxes) of each of the identified cargo entrance locks, and generates an approximation line for these center point coordinates. As shown in Fig. 18, the control device 60 sets both end points on the approximation line as the cargo entrance both end lock positions (u1, V1,) (u2, v2) (step S173). By using both end points on the approximation line, rather than using the positions of the identified both end locks themselves, the accuracy of the lock positions can be improved.

[0120] Based on the both end lock positions (u1, V1,) (u2, v2), the control device 60 uses the fact that the distance between both end locks (lock interval) is known from the lock data to calculate the both end lock heights from the both end lock positions in the camera image coordinate system. FIG. 18 shows the relationship between the camera image coordinates and the camera coordinates. In equations (5) and (6) in FIG. 18, f, Cx, and Cy are fixed values ​​due to the camera characteristics. The both end lock positions (u1, V1,) (u2, v2), which are values ​​in the camera image coordinate system (u, v), are expressed as lock positions in the camera coordinate system (x, y, z) by using equations (5) and (6). The lock positions x, y, and z can be expressed using only z.

[0121] FIG. 19 shows the relationship between the camera coordinates and the HL coordinates (XY coordinates in FIG. 15). In equation (7) in FIG. 19, R and t are determined by the camera characteristics and installation position and are known. The lock positions X, Y, and Z in the HL coordinate system can be expressed using z only. The control device 60 expresses the lock positions P1 (X1, Y1, Z1) and P2 (X2, Y2, Z2) in FIG. 19 using (u1, v1), (u2, v2), and z, using the relationships in equations (5), (6), and (7).

[0122] Here, since the ground support equipment 10 faces the cargo entrance, X1=X2, and since the inter-lock distance L is obvious, Y1-Y2=L. Using these conditions, the control device 60 calculates the lock height Z l , Z2 is calculated (step S174). l , Z2 is the lock height as seen from the ground support equipment 10 (high lift loader: HL), so the control device 60 l , Z2 are inverted and the height (including the inclination of the HL) of the ground support equipment 10 (high lift loader: HL) from the lock center is estimated (step S175).

[0123] Returning to FIG. 9, when it is determined in step S91 in FIG. 9 that the cargo door is closed, the second control (door closed state control) is executed.

[0124] In the second control, in step S921, a process of recognizing the position and angle of the ground support equipment 10 relative to the cargo entrance is executed by a second logic different from the first logic shown in Fig. 10. Fig. 20 shows the recognition process (step S921) by the second logic. The second logic also uses images acquired by the cameras 51, 52, and 53 (second sensors) and point cloud data acquired by the LiDAR sensor 55 (first sensor).

[0125] The entrance identifier included in the control device 60 detects a closed cargo entrance from the images captured by the cameras 51, 52, and 53 (step S201). This results in a bounding box being obtained as a rectangular area (entrance area) that includes the cargo entrance (see FIG. 21). The detection of this cargo entrance is similar to the first logic in the first control.

[0126] The control device 60 extracts partial point cloud data (fourth partial point cloud data. The fourth partial point cloud data is the same as the second partial point cloud data described above) within the closed cargo entrance range (step S202), and calculates the position and angle of the cargo entrance based on the fourth partial point cloud data (steps S213, S223). When the cargo door is closed, the fourth partial point cloud data is a measurement of the distance to the cargo entrance, so the center of gravity of the entrance range is determined as the position of the cargo entrance (step S213). In addition, the plane of the closed cargo door is detected using the fourth partial point cloud data, and the normal of the plane (direction in the horizontal plane) is determined as the angle of the cargo entrance (step S223).

[0127] The control device 60 performs coordinate conversion on the carry-in position calculated in step S213 and the carry-in angle calculated in step S223 (step S204). The method of coordinate conversion in step S204 is similar to the coordinate conversion in step 104 of Fig. 10. Note that the carry-in position in the second control preferably includes the height.

[0128] Returning to FIG. 9, once the position and angle of the ground support equipment 10 relative to the cargo entrance are determined (step S921), the control device 60 generates a route for approaching the cargo entrance by automatic driving (step S922). The control device 60 outputs an instruction to the traveling device 80 to move the ground support equipment 10 along the generated route (step S923). The control device 60 controls the traveling device 80 so that the ground support equipment 10 arrives at the cargo entrance while generally facing the cargo entrance. Note that by reacquiring sensor data during movement and repeatedly executing steps S921 to S923, the cargo entrance can be reached with high accuracy.

[0129] Furthermore, the control device 60 controls the height of the lifting unit 21 based on the loading entrance position including the height, and adjusts the lifting unit 21 to approximately the height of the loading entrance position. The worker gets into the cab 22 of the ground support equipment 10 and operates a control panel provided on the aircraft near the cargo loading entrance to open the cargo door 103.

[0130] The position and angle obtained in step S921 are less accurate than the position and angle obtained in step S911, but in the second control, it is sufficient that the ground support equipment 10 reaches a position where a worker who has boarded the ground support equipment 10 can operate the cargo door control panel. Therefore, in the second control, the accuracy of the position that the ground support equipment 10 should reach may be lower than that in the first control. Therefore, in the second control, the position and angle of the cargo entrance detected may be less accurate.

[0131] When the worker who got on the ground support equipment 10 that reached the cargo entrance in the second control completes the cargo door opening operation, he performs an operation to give a movement instruction (reverse instruction) to the control device 60. The control device 60 then performs position control to perform automatic travel to reverse along the route from the waypoint to the cargo entrance (step S925). This causes the ground support equipment 10 to return to the waypoint. When the ground support equipment 10 returns to the waypoint, the control device 60 executes the first control (steps S911 to S915). In this way, even if the cargo door is closed, the cargo door can be opened via the second control, thereby executing the first control for when the cargo door is open.

[0132] <3. Notes> The present invention is not limited to the above-described embodiment, and various modifications are possible. [Explanation of symbols]

[0133] 10. Ground Support Equipment 10A Ground Support Equipment 10B Ground Support Equipment 10C Ground Support Equipment 10C-1 Ground Support Equipment 10D Ground Support Equipment 10D-1 Ground Support Equipment 10E Ground Support Equipment 10E-1 Ground Support Equipment 10F Ground Support Equipment 10F-1 Ground Support Equipment 15 First Run 16 2nd Run 20 front 21 Liftable platform 22 Cab 30 Rear 31 Liftable Deck 32 Working Platform 50 Sensor device 51 Camera No. 1 52 Second Camera 53 Third Camera 54 4th Camera 55 LiDAR sensors 60 Control device 61 Initial Drive Control 62 Approach and height adjustment control 62A Door open status control (first control) 62B Door close state control (second control) 65 Lock Identifier 70 Databases 80 Running gear 91 Induction Line Detector 92 Aircraft symbol reading camera 100 aircraft 101 Aircraft (fuselage) 102 Cargo Entrance 102A Front cargo entrance 102B Rear cargo entrance 103 Cargo Door 105 Cargo hold 106 Cargo compartment floor 107 Rock 110 Wing 200 Induction Wire 201 1st guide line 202 2nd guide line 203 Third guide line 204 4th guide line 205 5th guide line 301 Equipment Restricted Area 302 Equipment waiting area C Container D. Detection range

Claims

1. A control device for controlling the travel of ground support equipment for an aircraft as the ground support equipment approaches the aircraft, based on point cloud data obtained by measuring the aircraft with a first sensor mounted on the ground support equipment, estimating the position of the end of the loading entrance in a state where the door for closing the loading entrance formed on the fuselage of the aircraft is open, and estimating the lateral direction of the fuselage, which is perpendicular to the horizontal plane with respect to the longitudinal direction of the fuselage, using the estimated position of the end and the lateral direction of the fuselage to determine the position of the ground support equipment and the attitude in the horizontal plane with reference to the end, and configured to control the travel of the ground support equipment to approach the loading entrance, Control device.

2. The loading entrance is a loading entrance for a cargo hold inside the fuselage, and the lateral direction of the fuselage is estimated based on the surface inside the cargo hold, The control device according to claim 1.

3. The surface inside the cargo hold is the inner wall of the cargo hold at the back of the cargo hold as seen from the loading entrance, a side wall of a container mounted in the cargo hold that is parallel to the inner wall of the cargo hold, or a plane of a side wall of a container mounted in the cargo hold that is perpendicular to the inner wall of the cargo hold, is, The control device according to claim 2.

4. The lateral direction of the fuselage is estimated based on the surface of the fuselage around the loading entrance, The control device according to claim 1.

5. The loading entrance is a loading entrance for a cargo hold inside the fuselage, and the lateral direction of the fuselage is estimated based on the surface inside the cargo hold and the surface of the fuselage around the loading entrance, The control device according to claim 1.

6. Estimating the position of the end is performed based on partial point cloud data within a partial region of the point cloud data that includes the end, where the partial region is a region that includes one of the both ends in the width direction of the loading entrance and does not include the other end, The control device according to claim 1.

7. The position of the end is estimated based on points in the partial point cloud data where the distance difference from adjacent points in the left-right direction is less than a threshold value, The control device according to claim 6.

8. The position of the end is estimated based on the point that is the point with the smallest distance difference from adjacent points in the left-right direction in the point cloud data and is on the one side, The control device according to claim 6.

9. Ground support equipment comprising the control device according to any one of Claims 1 to 8.

10. A method for controlling the travel of ground support equipment for an aircraft as the ground support equipment approaches the aircraft, estimating the position of an end portion of a loading entrance in a state where a door for closing the loading entrance formed in the fuselage of the aircraft is open, based on point cloud data obtained by measuring the aircraft with a first sensor mounted on the ground support equipment, and estimating a lateral direction of the fuselage that is perpendicular to a horizontal plane with respect to the longitudinal direction of the fuselage, based on the point cloud data, using the estimated position of the end portion and the lateral direction of the fuselage to obtain the position of the ground support equipment and the attitude in the horizontal plane with reference to the end portion, and controlling the travel of the ground support equipment so as to approach the loading entrance The control method comprising the above.

Images