Control device for ground support equipment based on aircraft loading dock location, ground support equipment, and control method

Abstract

To grasp, as accurately as possible, a loading dock of an aircraft for automatic traveling of ground support equipment.SOLUTION: A control apparatus disclosed is for controlling automatic traveling of ground support equipment of an aircraft to reach the aircraft, and is configured to perform first control comprising: detecting, on the basis of point cloud data of the aircraft measured by a first sensor mounted on the ground support equipment, a position of an edge of a loading dock in a state where a door for closing the loading dock formed on an airframe of the aircraft is open; and using the detected position of the edge to control the traveling of the ground support equipment to the loading dock.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] This disclosure relates to a control device for ground support equipment, ground support equipment, and a control method based on the aircraft loading dock location. [Background technology]

[0002] Patent Document 1 discloses a travel path setting system for setting travel paths for multiple ground support devices that perform ground support work on an aircraft. This travel path setting system determines the working position of each ground support device based on aircraft type information, etc., and sets a travel path from the current position of the ground support device to the working position. [Prior art documents] [Patent Documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2021-178533 [Overview of the project]

[0004] In Patent Document 1, the working position of the ground support equipment is determined based on aircraft type information, etc. While the approximate position of the aircraft can be determined using aircraft type information, its exact position cannot be determined. For loading and unloading cargo, etc., the aircraft's ground support equipment needs to arrive at the aircraft's entrance in a state where it is precisely aligned with the entrance.

[0005] Therefore, for the automated operation of ground support equipment, it is desirable to accurately determine the aircraft's loading dock location.

[0006] One aspect of the present disclosure relates to a control device. The disclosed control device is a control device for controlling automatic travel of aircraft ground support equipment to reach the aircraft. The disclosed control device is configured to perform a first control that comprises: detecting, based on point cloud data obtained by measuring the aircraft using a first sensor mounted on the ground support equipment, a position of an end portion of a cargo access opening in a state where a cargo door for closing the cargo access opening formed in the airframe of the aircraft is open; and controlling travel of the ground support equipment to the cargo access opening using the detected position of the end portion.

[0007] Another aspect of the present disclosure relates to ground support equipment. The disclosed ground support equipment includes a control device.

[0008] Yet another aspect of the present disclosure relates to a control method. The disclosed control method is a control method for automatic travel of aircraft ground support equipment to reach the aircraft. The disclosed method comprises: detecting, based on point cloud data obtained by measuring the aircraft using a first sensor mounted on the ground support equipment, a position of an end portion of a cargo access opening in a state where a door for closing the cargo access opening formed in the airframe of the aircraft is open; and controlling travel of the ground support equipment to the cargo access opening using the detected position of the end portion.

[0009] Further details will be described in the embodiments set forth later. [Brief Description of Drawings]

[0010] [Figure 1] Figure 1 is a configuration diagram of a sensor device and a control device. [Figure 2] Figure 2 is a side view of ground support equipment. [Figure 3] Figure 3 is a flowchart of automatic travel control. [Figure 4] Figure 4 is a flowchart of waypoint determination processing. [Figure 5] Figure 5 is an explanatory diagram of a waypoint and a guide line. [Figure 6] Figure 6 is an explanatory diagram of a detection range of a sensor device. [Figure 7] Figure 7 is an explanatory diagram of an automatic travel route. [Figure 8] Figure 8 is an explanatory diagram of an automatic travel route. [Figure 9] Figure 9 is a flowchart of control based on sensor data. [Figure 10] Figure 10 is a flowchart of a process for recognizing the vehicle body position and angle by the first logic. [Figure 11] Figure 11 is a diagram showing the relationship between the camera field of view and LiDAR reference coordinates. [Figure 12] Figure 12 is a diagram showing a camera image in which a cargo loading / unloading port is identified. [Figure 13] Figure 13 is an explanatory diagram of a method for obtaining the right end of the loading / unloading port. [Figure 14] Figure 14 is an explanatory diagram of a method for obtaining a normal line. [Figure 15] Figure 15 is an explanatory diagram of a coordinate system. [Figure 16] Figure 16 is an explanatory diagram of lock identification. [Figure 17] Figure 17 is a flowchart of cargo loading / unloading port height recognition. [Figure 18] Figure 18 is a diagram showing the relationship between a camera image coordinate system and camera coordinates. [Figure 19] Figure 19 is a diagram showing the relationship between a camera coordinate system and an HL coordinate system. [Figure 20] Figure 20 is a flowchart of a process for recognizing the vehicle body position and angle by the second logic. [Figure 21] Figure 21 is a diagram showing a camera image in which a loading / unloading port range is identified. [Mode for Carrying Out the Invention]

[0011] <1. Control device for aircraft ground support equipment, ground support equipment, and control method>

[0012] (1) The control device according to the embodiment is a control device for controlling the automatic movement of ground support equipment for an aircraft to reach the aircraft, and is configured to perform a first control that includes detecting the position of the end of an entrance formed in the aircraft's fuselage when the door for closing the entrance is open, based on point cloud data measured by a first sensor mounted on the ground support equipment, and controlling the movement of the ground support equipment to the entrance using the detected position of the end. The position of the end of an entrance with an open door has clearer shape characteristics than when the door is closed, so the position can be detected with high accuracy. As a result, the entrance can be accurately identified, and the accuracy of the movement of the ground support equipment can be improved. The entrance is, for example, a cargo entrance. A cargo entrance can also be used for loading and unloading cargo. The entrance includes a boarding gate for people such as passengers to enter and exit.

[0013] (2) It is preferable that the position of the end be detected based on the partial point cloud data within the partial region that includes the end. In this case, the end can be efficiently detected by the partial point cloud data within the partial region.

[0014] (3) Preferably, the partial region includes one of the widthwise ends of the entrance but does not include the other end.

[0015] (4) Preferably, the partial region is determined by detecting the entire range of the entrance and shifting the range in the width direction of the entrance. Note that the entire entrance only needs to be approximately the entire area, and does not need to be the entire area. Also, the method of shifting only needs to be approximately in the width direction, and does not need to be strictly in the width direction. For example, shifting the image horizontally will result in a shift approximately in the width direction.

[0016] (5) The range of the entire entrance is preferably detected by determining the entrance in image data obtained by a second sensor whose positional relationship with the first sensor is known. The entrance may be determined, for example, by a machine learning model described later, or by image recognition processing such as pattern matching.

[0017] (6) Preferably, the entrance in the image data is determined using a machine learning model that has been trained to identify the entrance from the image data.

[0018] (7) The first control preferably further comprises detecting a direction directly facing the entrance based on the point cloud data. The movement of the ground support equipment to the entrance is preferably controlled using the direction.

[0019] (8) Controlling the movement of the ground support equipment to the entrance is preferably done by controlling the movement of the ground support equipment so that it reaches the entrance while facing the entrance directly.

[0020] (9) The detection of a direction directly facing the loading entrance comprises at least one of the following: detecting a first direction perpendicular to the first surface based on the point cloud data, and detecting a second direction parallel to the second surface based on the point cloud data, wherein the first surface is preferably at least one of the interior wall of the cargo compartment located at the back of the cargo compartment as seen from the loading entrance, and a first side wall of a container loaded in the cargo compartment that is parallel to the interior wall of the cargo compartment, and the second surface is preferably a side wall of a container loaded in the cargo compartment that is perpendicular to the interior wall of the cargo compartment.

[0021] (10) The ground support equipment is preferably equipped with a height-adjustable lifting section, and the first control further comprises detecting the height of the entrance and controlling the height of the lifting section according to the detected height of the entrance. The height control is preferably performed after the ground support equipment has reached the entrance.

[0022] (11) The detection of the height of the loading entrance preferably includes detecting a plurality of parts located near the loading entrance whose relative positional relationship is known, and calculating the height of the loading entrance based on the positions of the detected plurality of parts and the relative positional relationship. The plurality of parts are preferably located on the floor of the cargo compartment. The plurality of parts are preferably located in the cargo compartment.

[0023] (12) The relative positional relationship is preferably determined based on the aircraft type information.

[0024] (13) When the door is closed, the system is configured to perform a second control different from the first control, the second control comprising detecting the position of the entrance based on point cloud data obtained by measuring the aircraft with the first sensor, and controlling the movement of the ground support equipment to the entrance using the detected position of the entrance, wherein the detection of the position of the entrance is performed based on partial point cloud data within a partial region of the point cloud data that includes the entrance with the closed door, and the partial region preferably includes the range of the entrance detected in image data obtained by the second sensor, whose positional relationship with the first sensor is known, when the aircraft with the closed door is captured.

[0025] (14) The ground support equipment according to the embodiment preferably includes the control device.

[0026] (15) A control method according to an embodiment is a control method for automatic driving of ground support equipment for an aircraft to reach the aircraft, comprising detecting the position of the end of a loading door formed in the aircraft's fuselage when the door for closing the loading door is open, based on point cloud data obtained by measuring the aircraft with a first sensor mounted on the ground support equipment, and controlling the driving of the ground support equipment to the cargo loading door using the detected position of the end.

[0027] <2. Examples of control devices, ground support equipment, and control methods for aircraft ground support equipment>

[0028] The following describes examples of control devices, ground support equipment, and control methods for aircraft ground support equipment, with reference to the drawings.

[0029] Figure 1 shows a control device 60 for ground support equipment 10 of an aircraft 100. The control device 60 is, for example, mounted on the ground support equipment 10 and controls the movement of the ground support equipment 10. The control device 60 may also be located outside the ground support equipment 10 and remotely control the ground support equipment 10. The control device 60 may include both a device mounted on the ground support equipment 10 and a device located outside the ground support equipment 10.

[0030] The control device 60 may be comprised of, for example, a computer that executes a computer program. The computer may include a processor and a storage device connected to the processor. The storage device may include, for example, a primary storage device and a secondary storage device. The primary storage device may be, for example, RAM. The secondary storage device may be, for example, a hard disk drive (HDD) or a solid-state drive (SSD). The memory contains a computer program executed by the processor. The processor reads and executes the computer program stored in the memory. The computer program has program code that causes the computer to execute processing for the control 61, 62 of the ground support equipment 10. Some or all of the functions of the control device 60 may be comprised of hardware logic circuits.

[0031] The control device 60 may, for example, perform initial travel control 61. The control device 60 may also, for example, perform approach and height adjustment control 62. Control 62 includes door open state control 62A (first control) and door closed state control 62B (second control). These controls will be described later.

[0032] Ground support equipment 10 controlled by control device 60 is equipment for ground handling operations of aircraft 100. Various types of ground support equipment exist depending on the type of work. Examples of ground support equipment 10 include cargo loaders, catering cars, passenger stairs, and cargo truck conveyors. Cargo loaders are used to load and unload cargo into the aircraft's cargo compartment. Catering cars are vehicles used to load beverages, food, and in-flight supplies onto the aircraft. Catering cars approach passenger boarding gates. Catering cars are equipped with a lifting mechanism that adjusts to the height of the boarding gate. The boarding gate that the catering car approaches is the right-side gate, not the left-side gate normally used for passenger entry and exit. The boarding gate is closed by the passenger door. Passenger stairs are vehicles used for passengers to board and disembark from the aircraft. Passenger stairs also approach boarding gates. The passenger stair car is adjusted to the height of the boarding gate. The boarding gate that the passenger stair car approaches is the one on the left side of the aircraft. Cargo truck conveyors can be used to approach the bulk cargo compartment of the aircraft. Cargo truck conveyor vehicles are equipped with a lift for workers to stand on, and the height of this lift is adjustable. The height is adjusted to a height that makes it easy for workers to load cargo into the cargo compartment. Below, as an example of ground support equipment 10 that approaches the loading entrance and has its height adjusted, a cargo loader used for loading and unloading cargo into the cargo compartment of an aircraft will be described. Cargo loaders include high-lift loaders, main deck loaders, belt loaders, and transporters. A high-lift loader is a vehicle used for loading and unloading containers into the lower deck cargo compartment or main deck cargo compartment of an aircraft 100.

[0033] Containers loaded and unloaded by high-lift loaders are also called Unit Load Devices (ULDs). ULDs are storage equipment specifically for aircraft. The cargo compartment where containers are loaded is also called the container compartment. In addition, the cargo compartment of aircraft 100 also has a bulk compartment where cargo is loaded loose. Cargo is loaded and unloaded into the bulk compartment by a belt loader.

[0034] Aircraft 100 may, for example, have a forward cargo compartment and an aft cargo compartment. The aft cargo compartment is located aft of the forward cargo compartment. The cargo compartments are located inside the fuselage of the aircraft. The entrance for loading cargo into the forward cargo compartment can be called the forward cargo entrance. The forward cargo entrance is located, for example, in front of the main wing. The entrance for loading cargo into the aft cargo compartment can be called the aft cargo entrance. The aft cargo entrance is located, for example, behind the main wing. The entrances can also be used for loading cargo out. Aircraft 100 may have one or more cargo entrances.

[0035] The forward and rear cargo loading ports may be located on the lower sides of the aircraft 100's fuselage. Generally, the forward and rear cargo loading ports are located on the lower right side of the fuselage. The right side of the fuselage refers to the right side of the fuselage when viewed from the rear of the aircraft 100 looking forward.

[0036] As shown in Figure 2, the cargo entrance 102 of the aircraft 100's fuselage 101 is provided with a door called a cargo door 103. The cargo entrance 102 is opened and closed by the cargo door 103. The opening and closing operation of the cargo door 103 is driven by, for example, an electric motor. A cargo door 103 provided on the side of the fuselage 101 is called a side cargo door. As shown in Figure 2, the side cargo door 103 opens outwards and upward from the fuselage 101. Note that since Figure 2 is a view of the aircraft 100 from the front looking towards the rear, the side cargo door 103 provided on the right side of the fuselage 101 is located on the left side of the fuselage 101 in Figure 2. Note that the cargo door may also be a nose cargo door, which is the nose section at the front of the fuselage 101. A nose cargo door opens by pushing the nose section upward. The cargo entrance opened and closed by a nose cargo door can be called a nose cargo entrance.

[0037] As shown in Figure 2, the high-lift loader 10 is used to load or unload container C with the side cargo door 103 open. The high-lift loader 10 approaches the cargo entrance facing the entrance for loading and unloading cargo.

[0038] The high-lift loader 10 comprises a front section 20 and a rear section 30. The front section 20 includes a lifting section 21 connected to the cargo loading entrance 102. The lifting section 31 is height-adjustable. The lifting section 21 is also called a lifting platform 21. The lifting platform 21 is height-adjustable to match the floor surface 106 of the cargo compartment 105.

[0039] The front section 20 of the vehicle is equipped with a driver's cab 22 that moves up and down together with the lifting section 21. The driver's cab 22 is used by an operator to manually operate the high-lift loader 10. Operations for manual operation include, for example, steering and switching between forward and reverse. An operator in the driver's cab 22 can also manually open and close the cargo door 103 by operating a control panel (not shown) located near the cargo door 103.

[0040] The front section 20 of the vehicle is equipped with a sensor device 50. For example, the sensor device 50 may be positioned in front of the driver's cab 22. The sensor device 50 can move up and down in accordance with the movement of the lifting section 21. The arrangement of the sensor device 50 in the ground support equipment 10 is not particularly limited.

[0041] The rear section 30 of the vehicle includes a liftable deck 31 located behind the front section 20 of the vehicle. The liftable deck 31 is used for loading and unloading cargo such as containers C. As shown in Figure 2, the height of the liftable deck 31 can be adjusted to the same height as the liftable platform 21 of the front section 20 of the vehicle. The rear section 30 of the vehicle includes a work platform 32 located behind the liftable deck 31. The work platform 32 is used, for example, for transferring cargo with a dolly (not shown).

[0042] Returning to Figure 1, the sensor device 50 will be described. The sensor device 50 acquires sensor data related to the aircraft 100 that the ground support equipment 10 should reach. The sensor data acquired by the sensor device 50 is used, for example, to control the automatic driving of the ground support equipment 10. The sensor device 50 shown in Figure 1 includes, as an example, cameras 51, 52, 53, and 54 for image acquisition, and a LiDAR sensor 55. The sensor device 50 includes, as an example, a plurality of cameras 51, 52, 53, and 54. The plurality of cameras include, as an example, a first camera 51, a second camera, a third camera, and a fourth camera.

[0043] The LiDAR sensor 55 irradiates the object to be measured with laser light to measure the distance to the object, etc. The LiDAR sensor 55 acquires point cloud data, which shows the three-dimensional coordinates of many points on the surface of the object to be measured. The type of sensor used to acquire the point cloud data is not particularly limited.

[0044] The image and point cloud data acquired by the sensor device 50 are provided to the control device 60 as sensor data. The control device 60 uses the acquired sensor data to control, for example, the travel device 80 and the lifting unit 21 of the ground support equipment 10. The travel device 80 is responsible for the movement of the ground support equipment 10 and includes a drive unit for movement, a steering mechanism, and wheels. Automatic movement by the travel device 80 is controlled by the control device 60. The control by the control device 60 includes steering and forward / reverse switching of the ground support equipment 10. The control by the control device 60 may also include speed adjustment.

[0045] For example, the control device 60 uses sensor data to detect a target (target destination) that the ground support equipment 10 should reach, and automatically drives the ground support equipment 10 to reach the target destination. In this case, the part of the aircraft 101 that is the target destination is, for example, a cargo loading hatch located on the side of the aircraft 101. In this case, the control device 60 determines the position of the cargo loading hatch that the ground support equipment 10 should reach, and controls the movement of the ground support equipment 10 to reach that cargo loading hatch. The control device 60 also detects the height of the cargo loading hatch and adjusts the height of the lifting unit 21 to match the height of the cargo loading hatch. The control device 60 can also adjust the height of the lifting deck 31 according to the height of the lifting unit 21.

[0046] The control device 60 can also acquire data or signals other than sensor data as needed for controlling the ground support equipment 10. As shown in Figure 2, other data that the control device 60 can acquire include, for example, the current position of the ground support equipment 10, data stored in the database 70, data detected by the guide line detector 91, data acquired by the aircraft identification number reading camera 92, the type of cargo loading dock that the ground support equipment 10 should reach, and at least one of the following selected data: flight schedule data. These data will be described later.

[0047] Figure 3 shows the procedure for automatic driving control of the ground support equipment 10 by the control device 60. The procedure shown in Figure 3 shows the process from when the ground support equipment 10 is stopped at a waiting position at the airport until it reaches the aircraft 100. In this example, the ground support equipment 10 reaches the aircraft 100 after passing through several intermediate points. Therefore, in the procedure shown in Figure 3, intermediate points are determined (step S33). In order to determine the intermediate points, necessary data is acquired prior to the determination of the intermediate points (steps S31, S32).

[0048] The data necessary to determine the waypoints is, for example, aircraft type information. Since the location of the cargo loading dock on aircraft 100 may differ depending on the aircraft type, aircraft type information is useful for appropriately navigating the aircraft automatically in accordance with various aircraft types. For this reason, in step S31, aircraft type information (aircraft type information) is acquired. Note that if the location of the cargo loading dock can be identified even without aircraft type information, step S31 may be omitted.

[0049] Aircraft type information is obtained, for example, by the control device 60 accessing a database containing flight schedule data (see Figure 1). Flight schedule data is stored, for example, on a database server for managing flight schedules. The flight schedule data includes aircraft type information of the aircraft 100 that the ground support equipment 10 is to reach. Aircraft type information may also be obtained by reading the aircraft registration number assigned to the aircraft 100 that the ground support equipment 10 is to reach using a camera 92 installed on the ground support equipment 10 or a camera 92 installed outside the ground support equipment 10 (see Figure 1).

[0050] The aircraft registration number is a symbol used to identify an aircraft as an individual. If the registration number includes a number or symbol that directly indicates the aircraft type, the aircraft type can be identified from the registration number. If the registration number does not directly indicate the aircraft type, the aircraft type can be identified by referring to a table that maps registration numbers to aircraft types. Note that the registration number may be attached to the aircraft in a partially abbreviated form.

[0051] The aircraft registration number can be displayed on various parts of the aircraft 101, such as the main wing, fuselage, or nose gear cover. The registration number read by camera 92 can be from any of these parts. For example, by recognizing the registration number attached to the nose gear cover with camera 92, aircraft type information can be obtained.

[0052] Furthermore, in order to determine the waypoint, the control device 60 may acquire the type of cargo entrance that the ground support equipment 10 (high-lift loader 10) should reach (step S32). Here, the type of cargo entrance is data indicating which of the multiple cargo entrances provided by the aircraft 100 it is. The types of cargo entrances are, for example, the forward cargo compartment (FWD) and the rear cargo compartment (AFT). For example, a high-lift loader 10 that should reach the forward cargo compartment acquires data indicating "forward cargo compartment (FWD)" as the type of cargo entrance. Similarly, a high-lift loader 10 that should reach the rear cargo compartment acquires data indicating "rear cargo compartment (AFT)" as the type of cargo entrance. The data indicating which high-lift loader 10 should reach which cargo entrance is either pre-set in the management computer (not shown) or can be appropriately set in the management computer by the work manager. The control device 60 can acquire the type of cargo entrance that the ground support equipment 10 should reach by accessing the management computer. Furthermore, if the type of cargo loading entrance to be reached is known to the control device 60, this step may be omitted.

[0053] In step S33, the control device 60 determines the waypoints. Figure 4 shows the procedure for determining the waypoints, which corresponds to step S33. First, the control device 60 acquires the current position of the ground support equipment 10 (step S41). The current position here is the waiting position of the ground support equipment 10 before it travels toward the aircraft 100. The current position is, for example, the position acquired by the GNSS (Global Navigation Satellite System) receiver mounted on the ground support equipment 10. If the ground support equipment 10 is parked at a predetermined waiting position, data indicating the waiting position may be acquired as the current position. If the waiting position is known to the control device 60, or if the waypoints can be determined even without the waiting position (current position), step S41 may be omitted.

[0054] The control device 60, as an example, determines the waypoints by referring to the model information table 71 stored in the database 70 (step S42). The table 71 contains, for example, waypoints P for each of the multiple model information. FWD1 ,P FWD2 ,P AFT This is the associated data. As shown in Figure 5, the transit points can be set according to the type of cargo loading entrance (location of the cargo loading entrance). For example, transit point P for forward cargo loading entrance 102A is an example of a transit point. FWD1 ,P FWD2 And, transit point P for rear cargo loading entrance 102B. AFT The following can be set. Depending on the location of the cargo loading dock, the control device 60 can determine an appropriate route to reach the cargo loading dock by selecting a waypoint.

[0055] Furthermore, the waypoints can be set according to the current position (standby position) of the ground support equipment 10. For example, the first waypoint P may be used as the waypoint for the forward cargo loading entrance 102A. FWD1 and the second waypoint P FWD2 The following can be set. By selecting a waypoint according to the current position (standby position) of the ground support equipment 10, the control device 60 can determine an appropriate waypoint according to the current position (standby position).

[0056] As shown in Figure 5, possible waiting positions for the ground support equipment 10 include, for example, positions 10C, 10D, 10E, and 10F shown in Figure 5. Here, the waiting positions for the ground support equipment 10 are located within the Equipment Staging Area (ESA) 302, 302 at the airport. The Equipment Staging Area is located outside the Equipment Restraint Area (ERA) 301 and is used as a waiting area for equipment used for ground handling operations of aircraft 100 parked at a spot. The spot is the location where the aircraft 100 is parked. In addition to the high-lift loader 10, other equipment such as dollies, catering vehicles, and refueling trucks may also be parked in the Equipment Staging Area 302. The Equipment Restraint Area 301 is located inside the Equipment Staging Area, and equipment is prohibited from entering or leaving the spot when aircraft 100 is entering or leaving.

[0057] In Figure 5, the standby ground support equipment 10C, 10D, 10E, and 10F are located within the equipment standby area 302 and are positioned in front of the cargo loading docks 102A and 102B of the parked aircraft 100. Furthermore, the standby ground support equipment 10C, 10D, 10E, and 10F are oriented in the same direction as the parked aircraft 100. Therefore, from the perspective of the standby ground support equipment 10, the cargo loading docks 102A and 102B of the parked aircraft 100 are located behind the ground support equipment 10. As a result, the sensor device 50 installed at the front of the ground support equipment 10 may have difficulty detecting the cargo loading docks 102A and 102B. Furthermore, because the distance from the standby ground support equipment 10C, 10D, 10E, and 10F to the cargo loading docks 102A and 102B is relatively large (several tens of meters or more), the sensor device 50 installed on the standby ground support equipment 10 may have difficulty accurately detecting the positions of the cargo loading docks 102A and 102B, which are far away and not particularly large.

[0058] Accordingly, the control device 60, as an example, does not start automatic traveling based on sensor data acquired by the sensor device 50 from the standby position of the ground support equipment 10, but temporarily starts the automatic traveling from a waypoint. The waypoint is set to a position where the cargo loading / unloading ports 102A, 102B that the ground support equipment 10 should reach are within the detection range of the sensor device 50 mounted on the ground support equipment 10. The ground support equipment 10 moves to the waypoint such that the cargo loading / unloading ports 102A, 102B fall within the detection range of the sensor device 50.

[0059] For example, as shown in FIG. 6, when the detection range D of the sensor device 50 is set to be directed from the front to the left of the ground support equipment 10, as in the ground support equipment 10A in FIG. 6, the cargo loading / unloading port 102 is positioned within the detection range D of the sensor device 50 by substantially directly facing the cargo loading / unloading port 102 at a position on the right side of the airframe 101. Further, as in the ground support equipment 10B in FIG. 6, the cargo loading / unloading port 102 is positioned within the detection range of the sensor device 50 by facing the front of the airframe in alignment with the airframe 101 at a position diagonally to the right rear of the cargo loading / unloading port 102. In the present embodiment, the waypoint is set to be the position of the ground support equipment 10A or 10B shown in FIG. 6 with respect to the cargo loading / unloading port 102 to be reached. The waypoint may be set within the equipment standby area 302, or may be set within the equipment restricted area 301.

[0060] Returning to FIG. 5, the first waypoint P for the front cargo loading / unloading port 102A FWD1 is a position corresponding to 10A in FIG. 6. The second waypoint P for the front cargo loading / unloading port 102A FWD2 and the waypoint P for the rear cargo loading / unloading port 102B AFT is a position corresponding to 10B in FIG. 6. As an example, the first waypoint P FWD1 is provided as a waypoint for the ground support equipment waiting at the position 10C in FIG. 5, and the second waypoint P FWD2 is provided as a waypoint for the ground support equipment waiting at the position 10E in FIG. 5. The waypoint P for the rear cargo loading / unloading port 102B AFTThese points are provided as waypoints for ground support equipment waiting at positions 10D and 10E in Figure 5. Note that these waypoints may be set within the equipment waiting area or within the equipment restriction area.

[0061] Based on the acquired model information (step S31), the control device 60 refers to the model information table 71 to determine the candidate waypoint P FWD1 ,P FWD2 ,P AFT Obtain the following. Also, if the obtained "Type of cargo loading dock" is "Forward cargo loading dock (FWD)", the candidate is location P. FWD1 ,P FWD2 This narrows it down to these. Also, if the current position of the standby ground support equipment (step S41) is at position 10C, then the waypoint P FWD1 If it is determined to be at position 10E, then waypoint P FWD2 The decision is made (step S42). If the acquired "type of cargo loading port" is "rear cargo loading port (AFT)", then the transit point P AFT This is determined (step S42). In this embodiment, the "waypoint" is also called the "turnaround position". For example, the ground support equipment 10 reaches the turnaround position from the waiting position by reversing, and then reaches the cargo loading entrance 102 from the turnaround position by moving forward. By reversing and then moving forward to approach the aircraft 100, the number of turns required to approach the aircraft 100 from the waiting position can be reduced.

[0062] Once the waypoint determination process S33 in Figure 4 is completed, the control device 60 performs automatic driving (initial driving; first driving) to move the ground support equipment 10 to the determined waypoint (step S34 in Figure 3). For example, driving to the waypoint is done in reverse. When the control device 60 reaches the waypoint, it controls the ground support equipment 10 to reach the cargo loading entrance based on the sensor data acquired by the sensor device 50 (step S35).

[0063] As shown in Figure 5, the equipment restriction area 301 and the equipment waiting area 302 are provided with guide lines 200 for guiding the ground support equipment 10 to the intermediate point. The guide lines 200 are markers painted on the airport ground, electromagnetic guidance lines embedded in the ground, or other route guidance markers. The ground support equipment 10 can move along the guide lines 200. The guide lines 200 are detected by a guide line detector 91 provided on the ground support equipment 10. The guide line detector 91 is, for example, a camera that images the guide lines 200, or a sensor that detects electromagnetic guidance lines.

[0064] The guide line 200 shown in Figure 5 includes, as an example, a first guide line 201, a second guide line 202 and a fourth guide line 204 branching off from the first guide line, a third guide line 203 branching off from the second guide line 202, and a fifth guide line 205 branching off from the fourth guide line 204. The first guide line 201 is located within the equipment restriction area 301, and there is a transit point P along its path. AFT There are several types of guide lines. The second guide line 202 extends from the equipment restriction area 301 to the equipment standby area 302 and is provided to reach position 10C from the first guide line 201. The third guide line 203 extends from the equipment restriction area 301 to the equipment standby area 302 and is provided to reach position 10D from the second guide line 202. The fourth guide line 204 extends from the equipment restriction area 301 to the equipment standby area 302 and is provided to reach position 10E from the first guide line 201. The fifth guide line 205 is provided within the equipment standby area 302 and is provided to reach position 10F from the fourth guide line 204.

[0065] Figure 7 shows an example of a travel route when the ground support equipment 10 is waiting at positions 10C and 10D in Figure 5. When the ground support equipment 10C in Figure 7 is heading towards the forward cargo loading entrance 102A, the route point is P FWD1 It is decided that in this case, ground support equipment 10C will be at waypoint P. FWD1 The first run 15 (initial run 15) is performed in reverse, and the vehicle is positioned to face the cargo loading entrance 102A almost directly, as shown by the ground support equipment 10C-1. This ground support equipment 10C-1 is located at point P.FWD1 The second run 16 is performed, moving forward and approaching the forward cargo loading entrance 102A. In the first run 15 of the ground support equipment 10C, for example, the initial part of the run is route-guided along the second guide line 202, and thereafter, the ground support equipment 10C is guided to a waypoint P by the GNSS signal received. FWD1 The vehicle will be guided to a specific location.

[0066] When the ground support equipment 10D in Figure 7 is heading towards the rear cargo loading entrance 102B, the intermediate point is P AFT This is determined. In this case, the ground support equipment 10D will be at waypoint P. AFT The first run 15 is performed in reverse, and the aircraft is oriented so that the cargo loading port 102B is located diagonally to the left front, as shown by the ground support equipment 10D-1, and is almost parallel to the aircraft 101 and facing forward. This ground support equipment 10D-1 is located at waypoint P AFT The second run 16 is performed, moving forward and approaching the rear cargo loading entrance 102B. The first run 15 of the ground support equipment 10D is, for example, a run guided along the third guide line 203, the second guide line 202, and the first guide line 201.

[0067] Figure 8 shows an example of a travel route when the ground support equipment 10 is waiting at positions 10E and 10F in Figure 5. When the ground support equipment 10E in Figure 8 is heading towards the forward cargo loading entrance 102A, the route point in the aircraft type information table is P FWD2 This is determined. In this case, the ground support equipment 10E will be at waypoint P FWD2 The first run 15 is performed in reverse, and the aircraft is oriented so that the cargo loading port 102A is located diagonally forward to the left, as shown by the ground support equipment 10E-1, and is facing forward, almost parallel to the aircraft 101. This ground support equipment 10E-1 is located at waypoint P FWD2 The second run 16 is performed, moving forward and approaching the forward cargo loading entrance 102A. The first run 15 of the ground support equipment 10E is, for example, a run guided along the fourth guide line and the first guide line 201.

[0068] When the ground support equipment 10F in Figure 8 is heading towards the rear cargo loading entrance 102B, the intermediate point is P AFTThis is determined. In this case, ground support equipment 10F will be at transit point P. AFT The first run 15 is performed in reverse, and the aircraft is oriented so that the cargo loading port 102B is located diagonally to the left front, as shown by the ground support equipment 10F-1, and is almost parallel to the aircraft 101 and facing forward. This ground support equipment 10F-1 is located at waypoint P AFT The second run 16 is performed, moving forward and approaching the rear cargo loading entrance 102B. The first run 15 of the ground support equipment 10F is, as an example, a run guided along the fifth guideline 205 and the first guideline 201.

[0069] When the ground support equipment reaches the positions 10C-1, 10D-1, 10E-1, and 10F-1 shown in Figures 7 and 8, even if the cargo loading docks were outside the detection range D of the sensor device 60 at the standby position, the ground support equipment can position the cargo loading docks 102A and 102B within the detection range D. Therefore, the control device 60 controls the second run, etc., based on the sensor data measured by the aircraft 100 after the waypoint (step S35 in Figure 3). Note that sensor data may also be utilized during the first run 15 as needed.

[0070] Figure 9 shows the procedure for control by the control device 60 based on sensor data (step S35). In the procedure shown in Figure 9, as an example, it is first determined whether the cargo doors of the cargo loading docks 102A and 102B that the ground support equipment 10 is about to reach are open or closed (step S91). Whether the cargo doors are open or closed before the ground support equipment 10 approaches the cargo loading docks 102A and 102B depends on the case. Whether they are open or closed depends, for example, on the type of aircraft 100. For example, on one type of aircraft, the cargo doors may be open before the ground support equipment 10 approaches the cargo loading docks 102A and 102B after parking. On another type of aircraft, the cargo doors remain closed after parking, and the operator on the ground support equipment 10 needs to operate a cargo door control panel (not shown) located on the aircraft near the cargo doors to open them. Therefore, in this embodiment, as an example, the control device 60 performs different controls depending on whether the cargo door is open or closed.

[0071] In the door opening / closing determination in step S91, for example, it is determined whether the cargo door is open or closed based on the aircraft type information acquired in step S31 in Figure 3. For door opening / closing determination, it is preferable that the aircraft type information table 71 contains door opening / closing information (not shown) associated with the aircraft type information. The door opening / closing information associated with the aircraft type information indicates whether the cargo door is open or closed for each aircraft type. The control device 60 refers to the aircraft type information table 71 containing the door opening / closing information and determines whether the cargo loading port of the aircraft 100 to which the ground support equipment 10 should reach is open or closed. If it is determined to be open, the first control (door open state control) is executed, and if it is determined to be closed, the second control (door closed state control) is executed. Note that if the second control is executed, the first control may be executed after the second control because the cargo door is open.

[0072] In the first control, step S911 executes a process to recognize the position and angle of the ground support equipment 10 relative to the cargo entrance. Similar recognition is performed in the second control, but in the first control, the recognition is performed by the first logic, while in the second control, the recognition is performed by a second logic that is different from the first logic. The difference between the first logic and the second logic stems from whether the cargo door is open or closed.

[0073] Figure 10 shows the recognition process by the first logic (step S911). Here, camera images acquired by the three cameras (second sensor) of the sensor device 50, namely the first camera 51, the second camera 52, and the third camera 53, and point cloud data acquired by the LiDAR sensor 55 (first sensor) are used.

[0074] In step S101 shown in Figure 10, the cargo entrance is detected from the camera image. For detecting the cargo entrance from the camera image, for example, an entrance identifier built into the control device 60 is used. The entrance identifier is a machine learning model that has been trained to identify the cargo entrance in an image when an image of an aircraft is input. In this case, the entrance identifier has been trained using both images with open and closed cargo doors for various types of aircraft, so that it can detect the cargo entrance whether the cargo door is open or closed. Therefore, the entrance identifier according to this embodiment can also be used for the second control, which is advantageous.

[0075] Furthermore, separate loading dock identifiers may be provided for the first control and the second control. In this case, the loading dock identifier for the first control may be machine-learned to identify cargo loading docks with open cargo doors, and the loading dock identifier for the second control may be machine-learned to identify cargo loading docks with closed cargo doors.

[0076] Figures 11 and 12 illustrate a method for determining the coordinates of the cargo loading entrance from camera images. Figure 11 shows the positional relationship between the camera coordinates and the LiDAR center polar coordinates. The three cameras 51, 52, and 53 are all at the same height, but are mounted on the sensor device 50 so that their fields of view are in different orientations. In Figure 11, the x-direction (upwards in Figure 11) corresponds to the front direction of the ground support equipment 10, and the y-direction corresponds to the left direction of the ground support equipment 10. The first camera 51 is positioned to face the x-direction (forward), and the third camera 53 is positioned to face the y-direction (left). The second camera 52 is positioned to face an intermediate direction between the first camera 51 and the third camera 53 (diagonally forward to the left). The fields of view of the three cameras partially overlap. The combined field of view of these three cameras 51, 52, and 53 constitutes the overall camera field of view. Furthermore, the LiDAR sensor 55 is designed to measure the distance to an object within a range that includes the entire camera field of view. The distance to the object can be appropriately determined within the overlapping range between the entire camera field of view and the detection range of the LiDAR sensor 55. Therefore, in this case, the entire camera field of view of the three cameras corresponds to the detection range D of the sensor device 50.

[0077] The loading dock identifier detects the cargo loading dock from each of the three images. This detection is performed, for example, by determining the coordinates of the rectangular area containing the loading dock in each image. This rectangular area is, for example, a bounding box indicating the loading dock identified by the loading dock identifier. The bounding box is obtained through bounding box annotation based on the identification results from the loading dock identifier.

[0078] In Figure 12, as an example, a cargo entrance with an open cargo door is shown in two separate images: a first camera image and a second camera image. As shown in Figure 12, the entrance identifier extracts a rectangular area containing the cargo entrance in both the first and second camera images. Since the coordinates of the rectangular area are obtained as camera image coordinates (u,v), the control device 60 converts the camera image coordinates of the rectangular area to LiDAR center polar coordinates (θ,φ). The coefficients for this conversion are determined from the relative positional relationship between the camera and the LiDAR. The relative positions of the camera and the LiDAR are known.

[0079] As shown in Figure 12, if the cargo entrance spans two camera images, the control device 60 calculates the range that includes the cargo entrance by adding the polar coordinate ranges representing the rectangular areas of the two images. For example, if the polar coordinate range θ of the rectangular area in the first camera image is from 20° to 35°, and the polar coordinate range θ of the rectangular area in the second camera image is from 30° to 40°, then the polar coordinate range θ of the area that includes the entire cargo entrance will be from 20° to 40°. The polar coordinate φ is calculated in a similar manner by adding them together. Through these processes, the range that includes almost the entire cargo entrance is extracted as coordinate values ​​based on the LiDAR sensor 55. That is, the azimuth angle range (range of θ) and elevation / depression angle range (range of φ) of the entire cargo entrance are extracted based on the LiDAR sensor (step S101).

[0080] Furthermore, if a cargo loading dock is detected in both images, and the bounding boxes representing those docks do not overlap, it may be determined that another cargo loading dock was mistakenly detected. In the event of a misdetection, a more likely cargo loading dock location can be adopted based on its positional relationship with the transit point. Alternatively, if a misdetection occurs, the ground support equipment 10 can be brought closer to the aircraft 100 from the transit point to prevent the other cargo loading dock from appearing in the image, at which point the cargo loading dock may be redetected.

[0081] Once the azimuth angle range (range of θ) and elevation / depression angle range (range of φ) of the cargo entrance are extracted, steps S112 and S113 estimate a predetermined reference position at the cargo entrance. The reference position here is the right end (right edge) of the cargo entrance with the cargo door open. The right end is the rightmost end of the periphery of the cargo entrance when viewed from outside the aircraft, facing the cargo entrance directly. The reference position may also be the left end of the cargo entrance. In other words, it is preferable that the reference position be one of the ends in the width direction of the cargo entrance.

[0082] The position of the cargo entrance detected in step S101 is a rough representation of the bounding box vertex coordinates. Therefore, in order to accurately connect the ground support equipment 10 to the cargo entrance 102, it is desirable to detect the position of the cargo entrance 102 more precisely. Also, since the cargo entrance with the door open is an opening, it is not possible to directly measure the distance to the opening cargo entrance. Therefore, it is preferable to use the periphery of the cargo entrance as the reference position. Accordingly, in steps S112 and S113, the position of the rightmost edge, which is the reference position at the cargo entrance, is estimated using the position of the cargo entrance detected in step S101 and the distance information measured by the LiDAR sensor 55.

[0083] To estimate the reference position (right end) of the cargo loading entrance, the control device 60 acquires point cloud data (first partial point cloud data) from all point cloud data acquired by the LiDAR sensor 55 within the region (first partial region) necessary for estimating the reference position (right end) of the cargo loading entrance. The first partial region is the region that includes the right end which serves as the reference position. The first partial region does not include the left end. By including only one end of the widthwise ends of the cargo loading entrance in the first partial region, it is not necessary to perform processing to prevent false detection of the other end, thus simplifying the process.

[0084] The first partial region is determined using the entire cargo entrance area (hereinafter simply referred to as the "entrance area") detected in step S101 of Figure 10. In the camera image shown in Figure 13, the first partial region is shown as a solid line, labeled as the "SIDE point cloud data area." Also in the camera image shown in Figure 13, the entrance area detected in step S101 is shown as a dotted rectangle.

[0085] The control device 60 determines the first partial region (SIDE point cloud data region) by, for example, shifting the entrance range to the right in the width direction of the entrance range by 0.5 of the entrance range's width. In the entrance range, which captures the entire entrance, the rightmost reference point is near the boundary of the entrance range, making it difficult to utilize information around the rightmost point when detecting the rightmost point. In contrast, in the first partial region, the rightmost point is located in the center of the first partial region in the width direction, allowing for the utilization of information around the rightmost point. Furthermore, the first partial region does not include the leftmost point due to the widthwise shift. Thus, the first partial region includes almost the entire rightmost point, from its top to its bottom.

[0086] The control device 60 extracts point cloud data within a first partial region (first partial point cloud data) from the point cloud data (step S112), and then detects the right edge of the cargo entrance based on this first partial point cloud data (step S113). By detecting the right edge based on the first partial point cloud data rather than the entire point cloud data, the processing load can be reduced. Furthermore, it is not always easy to identify the cargo entrance from the entire point cloud data, which includes information on other parts besides the cargo entrance, without utilizing the first partial region. However, by determining the first partial region including the right edge (reference position) of the cargo entrance from image data that can easily identify the cargo entrance, and then using the distance information contained in the point cloud data within that first partial region, the position of the right edge can be determined with high accuracy. Note that while image data is suitable for identifying the cargo entrance, it does not contain distance information, so to determine the position of the right edge with high accuracy, it is preferable to use point cloud data that contains distance information.

[0087] The rightmost point, which is the reference point for the cargo entrance, can be detected by utilizing the fact that a relatively large distance difference can occur between the distance from the LiDAR sensor 55 to the aircraft surface around the cargo entrance (especially the aircraft surface to the right of the rightmost point) and the distance measured for the area inside the periphery of the cargo entrance when the cargo door is open (especially the area to the left of the rightmost point). When the cargo door is open, the LiDAR sensor 55 measures the distance to the interior of the cargo compartment, and a relatively large distance difference tends to occur between the distance to the interior of the cargo compartment and the distance to the aircraft surface. Therefore, the reference position (rightmost point) can be easily detected. On the other hand, when the cargo door is closed, even if the LiDAR sensor 55, which is far from the aircraft, measures the distance to the cargo entrance and its vicinity, the difference between the distance to the cargo entrance and its vicinity is small, making it difficult to detect the reference position (rightmost point). For this reason, the second logic detects the position of the cargo entrance by a different method.

[0088] Furthermore, detecting the right edge of the cargo loading port within the first sub-region can also be said to be equivalent to detecting the left edge of the aircraft surface within the first sub-region.

[0089] The following describes an example of how to determine the rightmost position from the first partial point cloud data within the first partial region. As shown in Figure 13, "Camera image with superimposed point cloud data," the point cloud data has multiple layers spaced at predetermined intervals in the height direction (vertical direction). For example, the first partial region (SIDE point cloud data region) in Figure 13 contains four layers. Each layer has multiple measurement points (hereinafter simply referred to as "points") arranged at predetermined intervals in the width direction (horizontal direction). The vertical spacing of the layers and the horizontal spacing of the points are determined by the measurement performance of the LiDAR sensor 55.

[0090] Figure 13 shows a method for detecting the rightmost position using the first partial point cloud data. In Figure 13, the first partial point cloud data is shown as "SIDE point cloud data". The control device 60 arranges the point cloud data of each layer within the first partial region from left to right and calculates the distance to the left and right points of the same layer for each point in that layer. In each layer, the control device 60 extracts multiple points ranging from points where the distance to the rightmost point is significantly longer than the distance to the leftmost point (for example, points where the difference in distance to the left and right adjacent points is greater than or equal to a threshold) to points where the distance to the rightmost point is significantly shorter than the distance to the leftmost point (for example, points where the difference in distance to the left and right adjacent points is less than a threshold). The control device 60 then detects the point one position to the right of these multiple points as a candidate for the rightmost position in that layer. The average position of the two rightmost points among the detected candidate rightmost points is estimated as the rightmost position in that layer. The rightmost position of each layer may be averaged and used as the rightmost position of the final cargo loading entrance, or the rightmost position of any one layer may be used as the rightmost position of the final cargo loading entrance.

[0091] In this way, by determining the position of the rightmost point, it is possible to determine the rightmost point in accordance with various situations within the cargo compartment. Typical situations within the cargo compartment include cases where there are no containers (cargo) in the cargo compartment, as shown in "CASE.1" in Figure 13, and cases where containers C1, C2, C3, C4, and C5 are present in the cargo compartment, as shown in "CASE.2" and "CASE.3".

[0092] In "CASE.1" in Figure 13, the first partial point cloud data within the first partial region consists of the distance measured to the aircraft surface B1 to the right of the right edge, and the distance measured to the cargo compartment wall B2 at the back of the cargo compartment, viewed from the cargo entrance, to the left of the right edge.

[0093] In "CASE.2" in Figure 13, the first partial point cloud data within the first partial region consists of the distance measured to the aircraft surface B1 to the right of the right edge, and the distance measured to the side wall of container C2 facing the cargo loading entrance to the left of the right edge. This side wall is parallel to the inner wall B2 at the rear of the cargo compartment, but is located in front of inner wall B2.

[0094] In "CASE.3" in Figure 13, the first partial point cloud data within the first partial region consists of the distance measured to the aircraft surface B1 to the right of the right edge, and the distance measured to the side wall of container C2 and the inner wall B2 at the back of the cargo compartment to the left of the right edge.

[0095] In cases 1 and 2, for example, for point P2, the distance to the adjacent point P1 on its left is approximately the same as the distance to the adjacent point P3 on its right. The same applies to points P3, P4, and P5. In contrast, for point P6, the distance to the adjacent point P7 on its right is significantly longer than the distance to the adjacent point P5 on its left. Also, for point P7, the distance to the adjacent point P8 on its right is significantly shorter than the distance to the adjacent point P6 on its left. Therefore, points P6 and P7 are extracted, and points P7 and P8, which are one point to the right of each of these points, are detected as candidates for the rightmost point. In this case, since two points, P7 and P8, are detected, these points P7 and P8 are determined to be the rightmost positions. The average position of points P7 and P8 is estimated to be the rightmost position in this layer.

[0096] In CASE 3, four points, P12, P13, P17, and P18, are detected as candidates for the rightmost point. Of these four points, the two rightmost points, P17 and P18, are determined to be the rightmost position. The average position of points P17 and P18 is estimated to be the rightmost position in this layer.

[0097] Furthermore, by taking the average of the two rightmost points among the detected points, it can be expected that one of them is the target point, and the actual rightmost position can be determined with high accuracy. Additionally, by taking the average of the rightmost positions of each layer, the rightmost position can be determined with even greater accuracy. If only one point is detected, that single point can be confirmed as the rightmost position.

[0098] Returning to Figure 10, in steps S122 and S123, the angle (θ) of the cargo entrance is estimated. Here, in the xy coordinate system based on LiDAR (see Figure 11), the angle in the direction directly facing the cargo entrance relative to the x direction is estimated as the angle (θ) of the cargo entrance.

[0099] Since an open cargo entrance is an opening, it is difficult to detect the surface of the cargo entrance itself. Therefore, the control device 60 detects some wall inside the cargo compartment measured through the open cargo entrance. The control device 60 calculates the normal of the wall inside the cargo compartment, and if the calculated normal (first normal) is within a predetermined angle (for example, within 30°) from the normal of the aircraft surface near the cargo entrance (second normal), the control device 60 selects the first normal as the angle of the cargo entrance (direction directly facing the cargo entrance) (step S123). Here, the normal is the line obtained by projecting the normal to a surface made up of point cloud data onto the horizontal plane (xy plane in Figure 11). That is, the normal here indicates the direction in the horizontal plane. Therefore, the angle of the cargo entrance is estimated as the angle in the horizontal plane.

[0100] Furthermore, since the aircraft surface is curved, accuracy decreases if the second normal itself is used as the angle of the cargo entrance. The cargo compartment has a relatively large number of planes that are parallel or perpendicular to the longitudinal direction of the aircraft, so it is advantageous to determine the direction directly facing the cargo compartment as the transverse direction of the aircraft, which is perpendicular to the longitudinal direction of the aircraft in the horizontal plane.

[0101] To determine the angle of the cargo loading entrance, the control device 60 acquires the second and third partial point cloud data, which are necessary for estimating the angle of the cargo loading entrance, from all the point cloud data acquired by the LiDAR sensor 55 (step S122). The second partial point cloud data is the point cloud data within the second partial region from all the point cloud data acquired by the LiDAR sensor 55. The third partial point cloud data is the point cloud data within the third partial region from all the point cloud data acquired by the LiDAR sensor 55.

[0102] The second sub-region is the area that includes the entire cargo loading entrance, and the loading entrance range detected in step S101 in Figure 10 is used as is. In the camera image shown in Figure 14, the second sub-region is shown as "in point cloud data region".

[0103] The third sub-region includes the aircraft surface near the cargo loading entrance but does not include the cargo loading entrance itself. It is determined based on the loading entrance range detected in step S101 in Figure 10. In the camera image shown in Figure 14, the third sub-region is indicated as the "out point cloud data region." The third sub-region is set at a predetermined distance (for example, loading entrance width × 0.2) away from the loading entrance range detected in step S101, in the direction of the loading entrance width (for example, to the right). The width of the third sub-region is set to, for example, loading entrance range × 0.4. By making the width of the third sub-region smaller than the width of the loading entrance range, a decrease in the accuracy of the estimated angle can be prevented.

[0104] The control device 60 uses the second partial point cloud data within the second partial region indicating the loading area to detect the walls inside the cargo compartment measured through the open cargo loading area, and uses the third partial point cloud data within the third partial region indicating the aircraft surface to detect the aircraft surface (step S123). The detection of planes such as the walls inside the cargo compartment and the aircraft surface is performed, for example, using an algorithm such as RANSAC (Random Sample Consensus) based on the point cloud data.

[0105] The variations in the interior walls of the cargo compartment detected using the second partial point cloud data include "CASE.1", [CASE.2], and [CASE.3] as shown in Figure 14. In CASE.1, the front of container C2 is detected as an interior wall of the cargo compartment, and the first normal vector N11 is calculated. In CASE.2, the inner wall B2 at the back of the cargo compartment is detected as an interior wall of the cargo compartment, and the first normal vector N12 is calculated. In CASE.3, the side wall of container C4 and the inner wall B2 at the back of the cargo compartment are detected, and the first normal vector N13 of the side wall of container C4 and the first normal vector N14 of the inner wall B2 are calculated. Depending on the orientation of the ground support equipment 10, the side wall of container C4 may also be captured, as in CASE.3. In all of "CASE.1", [CASE.2], and [CASE.3], the second normal vector N2 of the aircraft surface is calculated using the third partial point cloud data.

[0106] In CASE 1, the first normal N11 is almost parallel (within 30°) to the second normal N2, so the first normal N11 is selected as the cargo compartment angle (step S123). In CASE 2, the first normal N12 is almost parallel to the second normal N2, so the first normal N12 is selected as the cargo compartment angle (step S123).

[0107] Regarding CASE 3, depending on the orientation of the ground support equipment 10, the first normal N14 may not be detected, and only the first normal N13 may be detected. To prepare for such cases, the first normal N13 perpendicular to the second normal N2 (for example, a normal at 90° ± 30° from the second normal N2) may also be selected. If the first normal N13 perpendicular to the second normal N2 is selected, the angle minus 90° is estimated as the cargo compartment angle.

[0108] As in CASE 3, if both a normal parallel to the second normal N2 and a normal perpendicular to it are obtained, the normal with the larger number of points in the plane corresponding to the normal can be selected as the cargo compartment angle (step S123).

[0109] As described above, the cargo compartment angle (the direction directly facing the cargo entrance) can be detected as either a first normal N11, N12, N14 perpendicular to a first surface that is roughly parallel to the opening of the cargo entrance, or as a direction parallel to a second surface that is roughly perpendicular to the opening of the cargo entrance (the direction obtained by subtracting 90° from the direction of the first normal N13). The first surface here refers to at least one of the following: the interior wall B2 of the cargo compartment located at the back of the cargo compartment as viewed from the cargo entrance, and the first side wall of the container C2 loaded in the cargo compartment that is parallel to the interior wall B2. The second surface here refers to the side wall of the container C4 loaded in the cargo compartment that is perpendicular to the interior wall B2.

[0110] Returning to Figure 10, the LiDAR-referenced loading dock position (x,y) obtained in step S113 and the LiDAR-referenced loading dock angle (θ) obtained in step S123 are transformed (step S104). Through the coordinate transformation, the LiDAR-referenced position (x,y) and angle (θ) are converted into the vehicle body position (X,Y) and angle (yaw) relative to the cargo loading dock (right end of the cargo loading dock). Here, the angle θ and yaw indicate the attitude of the ground support equipment 10.

[0111] Figure 15 shows the xy coordinates relative to the LiDAR, i.e., the xy coordinates relative to the ground support equipment 10, and the XY coordinates relative to the right end of the cargo entrance. In the coordinate transformation of step S104, the position (x,y) and angle (θ) relative to the LiDAR are transformed into the vehicle position (X,Y) and angle (yaw) in the world coordinate system relative to the cargo entrance (right end of the cargo entrance) according to equations (1) to (4) shown in Figure 15. In the world coordinate system, the X-axis direction is the forward direction of the vehicle, and the Y-axis direction is the left side of the vehicle. The angle (yaw) is the angle of the x-axis as seen from the X-axis.

[0112] Returning to Figure 9, once the position and angle of the ground support equipment 10 relative to the cargo entrance are determined as described above (step S911), the control device 60 generates a path for approaching the cargo entrance by automatic driving (step S912). The control device 60 outputs an instruction to the driving device 80 to move the ground support equipment 10 along the generated path (step S913). The control device 60 controls the driving device 80 so that it reaches the cargo entrance facing the cargo entrance. Therefore, the ground support equipment 10 can reach the cargo entrance in a posture suitable for loading and unloading cargo. Furthermore, by reacquiring sensor data during movement and repeatedly executing steps S911 to S913, the equipment can reach the cargo entrance with high accuracy.

[0113] When the ground support equipment 10 reaches the cargo entrance, the control device 60 detects the height of the cargo entrance (step S914). Here, the height of the bottom of the cargo compartment is detected as the height of the cargo entrance. For example, the fourth camera 54 in the sensor device 50 is used as a sensor for detecting the height of the cargo entrance. The control device 60 controls the height of the lifting unit 21 (and the lifting deck 31 if necessary) according to the detected height of the cargo entrance (step S915). The detection of the cargo entrance height may be performed while the ground support equipment 10 is moving to reach the cargo entrance. Furthermore, workers responsible for loading and unloading cargo may board the ground support equipment 10 after it has approached the cargo entrance. The workers may board before or after the height adjustment of the lifting unit 21.

[0114] In this embodiment, as an example, multiple locations (reference locations) within the cargo compartment whose relative positions are known are used to detect the height of the cargo entrance. The control device 60 uses the positions of the multiple locations whose relative positions are known in images captured by the camera 54, and the relative positions of these locations, to determine the height of the cargo entrance.

[0115] Here, as an example of a reference point, a part 107 located on the bottom surface 106 (see Figure 1) inside the cargo compartment 105, near the lower end of the cargo entrance, is used. This part 107 is, for example, a cargo entrance lock located near the cargo entrance, as shown in the entrance image in Figure 17. Such a lock is, for example, a lock for closing the cargo door or a lock for securing a container. Multiple such locks (generally four or two) are arranged in a straight line in the width direction of the cargo entrance within the cargo compartment near the cargo entrance, and their spacing is known, although it varies depending on the model. The height of the cargo entrance is detected using these.

[0116] More specifically, the control device 60 uses a lock identifier 65 (reference part identifier) ​​to identify locks (reference parts) from an image captured by the camera 54 near the cargo loading entrance. The lock identifier 65 is a machine learning model that has been trained to identify locks (reference parts) in an image when, for example, an image of the area near the cargo loading entrance is input. The lock identifier 65 has been trained to identify locks of various types. The lock identifier 65 determines the position coordinates of each of the multiple locks in the image. Since the relative positional relationship of each of the multiple locks is known, the control device 60 can determine the height of the locks relative to the camera 54 as the height of the cargo loading entrance.

[0117] Figure 17 shows an example of the procedure for detecting the height of the cargo loading entrance (step S914). First, the control device 60 acquires lock data (reference part data) based on the model information acquired in step S31 of Figure 3 (step S171). Preferably, the aforementioned model information table 71 contains lock data (not shown) associated with the model information. The lock data associated with the model information contains data indicating the relative positional relationship between multiple locks, such as the lock interval for each model. Based on the model information, the control device 60 refers to the model information table 71 and acquires the lock data.

[0118] Then, the control device 60 detects the position of the lock from the camera image captured by the lock identifier 65 (step S172: see Figure 16). In the image of the lock identification results in Figure 16, the four white rectangular bounding boxes indicate the positions of the four identified cargo entrance locks.

[0119] The control device 60 calculates the center point coordinates (center points of the rectangular bounding boxes) of each identified cargo entrance lock and generates an approximate straight line for those center point coordinates. As shown in Figure 18, the control device 60 uses the endpoints on the approximate straight line as the cargo entrance end lock positions (u1, V1,)(u2, v2) (step S173). By using the endpoints on the approximate straight line rather than the positions of the identified end locks themselves, the accuracy of the lock positions can be improved.

[0120] The control device 60 calculates the lock height from the lock positions in the camera image coordinate system, based on the lock positions at both ends (u1, V1,)(u2, v2), by utilizing the fact that the distance between the locks at both ends (lock interval) is known from the lock data. Figure 18 shows the relationship between camera image coordinates and camera coordinates. In equations (5) and (6) in Figure 18, f, Cx, and Cy are fixed values ​​due to the camera characteristics. The lock positions at both ends (u1, V1,)(u2, v2), which are values ​​in the camera image coordinate system (u,v), can be expressed as the lock positions in the camera coordinate system (x,y,z) by using equations (5) and (6). The lock positions x,y,z can be expressed using only z.

[0121] Figure 19 shows the relationship between camera coordinates and HL coordinates (XY coordinates in Figure 15). In equation (7) in Figure 19, R and t are determined by the camera characteristics and installation position and are known. The lock position X, Y, Z in the HL coordinate system can be expressed using only z. The control device 60 uses the relationship in equations (5), (6), and (7) to express the lock position P1 (X1, Y1, Z1) and lock position P2 (X2, Y2, Z2) in Figure 19 using (u1, v1), (u2, v2), and z.

[0122] Here, since the ground support equipment 10 is directly facing the cargo loading entrance, X1=X2, and the distance L between locks is obvious, so Y1-Y2=L. The control device 60 uses these conditions to determine the lock height Z l ,Calculate Z2 (step S174). Lock height Z l ,Z2 is the lock height as seen from the ground support equipment 10 (high-lift loader: HL), therefore the control device 60 is Z l The sign of Z2 is reversed in the calculation, and the height of the ground support equipment 10 (high-lift loader: HL) from the center of the lock (including the inclination of HL) is estimated (step S175).

[0123] Returning to Figure 9, if it is determined that the cargo door is closed in step S91 of Figure 9, the second control (door closed state control) is executed.

[0124] In the second control, in step S921, a second logic, different from the first logic shown in Figure 10, is used to recognize the position and angle of the ground support equipment 10 relative to the cargo loading entrance. Figure 20 shows the recognition process by the second logic (step S921). The second logic also uses images acquired by cameras 51, 52, and 53 (second sensors) and point cloud data acquired by the LiDAR sensor 55 (first sensor).

[0125] The loading dock identifier in the control device 60 detects a closed cargo loading dock from the images acquired by cameras 51, 52, and 53 (step S201). This results in a bounding box as a rectangular area (loading dock area) that includes the cargo loading dock (see Figure 21). The detection of this cargo loading dock is the same as the first logic in the first control.

[0126] The control device 60 extracts partial point cloud data (fourth partial point cloud data; the fourth partial point cloud data is the same as the second partial point cloud data described above) within the closed loading door range (step S202), and calculates the position and angle of the cargo loading door based on the fourth partial point cloud data (steps S213, S223). When the cargo door is closed, the fourth partial point cloud data measures the distance to the cargo loading door, so the centroid position of the loading door range is determined as the position of the loading door (step S213). In addition, the plane of the closed cargo door is detected using the fourth partial point cloud data, and the normal (direction in the horizontal plane) of that plane is determined as the angle of the cargo loading door (step S223).

[0127] The control device 60 performs a coordinate transformation on the entrance position determined in step S213 and the entrance angle determined in step S223 (step S204). The method of coordinate transformation in step S204 is the same as the coordinate transformation in step 104 in Figure 10. It is preferable that the entrance position in the second control includes height.

[0128] Returning to Figure 9, once the position and angle of the ground support equipment 10 relative to the cargo entrance are determined (step S921), the control device 60 generates a path for automatically approaching the cargo entrance (step S922). The control device 60 outputs an instruction to the travel device 80 to move the ground support equipment 10 along the generated path (step S923). The control device 60 controls the travel device 80 so that it reaches the cargo entrance while generally facing the cargo entrance. By reacquiring sensor data during movement and repeatedly executing steps S921 to S923, the cargo entrance can be reached with high accuracy.

[0129] Furthermore, the control device 60 controls the height of the lifting unit 21 based on the entrance position, including its height, and adjusts the lifting unit 21 to roughly the height of the entrance position. The operator gets into the driver's cab 22 of the ground support equipment 10 and operates the control panel located on the machine near the cargo entrance to open the cargo door 103.

[0130] The position and angle determined in step S921 are less precise than those determined in step S911. However, in the second control, the ground support equipment 10 only needs to reach a position where the worker who has boarded the ground support equipment 10 can operate the cargo door control panel. Therefore, in the second control, the precision of the position to which the ground support equipment 10 should reach can be lower than in the first control. Consequently, in the second control, the detected position and angle of the cargo entrance can be less precise.

[0131] In the second control, the worker who boarded the ground support equipment 10, which had reached the cargo loading entrance, gives a movement instruction (reverse instruction) to the control device 60 once the cargo door opening operation is complete. The control device 60 then performs position control to cause the ground support equipment 10 to automatically move backward along the route it took from the waypoint to the cargo loading entrance (step S925). As a result, the ground support equipment 10 returns to the waypoint. Once the ground support equipment 10 returns to the waypoint, the control device 60 executes the first control (steps S911 to 915). In this way, even if the cargo door is closed, by opening the cargo door via the second control, the first control for when the cargo door is open can be executed.

[0132] <3. Addendum> The present invention is not limited to the above embodiments, and various modifications are possible. [Explanation of Symbols]

[0133] 10 Ground support equipment 10A Ground support equipment 10B Ground support equipment 10C Ground support equipment 10C-1 Ground support equipment 10D Ground support equipment 10D-1 Ground support equipment 10E Ground Support Equipment 10E-1 Ground support equipment 10F Ground support equipment 10F-1 Ground Support Equipment 15. First Run 16. Run 2 20 front 21 Elevating platform 22 Driver's cab 30 Rear 31. Elevating deck 32 work platforms 50 Sensor device 51. Camera 1 52. Second camera 53 Third Camera 54. Camera No. 4 55 LiDAR sensors 60 Control device 61 Initial Driving Control 62 Approach and height adjustment control 62A Door open state control (first control) 62B Door closed state control (second control) 65 Lock Identifier 70 Databases 80 Traveling device 91 Induction wire detector 92 Aircraft registration reading camera 100 aircraft 101 Aircraft (fuselage) 102 Cargo loading dock 102A Front cargo entrance 102B Rear cargo entrance 103 Cargo Door 105 Cargo hold 106 Cargo compartment floor 107 Rock 110 Main Wing 200 guide wires 201 1st guide line 202 2nd guide line 203 Third guide line 204 4th guide line 205 5th guide line 301 Equipment Restriction Area 302 Equipment Standby Area C Container D Detection range

Claims

1. A control device for controlling the approach of ground support equipment for an aircraft to the aircraft, Based on point cloud data obtained by the first sensor mounted on the ground support equipment, the position of the end of the entrance formed in the aircraft's fuselage is estimated when the door for closing the entrance is open, and based on the point cloud data, the lateral direction of the aircraft, which is perpendicular to the longitudinal direction of the aircraft in the horizontal plane, is estimated. The system is configured to determine the position of the ground support equipment and its orientation in the horizontal plane relative to the end using the estimated position of the end and the lateral direction of the aircraft, and to control the movement of the ground support equipment so that it approaches the entrance. Control device.

2. The loading entrance is a loading entrance for a cargo compartment inside the aircraft, The lateral direction of the aircraft is estimated based on the surface of the cargo compartment. The control device according to claim 1.

3. The surface inside the cargo compartment is The interior wall of the cargo compartment, located at the back of the cargo compartment as viewed from the aforementioned entrance, Among the side walls of a container loaded in the cargo compartment, the side wall parallel to the interior wall of the cargo compartment, or, Of the side walls of a container loaded in the cargo compartment, the surface perpendicular to the interior wall of the cargo compartment, That is, The control device according to claim 2.

4. The lateral direction of the aircraft is estimated based on the surface of the aircraft around the entrance, The control device according to claim 1.

5. The loading entrance is a loading entrance for a cargo compartment inside the aircraft, The lateral direction of the aircraft is estimated based on the surfaces of the cargo compartment and the aircraft surface around the loading entrance. The control device according to claim 1.

6. The position of the end is estimated based on the partial point cloud data within the partial region that includes the end, The aforementioned partial region includes one end of the widthwise ends of the entrance, but does not include the other end. The control device according to claim 1.

7. The position of the end is estimated based on the distance difference between adjacent points in the left-right direction in the partial point cloud data and points that is less than a threshold. The control device according to claim 6.

8. The position of the end is estimated based on the point in the point cloud data where the distance difference between adjacent points in the left-right direction is less than a threshold, and which is the point on the furthest side. The control device according to claim 6.

9. Ground support equipment comprising the control device described in any one of Claims 1 to 8.

10. A method for controlling the movement of ground support equipment for an aircraft towards the aircraft, Based on point cloud data obtained by the first sensor mounted on the ground support equipment, the position of the end of the entrance formed in the aircraft's fuselage is estimated when the door for closing the entrance is open, and based on the point cloud data, the lateral direction of the aircraft, which is perpendicular to the longitudinal direction of the aircraft in the horizontal plane, is estimated. Using the estimated position of the end and the lateral direction of the aircraft, the position and orientation of the ground support equipment in the horizontal plane are determined relative to the end, and the movement of the ground support equipment is controlled to move closer to the entrance. A control method that includes the following.

Images