Remote controller

Abstract

To provide a technology capable of shortening the processing time required to detect the position of a moving body through matching of a model.SOLUTION: A remote controller includes: a point group data acquisition unit that acquires three-dimensional point group data measured using a range-finding device; a position estimation unit that estimates at least one of the position and orientation of a moving body in the three-dimensional point group data by matching a template point group which represents the moving body to the three-dimensional point group data; an initiation position determination unit that determines an initiation position where matching of the template point group to the three-dimensional point group data is initiated by the position estimation unit; and a remote control unit that uses at least one of the estimated position and orientation of the moving body to generate a control command with which the moving body is remotely controlled and transmit the control command to the moving body.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to a remote control device. [Background technology]

[0002] There are known techniques for detecting the position of an object in space. For example, Patent Document 1 discloses a technique in which a two-dimensional projection image showing the side shape of a vehicle is generated from an input image acquired by a camera, and a two-dimensional model stored in a memory is positioned at the leading position of the generated projection image, and then sequentially scanned, and matching is performed for each scanning position to recognize the position of the vehicle. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 11-259658 Summary of the Invention [Problem to be solved by the invention]

[0004] However, in conventional techniques, the 2D model stored in memory is positioned at the beginning of the 2D projected image and then scanned sequentially, which results in a problem that it takes a long time to complete matching. Therefore, in techniques for detecting the position of a moving object by matching a model stored in memory, a technique is desired that shortens the time until the position of the moving object is detected. [Means for solving the problem]

[0005] The present disclosure can be realized in the following forms.

[0006] (1) According to one aspect of the present disclosure, there is provided a remote control device comprising: a point cloud data acquisition unit that acquires three-dimensional point cloud data measured using a distance measuring device; a position estimation unit that matches a template point cloud representing a moving object to the three-dimensional point cloud data to estimate at least one of a position and an orientation of the moving object in the three-dimensional point cloud data; a start position determination unit that determines a start position at which the position estimation unit starts matching the template point cloud to the three-dimensional point cloud data; and a remote control unit that uses the estimated at least one of the position and orientation of the moving object to generate a control command for remotely controlling the moving object and transmit the control command to the moving object. According to this form of remote control device, by determining the starting position for matching to be near the moving body, the position of the moving body can be detected early, and the processing time required to detect the position of the moving body by template matching can be shortened. (2) In the remote control device of the above aspect, the start position determination unit may determine the start position using information about the position of the moving object in the three-dimensional point cloud data. According to this form of remote control device, by using information about the position of the moving body to determine the starting position for template matching, the position of the moving body can be detected early, thereby shortening the processing time required for template matching. (3) In the remote control device of the above form, the start position determination unit may acquire the previous matching position where matching between the three-dimensional point cloud data and the template point cloud was completed last time as information regarding the position of the moving body, and may determine the start position using the acquired previous matching position. According to this form of remote control device, by using the previous matching position which is likely to be near the position of the moving body at the time of current matching, the processing time required for template matching can be shortened through simple processing. (4) In the remote control device of the above form, the start position determination unit may further estimate the position of the moving object after it has moved from the previous matching position between the time when the previous matching was completed and the time when the current matching is performed, and may determine the start position using the estimated position after it has moved. According to this form of remote control device, by estimating the position of the moving body after movement, the starting position of matching can be set to a position closer to the position of the moving body at the time of current matching, thereby further shortening the processing time required for template matching. (5) In the remote control device of the above form, a memory device may be further provided for storing a target route of the moving body, which is information regarding the position of the moving body, and the start position determination unit may determine any position on the target route as the start position. According to this form of remote control device, by setting a position on the target route where a moving body is likely to be located as the starting position, the position of the moving body can be detected early, thereby shortening the processing time required for template matching. (6) In the remote control device of the above form, the start position determination unit may acquire a previous matching position where matching between the 3D point cloud data and the template point cloud was completed on the target route the previous time, and may determine the start position using the acquired previous matching position on the target route. According to this type of remote control device, the position of the moving object can be detected earlier by using the previous matching position on the target route where the moving object is more likely to exist. (7) In the remote control device of the above form, the start position determination unit may determine the start position using either the position of the moving body detected from the three-dimensional point cloud data by a predetermined object detection method or the position of the moving body detected by the difference between point cloud data corresponding to the background of the three-dimensional point cloud data and point cloud data corresponding to the moving body as information regarding the position of the moving body. According to this type of remote control device, the provisional position of a moving body can be extracted using object detection, and the position of the moving body can be detected early using a simple method, thereby shortening the processing time required for template matching. (8) In the remote control device of the above form, when multiple positions of the moving body are detected, the start position determination unit may further use moving body identification information for identifying the moving body, and determine the start position using the position of the moving body that corresponds to the moving body identification information among the multiple detected positions of the moving body. According to this type of remote control device, even if the tentative positions of multiple moving bodies are detected by object detection, the position of the moving body to be controlled can be extracted by using the moving body identification information. The present disclosure can also be realized in various forms other than a remote control device, such as a remote control system, a method for moving a moving object, a method for manufacturing a moving object, a method for detecting a moving object, a moving object detection device, a method for controlling a remote control system, a method for controlling a remote control device, a computer program for implementing these control methods, a non-transitory recording medium on which the computer program is recorded, etc. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 1 is an explanatory diagram showing a schematic configuration of a remote control system. [Figure 2] FIG. 2 is an explanatory diagram illustrating a vehicle driving method. [Figure 3] 4 is a flowchart showing a processing routine for remotely controlled vehicle driving processing. [Figure 4] 6 is a flowchart showing a processing routine of start position determination processing according to the first embodiment. [Figure 5] FIG. 10 is an explanatory diagram showing an estimated position of a vehicle after traveling from a previous matching position. [Figure 6] FIG. 10 is a block diagram showing the internal functionality of a remote control device according to a second embodiment. [Figure 7] 10 is a flowchart showing a start position determination process according to the second embodiment. [Figure 8] FIG. 10 is an explanatory diagram showing template matching in the start position determination process according to the second embodiment. [Figure 9] FIG. 11 is an explanatory diagram showing an example of the overall configuration of a remote control device according to a third embodiment, as viewed from above. [Figure 10] FIG. 10 is a block diagram showing the internal functional configuration of a remote control device according to a third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0008] A. First embodiment: 1 is an explanatory diagram showing a schematic configuration of a remote control system 500 including a remote control device 300 according to a first embodiment of the present disclosure. The remote control device 300 generates a control command for causing a vehicle 100 to travel automatically by remote control and transmits the control command to the vehicle 100. The remote control system 500 is used, for example, in a factory that manufactures vehicles 100 that can travel by remote control.

[0009] The vehicle 100 may be, for example, a passenger car, a truck, a bus, or a construction vehicle. The vehicle 100 is preferably an electric vehicle (BEV: Battery Electric Vehicle). The vehicle 100 is not limited to an electric vehicle, and may be, for example, a gasoline-powered vehicle, a hybrid vehicle, or a fuel cell vehicle. The vehicle 100 includes a vehicle communication device 190, an actuator 140, and an ECU (Electronic Control Unit) 200.

[0010] The ECU 200 is mounted on the vehicle 100 and executes various controls of the vehicle 100. The ECU 200 includes a storage device 220 such as an HDD (hard disk drive), an SSD (solid state drive), an optical recording medium, or a semiconductor memory, a CPU 210 as a central processing unit, and an interface circuit 230. The CPU 210, the storage device 220, and the interface circuit 230 are connected via an internal bus to enable bidirectional communication. The interface circuit 230 is connected to an actuator 140 and a vehicle communication device 190. The vehicle communication device 190 wirelessly communicates with devices external to the vehicle 100 that are connected to a network, such as a remote control device 300, via an access point in a factory or the like.

[0011] Computer programs for realizing at least some of the functions provided in this embodiment are stored in a readable / writable area of ​​the storage device 220. The CPU 210 executes various computer programs stored in the memory to realize functions such as the operation control unit 212.

[0012] The driving control unit 212 executes driving control of the vehicle 100. "Driving control" refers to various controls for driving the actuators 140 that perform the functions of "running," "turning," and "stopping" of the vehicle 100, such as adjusting acceleration, speed, and steering angle. In this embodiment, the actuators 140 include an actuator for a driving device for accelerating the vehicle 100, an actuator for a steering device for changing the direction of travel of the vehicle 100, and an actuator for a braking device for decelerating the vehicle 100. The driving device includes a battery, a traction motor driven by battery power, and drive wheels rotated by the traction motor. The actuator for the driving device includes the traction motor. The actuators 140 may further include an actuator for swinging the wipers of the vehicle 100, an actuator for opening and closing the power windows of the vehicle 100, etc.

[0013] When a driver is on board the vehicle 100, the driving control unit 212 controls the actuator 140 in accordance with the driver's operation, thereby causing the vehicle 100 to travel. The driving control unit 212 can also control the actuator 140 in accordance with a control command transmitted from the remote control device 300, thereby causing the vehicle 100 to travel, regardless of whether a driver is on board the vehicle 100 or not.

[0014] The remote control system 500 includes the remote control device 300 and one or more vehicle detectors 80. The vehicle detector 80 is a device for measuring various data used to estimate at least one of the position and orientation of the vehicle 100. The vehicle detector 80 uses a LiDAR (Light Detection and Ranging) ranging device that measures three-dimensional point cloud data of the vehicle 100. The three-dimensional point cloud data is data that indicates the three-dimensional positions of a point cloud. By using the LiDAR, it is possible to acquire highly accurate three-dimensional point cloud data. Note that the orientation and traveling direction of the vehicle 100 may be estimated by acquiring only the position of the vehicle 100 using the vehicle detector 80 and acquiring changes in the vehicle 100 over time, etc.

[0015] The vehicle detector 80 is communicatively connected to the remote control device 300 via wireless or wired communication. The remote control device 300 can acquire the relative position and orientation of the vehicle 100 with respect to the target route RT in real time by acquiring 3D point cloud data from the vehicle detector 80. In this embodiment, the position of the vehicle detector 80 is fixed near the travel path SR, and the relative relationship between the reference coordinate system Σr and the device coordinate system of the vehicle detector 80 is known. A coordinate transformation matrix for mutually converting the coordinate values ​​of the reference coordinate system Σr and the coordinate values ​​of the vehicle detector 80 and the device coordinate system is stored in advance in the remote control device 300.

[0016] The remote control device 300 performs driving control of the vehicle 100 by remote control. For example, the remote control device 300 transports the vehicle 100 in a manufacturing process within a factory by remotely controlling the vehicle 100 to automatically travel. Transporting the vehicle 100 using automatic travel by remote control is also called "self-propelled transport." Self-propelled transport allows the vehicle 100 to move without using transport devices such as a crane or conveyor. The remote control device 300 may also monitor the surroundings of the vehicle 100.

[0017] The remote control device 300 includes a CPU 310 as a central processing unit, a storage device 340, an interface circuit 350, and a remote communication device 390. The CPU 310, the storage device 340, and the interface circuit 350 are connected via an internal bus to enable bidirectional communication. The interface circuit 350 is connected to the remote communication device 390. The remote communication device 390 communicates with the vehicle 100 via a network or the like.

[0018] The storage device 340 is, for example, a RAM, a ROM, a HDD, or an SSD. Vehicle point cloud data VP, a target route RT, an actuator drive history AC, and a previous matching position BM are stored in a readable / writable area of ​​the storage device 340. The target route RT is a predetermined travel path of the vehicle 100 on the travel path SR.

[0019] The vehicle point cloud data VP functions as a template point cloud for estimating at least one of the position and orientation of the vehicle 100. For example, three-dimensional CAD data of the vehicle 100 can be used as the vehicle point cloud data VP. The vehicle point cloud data VP may include information for identifying the orientation of the vehicle 100. Template matching using the vehicle point cloud data VP makes it possible to estimate the position and orientation of the vehicle 100 in the three-dimensional point cloud data with high accuracy.

[0020] The actuator drive history AC is a history of input / output values ​​of each actuator 140 of the vehicle 100. The actuator drive history AC may be, for example, a history of control command values ​​transmitted from the remote control device 300 to the vehicle 100, or may be actual measured values ​​obtained from sensors on the vehicle 100, such as vehicle speed, steering angle, braking force, and rotation angle. The previous matching position BM is the coordinate value of the position where template matching between the three-dimensional point cloud data and the vehicle point cloud data VP previously performed by the position estimation unit 318 was completed, as will be described later.

[0021] The storage device 340 stores a computer program for realizing at least some of the functions provided in this embodiment. When the computer program stored in the storage device 340 is executed by the CPU 310, the CPU 310 functions as a remote control unit 312, a point cloud data acquisition unit 314, a start position determination unit 316, and a position estimation unit 318. However, some or all of these functions may be configured by hardware circuits. The point cloud data acquisition unit 314 acquires three-dimensional point cloud data measured by the vehicle detector 80.

[0022] The start position determination unit 316 determines a start position for starting matching of the vehicle point cloud data VP with the acquired 3D point cloud data. From the viewpoint of completing template matching as quickly as possible, the start position for starting matching is preferably the position of the vehicle 100 to be detected in the 3D point cloud data or a position nearby.

[0023] In this embodiment, the start position determination unit 316 determines the start position of template matching using information about the position of the vehicle 100 in the three-dimensional point cloud data. "Information about the position of the vehicle 100 in the three-dimensional point cloud data" is data used to estimate the position of the vehicle 100 or a position near the vehicle 100 in the three-dimensional point cloud data. "Information about the position of the vehicle 100 in the three-dimensional point cloud data" is acquired by a method different from template matching and is used in processing before template matching is performed. In order to increase the processing speed of template matching, it is preferable that the "information about the position of the vehicle 100 in the three-dimensional point cloud data" be small-volume data or data obtained by simple processing.

[0024] The position estimation unit 318 estimates the position and orientation of the vehicle 100 in the acquired three-dimensional point cloud data. In this embodiment, the position estimation unit 318 estimates the position and orientation of the vehicle 100 in the three-dimensional point cloud data by performing template matching on the three-dimensional point cloud data using the vehicle point cloud data VP. Template matching of the vehicle point cloud data VP with the three-dimensional point cloud data can use, for example, an iterative closest point (ICP) algorithm or a normal distribution transform (NDT) algorithm.

[0025] The remote control unit 312 uses the estimated position and orientation of the vehicle 100 to generate a control command for remote control and transmit it to the vehicle 100. This control command is a command to cause the vehicle 100 to travel along the target route RT stored in the storage device 340. The control command can be generated as a command including a driving force or braking force and a steering angle. Alternatively, the control command may be generated as a command including at least one of the position and orientation of the vehicle 100 and a future travel route. When the vehicle 100 receives a request for remote control, driving control is implemented by the driving control unit 212 of the ECU 200, and as a result, the vehicle 100 travels automatically.

[0026] FIG. 2 is an explanatory diagram that schematically shows a method for driving the vehicle 100. FIG. 2 shows how the vehicle 100 is self-propelled and transported in a factory FC where the vehicle 100 is manufactured. The factory FC has a first process 50 and a second process 60. The first process 50 is, for example, a location where the vehicle 100 is assembled, and the second process 60 is, for example, a location where the vehicle 100 is inspected. The first process 50 and the second process 60 are connected by a travel path SR along which the vehicle 100 can travel. Any position within the factory FC is expressed by xyz coordinate values ​​in a reference coordinate system Σr.

[0027] The remote control device 300 causes the vehicle 100 to travel along a target route RT while estimating, at predetermined time intervals, the position and orientation of the vehicle 100 in the 3D point cloud data acquired by the vehicle detector 80. In the example of Fig. 3, the target route RT set on the travel path SR includes consecutive target routes RT1, RT2, and RT3. A plurality of vehicle detectors 80 are installed around the travel path SR, including vehicle detectors 80A, 80B, and 80C corresponding to the target routes RT1, RT2, and RT3, respectively.

[0028] The remote control device 300 switches the vehicle detector 80 appropriately depending on the position of the vehicle 100 to estimate the position and orientation of the vehicle 100 and causes the vehicle 100 to travel along the target route RT. In the example of FIG. 3, the vehicle 100 that departs from the first step 50 travels along the target route RT1 by remote control using 3D point cloud data acquired by the vehicle detector 80A. When the vehicle 100 reaches the target route RT2, the vehicle detector 80A is switched to the vehicle detector 80B that is suitable for detecting the vehicle 100 traveling along the target route RT2. When the vehicle 100 reaches the target route RT3, the vehicle detector 80B is similarly switched to the vehicle detector 80C that is suitable for detecting the vehicle 100.

[0029] 3 is a flowchart showing a processing routine for the travel processing of the vehicle 100 by remote control according to this embodiment. This flow starts when the vehicle 100 starts traveling on the travel route SR. This flow can be repeatedly executed each time the point cloud data acquisition unit 314 acquires new 3D point cloud data from the vehicle detector 80 that is responsible for measuring the vehicle 100 to be controlled.

[0030] In step S100, the point cloud data acquisition unit 314 acquires three-dimensional point cloud data from the vehicle detector 80. In step S200, the start position determination unit 316 executes a start position determination process. The "start position determination process" is a process for determining a start position at which to start matching the vehicle point cloud data VP with the acquired three-dimensional point cloud data.

[0031] In step S300, template matching is started from the start position determined by the start position determination process. If matching with the vehicle point cloud data VP cannot be completed at the start position, matching is performed sequentially on the surrounding area starting from the start position. Template matching is repeatedly performed while changing the matching position until completion. In step S400, the position estimation unit 318 estimates the position and orientation of the vehicle 100 in the 3D point cloud data by performing template matching using the vehicle point cloud data VP. In step S500, the remote control unit 312 uses the estimated position and orientation of the vehicle 100 to generate a control command to cause the vehicle 100 to travel along the target route RT and transmits the control command to the vehicle 100.

[0032] 4 is a flowchart showing an example of a processing routine for the start position determination process according to the first embodiment. In this embodiment, depending on whether or not there is a previous matching result, either the previous matching position BM where the previous matching between the three-dimensional point cloud data and the vehicle point cloud data VP was completed, or the target route RT of the vehicle 100 is selectively used as the "information relating to the position of the vehicle 100 in the three-dimensional point cloud data."

[0033] In step S210, start position determination unit 316 checks whether or not there is a matching result from the previous template matching. Whether or not there is a previous matching result can be checked, for example, by checking whether or not the previous matching result is stored in the processing history stored in storage device 340, or by checking whether or not the previous matching position BM is stored in storage device 340.

[0034] If there is no previous matching result (S210: NO), the start position determination unit 316 proceeds to step S250. In this case, the target route RT of the vehicle 100 is used as "information related to the position of the vehicle 100 in the 3D point cloud data." In step S250, the start position determination unit 316 acquires the target route RT stored in the storage device 340.

[0035] In step S260, the travel start position of the vehicle 100 on the target route RT is acquired. The travel start position of the vehicle 100 on the target route RT is, for example, associated with the target route RT and stored in advance in the storage device 340. In the example of FIG. 2, the travel start position of the vehicle 100 is near the exit of the first process 50.

[0036] In step S270, the start position determination unit 316 determines the start position of template matching to be the travel start position of the acquired target route RT, and ends this flow. If there is no previous matching result, it is highly likely that the vehicle 100 has not started traveling and is currently at the travel start position. Therefore, if there is no previous matching result, the processing time for template matching can be reduced by setting the travel start position of the vehicle 100 on the target route RT as the start position.

[0037] In step S210, if there is a previous matching result (S210: YES), the start position determination unit 316 proceeds to step S220. In this case, the previous matching position BM at which the previous matching between the three-dimensional point cloud data and the vehicle point cloud data VP was completed is used as "information related to the position of the vehicle 100 in the three-dimensional point cloud data." Furthermore, in this embodiment, the previous matching position BM is used to estimate the position of the vehicle 100 after traveling at the time of performing the current matching, and this position after traveling is used as "information related to the position of the vehicle 100 in the three-dimensional point cloud data."

[0038] In step S220, the start position determination unit 316 acquires the actuator drive history AC of the vehicle 100. The acquired actuator drive history AC includes at least the drive history of the actuator 140 related to the travel of the vehicle 100 from the time when the previous matching was completed until the current matching is performed. In step S230, the start position determination unit 316 estimates the position of the vehicle 100 after travel, which has traveled between the time when the previous matching was completed and the time when the current matching is performed. Specifically, the start position determination unit 316 estimates the position of the vehicle 100 after travel by estimating the travel route from the previous matching position BM using the drive history of the actuator 140. In step S240, the start position determination unit 316 determines the estimated position of the vehicle 100 after travel as the start position of template matching, and ends this flow.

[0039] Fig. 5 is an explanatory diagram showing the estimated position of the vehicle 100 after traveling from the previous matching position. Fig. 5 shows an example of three-dimensional point cloud data PD1 acquired from the vehicle detector 80. The three-dimensional point cloud data PD1 schematically shows the position BM1 of the vehicle 100 at the time of the previous matching, which is an example of the previous matching position BM, and the position AM of the vehicle 100 after traveling along the traveling route AR from the previous matching position.

[0040] The position AM of the vehicle 100 after traveling can be obtained, for example, by estimating the traveling route AR or the position AM of the vehicle 100 obtained from the actuator drive history AC, with the position BM1 at the time of the previous matching as the starting point. In another embodiment, the start position determination unit 316 may estimate the traveling route AR or the position AM of the vehicle 100 from output values ​​of a detector that acquires the vehicle speed, steering angle, braking force, rotation angle, etc., instead of the actuator drive history AC. Alternatively, the current position of the vehicle 100 acquired from a GNSS (Global Navigation Satellite System) receiver or the like may be used as the position AM of the vehicle 100 after traveling.

[0041] The start position determination unit 316 determines the estimated position AM of the vehicle 100 after traveling as the start position of template matching, and the position estimation unit 318 starts template matching using the vehicle point cloud data VP from position AM. By using the position AM of the vehicle 100 after traveling as the start position of template matching, the actual position of the vehicle 100 or its vicinity can be set as the start position. This reduces the processing time required for template matching. Furthermore, compared to when template matching is started at a position far from position AM, this reduces the problem of matching completing at a position different from the actual position of the vehicle 100 due to falling into a local solution using a gradient descent method or the like, thereby improving the detection accuracy of the vehicle 100.

[0042] As described above, the remote control device 300 of this embodiment includes a point cloud data acquisition unit 314 that acquires three-dimensional point cloud data measured using the vehicle detector 80, a position estimation unit 318 that estimates the position and orientation of the vehicle 100 in the three-dimensional point cloud data by matching the vehicle point cloud data VP to the three-dimensional point cloud data, a start position determination unit 316 that determines a start position at which template matching for the three-dimensional point cloud data is started by the position estimation unit 318, and a remote control unit 312 that generates a control command for remotely controlling the vehicle 100 using the estimated position and orientation of the vehicle 100 and transmits it to the vehicle 100. According to the remote control device 300 of this embodiment, by setting the matching start position near the vehicle 100, the position of the vehicle 100 can be detected early and the processing time required for template matching can be shortened.

[0043] According to the remote control device 300 of this embodiment, the start position determination unit 316 determines the start position using information about the position of the vehicle 100 in the three-dimensional point cloud data acquired by the point cloud data acquisition unit 314. By setting the start position of template matching to the position of the vehicle 100 using the information about the position of the vehicle 100, the position of the vehicle 100 can be detected early and the processing time required for template matching can be shortened.

[0044] According to the remote control device 300 of this embodiment, the start position determination unit 316 acquires the previous matching position BM and determines the start position for template matching using the acquired previous matching position BM. By using the previous matching position BM, which is likely to be near the position of the vehicle 100 at the time of current matching, the processing time required for template matching can be shortened by simple processing.

[0045] According to the remote control device 300 of this embodiment, the start position determination unit 316 further estimates the post-travel position AM of the vehicle 100 that has traveled from the previous matching position BM between the time the previous matching was completed and the time the current matching is performed, and determines the estimated post-travel position AM as the start position of template matching. By estimating the position of the vehicle 100 after traveling, the start position of matching can be set to a position closer to the position of the vehicle 100 at the time of the current matching, and the processing time required for template matching can be further reduced.

[0046] According to the remote control device 300 of this embodiment, if there is no previous matching result, the start position determination unit 316 determines the travel start position of the target route RT as the start position of template matching. By setting the start position to be on the target route RT where the vehicle 100 is likely to be located, the position of the vehicle 100 can be detected early and the processing time required for template matching can be shortened. In particular, if there is no previous matching result, the processing time required for template matching can be further shortened by setting the start position to be a travel start position where the vehicle 100 is likely to be located.

[0047] B. Second embodiment: 6 is a block diagram showing the internal functionality of a remote control device 300b according to the second embodiment. The remote control device 300b of this embodiment differs from the remote control device 300 of the first embodiment in that, instead of the CPU 310, it further includes a CPU 310b that functions as an object detection unit 322, and, instead of the storage device 340, it further includes a storage device 340b that stores vehicle identification information VI and a learned model LM. Otherwise, the configuration is the same.

[0048] The trained model LM is a machine learning model that performs object detection using a predetermined object detection method that utilizes deep learning. The trained model LM has been fully trained using a predetermined dataset. For example, various methods using convolutional neural networks (CNNs) can be used for the trained model LM. Deep learning methods such as Region-Convolutional Neural Networks (R-CNNs), Fast R-CNNs, Faster R-CNNs, versions of YOKO (You Only Lock Once), and Single Shot Detectors (SSDs) are particularly suitable for object detection.

[0049] The object detection unit 322 uses the trained model LM to perform annotation of the 3D point cloud data and detect the tentative position of the vehicle in the 3D point cloud data. In this case, "vehicle" refers to a vehicle as an attribute (class) and may include vehicles other than the vehicle 100 to be controlled. Specifically, the object detection unit 322 inputs the detected 3D point cloud data into the trained model LM to perform object detection and generate a 3D bounding box for the area recognized as a vehicle. The "bounding box" refers to a partial area extracted from the external area of ​​the 3D point cloud data by enclosing the object area of ​​the vehicle with the smallest rectangular parallelepiped. In other words, the object detection unit 322 extracts the area occupied by the vehicle in the 3D point cloud data by forming a bounding box.

[0050] The vehicle identification information VI refers to various information that can identify each vehicle 100. The vehicle identification information VI includes, for example, ID information assigned to each vehicle 100, such as a vehicle identification number (VIN), specification information of the vehicle 100, such as the model, color, shape, and size, and production management information of the vehicle 100, such as the name of the process in progress. The vehicle identification information VI may also include the size of a bounding box that can be detected. The vehicle identification information VI can be acquired, for example, via short-range wireless communication from an RF-ID (Radio Frequency Identification) tag or the like attached to the vehicle 100.

[0051] 7 is a flowchart showing the start position determination process according to the second embodiment. In step S600, the object detection unit 322 performs object detection using the acquired three-dimensional point cloud data and acquires the tentative position of the vehicle 100 in the three-dimensional point cloud data. More specifically, the object detection unit 322 inputs the acquired three-dimensional point cloud data into the trained model LM to generate a bounding box of the vehicle in the three-dimensional point cloud data.

[0052] In step S610, the object detection unit 322 acquires the vehicle identification information VI of the vehicle 100 to be controlled. In step S620, the object detection unit 322 uses the vehicle identification information VI to extract a bounding box corresponding to the vehicle 100 to be controlled. The object detection unit 322 extracts, from the generated bounding box, a bounding box whose size matches the size of the bounding box associated with the vehicle identification information VI of the vehicle 100 to be controlled or the size of the vehicle 100. Instead of the size of the bounding box or the size of the vehicle 100, vehicle identification information VI such as the shape of part or all of the vehicle 100, or the length, width, and height of part or all of the vehicle 100 may be used. In step S630, the start position determination unit 316 determines the position of the extracted bounding box as the start position of template matching, and this flow ends.

[0053] 8 is an explanatory diagram showing template matching in the start position determination process according to the second embodiment. FIG. 8 shows an example of three-dimensional point cloud data PD2 acquired from the vehicle detector 80. The three-dimensional point cloud data PD2 includes three-dimensional point cloud data of the vehicle 100, which is the control target, and three-dimensional point cloud data of the vehicle 100R, which is not the control target. The object detection unit 322 inputs the three-dimensional point cloud data PD2 into the trained model LM, thereby generating bounding boxes BB1 and BB2 for the vehicles 100 and 100R, which are recognized as vehicles.

[0054] Next, the object detection unit 322 acquires vehicle identification information VI. In this case, the vehicle identification information VI is preferably information related to the size and width of the vehicles 100 and 100R, or the size of the bounding box, as information that facilitates distinguishing between the vehicles 100 and 100R, which are of different sizes. From the acquired vehicle identification information VI, the bounding box BB1 of the vehicle 100 is extracted from the bounding boxes BB1 and BB2. The start position determination unit 316 determines the position of the extracted bounding box BB1 as the start position of template matching, and the position estimation unit 318 starts template matching using the vehicle point cloud data VP from the position of the bounding box BB1 as the start position.

[0055] As described above, according to the remote control device 300b of this embodiment, the start position determiner 316 determines, as the start position, the position of the vehicle 100 detected from the three-dimensional point cloud data PD2 by a predetermined object detection method using deep learning. The position of the vehicle 100 can be extracted from the three-dimensional point cloud data PD2 by using a known method of object detection using deep learning. Furthermore, by using the position of the vehicle 100 in the three-dimensional point cloud data PD2, the start position can be set near the vehicle 100, thereby reducing the processing time required for template matching.

[0056] According to the remote control device 300b of this embodiment, when the positions of multiple vehicles 100, 100R are detected, the start position determination unit 316 further uses vehicle identification information VI for identifying the vehicles 100 to determine the start position using the position of the vehicle 100 that corresponds to the vehicle identification information VI among the detected positions of the multiple vehicles 100. Therefore, even when multiple vehicles 100, 100R are detected by an object detection method using deep learning, the position of the vehicle 100 to be controlled can be extracted.

[0057] C. Third embodiment: FIG. 9 is an explanatory diagram showing an example of the overall configuration of a remote control device 300c according to a third embodiment, as viewed from above. The remote control device 300c of this embodiment is an unmanned guided vehicle, also known as guide mobility. The remote control device 300c has a function to automatically drive the vehicle 100 by remote control, as in the first embodiment, and also has a function to automatically drive the main body of the remote control device 300c on the travel path SR. In other words, the remote control device 300c automatically drives the following vehicle 100 by remote control, while guiding it along the travel path SR.

[0058] The remote control device 300c includes self-propelled driving wheels 142, a self-propelled detector 70, and a vehicle detector 80. The function of the vehicle detector 80 is similar to that of the vehicle detector 80 described in the first embodiment, and therefore a description thereof will be omitted. The self-propelled detector 70 is a distance measurement device such as a camera or LiDAR. Data acquired by the self-propelled detector 70 is used for simultaneous self-position estimation and environmental map creation (SLAM: Simultaneous Localization and Mapping) for the remote control device 300c to travel automatically. The camera may be a stereo camera, a monocular camera, an RGB-D camera (depth camera), or another camera. A ToF sensor or the like may be used instead of the camera or LiDAR. Note that, although the vehicle detector 80 is only mounted on the remote control device 300c in the example of FIG. 9, it may also be installed around the travel path SR.

[0059] 10 is a block diagram showing the internal functional configuration of a remote control device 300c according to the third embodiment. The internal functional configuration of the remote control device 300c is similar to that of the remote control device 300 of the first embodiment, except that, instead of the CPU 310, the remote control device 300c further includes a CPU 310c that functions as a SLAM unit 328 and an autonomous driving control unit 330, and instead of the storage device 340, the remote control device 300c further includes a storage device 340c that stores a guide vehicle route GR. The guide vehicle route GR is a predetermined target route along which the remote control device 300c should travel.

[0060] The actuator 360 includes actuators for the drive device, steering device, and braking device for automatic travel of the remote control device 300c. The drive device includes a battery, a travel motor driven by battery power, and drive wheels 142 rotated by the travel motor.

[0061] The SLAM unit 328 executes SLAM using data detected by the self-propelled detector 70 to generate a map for automatic driving of the remote control device 300c. The automatic driving control unit 330 controls the actuator 360 to cause the remote control device 300c to automatically drive. Specifically, the automatic driving control unit 330 uses the map generated by the SLAM unit 328 to cause the remote control device 300c to automatically drive according to the guide vehicle route GR stored in the storage device 340c.

[0062] As shown in Fig. 9, when the remote control device 300c travels along the guide vehicle route GR, the vehicle 100 is guided by the remote control of the remote control device 300c and travels along the same travel route as the remote control device 300c. In this embodiment, as in the first embodiment, template matching is performed on the 3D point cloud data acquired from the vehicle detector 80, and the start position determiner 316 executes a start position determination process to determine the start position of template matching. In this way, in the present embodiment, as in the first embodiment, the processing time required for template matching can be reduced by using information about the position of the vehicle 100 to set the start position of template matching near the vehicle 100.

[0063] D. Other Embodiments: (D1) In the second embodiment, a bounding box for a vehicle is generated using an object detection technique that utilizes deep learning, and the position of the vehicle 100 extracted using the generated bounding box is determined as the start position. Instead of generating a bounding box, the object detection unit 322 may perform preprocessing to remove background point cloud data representing stationary objects from the acquired 3D point cloud data. This preprocessing is also called "background subtraction processing." In this case, the start position determination unit 316 determines the tentative position of the vehicle 100 detected by subtracting point cloud data corresponding to the background of the 3D point cloud data from point cloud data corresponding to the vehicle 100 as the detection target as the start position of template matching. The remote control device 300 configured in this manner can determine the start position using a simple process called background subtraction.

[0064] (D2) In the first embodiment described above, an example was shown in which the start position determination unit 316 set the travel start position of the vehicle 100 on the target route RT as the start position when there was no previous matching result. In contrast, the start position determination unit 316 may set the travel start position of the vehicle 100 on the target route RT as the start position regardless of whether there was a previous matching result. Alternatively, instead of the travel start position of the target route RT, any position on the target route RT may be determined as the start position. In this case, step S260 shown in FIG. 4 can be omitted. According to this embodiment of the remote control device 300, the processing time required for template matching can be reduced by setting the start position to a position on the target route RT where the vehicle 100 is likely to be present. In this case, if the vehicle 100 cannot be detected at the start position, it is preferable to perform template matching along the target route RT.

[0065] Alternatively, the start position determination unit 316 may previously acquire a previous matching position BM on the target route RT, and determine the start position using the acquired previous matching position BM on the target route RT. According to this form of the remote control device 300, by using the previous matching position BM on the target route RT where the vehicle 100 is more likely to be present, it is possible to further reduce the processing time required for template matching in a simple manner.

[0066] (D3) In the above embodiments, examples have been given in which the vehicle 100 is a passenger car, truck, bus, construction vehicle, or the like. However, the vehicle 100 is not limited to these, and may include various automobiles such as two-wheeled vehicles and four-wheeled vehicles, as well as trains. It may also be various moving bodies other than the vehicle 100. A "moving body" refers to an object that can move. Examples of moving bodies include electric vertical take-off and landing aircraft (so-called flying cars), ships, aircraft, robots, linear motor cars, and the like. In this case, the terms "vehicle" and "car" in this disclosure may be appropriately replaced with "moving body," and the term "traveling" may be appropriately replaced with "moving."

[0067] (D4) The vehicle 100 may have a configuration that allows it to move by remote control. For example, it may be in the form of a platform having the configuration described below. Specifically, the vehicle 100 may have a configuration that allows it to perform the functions of "running," "turning," and "stopping" by remote control. In other words, a "vehicle 100 that can move by remote control" may not have at least some of its interior parts, such as a driver's seat or dashboard, at least some of its exterior parts, such as a bumper or fenders, or a body shell. In this case, the remaining parts, such as the body shell, may be attached to the vehicle 100 before it is shipped from the factory, or the remaining parts, such as the body shell, may be attached to the vehicle 100 after it is shipped from the factory without the remaining parts, such as the body shell.

[0068] (D5) In the first embodiment described above, an example was shown in which the start position determination unit 316 further estimates the position AM of the vehicle 100 after traveling from the previous matching position BM. In contrast, the start position may be a position other than the position AM after traveling, and for example, the previous matching position BM may be used. By using the previous matching position BM, which is likely to be near the position of the vehicle 100, the processing time required for template matching can be shortened by simple processing.

[0069] (D6) In the first embodiment described above, an example was shown in which the start position determination unit 316 determined the start position using the previous matching position BM, and an example was shown in which the start position determination unit 316 determined any position on the target route RT as the start position. In contrast, the start position determination unit 316 may determine a position other than the previous matching position BM or a position different from the target route RT as the start position. For example, the start position determination unit 316 may determine, as the start position, a tentative position of the vehicle 100 acquired from detection results by a detector other than the vehicle detector 80.

[0070] (D7) In the above third embodiment, an example was shown in which the object detection unit 322 extracts the vehicle 100 to be controlled using the vehicle identification information VI. In contrast, for example, if a single bounding box is generated in step S600, that is, if a bounding box for only the vehicle 100 to be controlled is generated, the vehicle identification information VI may not be used. Also, if the vehicle 100 to be controlled can be extracted using information other than the vehicle identification information VI, such as the position of the vehicle 100, the object detection unit 322 may not need to acquire the vehicle identification information VI. In this case, steps S610 and S620 shown in FIG. 7 may be omitted.

[0071] The control and methods described herein may be implemented by a special-purpose computer configured with a processor and memory programmed to perform one or more functions embodied in a computer program. Alternatively, the control unit and methods described herein may be implemented by a special-purpose computer configured with a processor comprising one or more dedicated hardware logic circuits. Alternatively, the control unit and methods described herein may be implemented by one or more special-purpose computers configured with a processor and memory programmed to perform one or more functions in combination with a processor configured with one or more hardware logic circuits. Furthermore, the computer program may be stored in a computer-readable non-transitory tangible storage medium as instructions executed by a computer.

[0072] The present disclosure is not limited to the above-described embodiments and can be realized in various configurations without departing from the spirit thereof. For example, the technical features in the embodiments corresponding to the technical features in each aspect described in the Summary of the Invention section can be appropriately replaced or combined to solve some or all of the above-described problems or achieve some or all of the above-described effects. Furthermore, if a technical feature is not described as essential in this specification, it can be appropriately deleted. [Explanation of symbols]

[0073] 50...first process, 60...second process, 70...self-propelled detector, 80, 80A, 80B, 80C...vehicle detector, 100, 100R...vehicle, 140...actuator, 142...drive wheel, 190...vehicle communication device, 200...ECU, 210...CPU, 212...driving control unit, 220...storage device, 230...interface circuit, 300, 300b, 300c...remote control device, 310, 310b, 310c...CPU, 312...remote control unit, 314...point cloud data acquisition unit, 316...start position determination unit, 318...position estimation unit, 322...object detection unit, 328...SLA M unit, 330...automatic driving control unit, 340, 340b, 340c...storage device, 350...interface circuit, 360...actuator, 390...remote communication device, 500...remote control system, AC...actuator driving history, AM...position, AR...driving route, BB1, BB2...bounding box, BM, BM1...previous matching position, FC...factory, GR...guided vehicle route, LM...model, PD1, PD2...3D point cloud data, RT, RT1, RT2, RT3...target route, SR...driving path, VI...vehicle identification information, VP...vehicle point cloud data

Claims

1. 1. A remote control device, comprising: a point cloud data acquisition unit that acquires three-dimensional point cloud data measured using a distance measuring device; a position estimation unit that estimates at least one of a position and an orientation of the moving object in the three-dimensional point cloud data by matching a template point cloud indicating the moving object to the three-dimensional point cloud data; a start position determination unit that determines a start position at which matching of the template point cloud with the three-dimensional point cloud data is started by the position estimation unit; and a remote control unit that uses at least one of the estimated position and orientation of the moving object to generate a control command for remotely controlling the moving object and transmits the control command to the moving object. Remote control device.

2. The remote control device according to claim 1 , wherein the start position determination unit determines the start position by using information on a position of the moving object in the three-dimensional point cloud data.

3. The remote control device according to claim 2, The start position determination unit is acquiring a previous matching position where matching between the three-dimensional point cloud data and the template point cloud was previously completed as information related to the position of the moving object; determining the starting position using the previously obtained matching position; Remote control device.

4. 4. The remote control device according to claim 3, The start position determination unit is Furthermore, a position after movement of the moving object that has moved from the previous matching position during the period from the time when the previous matching was completed to the time when the current matching is performed is estimated; determining the starting position using the estimated post-movement position; Remote control device.

5. 3. The remote control device according to claim 2, Further, a storage device is provided for storing a target route of the moving object, which is information regarding the position of the moving object, The start position determination unit determines any position of the target route as the start position. Remote control device.

6. 6. The remote control device according to claim 5, The start position determination unit is acquiring a previous matching position where matching between the three-dimensional point cloud data and the template point cloud was completed on the target route the previous time; determining the start position using the previous matching position on the acquired target route; Remote control device.

7. 3. The remote control device according to claim 2, the start position determination unit determines the start position by using, as information regarding the position of the moving body, either the position of the moving body detected from the three-dimensional point cloud data by a predetermined object detection method or the position of the moving body detected based on a difference between point cloud data corresponding to a background of the three-dimensional point cloud data and point cloud data corresponding to the moving body; Remote control device.

8. 8. The remote control device according to claim 7, The start position determination unit is When a plurality of positions of the moving object are detected, and determining the start position by using moving object identification information for identifying the moving object and using a position of the moving object corresponding to the moving object identification information among the detected positions of the moving objects. Remote control device.

Images