Remote control device
By using three-dimensional point cloud data to match template point clouds in the remote control device, combined with the previous matching position and object detection information, the problem of long detection of moving objects in the prior art is solved, and a fast and efficient position is achieved Detection and control.
Patent Information
- Application Number
- CN202410677390.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2023-05-31
- Filing Date
- 2024-05-29
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2044-05-29
AI Technical Summary
In the prior art, by placing the two-dimensional model stored in the memory at the front point of the two-dimensional projected image, the time required to detect the position of the moving object is longer.
Remote control device is adopted to obtain the three-dimensional point cloud data to match the template point cloud, estimate the position and orientation of the moving body, and generate control commands to shorten the matching time. The device can use the previous matching position, target route and object detection information to determine the start position of the match, thereby improving matching efficiency.
The processing time required to detect the position of the moving body is significantly shortened, the matching efficiency is improved, and the position of the moving body can be quickly detected and controlled.
Smart Images

Figure CN119065280A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to a remote control device. Background Art
[0002] Technologies for detecting the position of an object in space are known. For example, Japanese Unexamined Patent Application Publication No. 11-259658 (JP 11-259658A) discloses a technology in which a two-dimensional projection image showing the side shape of a vehicle is generated based on an input image acquired from a camera, the generated projection images are sequentially scanned after a two-dimensional model stored in a memory is placed at the foremost point of the two-dimensional projection image, and the position of the vehicle is recognized by performing matching for each scanned position. Summary of the invention
[0003] In the related art, scanning is performed sequentially after placing a two-dimensional model stored in a memory at the foremost point of a two-dimensional projection image. Therefore, there is a problem that a long period of time is required to complete the matching. Therefore, in the technology for detecting the position of a moving object by matching a model stored in a memory, a technology for shortening the period of time required to detect the position of the moving object is required.
[0004] The present disclosure may be implemented as follows.
[0005] A remote control device includes: at least one processor, the at least one processor being configured to perform processing, the processing including: acquiring three-dimensional point cloud data measured using a distance measuring device; estimating at least one of a position and an orientation of a moving body in the three-dimensional point cloud data by matching a template point cloud indicating the moving body with the three-dimensional point cloud data; determining a starting position to start matching the template point cloud with the three-dimensional point cloud data; and generating a control command for remotely controlling the moving body using at least one of the estimated position and the estimated orientation of the moving body, and transmitting the control command to the moving body. The remote control device of the present invention can quickly detect the position of the moving object by determining the start position of matching near the moving object. Therefore, the processing period required for detecting the position of the moving object by template matching can be shortened.
[0006] In the remote control device of the above aspect, the processing may further include: determining the start position using information on the position of the mobile body in the three-dimensional point cloud data. The remote control device of the present invention can quickly detect the position of the moving object by determining the starting position of template matching using information on the position of the moving object. Therefore, the processing period required for template matching can be shortened.
[0007] In the remote control device of the above aspect, the processing may further include: acquiring a previous matching position where a previous matching between the three-dimensional point cloud data and the template point cloud was completed as information about the position of the moving body, and determining the start position using the acquired previous matching position. With the remote control device of the present aspect, it is possible to shorten the processing period required for template matching by simple processing using a previous matching position that is highly likely to be near the position of the mobile body at the time of current matching.
[0008] In the remote control device of the above aspect, the processing may further include: estimating a moved position of the mobile body that has moved from a previous matching position during a period from a time when the previous matching is completed to a time before the current matching is performed; and determining the start position using the estimated moved position. The remote control device of the present aspect can set the start position of matching to be closer to the position of the moving body at the time of current matching by estimating the position of the moving body after the movement. Therefore, the processing period required for template matching can be further shortened.
[0009] The remote control device of the above aspect may further include: a storage device configured to store a target route of the moving body as information on the position of the moving body, and the processing may further include: determining an arbitrary position on the target route as the starting position. The remote control device of the present invention can quickly detect the position of a moving object by setting a starting position on a target route where the moving object is likely to exist. Therefore, the processing period required for template matching can be shortened.
[0010] In the remote control device of the above aspect, the processing may further include: obtaining a previous matching position where the previous matching between the three-dimensional point cloud data on the target route and the template point cloud has been completed, and determining the starting position using the obtained previous matching position on the target route. The remote control device of the present aspect can detect the position of the moving object more quickly by using the previous matching position on the target route where the moving object is more likely to exist.
[0011] In the remote control device of the above aspect, the processing may also include: using the position of the moving object detected from the three-dimensional point cloud data by a predetermined object detection method, or the position of the moving object detected based on the difference between the point cloud data associated with the background of the three-dimensional point cloud data and the point cloud data associated with the moving object, as information about the position of the moving object, to determine the starting position. The remote control device of the present invention can extract the temporary position of the moving body using object detection, and quickly detect the position of the moving body by a simple method. Therefore, the processing period required for template matching can be shortened.
[0012] In the remote control device of the above aspect, the processing may further include: when positions of a plurality of moving bodies are detected, determining the start position using a position of a moving body associated with moving body identification information for identifying the moving body among the detected positions of the moving bodies. The remote control device of the present aspect can extract the position of the moving object to be controlled using the moving object identification information even when the temporary positions of a plurality of moving objects are detected by object detection.
[0013] In the remote control device of the above aspect, the distance measuring device may be configured to communicate with at least one processor, and the distance measuring device may be configured to transmit the three-dimensional point cloud data to the remote control device.
[0014] In the remote control device of the above aspect, the distance measuring device may be installed around a travel path of the moving body.
[0015] The remote control device of the above aspect may further include a distance measuring device.
[0016] The present disclosure may also be implemented in various forms other than the remote control device. For example, the present disclosure may be implemented in the form of a remote control system, a method for moving a mobile body, a method for manufacturing a mobile body, a method for detecting a mobile body, a mobile body detection device, a method for controlling a remote control system, a method for controlling a remote control device, a computer program for implementing these control methods, or a non-transitory recording medium recording the computer program. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] Features, advantages, and technical and industrial significance of exemplary embodiments of the present invention will be described below with reference to the accompanying drawings, in which like numerals represent like elements, and in which: Figure 1 shows a schematic configuration of a remote control system; Figure 2 A vehicle driving method is schematically shown; Figure 3 is a flowchart showing a processing routine of a vehicle travel process by remote control; Figure 4 is a flowchart showing a processing routine of a start position determination process according to the first embodiment; Figure 5 shows the estimated position of the vehicle after traveling from the previous matched position; Figure 6 is a block diagram showing an internal functional configuration of a remote control device according to a second embodiment; Figure 7 is a flowchart showing a start position determination process according to the second embodiment; Figure 8 shows template matching in the start position determination process according to the second embodiment; Fig. 9 is a top view showing an example of the overall configuration of a remote control device according to a third embodiment; and Fig.10 is a block diagram showing an internal functional configuration of a remote control apparatus according to a third embodiment. DETAILED DESCRIPTION A. First Embodiment
[0018] Figure 1 1 shows a schematic configuration of a remote control system 500 including a remote control device 300 according to a first embodiment of the present disclosure. The remote control device 300 generates a control command for autonomously driving a vehicle 100 by remote control, and transmits it to the vehicle 100. For example, the remote control system 500 is used in a factory for manufacturing vehicles 100 that can travel by remote control.
[0019] Examples of the vehicle 100 include passenger cars, trucks, buses, and construction vehicles. The vehicle 100 is preferably a pure electric vehicle (Battery Electric Vehicle, BEV). The vehicle 100 is not limited to a pure electric vehicle, but may be, for example, a gasoline vehicle, a hybrid electric vehicle, or a fuel cell electric vehicle. The vehicle 100 includes a vehicle communication device 190, an actuator 140, and an electronic control unit (Electronic Control Unit, ECU) 200.
[0020] The ECU 200 is mounted on the vehicle 100, and performs various types of control in the vehicle 100. The ECU 200 includes a storage device 220 (such as a hard disk drive (HDD), a solid state drive (SSD), an optical recording medium, or a semiconductor memory), a central processing unit (CPU) 210, and an interface circuit 230. The CPU 210, the storage device 220, and the interface circuit 230 are connected via an internal bus in a manner that enables them to communicate bidirectionally. The actuator 140 and the vehicle communication device 190 are connected to the interface circuit 230. The vehicle communication device 190 wirelessly communicates with a device outside the vehicle 100 connected to a network (such as a remote control device 300) via an access point in a factory or the like.
[0021] The read / write area of the storage device 220 stores computer programs for implementing at least a part of the functions provided in the present embodiment. Functions such as the driving control unit 212 are implemented by the CPU 210 executing various computer programs stored in the memory.
[0022] The driving control unit 212 performs driving control of the vehicle 100. "Driving control" refers to various types of control of the actuator 140 used to drive the "driving", "steering" and "stopping" functions of the vehicle 100, typically the adjustment of acceleration, speed and steering angle. In the present embodiment, the actuator 140 includes an actuator of a driving device for accelerating the vehicle 100, an actuator of a steering device for changing the driving direction of the vehicle 100, and an actuator of a braking device for decelerating the vehicle 100. The driving device includes a battery, a driving motor driven by battery power, and a driving wheel rotated by the driving motor. The actuator of the driving device includes a driving motor. The actuator 140 may also include an actuator for shaking the wipers of the vehicle 100 and an actuator for opening and closing the electric windows of the vehicle 100, etc.
[0023] When the driver is in the vehicle 100, the driving control unit 212 can drive the vehicle 100 by controlling the actuator 140 in response to the driver's operation. The driving control unit 212 can also drive the vehicle 100 by controlling the actuator 140 in response to a control command transmitted from the remote control device 300, regardless of whether the driver is in the vehicle 100.
[0024] The remote control system 500 includes one or more vehicle detectors 80 and the remote control device 300. The vehicle detector 80 is a device for measuring various types of data to be used to estimate at least one of the position of the vehicle 100 and the orientation of the vehicle 100. The vehicle detector 80 is a light detection and ranging (Light Detection And Ranging, LiDAR) sensor, which is a distance measuring device that measures three-dimensional point cloud data of the vehicle 100. The three-dimensional point cloud data indicates the three-dimensional position of the point cloud. Using the LiDAR sensor, high-precision three-dimensional point cloud data can be acquired. The orientation and driving direction of the vehicle 100 can be estimated by acquiring only the position of the vehicle 100 using the vehicle detector 80 and acquiring the change of the vehicle 100 over time.
[0025] The vehicle detector 80 is connected to the remote control device 300 to communicate with the remote control device 300 wirelessly or by wire. By acquiring the three-dimensional point cloud data from the vehicle detector 80, the remote control device 300 can acquire the position and orientation of the vehicle 100 relative to the target route RT in real time. In this embodiment, the position of the vehicle detector 80 is fixed near the driving road SR, and the relative relationship between the reference coordinate system Σr and the device coordinate system of the vehicle detector 80 is known. A coordinate transformation matrix for mutually transforming the coordinate values of the reference coordinate system Σr and the coordinate values of the device coordinate system of the vehicle detector 80 is pre-stored in the remote control device 300.
[0026] The remote control device 300 performs driving control of the vehicle 100 by remote control. The remote control device 300 transports the vehicle 100 by, for example, autonomously driving the vehicle 100 by remote control during a manufacturing process in a factory. The transportation of the vehicle 100 using autonomous driving by remote control is also referred to as "autonomous driving transportation". Autonomous driving transportation allows the vehicle 100 to be moved without using a transportation device such as a crane or a conveyor belt. The remote control device 300 can monitor the surroundings of the vehicle 100.
[0027] The remote control device 300 includes a central processing unit (CPU) 310, a storage device 340, an interface circuit 350, and a remote communication device 390. The CPU 310, the storage device 340, and the interface circuit 350 are connected via an internal bus in a manner that enables bidirectional communication. The remote communication device 390 is connected to the interface circuit 350. The remote communication device 390 communicates with the vehicle 100 via a network or the like.
[0028] The storage device 340 is, for example, a random access memory (RAM), a read only memory (ROM), a HDD or an SSD. The read / write area of the storage device 340 stores the vehicle point cloud data VP, the target route RT, the actuator driving history AC and the previous matching position BM. The target route RT is a predetermined driving route of the vehicle 100 on the driving road SR.
[0029] The vehicle point cloud data VP is used as a template point cloud for estimating at least one of the position and orientation of the vehicle 100. For example, three-dimensional computer-aided design (CAD) data of the vehicle 100 may be used as the vehicle point cloud data VP. The vehicle point cloud data VP may include information for determining the orientation of the vehicle 100. By performing template matching using the vehicle point cloud data VP, the position and orientation of the vehicle 100 in the three-dimensional point cloud data may be estimated with high accuracy.
[0030] The actuator driving history AC is a history of input values and output values of each actuator 140 of the vehicle 100. The actuator driving history AC may be, for example, a history of control command values transmitted from the remote control device 300 to the vehicle 100, or actual measurement values such as vehicle speed, steering angle, braking force, and rotation angle acquired from a detector of the vehicle 100. As described later, the previous matching position BM is a coordinate value of a position where the previous template matching between the three-dimensional point cloud data and the vehicle point cloud data VP performed by the position estimation unit 318 has been completed.
[0031] The storage device 340 stores a computer program for implementing at least a portion of the functions provided in the present embodiment. The CPU 310 executes the computer program stored in the storage device 340 to function as a remote control unit 312, a point cloud data acquisition unit 314, a start position determination unit 316, and a position estimation unit 318. Some or all of these functions may be implemented by hardware circuits. The point cloud data acquisition unit 314 acquires three-dimensional point cloud data measured by the vehicle detector 80.
[0032] The starting position determination unit 316 determines the starting position for matching the vehicle point cloud data VP with the acquired three-dimensional point cloud data. From the perspective of quickly completing template matching, the starting position for matching is preferably the position of the vehicle 100 to be detected in the three-dimensional point cloud data or near the position of the vehicle 100.
[0033] In the present embodiment, the starting position determination unit 316 uses the information about the position of the vehicle 100 in the three-dimensional point cloud data to determine the starting position of the template matching. The "information about the position of the vehicle 100 in the three-dimensional point cloud data" is data to be used to estimate the position of the vehicle 100 in the three-dimensional point cloud data or a position near the position of the vehicle 100. The "information about the position of the vehicle 100 in the three-dimensional point cloud data" is obtained by a method different from template matching, and it is used in processing before performing template matching. The "information about the position of the vehicle 100 in the three-dimensional point cloud data" is preferably small-size data, data obtained by simple processing, etc., to increase the processing speed of template matching.
[0034] The position estimation unit 318 estimates the position and orientation of the vehicle 100 in the acquired three-dimensional point cloud data. In this embodiment, the position estimation unit 318 estimates the position and orientation of the vehicle 100 in the three-dimensional point cloud data by performing template matching using the vehicle point cloud data VP on the three-dimensional point cloud data. For example, an iterative closest point (ICP) algorithm or a normal distribution transform (NDT) algorithm may be used to perform template matching between the vehicle point cloud data VP and the three-dimensional point cloud data.
[0035] The remote control unit 312 generates a control command for remote control using the estimated position and orientation of the vehicle 100 and transmits it to the vehicle 100. The control command is a command for causing the vehicle 100 to travel along the target route RT stored in the storage device 340. The control command may be generated as a command including a driving force or a braking force and a steering angle. Alternatively, the control command may be generated as a command including at least one of the position and orientation of the vehicle 100 and a future travel route. When the vehicle 100 receives the remote control request, the driving control is implemented by the driving control unit 212 of the ECU 200, and as a result, the vehicle 100 automatically travels.
[0036] Figure 2 A driving method for vehicle 100 is schematically shown. Figure 2 It shows how to perform autonomous driving transportation of the vehicle 100 in the factory FC that manufactures the vehicle 100. The factory FC includes a first process 50 and a second process 60. The first process 50 is, for example, a place where the vehicle 100 is assembled, and the second process 60 is, for example, a place where the vehicle 100 is inspected. The first process 50 and the second process 60 are connected by a driving road SR, and the vehicle 100 can travel on the driving road SR. Any position in the factory FC is represented by the xyz coordinate value of the reference coordinate system Σr.
[0037] The remote control device 300 causes the vehicle 100 to travel along the target route RT while estimating the position and orientation of the vehicle 100 in the three-dimensional point cloud data acquired by the vehicle detector 80 at predetermined time intervals. Figure 2 In the example of , the target route RT set on the driving road SR includes mutually continuous target routes RT1, RT2, and RT3. A plurality of vehicle detectors 80 are installed around the driving road SR, including vehicle detectors 80A, 80B, and 80C respectively associated with the target routes RT1, RT2, and RT3.
[0038] The remote control device 300 appropriately switches the vehicle detector 80 based on the position of the vehicle 100, estimates the position and orientation of the vehicle 100, and causes the vehicle 100 to travel along the target route RT. Figure 2 In the example of FIG. 5 , the vehicle 100 that has departed from the first process 50 travels along the target route RT1 by remote control using the three-dimensional point cloud data acquired by the vehicle detector 80A. When the vehicle 100 has reached the target route RT2, the vehicle detector 80A switches to the vehicle detector 80B that is suitable for detecting the vehicle 100 traveling on the target route RT2. When the vehicle 100 has reached the target route RT3, similarly, the vehicle detector 80B switches to the vehicle detector 80C that is suitable for detecting the vehicle 100.
[0039] Figure 3 1 is a flowchart showing a processing routine of a driving process of the vehicle 100 by remote control according to the present embodiment. This process starts when the vehicle 100 starts to travel on the driving road SR. This process can be repeated each time the point cloud data acquisition unit 314 newly acquires three-dimensional point cloud data from the vehicle detector 80 that performs measurement of the vehicle 100 to be controlled.
[0040] In step S100, the point cloud data acquisition unit 314 acquires three-dimensional point cloud data from the vehicle detector 80. In step S200, the start position determination unit 316 performs a start position determination process. The "start position determination process" is a process for determining a start position at which to start matching the vehicle point cloud data VP with the acquired three-dimensional point cloud data.
[0041] In step S300, template matching is started from the starting position determined by the starting position determination process. When matching with the vehicle point cloud data VP cannot be completed at the starting position, matching is performed sequentially near the starting position. Template matching is repeated while changing the matching position until it is completed. In step S400, the position estimation unit 318 estimates the position and orientation of the vehicle 100 in the three-dimensional point cloud data by performing template matching using the vehicle point cloud data VP. In step S500, the remote control unit 312 generates a control command for causing the vehicle 100 to travel along the target route RT using the estimated position and orientation of the vehicle 100, and transmits it to the vehicle 100.
[0042] Figure 4 1 is a flowchart showing an example of a processing routine for starting position determination processing according to the first embodiment. In the present embodiment, depending on whether there is a previous matching result, the previous matching position BM where the previous matching between the three-dimensional point cloud data and the vehicle point cloud data VP has been completed or the target route RT of the vehicle 100 is selectively used as the "information on the position of the vehicle 100 in the three-dimensional point cloud data".
[0043] In step S210, the start position determination unit 316 checks whether there is a matching result of the previous template matching. For example, it can be checked whether there is a previous matching result based on whether there is a previous matching result stored in the processing history stored in the storage device 340 or whether there is a previous matching position BM stored in the storage device 340.
[0044] When there is no previous matching result (S210: No), the start position determination unit 316 advances the process to step S250. In this case, the target route RT of the vehicle 100 is used as the "information about the position of the vehicle 100 in the three-dimensional point cloud data". In step S250, the start position determination unit 316 obtains the target route RT stored in the storage device 340.
[0045] In step S260, the travel start position of the vehicle 100 on the target route RT is acquired. The travel start position of the vehicle 100 on the target route RT is stored in the storage device 340 in advance, for example, in association with the target route RT. Figure 2 In the example of FIG. 5 , the travel start position of the vehicle 100 is near the exit of the first step 50 .
[0046] In step S270, the start position determination unit 316 determines the start position of template matching as the travel start position on the acquired target route RT, and terminates the process. When there is no previous matching result, the vehicle 100 is likely not to have started traveling and is still at the travel start position. Therefore, when there is no previous matching result, the processing period of template matching can be reduced by setting the travel start position of the vehicle 100 on the target route RT as the start position.
[0047] When there is a previous matching result in step S210 (S210: Yes), the start position determination unit 316 advances the processing to step S220. In this case, the previous matching position BM where the previous matching between the three-dimensional point cloud data and the vehicle point cloud data VP was completed is used as the "information on the position of the vehicle 100 in the three-dimensional point cloud data". In this embodiment, the position of the vehicle 100 after traveling at the time when the current matching is performed is further estimated using the previous matching position BM, and the position after traveling is used as the "information on the position of the vehicle 100 in the three-dimensional point cloud data".
[0048] In step S220, the start position determination unit 316 acquires the actuator drive history AC of the vehicle 100. The acquired actuator drive history AC includes at least the drive history of the actuator 140 related to the travel of the vehicle 100 during the period from the moment when the previous match was completed to the moment before the current match was performed. In step S230, the start position determination unit 316 estimates the position of the vehicle 100 after traveling during the period from the moment when the previous match was completed to the moment before the current match was performed. Specifically, the start position determination unit 316 estimates the position of the vehicle 100 after traveling by estimating the travel route from the previous matching position BM using the drive history of the actuator 140. In step S240, the start position determination unit 316 determines the estimated position of the vehicle 100 after traveling as the start position of the template matching, and terminates the process.
[0049] Figure 5 The estimated position of the vehicle 100 after traveling from the last matched position is shown. Figure 5 An example of three-dimensional point cloud data PD1 acquired from the vehicle detector 80 is shown. The three-dimensional point cloud data PD1 schematically shows a position BM1 of the vehicle 100 at the time of the previous match as an example of the previous match position BM, and a position AM of the vehicle 100 after traveling on the travel route AR from the previous match position.
[0050] For example, starting from the position BM1 at the time of the previous match, by estimating the travel route AR or position AM of the vehicle 100 obtained from the actuator drive history AC, the traveled position AM of the vehicle 100 can be obtained. In another embodiment, the starting position determination unit 316 can estimate the travel route AR or position AM of the vehicle 100 based on the output value from the detector for obtaining the vehicle speed, steering angle, braking force, rotation angle, etc. instead of the actuator drive history AC. Alternatively, the current position of the vehicle 100 obtained from a Global Navigation Satellite System (GNSS) receiver or the like can be used as the traveled position AM of the vehicle 100.
[0051] The starting position determination unit 316 determines the estimated position AM of the vehicle 100 after traveling as the starting position of template matching, and the position estimation unit 318 starts template matching using the vehicle point cloud data VP from the position AM. By setting the position AM of the vehicle 100 after traveling as the starting position of template matching, the actual position of the vehicle 100 or its vicinity can be set as the starting position. Therefore, the processing period required for template matching can be shortened. Compared with the case where template matching is started at a position far away from the position AM, this can reduce the problem that the matching is completed at a position different from the actual position of the vehicle 100 due to obtaining a local solution through gradient descent, etc. Therefore, the detection accuracy of the vehicle 100 can be improved.
[0052] As described above, the remote control device 300 of the present embodiment includes: a point cloud data acquisition unit 314 configured to acquire three-dimensional point cloud data measured using the vehicle detector 80; a position estimation unit 318 configured to estimate the position and orientation of the vehicle 100 in the three-dimensional point cloud data by matching the vehicle point cloud data VP with the three-dimensional point cloud data; a start position determination unit 316 configured to determine the start position at which template matching of the three-dimensional point cloud data is to be started by the position estimation unit 318; and a remote control unit 312 configured to generate a control command for remotely controlling the vehicle 100 using the estimated position and orientation of the vehicle 100, and transmit the control command to the vehicle 100. The remote control device 300 of the present embodiment can quickly detect the position of the vehicle 100 by setting the start position of matching near the vehicle 100. Therefore, the processing period required for template matching can be shortened.
[0053] In the remote control device 300 of the present embodiment, the start position determination unit 316 determines the start position using information about the position of the vehicle 100 in the three-dimensional point cloud data acquired by the point cloud data acquisition unit 314. By setting the start position of template matching to the position of the vehicle 100 using the information about the position of the vehicle 100, the position of the vehicle 100 can be quickly detected. Therefore, the processing period required for template matching can be shortened.
[0054] In the remote control device 300 of the present embodiment, the start position determination unit 316 acquires the previous matching position BM and determines the start position of the template matching using the acquired previous matching position BM. By using the previous matching position BM which is likely to be near the position of the vehicle 100 at the time of the current matching, the processing period required for the template matching can be shortened by simple processing.
[0055] In the remote control device 300 of the present embodiment, the start position determination unit 316 estimates the position AM of the vehicle 100 after traveling from the previous matching position BM during the period from the time when the previous matching is completed to the time before the current matching is performed, and determines the estimated position AM after traveling as the start position of template matching. By estimating the position of the vehicle 100 after traveling, the start position of matching can be set to be closer to the position of the vehicle 100 at the time of the current matching. Therefore, the processing period required for template matching can be further shortened.
[0056] In the remote control device 300 of the present embodiment, when there is no previous matching result, the starting position determination unit 316 determines the travel start position on the target route RT as the start position of the template matching. By setting the start position on the target route RT where the vehicle 100 is likely to be present, the position of the vehicle 100 can be quickly detected. Therefore, the processing period required for the template matching can be shortened. In particular, when there is no previous matching result, the travel start position where the vehicle 100 is likely to be present is set as the start position. Therefore, the processing period required for the template matching can be further shortened. B. Second Embodiment
[0057] Figure 6 is a block diagram showing the internal functional configuration of a remote control device 300b according to a second embodiment. The remote control device 300b of this embodiment is different from the remote control device 300 of the first embodiment in that a CPU 310b further serving as an object detection unit 322 is provided instead of the CPU 310, and a storage device 340b further storing vehicle identification information VI and a trained model LM is provided instead of the storage device 340. The other configurations are the same.
[0058] The trained model LM is a machine learning model that performs object detection by a predetermined object detection method using deep learning. The trained model LM has been fully trained using a predetermined data set. For example, various methods using a convolutional neural network (CNN) can be used for the trained model LM. For object detection, it is particularly preferred to adopt a method using deep learning, such as a region-convolutional neural network (R-CNN), a fast R-CNN, a fast R-CNN, various versions of You Only Look Once (YOLO) or a single shot detector (SSD).
[0059] The object detection unit 322 performs annotation on the three-dimensional point cloud data using the trained model LM, and detects the temporary position of the vehicle in the three-dimensional point cloud data. In this case, the term "vehicle" refers to an attribute (category) as a vehicle, which may include vehicles other than the vehicle 100 to be controlled. Specifically, the object detection unit 322 performs object detection by inputting the detected three-dimensional point cloud data into the trained model LM, and generates a 3D bounding box for the area identified as a vehicle. "Bounding box" refers to a partial area extracted from the outer area of the three-dimensional point cloud data by using a minimum rectangular box to surround the object area of the vehicle. That is, the object detection unit 322 extracts the area occupied by the vehicle in the three-dimensional point cloud data by forming a bounding box.
[0060] The vehicle identification information VI refers to various types of information capable of identifying each vehicle 100. The vehicle identification information VI includes, for example, identification (ID) information assigned to each vehicle 100 (such as a vehicle identification number (VIN)), specification information of the vehicle 100 (such as vehicle type, color, shape, and size), and production management information of the vehicle 100 (such as the name of an ongoing process). The vehicle identification information VI may include the size of a bounding box that can be detected. For example, the vehicle identification information VI may be obtained from a radio frequency identification (RF-ID) tag attached to the vehicle 100 through short-range wireless communication.
[0061] Figure 7 is a flowchart showing the start position determination process according to the second embodiment. In step S600, the object detection unit 322 performs object detection using the acquired three-dimensional point cloud data and acquires the temporary position of the vehicle 100 in the three-dimensional point cloud data. More specifically, the object detection unit 322 generates a bounding box of the vehicle in the acquired three-dimensional point cloud data by inputting the three-dimensional point cloud data into the trained model LM.
[0062] In step S610, the object detection unit 322 acquires the vehicle identification information VI of the vehicle 100 to be controlled. In step S620, the object detection unit 322 extracts a bounding box associated with the vehicle 100 to be controlled using the vehicle identification information VI. For example, the object detection unit 322 extracts a bounding box that matches the size of the bounding box associated with the vehicle identification information VI of the vehicle 100 to be controlled or the size of the vehicle 100 from the generated bounding box. Instead of the size of the bounding box or the size of the vehicle 100, the vehicle identification information VI may also indicate a partial or complete shape of the vehicle 100, or a partial or complete length, width, and height of the vehicle 100. In step S630, the starting position determination unit 316 determines the position of the extracted bounding box as the starting position for template matching, and terminates the process.
[0063] Figure 8 Template matching in the start position determination process according to the second embodiment is shown. Figure 8 An example of three-dimensional point cloud data PD2 acquired from the vehicle detector 80 is shown. The three-dimensional point cloud data PD2 includes three-dimensional point cloud data of the vehicle 100 to be controlled and three-dimensional point cloud data of the vehicle 100R not to be controlled. The object detection unit 322 generates bounding boxes BB1, BB2 of the vehicle 100 and the vehicle 100R recognized as vehicles by inputting the three-dimensional point cloud data PD2 into the trained model LM.
[0064] Next, the object detection unit 322 acquires the vehicle identification information VI. In this case, the vehicle identification information VI is preferably information such as the size and width of the vehicle 100, 100R or the size of the bounding box, as information that facilitates distinguishing vehicles 100, 100R with different sizes. Based on the acquired vehicle identification information VI, the bounding box BB1 of the vehicle 100 is extracted from the bounding boxes BB1, BB2. The starting position determination unit 316 determines the position of the extracted bounding box BB1 as the starting position of the template matching, and the position estimation unit 318 performs template matching using the vehicle point cloud data VP starting from the position of the bounding box BB1 as the starting position.
[0065] As described above, in the remote control device 300b of the present embodiment, the start position determination unit 316 determines the position of the vehicle 100 detected from the three-dimensional point cloud data PD2 by a predetermined object detection method using deep learning as the start position. The position of the vehicle 100 can be extracted from the three-dimensional point cloud data PD2 using a known method, which is object detection using deep learning. The position of the vehicle 100 in the three-dimensional point cloud data PD2 can be used to set the start position near the vehicle 100. Therefore, the processing period required for template matching can be shortened.
[0066] In the remote control device 300b of the present embodiment, when the positions of a plurality of vehicles 100, 100R are detected, the start position determination unit 316 also uses the vehicle identification information VI for identifying the vehicle 100 to determine the start position using the position of the vehicle 100 associated with the vehicle identification information VI among the detected positions of the plurality of vehicles 100. Therefore, even when a plurality of vehicles 100, 100R are detected by an object detection method using deep learning, the position of the vehicle 100 to be controlled can be extracted. C. Third Embodiment
[0067] Fig. 9 3 is a top view showing an example of the overall configuration of the remote control device 300c according to the third embodiment. The remote control device 300c of the present embodiment is an automatic guided vehicle also called "guided mobility". As in the first embodiment, the remote control device 300c has a function of autonomously driving the vehicle 100 by remote control, and also has a function of autonomously driving the main body of the remote control device 300c on the driving road SR. That is, the remote control device 300c autonomously drives the following vehicle 100 by remote control to guide the following vehicle 100 while autonomously driving on the driving road SR.
[0068] The remote control device 300c includes a driving wheel 142 for autonomous driving, an autonomous driving detector 70, and a vehicle detector 80. The function of the vehicle detector 80 is the same as that of the vehicle detector 80 shown in the first embodiment, so its description will be omitted. The autonomous driving detector 70 is a distance measuring device, such as a camera or a LiDAR sensor. The data acquired by the autonomous driving detector 70 is used for simultaneous localization and mapping (SLAM) of the autonomous driving of the remote control device 300c. The camera can be a stereo camera, a monocular camera, an RGB-D camera (depth camera), etc. A time-of-flight (ToF) sensor or the like can be used instead of a camera or a LiDAR sensor. Fig. 9 In the example of FIG. 3 , the vehicle detector 80 is installed only on the remote control device 300 c. The vehicle detector 80 may also be installed around the travel road SR.
[0069] Fig.103 is a block diagram showing the internal functional configuration of the remote control device 300c according to the third embodiment. The internal functional configuration of the remote control device 300c is different from the internal functional configuration of the remote control device 300 of the first embodiment in that a CPU 310c further serving as a SLAM unit 328 and an autonomous driving control unit 330 is provided instead of the CPU 310, and a storage device 340c further storing a guide vehicle route GR is provided instead of the storage device 340. The other configurations are the same. The guide vehicle route GR is a predetermined target route on which the remote control device 300c is expected to travel.
[0070] The actuator 360 includes actuators of a driving device, a steering device, and a braking device for autonomous driving of the remote control device 300c. The driving device includes a battery, a travel motor driven by battery power, and a driving wheel 142 rotated by the travel motor.
[0071] The SLAM unit 328 performs SLAM using the data detected by the autonomous driving detector 70 and generates a map for autonomous driving of the remote control device 300c. The autonomous driving control unit 330 controls the actuator 360 to make the remote control device 300c autonomously drive. Specifically, the autonomous driving control unit 330 uses the map generated by the SLAM unit 328 to make the remote control device 300c autonomously drive along the guided vehicle route GR stored in the storage device 340c.
[0072] like Fig. 9 As shown, when the remote control device 300c travels along the guided vehicle route GR, the vehicle 100P is guided by the remote control of the remote control device 300c and travels on the same travel route as the remote control device 300c. In the present embodiment and the first embodiment, template matching is performed on the three-dimensional point cloud data acquired from the vehicle detector 80, and the start position determination unit 316 performs a start position determination process for determining the start position of the template matching. In the present embodiment and the first embodiment, the remote control device 300c for autonomous driving also uses the information on the position of the vehicle 100 to set the start position of the template matching near the vehicle 100. Therefore, the processing period required for the template matching can be shortened. D. Other embodiments:
[0073] (D1) The second embodiment shows an example in which a bounding box of a vehicle is generated by an object detection method using deep learning, and the position of the vehicle 100 extracted using the generated bounding box is determined as the starting position. Instead of generating a bounding box, the object detection unit 322 may also perform preprocessing for removing background point cloud data representing a stationary object from the acquired three-dimensional point cloud data. Such preprocessing is also referred to as "background subtraction". In this case, the starting position determination unit 316 determines the temporary position of the vehicle 100 detected based on the difference between the point cloud data associated with the background of the three-dimensional point cloud data and the point cloud data associated with the vehicle 100 to be detected as the starting position for template matching. The remote control device 300 configured in this manner can determine the starting position by a simple process as background subtraction.
[0074] (D2) The first embodiment shows an example in which the start position determination unit 316 sets the travel start position of the vehicle 100 on the target route RT as the start position when there is no previous matching result. The start position determination unit 316 may set the travel start position of the vehicle 100 on the target route RT as the start position regardless of whether there is a previous matching result. Instead of the travel start position on the target route RT, an arbitrary position on the target route RT may be determined as the start position. In this case, the start position may be omitted. Figure 4 . With the remote control device 300 configured in this manner, by setting the starting position on the target route RT where the vehicle 100 is likely to appear, the processing period required for template matching can be shortened. When the vehicle 100 cannot be detected at the starting position in this case, template matching is preferably performed along the target route RT.
[0075] The start position determination unit 316 can acquire the previous matching position BM on the target route RT of the last time, and determine the start position using the acquired previous matching position BM on the target route RT. With the remote control device 300 configured in this manner, the processing period required for template matching can be shortened by a simple method using the previous matching position BM on the target route RT where the vehicle 100 is more likely to appear.
[0076] (D3) Each of the above-mentioned embodiments shows an example in which the vehicle 100 is a passenger car, a truck, a bus, a construction vehicle, etc. However, the vehicle 100 is not limited thereto and may include various motor vehicles such as a two-wheeled vehicle, a four-wheeled vehicle, a train, etc. Various moving bodies other than the vehicle 100 may be used. A "moving body" refers to an object that can move. Examples of moving bodies include electric vertical take-off and landing aircraft (so-called flying cars), ships, airplanes, robots, and maglev trains. In this case, the terms "vehicle" and "car" in the present disclosure may be replaced with "moving body" as appropriate, and the term "traveling" may be replaced with "moving" as appropriate.
[0077] (D4) The vehicle 100 only needs to have a configuration that enables the vehicle 100 to be moved by remote control. For example, the vehicle 100 may be in the form of a platform having the following configuration. Specifically, the vehicle 100 only needs to have a configuration that performs the functions of "moving", "turning" and "stopping" by remote control. That is, the "vehicle 100 that can be moved by remote control" does not need to have at least a portion of the internal components such as the driver's seat and the dashboard, at least a portion of the external components such as the bumper and the fender, or the body shell. In this case, the remaining components (such as the body shell) may be installed on the vehicle 100 before the vehicle 100 is shipped from the factory, or the remaining components (such as the body shell, etc.) may be installed on the vehicle 100 after the vehicle 100 is shipped from the factory without being installed on the vehicle 100.
[0078] (D5) The first embodiment shows an example in which the start position determination unit 316 further estimates the position AM after the vehicle 100 travels from the previous matching position BM. The start position may be a position other than the position AM after the travel. For example, the previous matching position BM may be used. By using the previous matching position BM which is likely to be near the position of the vehicle 100, the processing period required for template matching can be shortened by simple processing.
[0079] (D6) The first embodiment shows an example in which the start position determination unit 316 uses the last matching position BM to determine the start position, and an example in which the start position determination unit 316 determines an arbitrary position on the target route RT as the start position. The start position determination unit 316 may determine a position other than the last matching position BM or a position different from the target route RT as the start position. For example, the start position determination unit 316 may determine the temporary position of the vehicle 100 acquired based on the detection result from a detector different from the vehicle detector 80 as the start position.
[0080] (D7) The third embodiment shows an example in which the object detection unit 322 uses the vehicle identification information VI to extract the vehicle 100 to be controlled. For example, when a single bounding box is generated in step S600, that is, when only the bounding box of the vehicle 100 to be controlled is generated, the vehicle identification information VI does not need to be used. In addition, when information other than the vehicle identification information VI (such as the position of the vehicle) can be used to extract the vehicle 100 to be controlled, the object detection unit 322 does not need to obtain the vehicle identification information VI. In this case, the vehicle identification information VI can be omitted. Figure 7 Steps S610 and S620 shown in FIG.
[0081] The control and method thereof described in the present disclosure may be implemented by a special-purpose computer provided by configuring a processor and a memory to be programmed to perform one or more functions implemented by a computer program. Alternatively, the control and method thereof described in the present disclosure may be implemented by a special-purpose computer provided by configuring a processor using one or more special-purpose hardware logic circuits. Alternatively, the control and method thereof described in the present disclosure may be implemented by one or more special-purpose computers, which are configured by a combination of a memory and a processor, the processor being programmed to perform one or more functions, and the processor being configured by one or more hardware logic circuits. The computer program may be stored in a computer-readable non-transitory tangible storage medium as instructions to be executed by a computer.
[0082] The present disclosure is not limited to the above-described embodiments, and can be implemented by various configurations without departing from the spirit of the present disclosure. For example, the technical features in the various embodiments corresponding to the technical features of each aspect described in the summary of the invention may be appropriately replaced or combined to solve some or all of the above-described problems, or to achieve some or all of the above-described effects. When a technical feature is not described as being necessary in this specification, such a feature may be omitted as appropriate.
Claims
1. Remote control device, including: a point cloud data acquisition unit, which acquires three-dimensional point cloud data measured using the distance measuring device; a position estimating unit that estimates at least one of a position and an orientation of the moving body in the three-dimensional point cloud data by matching a template point cloud indicating the moving body with the three-dimensional point cloud data; a starting position determining unit, which determines a starting position at which the position estimating unit starts matching the template point cloud with the three-dimensional point cloud data; as well as a remote control unit that generates a control command for remotely controlling the mobile body using at least one of the estimated position and orientation of the mobile body, and transmits the control command to the mobile body, wherein The starting position determination unit determines the starting position using information about the position used for the matching and information about the driving history of the actuator set in the mobile body included in the control command, and uses information about the position and orientation of the mobile body used to generate the control command and at least one of the target route predetermined as the driving route of the mobile body through the remote control.
2. The remote control device according to claim 1, wherein: The start position determination unit determines the start position using information on the position of the moving object in the three-dimensional point cloud data.
3. The remote control device according to claim 2, wherein The starting position determining unit acquiring a previous matching position at which a previous matching between the three-dimensional point cloud data and the template point cloud has been completed as information on the position of the moving object, and The start position is determined using the acquired previous matching position.
4. The remote control device according to claim 3, wherein The starting position determining unit further estimating the moved position of the moving body from the previous matching position during the period from the time when the previous matching was completed to the time before the current matching was performed, and The starting position is determined using the estimated position after the shift.
5. The remote control device according to claim 2, Also included is a memory storing the target route as information on the position of the mobile body, wherein The start position determination unit determines an arbitrary position on the target route as the start position.
6. The remote control device according to claim 5, wherein The starting position determining unit Acquire the last matching position where the last matching between the three-dimensional point cloud data on the target route and the template point cloud has been completed, and The start position is determined using the acquired previous matching position on the target route.
7. The remote control device according to claim 2, wherein The starting position determination unit determines the starting position by using any one of the position of the moving object detected from the three-dimensional point cloud data by a predetermined object detection method and the position of the moving object detected based on the difference between the point cloud data associated with the background of the three-dimensional point cloud data and the point cloud data associated with the moving object as information about the position of the moving object.
8. The remote control device according to claim 7, wherein The starting position determining unit, When the positions of a plurality of said moving objects are detected, The start position is further determined using the mobile body identification information for identifying the mobile body, and using the position of the mobile body associated with the mobile body identification information among the detected positions of the plurality of mobile bodies.
Citation Information
Patent Citations
Servo motor-based three-dimensional point cloud data matching method and system and controller
CN109579765A
Control method and control device of disinfection robot
CN114371690A
Pose estimation and dynamic vehicle detection method based on normal vector histogram features
CN114612665A
Information processing apparatus, control method, program, and storage medium
CN114867989A
Vehicle with remote-controlled operating mode
US20180339703A1
