Remote control system and remote control method

Abstract

To allow an operator to recognize whether position information of a mobile body transmitted with peripheral environment information of the mobile body is correct or not while narrowing down the peripheral environment information under a poor communication environment.SOLUTION: A remote control system 100 remotely controls a mobile body 10 comprising: a LiDAR 40 which observes a peripheral environment of the mobile body 10 as point group data; and a camera 30 which captures an image of the peripheral environment as image data. The mobile body 10 transmits an extracted image ID, an estimated position posture of the mobile body 10, and a position of a detected obstacle to a remote control device 20 as current peripheral environment information, and a rendering screen generation unit 22 generates: a first screen in which the position posture of the mobile body 10 and the position of the obstacle out of the current peripheral environment information received from the mobile body 10 are arranged in map point group data; and a second screen in which past image data in a storage device 21 corresponding to the image ID out of the current peripheral environment information received from the mobile body 10 is presented.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a remote control system and a remote control method.

Background Art

[0002] Mobile bodies such as robots are utilized for purposes such as inspecting social infrastructure such as substations and construction sites. As a method for setting the movement path of a mobile body, it is classified into a method in which the mobile body autonomously moves along the movement path and a method in which an operator remotely controls it. In the method of remote control, fine movement control is possible, so it is efficient. On the other hand, in order to remotely control a mobile body located inside a facility where the housing of the mobile body cannot be directly visually observed, information on the position of the mobile body inside the facility and the surrounding environment as seen from the mobile body is required. In particular, the surrounding environment information is essential for controlling the mobile body to avoid obstacles. The surrounding environment information is, for example, information obtained from external recognition sensors such as LiDAR (Light Detection And Ranging) and cameras mounted on the mobile body.

[0003] Patent Document 1 describes a method having a function of assisting the self-position estimation of a mobile body in order to cope with unprogrammed operations and abnormal situations. In this method, the operator processes the surrounding environment information by adding information on noise obstacles to the surrounding environment information transmitted from the mobile body and then replying. According to this method, if the surrounding environment information such as images and point clouds acquired on the mobile body side can be transmitted to the operator side, situations not assumed by the interaction with the mobile body can be avoided.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, in an environment where communication facilities are not well - maintained, such as a substation deep in the mountains, sufficient communication speed cannot be ensured, so the surrounding environment information of the moving object cannot be transmitted to the operator side. When the surrounding environment information is not presented to the operator, the operator cannot avoid obstacles around the moving object and may collide with them. Conventional technologies such as Patent Document 1 do not assume communication in a place with a poor communication environment where only a small amount of data communication is possible.

[0006] Therefore, a method of narrowing down the surrounding environment information to only the information necessary for operation from the information obtained by the external recognition sensor of the moving object and then notifying the operator of the small amount of surrounding environment information is considered. For example, only the position information (latitude, longitude, altitude, etc.) of the moving object estimated from the point - cloud data of the three - dimensional space obtained by LiDAR is notified to the operator. Thereby, the operator may be able to operate based on the position information and the map data on the operator's side.

[0007] However, if the estimated position information of the moving object is at a position different from the actual current position, that is, if the position estimation fails, the operator may give an instruction to the moving object relying on the incorrect position information and there is a risk of crashing into a wall.

[0008] Therefore, in the present invention, while narrowing down the surrounding environment information of the moving object in a situation with a poor communication environment, the main problem is to make the operator recognize the correctness or incorrectness of the position information of the moving object notified with the surrounding environment information.

Means for Solving the Problem

[0009] In order to solve the above problems, the remote - control system of the present invention has the following features. The present invention is a remote - control system for remotely controlling the moving object provided with a LiDAR that observes the surrounding environment of the moving object as point - cloud data and a camera that photographs the surrounding environment as image data. The mobile body has a self-position estimation unit, an obstacle detection unit, and an identical image extraction unit, and stores past feature points extracted from past image data captured by the camera in the past and map point cloud data observed by the LiDAR in the past in a first storage device. The remote control device has a drawing screen generation unit, and stores the past image data for each image ID and the map point cloud data in a second storage device. Based on a location on the map point cloud data in the first storage device that is similar to the current point cloud data observed by the LiDAR at the current time, the self-position estimation unit estimates the position and orientation of the mobile body. Based on a location where there is a difference between the current point cloud data and the map point cloud data in the surrounding environment, the obstacle detection unit detects the position of an obstacle that does not exist in the map point cloud data but exists in the current point cloud data. By comparing the current feature points extracted from the current image data captured by the camera at the current time with the past feature points in the first storage device, the identical image extraction unit obtains the past image data of the past feature points similar to the current feature points, and extracts the image ID corresponding to the past image data. The mobile body transmits the extracted image ID, the estimated position and orientation of the mobile body, and the position of the detected obstacle to the remote control device as the surrounding environment information at the current time. The drawing screen generation unit A first screen in which the position and orientation of the mobile body and the position of the obstacle among the surrounding environment information at the current time received from the mobile body are arranged in the map point cloud data, and A second screen for presenting the past image data in the second storage device corresponding to the image ID among the surrounding environment information at the current time received from the mobile body, are generated. Other means will be described later.

Advantages of the Invention

[0010] According to the present invention, it is possible to narrow down the surrounding environment information of the mobile body in a situation with a poor communication environment and enable the operator to recognize the correctness of the position information of the mobile body notified with the surrounding environment information.

Brief Description of the Drawings

[0011]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Figure 24

Figure 25

Modes for Carrying Out the Invention

[0012] Hereinafter, each example for carrying out the present invention will be described in detail with reference to the drawings and the like.

Examples

[0013] FIG. 1 is a configuration diagram of a remote control system 100. The remote control system 100 is configured by communicatively connecting a mobile body 10 and a remote control device 20. The mobile body 10 is equipped with a camera 30 and a LiDAR 40 as external environment recognition sensors for recognizing the external environment, and data is acquired from these external environment recognition sensors at regular intervals. In addition, the mobile body 10 receives control information regarding movement from the remote control device 20, and this is also input at regular intervals. The remote control device 20 is connected to a controller 50 into which an operator inputs a signal for controlling the mobile body 10 and an external terminal 60 such as a display for allowing the operator to view information about the mobile body 10. The controller 50 generates a control signal at regular intervals.

[0014] Here, the terms used in this embodiment are defined as follows. "Map point cloud data" is map data in which point clouds are arranged in a three-dimensional space showing the real environment as a result of observing the real environment (the surrounding environment of the mobile body 10) in the past with the LiDAR 40. The "observed in the past" here refers to a time point in the past compared to the current time point when remotely controlling the mobile body 10 from the remote control device 20. For example, the map point cloud data described later in FIG. 3 is observed from the real environment described later in FIG. 2. "Current point cloud data" is point cloud data obtained as a result of observing with the LiDAR 40 at the current time point.

[0015] "Past image data" is image data obtained by photographing the real environment in the past with the camera 30. For example, since the second screen 558 in FIG. 15 was photographed in the past, passersby etc. who exist at the current time point are not shown. "Past feature points" are feature points extracted from the past image data. Feature points are, for example, SIFT (Scale-Invariant Feature Transform), SURF (Speeded Up Robust Features), ORB (Oriented FAST and Rotated BRIEF), FAST (Features from Accelerated Segment Test), Harris.

[0016] The "current image data" refers to the image data that is the result of photographing the actual environment with the camera 30 at the current time. For example, since the second screen 561 in FIG. 19 was photographed at the current time, a human 451 who is a passerby is shown in it. The "current feature points" are the feature points extracted from the current image data, similar to the past feature points. In the search process for similar images between the past image data and the current image data, instead of processing the image data itself, by searching for the similarity relationship between the past feature points and the current feature points, the processing time and the data capacity required for the processing can be reduced.

[0017] The "image ID" is a uniquely assigned ID for the past image data photographed at the shooting position of the camera 30 and the posture of the camera 30 (the direction of the camera 30) at that shooting position, regardless of the type and number of objects (such as passersby) shown in the image data. That is, by associating and recording the shooting position of the camera 30 and the posture of the camera 30 with the image ID at the time of shooting the past image data, the shooting position of the camera 30 and the posture of the camera 30 can be obtained from the image ID.

[0018] The "obstacle" is an object that inhibits the movement of the mobile body 10 at the current time when remotely operating the mobile body 10. In this embodiment, the obstacle is detected by the following procedure. (Step 1) The past actual environment in a state where there is no obstacle is stored in advance as map point cloud data. (Step 2) Observe the current actual environment in a state where an obstacle may exist as current point cloud data. (Step 3) Compare the map point cloud data and the current point cloud data, and detect an object that does not exist in the map point cloud data in Step 1 but exists in the current point cloud data in Step 2 as an obstacle. The mobile body 10 transmits a signal including the position of the mobile body 10, the obstacle position, and the image ID to the remote control device 20 as information for the operator to remotely control in real time.

[0019] The mobile body 10 includes a self-position estimation unit 11, an obstacle detection unit 12, and an identical image extraction unit 13, and is equipped with a camera 30 and a LiDAR 40 as input means. The remote control device 20 includes a drawing screen generation unit 22, and is equipped with a controller 50 as input means and an external terminal 60 as output means. The external terminal 60 presents the drawing screen generated by the remote control device 20 to the operator. The mobile body 10 is equipped with a storage device 14. The storage device 14 stores past feature points extracted from past image data captured by the camera 30 in the past and their image IDs, and map point cloud data observed by the LiDAR 40 in the past. The remote control device 20 is equipped with a storage device 21. The storage device 21 stores past image data for each image ID and map point cloud data.

[0020] Data is input to the mobile body 10 from sensors (camera 30, LiDAR 40) attached to itself via a USB (Universal Serial Bus) or a LAN (Local Area Network). The input data is acquired according to the data acquisition cycles of various sensors. Here, an image is acquired from the camera 30 and three-dimensional point cloud information is acquired from the LiDAR 40 and input respectively.

[0021] The controller 50 attached to the remote control device 20 is assumed to be, for example, a keyboard. The moving direction is input by pressing the cross keys, and the functions provided in the mobile body 10 are activated by pressing various alphabet keys. The external terminal 60 notifies the operator of the result and is assumed to be a display or the like. Further, the remote control device 20 may be realized in a state where the controller 50 and the external terminal 60 are integrated, such as in a tablet terminal.

[0022] The self-position estimation unit 11 estimates the position of the moving body 10 on the map point cloud data by aligning the map point cloud data of the driving environment pre-recorded in the storage device 14 with the current point cloud data newly acquired by the LiDAR 40. That is, the self-position estimation unit 11 estimates the position and orientation of the moving body 10 based on a location on the map point cloud data in the storage device 14 that is similar to the current point cloud data observed by the LiDAR 40 at the current time.

[0023] The obstacle detection unit 12 extracts a difference based on the result of the self-position estimation unit 11 and the alignment result between the map point cloud data of the driving environment pre-recorded in the storage device 14 and the current point cloud data. The obstacle detection unit 12 detects obstacles such as three-dimensional objects and grooves that have newly appeared in the driving environment from locations where there are differences. That is, the obstacle detection unit 12 detects the positions of obstacles that do not exist in the map point cloud data but exist in the current point cloud data based on locations where differences exist between the current point cloud data and the map point cloud data in the surrounding environment. In addition, the obstacle detection unit 12 detects a detection target registered in advance from the image acquired by the camera 30 and calculates the position of the obstacle on the three-dimensional point cloud in combination with the point cloud information.

[0024] The same image extraction unit 13 compares the past feature points of the past image data recorded in the storage device 14 with the current feature points extracted from the current image data newly acquired by the camera 30. As a comparison result, the same image extraction unit 13 extracts the image ID of the past image data with a large number of matching feature amounts as the image data closest to (identical to) the current image data. That is, the same image extraction unit 13 collates the current feature points extracted from the current image data captured by the camera 30 at the current time with the past feature points in the storage device 14 to obtain past image data of past feature points similar to the current feature points, and extracts the image ID corresponding to the past image data. The moving body 10 transmits the extracted image ID, the estimated position and orientation of the moving body 10, and the detected positions of the obstacles to the remote control device 20 as the surrounding environment information at the current time.

[0025] The drawing screen generation unit 22 generates the first screen 551 and the second screen 558 in FIG. 15. The first screen 551 is a screen in which the position and orientation of the moving body 10 and the positions of obstacles among the current surrounding environment information received from the moving body 10 are arranged in the map point cloud data. The second screen 558 is a screen that presents the past image data in the storage device 21 corresponding to the image ID among the current surrounding environment information received from the moving body 10. In addition, the drawing screen generation unit 22 draws whether there is an abnormality in the moving body 10 based on the time when data was last acquired by various sensors included in the information transmitted by the moving body 10 and the state of the robot, and draws it to the external terminal 60. The external terminal 60 displays the screen generated by the drawing screen generation unit 22 and presents it to the operator.

[0026] FIG. 2 is a diagram showing an example of the actual environment of the 3D map. The moving body 10 uses an external recognition sensor (102 in FIG. 2) such as the mounted LiDAR 40 to acquire three-dimensional point cloud information from the actual environment in FIG. 2.

[0027] FIG. 3 is a diagram showing the map point cloud data generated from the actual environment in FIG. 2. The moving body 10 generates map point cloud data by arranging it in the space of the map point cloud data based on the position information and orientation information (hereinafter "position and orientation") of the moving body 10 (camera 30) when data was acquired in FIG. 2. The generated map point cloud data has points positioned so as to have the same size and shape as the actual environment as shown in 101 in FIG. 3. The map point cloud data reproduces the shape of the actual environment in FIG. 2 by arranging a large number of points as shown in 105. In addition, this map point cloud data has a coordinate system with an arbitrary location in the map as the origin as shown in 104 in FIG. 3. At this time, a different position may be used as the origin from the coordinate system of the LiDAR 40 as shown in 103 in FIG. 2. Although the map point cloud data has been defined as above, it may be replaced with information that can estimate its own position using newly acquired data including position information such as CAD or a 2D map.

[0028] FIG. 4 shows the surrounding environment information that the moving body 10 transmits to the remote control device 20. The surrounding environment information is broadly classified into transmission time, mobile body information, image information, obstacle information, and information of the LiDAR 40. · The transmission time stores the time when the mobile body 10 transmits information. · The mobile body information is composed of a mobile body ID, battery information, information indicating a state abnormality of the mobile body 10, and position information on the map point cloud data. · The image information is composed of the time when an image is captured on the mobile body side, the image ID of the past image data that is most similar to the current image data, and the similarity at that time. The obstacle information is composed of the number of obstacles, IDs indicating the type of each obstacle, the position on the map point cloud data, and information indicating sizes such as width and height. · The information of the LiDAR 40 stores the time when the LiDAR 40 mounted on the mobile body 10 last acquired a point cloud. Note that the information transmitted from the remote control device 20 to the mobile body 10 is the speed of forward / backward movement and left / right turning, and commands that serve as triggers for executing various functions. Since general items are assumed, it is not particularly defined.

[0029] In this way, the surrounding environment information transmitted in real time during operation does not include the large-volume image data itself, but instead includes a small-volume image ID. Thereby, a remote control screen that can be remotely controlled can be presented even in a poor communication environment. On the other hand, for the inspection work of the actual environment, it is necessary to visually check the image data. This is not only for operation information such as the presence or absence of obstacles, but also for the inspector to check the actual environment in detail, such as whether the inspection object is cracked or whether the cable is disconnected. Therefore, for the large-volume current image data, the mobile body 10 continuously records it in its own storage means instead of transmitting it to the remote control device 20 in real time. After the operation in the actual environment is completed and the mobile body 10 is recovered by the inspector, the inspector can browse the current image data recorded in the mobile body 10 afterwards to perform the inspection.

[0030] Figure 5 shows the image information recorded in the storage device 14. The image information is composed of the shooting date and time, weather ID, position and orientation on the map point cloud data, and image feature amounts. The image information is configured for each image. Here, the image feature amount refers to a histogram with the number of Visual-Words as the number of dimensions, where the feature amounts extracted from each image by SIFT or SURF are classified into Visual-Words used in Bag-of-Visual-Words. The image information in the storage device is updated together with the images of the remote control device 20 when the moving body 10 returns to the charging device. The update is performed by first transferring the newly acquired images stored in the moving body 10 to the remote control device 20, recomputing the image feature amounts for all the images recorded in the remote control device, and then transferring the image information from the remote control device 20 to the moving body 10. As a result, image information corresponding to a new date and weather is acquired every time the vehicle travels.

[0031] Figure 6 shows a flowchart of the main process executed by the moving body 10. The moving body 10 receives control information transmitted from the remote control device 20 (S101). The moving body 10 converts the information that can control itself, such as the motor rotation speed, etc., and controls itself (S102).

[0032] After controlling itself in S102, the moving body 10 determines whether or not there has been a change in the position and orientation (S103). If there has been a change in the position and orientation (Yes in S103), the moving body 10 estimates its own position on the map point cloud data based on the information acquired from the LiDAR 40 (S104, details are shown in Figure 7), and extracts the same images (S105, details are shown in Figure 8). If there has been no change in the position and orientation (No in S103), the process proceeds to S106. The moving body 10 transmits the surrounding environment information (details are shown in Figure 4) to the remote control device 20.

[0033] Figure 7 is a flowchart showing the details of the process (S104) in which the self-position estimation unit 11 estimates the position of the moving body 10. This time, the self-position estimation will be described assuming the ICP (Iterative Closest Point), which is a type of point cloud matching method. ICP is a method for estimating a rigid body transformation that minimizes the error of the closest points obtained from two input point clouds. A rigid body transformation refers to a transformation that moves a point cloud only by rotation and translation without deforming the shape of the point cloud. For example, when an arbitrary point position p, a rotation matrix R represented by a 3×3 orthogonal matrix, and a translation vector t are given, the transformation from an arbitrary point p to p' is as shown in (Equation 1). Here, estimating the rigid body transformation corresponds to estimating the rotation matrix R and the translation vector t. p → p' = pR + t …(Equation 1)

[0034] This time, using the map point cloud data as the reference point cloud and the point cloud newly acquired by LiDAR40 as the input point cloud, a transformation for aligning the point cloud newly acquired by LiDAR40 with the map point cloud data is estimated. First, the self-position estimation unit 11 transforms the input point cloud so that it matches the coordinate system of the map point cloud data based on the position and orientation on the latest map point cloud data (S201).

[0035] The self-position estimation unit 11 estimates the points included in the reference point cloud that are the closest points as seen from the input point cloud transformed in S201 (S202). Detection of the closest points can be realized by using a general algorithm such as the k-nearest neighbor method.

[0036] The self-position estimation unit 11 calculates the distances of the associated closest points and removes points that are separated by a certain distance or more, for example, 1 m or more, as invalid correspondences (S203). ICP estimates the transformation by repeating the process of minimizing the distance between points. Therefore, since correct convergence does not occur if points with large errors are included, the self-position estimation unit 11 executes the process of S203.

[0037] The self-position estimation unit 11 estimates the transformation and calculates the error (S204). For calculating the error, first, the self-position estimation unit 11 calculates the centroids of the corresponding points of the input point cloud and the reference point cloud remaining in the process of S203, and respectively μ a , μ bLet it be so. Then, the self - position estimation unit 11 uses each centroid to calculate the covariance matrix Σ ab Let it be so. The centroid is calculated by (Equation 2), and the covariance matrix is calculated by (Equation 3).

[0038]

Equation

[0039] Here, μ is the centroid, p i is a point in the point cloud, Σ ab is the covariance matrix, a i is a point in the reference point cloud, b i is a point in the input point cloud, μ a is the centroid of the reference point cloud where the nearest neighbor point is obtained, μ b is the centroid of the input point cloud where the nearest neighbor point is obtained, respectively. Next, the calculated covariance matrix is subjected to singular value decomposition to obtain U and V. Then, the rotation matrix R and the translation vector t are calculated from U and V obtained here. Finally, the error between the corresponding point clouds is obtained using the rotation matrix R and the translation vector t. Here, the singular value decomposition, the rotation matrix, the translation vector, and the error are calculated by (Equations 4) to (Equation 7), respectively. Here, R is the rotation matrix, t is the translation vector, and E is the error.

[0040]

Equation

[0041] The self - position estimation unit 11 performs convergence determination (S205). In this explanation, two convergence conditions are set. One is that the change amount of the error becomes less than or equal to a preset value, and the other is that the number of iterations reaches a certain number. The change amount of the error is calculated by storing the error value calculated in the previous iteration and calculating the change amount. The self - position estimation unit 11 estimates the position of the moving object on the map point cloud data by repeating the processes of S201 to S205 until the convergence conditions are satisfied.

[0042] FIG. 8 is a flowchart showing details of the same-image extraction process (S105) by the same-image extraction unit 13. The same-image extraction unit 13 extracts image feature points from the newly captured image and describes local feature amounts (S301). The same-image extraction unit 13 determines to which feature cluster of which VisualWord the extracted local feature amount belongs, and generates a feature histogram of the VisualWord based on the result. The feature cluster of the VisualWord is calculated based on the images recorded in the storage device 21. Since the Bag-of-Visual-Word described here is a known method, details are omitted.

[0043] The same-image extraction unit 13 compares the feature histogram generated in S302 with the image feature histogram recorded in the storage device 14 and calculates the similarity (S303). For example, there is Histogram Intersection that totals the overlapping parts of two histograms for calculating the similarity.

[0044] The same-image extraction unit 13 extracts the image ID of the past image data that is closest (has a high similarity) to the current image data (S304). At this time, when there are images captured while driving at different dates and times or in different weather conditions, the top 10 images with high similarity may be extracted, and an image may be selected from those with a close date and time, matching weather, and close position and orientation stored in the image information. Also, before performing remote control, only image information with a close date and time and weather may be selected in advance to create a feature cluster and a histogram of the feature cluster and store them in the storage device 14. That is, in the storage device 14, for the past image data for each image ID, the shooting conditions including the weather and date and time at the time of shooting are also stored in association. And in S304, the same-image extraction unit 13 obtains past image data with similar shooting conditions in the process of collating past feature points similar to the current feature points. Thereby, even when photographing the same object, it is possible to prevent the image data from greatly differing depending on the shooting conditions (morning or night, sunny or rainy, etc.) and the accuracy degradation of the similarity calculation process from occurring.

[0045] FIG. 9 is a flowchart showing details of the obstacle detection process (S106) by the obstacle detection unit 12. The obstacle detection unit 12 converts the newly acquired point cloud so as to match the coordinate system of the map point cloud data, using the self-position of the moving body 10 estimated in S104 (S401).

[0046] The obstacle detection unit 12 detects the difference between the map point cloud data and the newly acquired point cloud (S402). The detection of the difference is performed using, for example, a KDTree. First, the map point cloud data is spatially divided by the KDTree. Then, for each point of the input point cloud, the nearest neighbor point is searched. If there is a point of the map point cloud data within 0.05 m, there is no difference. If there is no point of the map point cloud data, it is extracted as a difference. Next, the point cloud of the difference extracted here is clustered to detect newly appeared obstacles on the map point cloud data.

[0047] FIG. 10 is a diagram showing an example of an obstacle to be detected and information to be drawn. By the process of S402, when the moving body 453 is facing the direction of the human 451 and the triangular cone 452, the point clouds of the human 451 and the triangular cone 452 are extracted as a difference (obstacle). The obstacle detection unit 12 detects a target (obstacle) to be detected during remote operation from the current image data (S403). For the detection of the target, a discriminator that has learned the target to be detected in advance may be used, such as SVN or deep learning. Since these methods are well-known, the description is omitted.

[0048] The obstacle detection unit 12 calculates the patrol target position detected in S403 (S404). The calculation of the patrol target position is, for example, the following procedure. (Step 1) Project the point cloud acquired by the LiDAR 40 onto the image captured by the camera 30. (Step 2) Set the center of gravity calculated from the values on the coordinate system of the map point cloud data of the point cloud projected onto the patrol target as the position of the obstacle. (Step 3) Set the value in the Z-axis direction of the point at the highest position as the height of the obstacle. (Step 4) Set the difference in the Y-axis direction as the width of the obstacle.

[0049] (Step 1) is performed using the following (Equation 8), (Equation 9), and (Equation 10). (Equation 8) is an equation for converting the coordinate system of the point cloud acquired by LiDAR 40 to the coordinate system of the image. (Equation 9) and (Equation 10) are equations for projecting the point cloud converted by (Equation 8) assuming an image captured by the camera 30 with an ideal pinhole model.

[0050]

Number

[0051] p l indicates the point of LiDAR 40 in the coordinate system of the map point cloud data. p c indicates the point position as seen from the camera 30. R lc indicates the rotation matrix for converting from the coordinate system of LiDAR 40 to the coordinate system of the camera 30. t lc indicates the translation amount for converting from the coordinate system of LiDAR 40 to the coordinate system of the camera 30. x i and y i indicate the coordinates on the image. x l y l z l indicate the point cloud coordinates of LiDAR 40 converted to the coordinate system as seen from the camera 30. f x f y indicate the focal length of the camera 30. c x c y indicate the center position of the image. Here, when the obstacle detected in S403 is projected onto the detection target in the image, assign the ID of the detection target to the obstacle. Otherwise, assign an ID indicating outside the detection target to the obstacle.

[0052] The obstacle detection unit 12 calculates the presence or absence of movement and the speed of the detected object (S405). The estimation of the presence or absence of movement calculates the difference from the position of the same object detected a certain time ago, classifies it as a moving obstacle when there is a change in position, and calculates the moving speed from the difference and the time. The association between obstacles may use, for example, the Hungarian algorithm or the Kalman filter. Since these methods are well-known, the explanation is omitted. The obstacle detection unit 12 stores the obstacle information in the storage area together with the time (S406).

[0053] FIG. 12 is a flowchart of the drawing screen generation unit 22. The drawing screen generation unit 22 receives the signal transmitted from the moving body 10 at S107 (S501). Based on the moving body information among the received information, the drawing screen generation unit 22 draws the position and orientation on the map point cloud data stored in the storage device 21 (S502). FIG. 11 is a diagram showing the map point cloud data 454 obtained by performing an aerial view conversion of the site in FIG. 10. In S502, the drawing screen generation unit 22 draws the moving body 457 on the map point cloud data 454 in a form in which the direction and position of the moving body 10 can be confirmed.

[0054] The drawing screen generation unit 22 draws obstacles based on the obstacle information (S503). For each ID, the obstacle draws a corresponding model or symbol on the map point cloud data based on the position, height, and width. In addition, a moving obstacle is drawn so that the moving direction is in the direction of the arrow and the speed is represented by the length based on the speed information. An example is shown in FIG. 11. For example, the triangular cone 452 detected in FIG. 10 is drawn as a triangle with the combined width and height as shown in 456 of FIG. 11. Also, the human 451 detected in FIG. 10 is drawn with an ellipse and an arrow representing the moving direction and speed as shown in FIG. 11.

[0055] FIG. 13 is a diagram showing the imaging range of an image when photographing the site in FIG. 10. The drawing screen generation unit 22 draws the imaging range 460 of the image as shown in FIG. 13 (S504). For example, when the imaging range 460 can be captured by the camera 30 mounted on the moving body 10, the color of the points of the map point cloud data included in the imaging range 460 is changed from the color of the points of the map point cloud data not included in the imaging range 460 and presented to the operator. The detection of the points whose color is to be changed is obtained by the following procedure. (Step 1) Convert the map point cloud data into a coordinate system with the position of the moving body 10 as the origin using (Equation 1). (Step 2) Project the point cloud extracted by the following (Equation 11) onto the image using the equations of (Equations 8) to (10). Here, p i is a point in the map point cloud data with the position and orientation as the origin, ||p i || is the distance from the origin of each point, and γ is a numerical value of 0 or less.

[0056]

Equation

[0057] FIG. 14 is a plan view of the imaging range 460 of the image in FIG. 13 after top-down conversion. Since the point cloud projected onto the image by the process of S504 is the point cloud within the range that can be captured by the camera 30, it is only necessary to change the color of this point cloud. Thereby, the drawing screen generation unit 22 can draw the imaging range 460 in FIG. 13 corresponding to the imaging range 461 in FIG. 14. This time, the imaging range is drawn by changing the color of the point cloud, but the drawing screen generation unit 22 may also draw a quadrangular pyramid of the imaging range 460 on the map based on the field of view angle of the camera 30.

[0058] The drawing screen generation unit 22 determines the communication status based on the time of the signal (S505). Specifically, the drawing screen generation unit 22 calculates the time taken for communication by comparing the timestamp recorded when the mobile body 10 transmitted information in the surrounding environment information with the timestamp when the remote control device 20 received it. The time taken for communication (delay or frame drop) is drawn on the screen and notified to the user. Also, the drawing screen generation unit 22 warns the user by drawing with an icon whether it is slower than expected compared to the communication time assumed by the system.

[0059] The drawing screen generation unit 22 determines a failure based on the acquisition time of the sensor (S506). Specifically, the drawing screen generation unit 22 checks whether the sensor is operating normally by comparing the difference between the data acquisition time of the sensor (camera 30, LiDAR 40) recorded in the surrounding environment information and the previous acquisition time. This is to confirm, for example, a malfunction such as returning an old image stored in the memory of the camera when the camera 30 is malfunctioning. The drawing screen generation unit 22 also warns the user about the sensor failure by drawing it with an icon. In this way, based on the timestamp recorded at the time of transmission and reception of the current surrounding environment information, the drawing screen generation unit 22 detects at least one of the poor state of the communication environment used for transmission and reception and the failure state of the LiDAR 40 or the camera 30, and displays the detected result state as a warning.

[0060] FIG. 15 is a screen diagram of an operation screen that presents correct surrounding environment information acquired from the site of FIG. 10 to the user. The drawing screen generation unit 22 reads an image corresponding to the image ID stored in the surrounding environment information from the storage device 21 and draws it on the remote control screen 550 (S507). The remote control screen 550 has a first screen 551 displayed on the left side, a second screen 558 displayed on the right side, and an icon group displayed on the lower side. The icon group is composed of the following icons. · Reference numeral 552 indicates the state of the mobile body 10. · Reference numeral 553 indicates the remaining battery level of the mobile body 10. · Symbol 554 indicates the presence or absence (degree) of communication delay. · Symbol 555 indicates the state (normal or abnormal) of the camera 30 mounted on the mobile body 10. · Symbol 556 indicates the state (normal or abnormal) of the LiDAR 40 mounted on the mobile body 10. · Symbol 557 indicates that an abnormality has occurred in the mobile body 10, sensors, etc.

[0061] Note that the abnormalities of the sensors (camera 30, LiDAR 40) of the mobile body 10 are, for example, abnormalities in the time of acquiring the latest data (data acquisition abnormality), and the availability of connection to the sensors (communication abnormality). The abnormalities of the mobile body 10 are, for example, abnormalities in the time of acquiring the latest data (data update abnormality), abnormalities in the battery state, and abnormalities in the motor. Also, although an abnormality is indicated by the icon shown by symbol 557 this time, it may be presented to the operator by changing the color of the abnormal part of the mobile body 10 shown by symbol 552.

[0062] The first screen 551 includes a mobile body display 457 that shows the position and orientation (the upward orientation in FIG. 15) received as the surrounding environment information. The first screen 551 also includes the shooting range 461 of FIG. 14 based on the position of the mobile body display 457. Furthermore, the first screen 551 also includes the following displays for each obstacle received as the surrounding environment information. (Display of the obstacle 1) The triangular cone 452 in FIG. 10 is displayed as the triangular cone display 456 in FIG. 15. This triangular cone display 456 is a triangular display imitating the triangular cone based on the obstacle ID = the ID of the triangular cone specified by the obstacle detection unit 12. In this way, in addition to the position of the obstacle, the obstacle detection unit 12 specifies the obstacle ID corresponding to the type of the obstacle and transmits the specification result to the remote control device 20. Then, the drawing screen generation unit 22 may draw the obstacle corresponding to the received obstacle ID on the first screen 551.

[0063] (Obstacle Display 2) The human 451 in Fig. 10 is displayed as the human display 455 in Fig. 15. This human display 455 also displays a right arrow indicating that the human is moving in the right direction based on the movement (right direction) of the obstacle identified by the obstacle detection unit 12. In this way, in addition to the position of the obstacle, the obstacle detection unit 12 detects the size of the obstacle, the shape of the obstacle, and the presence or absence of the movement of the obstacle, and transmits the detection results to the remote control device 20. Then, the drawing screen generation unit 22 may draw the received detection results on the first screen 551.

[0064] On the second screen 558, past image data corresponding to the image ID stored in the peripheral environment information is displayed. Here, the operator can recognize the correctness of the position and orientation of the moving object display 457 in the first screen 551 by comparing the first screen 551 based on the observation data of the LiDAR 40 with the second screen 558 based on the shooting data of the camera 30. For example, the operator confirms the corner 551C of the room at the left end of the shooting range 461 from the first screen 551. Next, the operator confirms the corner 558C of the room on the left side of the past image data from the second screen 558. Therefore, the operator can confirm that the position and orientation of the moving object display 457 in the first screen 551 are correct by the alignment of the position of the corner 551C in the shooting range 461 and the position of the corner 558C.

[0065] Fig. 16 is a screen diagram of a control screen that presents incorrect peripheral environment information at the same site as Fig. 15 to the user. The correct moving object display 457 in Fig. 15 is replaced by the incorrect moving object display 457X in Fig. 16. The correct shooting range 461 in Fig. 15 is replaced by the incorrect shooting range 461X in Fig. 16. For example, the operator checks from the first screen 551 that the corner 551C of the room is not included in the shooting range 461X. Next, the operator checks the corner 558C of the room on the left side of the past image data from the second screen 558. Therefore, since the position of the corner 551C outside the shooting range 461X and the position of the corner 558C are inconsistent, the operator can confirm that the moving body display 457X and the shooting range 461X in the first screen 551 are incorrect.

Embodiment

[0066] FIG. 17 is a configuration diagram of the remote control system 100 according to Embodiment 2. The remote control system 100 in FIG. 17 adds a communication environment determination unit 15 for the moving body 10 and a communication delay determination unit 23 for the remote control device 20 to the remote control system 100 in FIG. 1. The communication environment determination unit 15 determines whether the communication environment of the surrounding environment is good (e.g., 5G) or bad (e.g., 3G). A good communication environment means, for example, a state where a communication capacity sufficient to transfer current image data can be ensured. Since there are known techniques for determining 3G, LTE (Long Term Evolution), 5G, etc. executed by the communication environment determination unit 15, the description is omitted. The determination process of the communication environment determination unit 15 is executed at regular intervals to monitor the communication environment. The moving body 10 in FIG. 17 performs the processing shown in Embodiment 1 in an environment with a bad communication environment. On the other hand, in an environment with a good communication environment, the moving body 10 does not execute S105 in FIG. 1 and directly transmits the captured current image data as the information of the surrounding environment in S107.

[0067] FIG. 18 shows the information that the moving body 10 transmits to the operator when it is determined that the communication environment is good. The communication environment determination unit 15 transmits the surrounding environment information with the current image data added to the surrounding environment information in FIG. 4 to the remote control device 20.

[0068] The communication delay determination unit 23 determines the time required for communication from the mobile body 10 to the remote control device 20 based on the timestamps at the time of transmission and reception. For example, assuming that the upper limit of the delay in the case of remote control is 50 ms, if the difference in timestamps is 50 ms or more, the communication environment determination unit 15 may switch so as not to transmit an image even in a 5G environment. Alternatively, even if the difference in timestamps is 50 ms or more, the communication environment determination unit 15 may calculate an approximate communication speed from the capacity of the surrounding environment information and the time taken for communication, and reduce the image size to be transmitted (from 1920×1080 to 640×480, etc.) so that it fits, and then transmit it.

[0069] FIG. 19 is a display screen diagram of the remote control device 20 during a period with good communication environment. The basic configuration of the screen in FIG. 19 is the same as that in FIG. 15. The reference numeral 550 indicates the entire remote control screen, and the first screen 551 is one in which the position and orientation and obstacle information are drawn on the map point cloud data. The drawing screen generation unit 22 presents the current image data received as surrounding environment information on the second screen 561 in FIG. 19 instead of the past image data presented on the second screen 558 in FIG. 15. On the second screen 561, objects that become obstacles when the mobile body 10 moves, such as triangular cones and humans, appear as the latest surrounding environment information. Also, the icon 560 in FIG. 19 indicates that the communication environment is 5G, and the user can recognize that the second screen 561 is current image data.

[0070] FIG. 20 is a display screen diagram of the remote control device 20 during a period with poor communication environment. The drawing screen generation unit 22 presents past image data on the second screen 571 in FIG. 20 as well, similar to the second screen 558 in FIG. 15. At present, although there are obstacles such as triangular cones and humans, those obstacles do not appear in the past image data on the second screen 571. Also, the icon 570 in FIG. 20 indicates that the communication environment is 3G, and the user can recognize that the second screen 571 is past image data. According to the above-described Example 2, even in a situation where the mobile body 10 moves back and forth between an indoor area with a good communication environment and an indoor area with a poor communication environment, the information necessary for remote control can be presented by a method suitable for the communication environment at the current position.

Example

[0071] FIG. 21 is a configuration diagram of Example 3. The remote control system 100 in FIG. 21 adds a self-position correction unit 16 and a position re-estimation unit 17 to the remote control system 100 in FIG. 1. The self-position correction unit 16 corrects the position of the mobile body based on the given position information (details will be described in the explanation of FIG. 22 below). That is, the self-position correction unit 16 reflects, on the first screen, the result of correcting the position and orientation of the mobile body 10 in the current surrounding environment information received from the mobile body 10 with the position and orientation of the mobile body 10 corresponding to the image ID in the current surrounding environment information received from the mobile body 10. Therefore, in the storage device 14, for the past image data for each image ID, the position and orientation of the mobile body 10 at the time of shooting the past image data are also stored in association.

[0072] The position re-estimation unit 17 re-estimates the position of the mobile body 10 when the user determines that the self-position estimation of the mobile body 10 has failed on the mobile body side. "Re-estimation" by the position re-estimation unit 17 is not a process of repeating the same estimation process, but a process of switching between two types of estimation processes. Specifically, the position re-estimation unit 17 switches from the self-position estimation process of the mobile body 10 based on the measurement data of the LiDAR 40 (the process of the self-position estimation unit 11 shown in FIG. 7) to the self-position estimation process of the mobile body 10 based on the shooting data (image ID) of the camera 30.

[0073] That is, the position re-estimation unit 17 receives an input from the user who has viewed the first screen and the second screen, indicating that the position and orientation of the moving object 10 in the first screen are incorrect, and obtains the position and orientation of the corresponding moving object 10 from the image ID of the past image data presented on the second screen. The position re-estimation unit 17 reflects the obtained position and orientation of the moving object 10 on the first screen. Therefore, in the storage device 14, for the past image data for each image ID, the position and orientation of the moving object 10 at the time of shooting the past image data are also stored in association with each other. Note that the measurement data of the LiDAR 40 has high accuracy in indoor areas and passages where objects exist in the surrounding environment of the moving object 10, but has low accuracy in open spaces with few objects in the wild. Therefore, in situations where the LiDAR 40's measurement data is not good, by using the two types of position estimation processes complementarily so as to perform position estimation using the shooting data of the camera 30, it is possible to prevent the position estimation from failing and making it difficult to operate.

[0074] FIG. 22 is a flowchart including the processing of the self-position correction unit 16. The flowchart of FIG. 22 adds S108 and S109 to the flowchart of FIG. 6. Since there are no changes in the processing of S101 to S107, the description thereof is omitted here.

[0075] The self-position correction unit 16 estimates the relative position and orientation from the extracted same image (S108). The calculation of the relative position and orientation is performed by extracting feature points from two sets of images and performing association. The association of the feature points may be performed by comparing the feature amounts of the image feature points or by comparing the pixel values around the image feature points. For example, in the case of ORB, the comparison of the feature amounts can be calculated by taking the exclusive logical sum of the feature amounts described in binary, and the smaller the difference, the more corresponding the feature points are. In the case of corner points detected by a method without feature amounts such as Harris, the association can be performed by comparing the surrounding pixels of the image feature points such as the SSD (Sum of Squared Difference) method and the KLT (Kanade-Lucas-Tomasi Feature Tracker) method. Since the methods described here are well-known, the description is omitted.

[0076] Next, a fundamental matrix is obtained using the five-point method from the obtained correspondence relationship of feature points, and a rotation matrix is obtained from the fundamental matrix. Here, since the movement distance cannot be obtained, it is replaced by the distance between the position where a new image is taken and the position of the image for gymnastics, which is obtained by subtracting the movement distance obtained from the distance obtained from the position one time step before and the position information attached to the image information, such as the motor rotation amount of the moving body 10.

[0077] By this series of processes, the relative position and orientation from the position where the extracted same image was taken to the position where a new image is taken can be estimated. By adding this relative movement amount to the position and orientation that the same image has as information, the position where a new image is taken can be estimated. Since the five-point method is also a known method, the explanation is omitted.

[0078] Also, although the relative position was calculated using the change in the pose estimated by the five-point method this time and the distance between the shooting positions, the position where a new image is taken as seen from the same image may be directly obtained by solving the Perspective-n-Point (PNP) problem. In this case, the three-dimensional positions of the past feature points included in the past image data are estimated in advance by Structure From Motion or the like. Then, based on the correspondence relationship when associating the current feature points extracted from the current image data with the image feature points of the same image, the three-dimensional positions of the image feature points estimated in advance are associated with the image feature points of the newly taken image. And based on the relationship between the image feature points in this image and the three-dimensional positions of the image feature points, the position and orientation can be obtained by decomposition. Since the position can be estimated by any method, any method can be used.

[0079] The self-position correction unit 16 integrates the position information estimated in S108 and the position information estimated in S104 (S109). For example, an extended Kalman filter may be used for the integration of the position information. The extended Kalman filter is a type of Kalman filter and is used for self-position estimation of a non-linear dynamic system. Since the extended Kalman filter is a well-known method, the description thereof is omitted. By integrating the position and orientation estimated based on the LiDAR 40 using the extended Kalman filter and the position and orientation estimated based on the position information given to the image information, a more robust position and orientation against disturbances can be obtained. That is, the storage device 14 holds the position and orientation corresponding to the past feature points at the time of past visits. Then, at the current time of re-visiting, the self-position correction unit 16 corrects the position and orientation displayed by the drawing screen generation unit 22 using the position and orientation held in the storage device 14.

[0080] FIG. 23 is a flowchart showing the processing of the position re-estimation unit 17. The position re-estimation unit 17 extracts the same image (S105) and estimates the relative position and orientation from the extracted same image (S108). The position re-estimation unit 17 estimates its own position on the map using the LiDAR 40 with the position and orientation estimated in S108 as the initial position (S104). The process of FIG. 23 is executed each time an arbitrary key on the keyboard connected as the controller 50 is input when the operator recognizes a discrepancy between the moving body position on the map point cloud data shown in FIG. 15 and the extracted same image.

[0081] FIG. 24 is a diagram showing an example of the layout of the drawing screen in the third embodiment. It may be implemented as a button on the remote control screen 550. Also, since the self-position correction unit 16 has few features obtained from the image and fails to estimate the position and orientation especially indoors, such as in a corridor, it may be made switchable between ON and OFF. As described above, more robust self-position estimation and the ability to return when the self-position estimation fails are achieved.

[0082] FIG. 25 is a hardware configuration diagram of the mobile body 10 and the remote control device 20 included in the configuration of the remote control system 100. The mobile body 10 and the remote control device 20 are each configured as a computer 900 having a CPU 901, a RAM 902, a ROM 903, an HDD 904, a communication I / F 905, an input / output I / F 906, and a media I / F 907. The communication I / F 905 is connected to an external communication device 915. The input / output I / F 906 is connected to an input / output device 916. The media I / F 907 reads and writes data from and to a recording medium 917. Further, the CPU 901 controls each processing unit by executing a program (also called an application or an app for short) read into the RAM 902. And this program can be distributed via a communication line or recorded on a recording medium 917 such as a CD-ROM and distributed. Note that some or all of the functions of the mobile body 10 and the remote control device 20 may be realized using hardware such as an FPGA (Field Programmable Gate Array) or an ASIC (application specific integrated circuit).

[0083] As described above, the embodiments of the present invention have been described. However, the present invention is not limited to the above-described embodiments, and various modifications can be made without departing from the scope described in the claims. For example, the above-described embodiments have described the present invention in detail, and it is not necessary to have all the configurations described. Also, it is possible to add the configurations of other embodiments to the configuration. In addition, for a part of the configuration, addition, deletion, and replacement are possible.

[0084] Also, for a part of the configuration of each embodiment, addition, deletion, and replacement with other configurations are possible. Further, each of the above configurations, functions, processing units, processing means, etc. may be realized in hardware, for example, by designing them with an integrated circuit or the like. In addition, each of the above-described configurations, functions, etc. may be implemented by software by a processor interpreting and executing a program for realizing each function.

[0085] Information such as programs, tables, and files for realizing each function can be stored in a memory, a recording device such as a hard disk or an SSD (Solid State Drive), or a recording medium such as an IC (Integrated Circuit) card, an SD card, or a DVD (Digital Versatile Disc). Also, the cloud can be utilized. In addition, the control lines and information lines show those considered necessary for explanation, and not all control lines and information lines are necessarily shown on the product. In practice, it may be considered that almost all components are interconnected. Furthermore, the communication means for connecting each device is not limited to a wireless LAN and may be changed to a wired LAN or other communication means.

Explanation of Signs

[0086] 10 Moving body 11 Self-position estimation unit 12 Obstacle detection unit 13 Same-image extraction unit 14 Storage device (first storage device) 15 Communication environment determination unit 16 Self-position correction unit 17 Position re-estimation unit 20 Remote control device 21 Storage device (second storage device) 22 Drawing screen generation unit 23 Communication delay determination unit 30 Camera 40 LiDAR 50 Controller 60 External terminal 100 Remote control system 551 First screen 558 Second screen

Claims

1. A remote control system for remotely controlling a mobile body equipped with a LiDAR that observes the surrounding environment of the mobile body as point cloud data and a camera that captures the surrounding environment as image data, wherein the mobile body has a self-position estimation unit, an obstacle detection unit, and an identical image extraction unit, and stores past feature points extracted from past image data captured by the camera in the past and map point cloud data observed by the LiDAR in the past in a first storage device, the remote control device has a drawing screen generation unit, and stores the past image data for each image ID and the map point cloud data in a second storage device, the self-position estimation unit estimates the position and orientation of the mobile body based on a location on the map point cloud data in the first storage device that is similar to the current point cloud data observed by the LiDAR at the current time, the obstacle detection unit detects the position of an obstacle that does not exist in the map point cloud data but exists in the current point cloud data based on a location where there is a difference between the current point cloud data and the map point cloud data in the surrounding environment, the identical image extraction unit obtains the past image data of the past feature points similar to the current feature points by comparing the current feature points extracted from the current image data captured by the camera at the current time with the past feature points in the first storage device, and extracts the image ID corresponding to the past image data, the mobile body transmits the extracted image ID, the estimated position and orientation of the mobile body, and the position of the detected obstacle to the remote control device as the surrounding environment information at the current time, the drawing screen generation unit, a first screen in which the position and orientation of the mobile body and the position of the obstacle among the surrounding environment information at the current time received from the mobile body are arranged on the map point cloud data, and a second screen that presents the past image data in the second storage device corresponding to the image ID among the surrounding environment information at the current time received from the mobile body, and is characterized by a remote control system.

2. In addition to the position of the obstacle, the obstacle detection unit detects the size of the obstacle, the shape of the obstacle, and the presence or absence of movement of the obstacle, and transmits the detection results to the remote control device, and the drawing screen generation unit is characterized by drawing the received detection results on the first screen The remote control system according to Claim 1.

3. In addition to the position of the obstacle, the obstacle detection unit identifies an obstacle ID corresponding to the type of the obstacle, and transmits the identification result to the remote control device. The drawing screen generation unit is characterized by drawing an obstacle corresponding to the received obstacle ID on the first screen. The remote control system according to claim 1.

4. In the first storage device, for the past image data for each image ID, the position and orientation of the moving body at the time of shooting the past image data are further stored in association therewith. The moving body further includes a position re-estimation unit. The position re-estimation unit receives an input from a user who has viewed the first screen and the second screen, indicating that the position and orientation of the moving body in the first screen are incorrect, obtains the corresponding position and orientation of the moving body from the image ID of the past image data presented on the second screen, and reflects the obtained position and orientation of the moving body on the first screen. The remote control system according to claim 1.

5. The drawing screen generation unit is characterized by arranging the shooting range of the camera from the position and orientation of the moving body on the first screen in addition to the position and orientation of the moving body. The remote control system according to claim 1.

6. Based on the time stamp recorded at the time of transmission and reception of the current surrounding environment information, the drawing screen generation unit detects at least one of a poor state of the communication environment used for transmission and reception and a failure state of the LiDAR or the camera, and displays the detected state as a warning. The remote control system according to claim 1.

7. In the first storage device, for the past image data for each image ID, the position and orientation of the moving body at the time of shooting the past image data are further stored in association therewith. The moving body further includes a self-position correction unit. The self-position correction unit reflects, on the first screen, a result of correcting the position and orientation of the moving body in the current surrounding environment information received from the moving body with the position and orientation of the moving body corresponding to the image ID in the current surrounding environment information received from the moving body. The remote control system according to claim 1.

8. A remote control method executed by a remote control system for remotely controlling a moving body equipped with a LiDAR for observing the surrounding environment of the moving body as point cloud data and a camera for shooting the surrounding environment as image data. The mobile body has a self-position estimation unit, an obstacle detection unit, and an identical image extraction unit, and stores past feature points extracted from past image data captured by the camera in the past and map point cloud data observed by the LiDAR in the past in a first storage device. The remote control device has a drawing screen generation unit, and stores the past image data for each image ID and the map point cloud data in a second storage device. The self-position estimation unit estimates the position and orientation of the mobile body based on a location on the map point cloud data in the first storage device that is similar to the current point cloud data observed by the LiDAR at the current time. The obstacle detection unit detects the positions of obstacles that do not exist in the map point cloud data but exist in the current point cloud data based on locations where there are differences between the current point cloud data and the map point cloud data in the surrounding environment. The identical image extraction unit obtains the past image data of the past feature points similar to the current feature points by comparing the current feature points extracted from the current image data captured by the camera at the current time with the past feature points in the first storage device, and extracts the image ID corresponding to the past image data. The mobile body transmits the extracted image ID, the estimated position and orientation of the mobile body, and the detected positions of the obstacles to the remote control device as the surrounding environment information at the current time. The drawing screen generation unit generates a first screen in which the position and orientation of the mobile body and the positions of the obstacles among the surrounding environment information at the current time received from the mobile body are arranged on the map point cloud data, and a second screen that presents the past image data in the second storage device corresponding to the image ID among the surrounding environment information at the current time received from the mobile body. A remote control method characterized by this.

Images