Moving body remote steering assistance system, moving body remote steering assistance method, and program

Abstract

Provided is a moving body remote steering assistance system for determining an imaging means for preferentially displaying on a control screen of an operator. This moving body remote steering assistance system comprises: a moving body position / orientation estimation means for estimating the position and orientation of a moving body; a conveyance object position / orientation estimation means for estimating the position and orientation of a conveyance object; a cargo handling work estimation means for estimating, in accordance with the estimated position and orientation of the moving body and the estimated position and orientation of the conveyance object, whether the moving body is performing cargo handling work; and a preferred imaging means determination means for determining, in response to an estimation that the moving body is performing cargo handling work, an imaging means for preferentially displaying on a control screen of an operator.

Description

Mobile body remote control support system, mobile body remote control support method and program

[0001] The present disclosure relates to a mobile object remote control assistance system, a mobile object remote control assistance method, and a program.

[0002] Patent Document 1 discloses a remote control system for a forklift that, when displaying an image captured by a camera on a display unit, also displays a display for selecting a detected pallet.

[0003] Japanese Patent Application Laid-Open No. 2019-204353

[0004] However, there is no disclosure of selecting an imaging means to be preferentially displayed from among multiple display screens. Therefore, an object of the present disclosure is to provide a mobile object remote control support system etc. that determines an imaging means to be preferentially displayed on an operator's control screen.

[0005] The mobile object remote control support system of the present disclosure is a mobile object remote control support system comprising: a mobile object position and attitude estimation means for estimating the position and attitude of a mobile object; a transport object position and attitude estimation means for estimating the position and attitude of a transport object; a loading and unloading operation estimation means for estimating whether the mobile object is performing loading and unloading operations based on the estimated position and attitude of the mobile object and the position and attitude of the transport object; and a priority image capture means determination means for determining an image capture means to be preferentially displayed on an operator's control screen based on the estimation that loading and unloading operations are in progress.

[0006] The mobile body remote control support method disclosed herein is a mobile body remote control support method in which an information processing device estimates the position and attitude of a mobile body, estimates the position and attitude of an object to be transported, estimates whether the mobile body is engaged in loading and unloading operations based on the estimated position and attitude of the mobile body and the position and attitude of the object to be transported, and determines an imaging means to be preferentially displayed on an operator's control screen based on the estimation that loading and unloading operations are being performed.

[0007] The program disclosed herein is a program that causes an information processing device to execute the following: estimate the position and attitude of a moving body; estimate the position and attitude of an object to be transported; estimate whether the moving body is engaged in loading and unloading operations based on the estimated position and attitude of the moving body and the position and attitude of the object to be transported; and determine an imaging means to be preferentially displayed on an operator's control screen based on the estimation that loading and unloading operations are in progress.

[0008] The present disclosure provides a mobile object remote control support system and the like that determines which imaging means to display preferentially on the operator's control screen.

[0009] FIG. 1 is a schematic diagram of a related mobile object control system and a mobile object remote control support system according to the present disclosure. FIG. 2 is a block diagram showing a first configuration of a mobile object remote control support system according to the present disclosure. FIG. 3 is a flowchart of a mobile object remote control support method according to the present disclosure. FIG. 4 is an example of an imaging means and an example of a display of a mobile object remote control support system according to the present disclosure. FIG. 5 is a diagram showing an example of determining whether a driving operation is in progress and whether a loading and unloading operation is in progress according to the present disclosure. FIG. 6 is a diagram showing an example of determining whether a driving operation is in progress and whether a loading and unloading operation is in progress according to the present disclosure. FIG. 7 is a diagram showing an example of determining whether a driving operation is in progress and whether a loading and unloading operation is in progress according to the present disclosure. FIG. 8 is a diagram showing an example of determining whether a driving operation is in progress and whether a loading and unloading operation is in progress according to the present disclosure. FIG. 9 is a diagram showing an example of a display by an imaging device selected during driving operation and an example of a display by an imaging device selected during loading and unloading operation according to the present disclosure. FIG. 10 is a block diagram showing a second configuration of a mobile object remote control support system according to the present disclosure. FIG. 11 is a diagram showing an example of a selection state of an imaging means of a mobile object remote control support system according to the present disclosure. FIG. 12 is a block diagram showing a configuration of an information processing device according to the present disclosure.

[0010] (Description of a Mobile Object Remote Control Support System According to a First Embodiment) FIG. 1 is a schematic diagram of a related mobile object control system and a mobile object remote control support system according to the present disclosure. FIG. 2 is a block diagram showing a first configuration of a mobile object remote control support system according to the present disclosure. FIG. 4 is a diagram showing an example of an imaging means and an example of a display of a mobile object remote control support system according to the present disclosure. FIG. 5A is a diagram showing an example of determining whether a traveling operation is in progress or a cargo handling operation is in progress according to the present disclosure. FIG. 5B is a diagram showing an example of determining whether a traveling operation is in progress or a cargo handling operation is in progress according to the present disclosure. FIG. 5C is a diagram showing an example of determining whether a traveling operation is in progress or a cargo handling operation is in progress according to the present disclosure. FIG. 5D is a diagram showing an example of determining whether a traveling operation is in progress or a cargo handling operation is in progress according to the present disclosure. FIG. 6 is a diagram showing an example of a display by an imaging device selected during traveling operation and an example of a display by an imaging device selected during cargo handling operation according to the present disclosure. The mobile object remote control support system according to the first embodiment will be described with reference to FIGS. 1, 2, and 4 to 6.

[0011] The mobile object is described as a forklift in this example. However, the mobile object is not limited to a forklift and may be other remotely controlled equipment such as a drone or heavy machinery. The object to be transported is, for example, a pallet.

[0012] The upper diagram of Fig. 1 illustrates a situation in which one operator rides a forklift for each forklift to perform work in a factory or the like. The lower diagram of Fig. 1 illustrates a remote control system in which one operator in a remote location in the factory operates multiple forklifts in the factory. As shown in the upper diagram of Fig. 1, the related mobile object control system requires an operator to ride a forklift to perform work in the field, such as in a factory. As shown in the lower diagram of Fig. 1, the mobile object remote control support system according to the present disclosure is a system for remotely controlling multiple mobile objects at a location away from the factory.

[0013] In remote control, it is important to capture and display the situation of the actual site. The mobile object remote control system according to the present disclosure relates to a method for displaying the situation of the actual site.

[0014] As shown in Figure 2, the mobile object remote control support system 200 according to the present disclosure includes a driving operation estimation unit 201, a mobile object position and attitude estimation unit 202, a transport object position and attitude estimation unit 203, a loading and unloading operation estimation unit 204, and a priority imaging means determination unit 205.

[0015] The driving operation estimation unit 201 estimates whether driving operation is in progress. Whether driving operation is in progress can be determined, for example, by obtaining a control command. Control commands include a throttle lever, brake pedal, steering, reach lever, tilt lever, and lift lever. The throttle lever is for moving forward or backward. The brake lever is for stopping. The steering lever is for changing the direction of movement. The reach lever is for extending or retracting the grip of the moving object. The tilt lever is for changing the angle of the grip. The lift lever is for lifting the grip. In remote work, control commands are issued, so it is possible to determine whether driving operation is in progress.

[0016] The forks of a forklift are the gripping parts that grip an object of the moving body. The holes in a pallet may be the gripped parts of the object to be transported. The gripped parts refer to specific parts of the object to be gripped by the moving body.

[0017] A moving body position and orientation estimation unit 202 estimates the position and orientation of a moving body. A transport target object position and orientation estimation unit 203 estimates the position and orientation of a transport target object. A cargo handling operation estimation unit 204 estimates whether cargo handling operation is in progress from the moving body position and orientation estimation unit 202 and the transport target object position and orientation estimation unit 203.

[0018] The moving object position and orientation estimation unit 202 estimates the position of the moving object by using the image capturing means. ｆｏｒｋ , and the attitude is θ ｆｏｌｋ and the transfer object position and orientation estimation unit 203 estimates the position of the transfer object as t ｐａｌｌｅｔ , and the attitude is θ ｐａｌｌｅｔ The position and orientation p is a vector p = [x, y, θ] that combines the position vectors x, y and the yaw angle θ. For simplicity, the position and orientation of the moving body when the object to be conveyed is set as the origin is is calculated, for example, as follows: Here, R(θ) is a rotation matrix that rotates around the origin by θ on the two-dimensional coordinate system.

[0019] Here, the first threshold value T １ = [xyθ] and the second threshold T ２ = [xyθ], and T １ <T ２ Let's say. In this case, it is assumed that loading and unloading work is in progress, and the loading and unloading imaging means is displayed preferentially. The loading and unloading imaging means is an imaging means used for loading and unloading, and is attached to the base of the gripping part of the moving body or the like, and captures an image of the object to be transported. In this case, the cargo handling operation estimation unit 204 estimates that there is a possibility that the vehicle is transitioning to cargo handling operation, and prioritizes displaying the traveling imaging means and the cargo handling imaging means. The traveling imaging means is an imaging means used for the vehicle to travel, and captures images of the front or rear of the vehicle. The traveling imaging means may be a fixed camera attached to the facility. In this case, the cargo handling operation estimation unit 204 estimates that cargo handling is not being performed. If it is estimated that cargo handling is not being performed, it may be estimated that traveling is being performed. However, the comparison operator is assumed to compare each element of the vector. For example, A<B means that A ｘ <B ｘ And A ｙ <B ｙ And A θ <B θ Shows.

[0020] FIG. 5A is a The moving body is closer to the object to be transported than the distance threshold value, and the angle difference between the gripping part of the moving body and the gripped part of the object to be transported is equal to or less than the angle threshold value, so the loading / unloading operation estimation unit 204 estimates that loading / unloading operation is in progress.

[0021] FIG. 5B is a Even if the moving body is closer to the object to be transported than the distance threshold, if the angle difference between the fork of the moving body and the gripped part of the object to be transported is greater than the angle threshold, it is considered that the moving body is moving while avoiding the object to be transported, and therefore the loading and unloading operation estimation unit 204 estimates that the moving body is not performing loading and unloading operation. If it is estimated that the moving body is not performing loading and unloading operation, it may be estimated that the moving body is traveling.

[0022] FIG. 5C is a Even if the distance between the moving body and the object to be transported is equal to or greater than the distance threshold, if the lateral deviation and the difference in angle between the fork of the moving body and the gripped part of the object to be transported are equal to or less than a certain value, the loading / unloading operation estimation unit 204 estimates that the operation is transitioning from traveling to loading / unloading. Lateral deviation refers to a state in which the moving body is deviated laterally from directly facing the object to be transported.

[0023] FIG. 5D is a In the case where there is a lateral shift between the moving body and the object to be transported or an angular difference between the forks of the moving body and the gripped part of the object to be transported is equal to or greater than a certain value, the moving body is considered to be moving to avoid the object to be transported, and the loading / unloading operation estimation unit 204 therefore estimates that loading / unloading operation is not being performed. If it is estimated that loading / unloading operation is not being performed, it may also be estimated that the moving body is traveling.

[0024] The position and orientation of the moving body are estimated using, for example, Visual SLAM, and the position and orientation of the object to be transported are estimated using deep learning. The position and orientation of the moving body are estimated using, for example, the method disclosed in International Publication WO2021 / 256103. The position and orientation of the pallet are estimated using, for example, Japanese Patent Application No. 2024-001645. However, the position and orientation of the moving body and the position and orientation of the object to be transported may also be estimated using methods other than these.

[0025] First threshold T １ and the second threshold T ２ The user inputs an arbitrary value for the first threshold T １ and a second threshold T ２ By correcting the value of , it is possible to make an estimation that learns the habits of the operator.

[0026] The priority image capturing means determination unit 205 determines the image capturing means to be preferentially displayed on the operator's control screen from the traveling operation estimation unit 201 and the loading / unloading operation estimation unit 204. As shown in the upper table of Fig. 4, the image capturing means may be a front camera, a rear camera, a gripper base camera, a floor transport object camera, or other fixed cameras. The camera is an image capturing device that enables a person to visually recognize the surroundings, such as an RGB camera, an RGBD camera, or an infrared camera.

[0027] The front camera takes camera images facing forward of the moving body. The front camera is placed, for example, near the driver's seat and takes images of the area ahead in the direction of travel. The images ahead are enlarged and displayed while the moving body is in motion. The rear camera takes camera images facing backward of the moving body. The rear camera is placed, for example, near the driver's seat and takes images of the area behind the moving body in the direction of travel. The images behind are enlarged and displayed while the moving body is in motion.

[0028] The gripping unit base camera is a camera for handling the transported object. The gripping unit base camera is placed at the base of the gripping unit and captures the transported object that is about to be handled. The gripping unit base camera is enlarged during the loading and unloading operation. The floor transported object camera is a camera for handling the transported object. The floor transported object camera is placed at the bottom of the moving body, between the floor and the moving body, and captures the transported object that is about to be handled. The floor transported object camera is enlarged during the loading and unloading operation.

[0029] The other fixed cameras are cameras for monitoring fixed locations. The other fixed cameras are installed on pillars, ceilings, etc. The other fixed cameras monitor moving objects passing through the passage.

[0030] An example of a remote control screen is shown in the lower diagram of Figure 4. As shown in the left diagram of the lower diagram of Figure 4, for example, an electronic map is displayed. The electronic map is map information. The electronic map displays information about the moving object itself and surrounding obstacles. Also, as shown in the right diagram of the lower diagram of Figure 4, images taken by multiple cameras, such as a front camera, a rear camera, a camera at the base of the gripper, a camera for the object being transported on the floor, and other fixed cameras, are usually all displayed at the same size.

[0031] The upper diagram in Figure 6 shows an example of a remote operation screen when it is estimated that loading and unloading work is in progress. For example, cameras for loading and unloading the object to be transported, such as the camera at the base of the gripping unit and the camera for the floor of the object to be transported, are displayed with priority. The lower diagram in Figure 6 shows an example of a remote operation screen when it is estimated that loading and unloading work is in progress. For example, the front camera and the camera for loading and unloading the object to be transported are displayed with priority. Priority display means, for example, an enlarged display, displayed alone or selected from multiple cameras.

[0032] The above configuration provides a mobile object remote control support system that determines which imaging means to prioritize display on the operator's control screen. The traveling operation estimation unit 201, the mobile object position and orientation estimation unit 202, the transported object position and orientation estimation unit 203, the loading and unloading operation estimation unit 204, and the priority imaging means determination unit 205 may be read as a traveling operation estimation means, a mobile object position and orientation estimation means, a transported object position and orientation estimation means, a loading and unloading operation estimation means, and a priority imaging means determination means. Furthermore, the above configuration may be implemented by a single device.

[0033] (Description of a mobile object remote control assistance method according to a first embodiment) Fig. 3 is a flowchart of a mobile object remote control assistance method according to the present disclosure. Fig. 9 is a block diagram showing a configuration of an information processing device according to the present disclosure. The mobile object remote control assistance method according to the first embodiment will be described with reference to Figs. 3 and 9 .

[0034] As shown in FIG. 3, first, it is estimated whether or not traveling work is in progress (step S301). The traveling work estimation unit 201 estimates whether or not traveling work is in progress. Next, the position and orientation of the moving body are estimated (step S302). The moving body position and orientation estimation unit 202 estimates the position and orientation of the moving body. Next, the position and orientation of the transported object are estimated (step S303). The transported object position and orientation estimation unit 203 estimates the position and orientation of the transported object. Next, it is estimated whether or not loading and unloading work is in progress (step S304). The loading and unloading work estimation unit 204 estimates whether or not loading and unloading work is in progress from the estimation of the position and orientation of the moving body and the estimation of the position and orientation of the transported object. Finally, the imaging means to be preferentially displayed is determined (step S305). The priority imaging means determination unit 205 determines the imaging means to be preferentially displayed on the operator's control screen from the estimation of traveling work and the estimation of loading and unloading work.

[0035] The above configuration provides a mobile object remote control assistance method for determining which imaging means to preferentially display on the operator's control screen. The mobile object remote control assistance method is realized by an information processing device executing processing. As shown in FIG. 9 , the information processing device 900 physically includes a memory 902 that stores a program and a processor 901 that executes processing by starting the program. The information processing device 900 may be configured as a single device or multiple devices. The information processing device 900 may also be a cloud server that executes some or all of its functions in a distributed manner.

[0036] The program includes instructions (or software code) that, when loaded into a computer, cause the computer to perform one or more functions described in the embodiments. The program may be stored on a non-transitory computer-readable medium or a tangible storage medium. By way of example and not limitation, computer-readable media or tangible storage media include random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drive (SSD) or other memory technologies, CD-ROM, digital versatile disc (DVD), Blu-ray disc or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage device. The program may also be transmitted on a transitory computer-readable medium or communication medium. By way of example and not limitation, transitory computer-readable media or communication media include electrical, optical, acoustic, or other forms of propagated signals.

[0037] (Description of a Mobile Object Remote Control Support System According to a Second Embodiment) Fig. 7 is a block diagram showing a second configuration of a mobile object remote control support system according to the present disclosure. A mobile object remote control support system 700 according to the second embodiment will be described with reference to Fig. 7 .

[0038] As shown in Figure 7, the mobile object remote control support system 700 includes an image acquisition unit 701, a mobile object position and attitude estimation unit 702, a transport object position and attitude estimation unit 703, a control command acquisition unit 704, a driving operation estimation unit 705, a loading and unloading operation estimation unit 706, and a priority imaging means determination unit 707.

[0039] The image acquisition unit 701 acquires images using an imaging means installed in the mobile body or an imaging means that monitors the periphery of the mobile body.

[0040] The moving object position and orientation estimation unit 702 has the same functions as the moving object position and orientation estimation unit 202. Furthermore, the moving object position and orientation estimation unit 702 estimates the position and orientation of the moving object using the images acquired by the image acquisition unit 701. The moving object position and orientation estimation unit 702 estimates the position and orientation of the moving object by performing calculations using Visual SLAM or the like using the acquired images.

[0041] The transfer object position and orientation estimation unit 703 has the same functions as the transfer object position and orientation estimation unit 203. Furthermore, the transfer object position and orientation estimation unit 703 estimates the position and orientation of the transfer object using the images acquired by the image acquisition unit 701. The transfer object position and orientation estimation unit 703 estimates the position and orientation of the pallet using deep learning or the like using the acquired images.

[0042] The control command acquisition unit 704 acquires control command information input from an external operator to the vehicle. The control commands include those for the throttle lever, brake pedal, steering, reach lever, tilt lever, lift lever, and the like.

[0043] The driving operation estimation unit 705 has the same functions as the driving operation estimation unit 201. The driving operation estimation unit 705 further estimates from the control command information acquired by the control command acquisition unit 704 whether driving operation is in progress.

[0044] The cargo handling operation estimation unit 706 has the same functions as the cargo handling operation estimation unit 204. The cargo handling operation estimation unit 706 further calculates the speed of the moving body using the control command information and the position and orientation of the moving body, and estimates whether or not a cargo handling transition is in progress. The cargo handling operation estimation unit 706 estimates whether or not a cargo handling operation is in progress based on the estimations of the moving body position and orientation estimation unit 702 and the transport target object position and orientation estimation unit 703.

[0045] The priority imaging means determination unit 707 has the same function as the priority imaging means determination unit 205. The priority imaging means determination unit 707 determines the imaging means to be prioritized based on the estimations of the traveling operation estimation unit 705 and the loading and unloading operation estimation unit 706.

[0046] The above configuration provides a mobile object remote control support method that determines which imaging means to prioritize display on the operator's control screen based on information from the image acquisition unit and the control command acquisition unit. The image acquisition unit 701 and the control command acquisition unit 704 may be read as image acquisition means and control command acquisition means, respectively.

[0047] Other Embodiments Fig. 8 is a diagram illustrating a selection state of the imaging means of the mobile object remote control support system according to the present disclosure. Other embodiments will be described with reference to Fig. 8 .

[0048] As shown in the upper diagram of Fig. 8, the imaging direction of each imaging means is shown on the electronic map. As shown in the lower diagram of Fig. 8, the imaging means currently capturing an image may be highlighted on the electronic map. For example, the imaging means displayed on the electronic map may be lit up to notify the operator which imaging means is being given priority in display.

[0049] In other embodiments, the priority display may be cancelled or the display may be fixed. The mobile object remote operation support system may cancel the priority display or fix the display according to the operator's preference.

[0050] Although the present disclosure has been described above with reference to the embodiments, the present disclosure is not limited to the above-described embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure. Furthermore, each embodiment can be combined with other embodiments as appropriate.

[0051] Each drawing is merely an example for describing one or more embodiments. Each drawing may not relate to only one particular embodiment, but may also relate to one or more other embodiments. As will be understood by those skilled in the art, various features or steps described with reference to any one drawing can be combined with features or steps shown in one or more other drawings to create, for example, an embodiment not explicitly shown or described. Not all features or steps shown in any one drawing are necessary to describe an exemplary embodiment, and some features or steps may be omitted. The order of steps described in any drawing may be changed as appropriate.

[0052] Some or all of the above embodiments can be described as, but are not limited to, the following supplementary notes. (Supplementary Note 1) A mobile body remote control support system comprising: a mobile body position and attitude estimation means for estimating a position and attitude of a mobile body; a transport object position and attitude estimation means for estimating a position and attitude of a transport object; a cargo handling operation estimation means for estimating whether the mobile body is engaged in cargo handling operation based on the estimated position and attitude of the mobile body and the position and attitude of the transport object; and a priority image capture means determination means for determining an image capture means to be preferentially displayed on an operator's control screen based on the estimation of whether cargo handling operation is being performed. (Supplementary Note 2) The mobile body remote control support system according to Supplementary Note 1 further comprises image capture means for acquiring images using an image capture means installed on the mobile body or an image capture means for monitoring the periphery of the mobile body, wherein the mobile body position and attitude estimation means estimates the position and attitude of the mobile body using the acquired images, and the transport object position and attitude estimation means estimates the position and attitude of the transport object using the acquired images. (Supplementary Note 3) The mobile body remote control support system according to Supplementary Note 1, wherein the cargo handling operation estimation means estimates that the mobile body is performing cargo handling operation when the distance between the mobile body and the object to be transported is equal to or less than a distance threshold related to distance and the angle difference between the gripping part of the mobile body and the gripped part of the object to be transported is equal to or less than an angle threshold related to angle. (Supplementary Note 4) The mobile body remote control support system according to Supplementary Note 1, wherein the cargo handling operation estimation means calculates the speed of the mobile body using control command information and the position and orientation of the mobile body, and estimates whether or not the mobile body is transitioning from moving operation to cargo handling operation. (Supplementary Note 5) The mobile body remote control support system according to Supplementary Note 2, wherein the imaging means displayed on an electronic map lights up to notify the operator which position of the imaging means is being preferentially displayed. (Supplementary Note 6) The preferential display can be canceled or the display can be fixed. The mobile body remote control support system according to Supplementary Note 1. (Supplementary Note 7) The mobile object remote operation support system according to Supplementary Note 1, wherein the loading and unloading operation estimation means learns and performs estimation according to the operator.(Supplementary Note 8) A mobile body remote control assistance method in which an information processing device estimates a position and attitude of a mobile body, estimates a position and attitude of a transport object, estimates whether the mobile body is performing cargo handling operations based on the estimated position and attitude of the mobile body and the position and attitude of the transport object, and determines an imaging means to be preferentially displayed on an operator's control screen based on the estimation that cargo handling operations are in progress. (Supplementary Note 9) The mobile body remote control assistance method according to Supplementary Note 8, in which an image is captured using an imaging means installed on the mobile body or an imaging means monitoring the periphery of the mobile body, estimates the position and attitude of the mobile body using the acquired image, and estimates the position and attitude of the transport object using the acquired image. (Supplementary Note 10) A program that causes an information processing device to execute the following steps: estimate a position and attitude of a mobile body, estimate a position and attitude of an object to be transported, estimate whether the mobile body is performing cargo handling operations based on the estimated position and attitude of the mobile body and the position and attitude of the object to be transported, and determine an imaging means to be preferentially displayed on an operator's control screen based on the estimation that cargo handling operations are in progress.

[0053] Some or all of the elements (e.g., configurations and functions) described in Supplementary Notes 2 to 7 that are dependent on Supplementary Note 1 (e.g., system) may also be dependent on Supplementary Note 8 (e.g., method) and Supplementary Note 10 (e.g., program) in the same dependency relationship as Supplementary Note 2 to Supplementary Note 7. Some or all of the elements described in any Supplementary Note may be applied to various hardware, software, recording means for recording software, systems, and methods.

[0054] This application claims priority based on Japanese Patent Application No. 2024-107668, filed July 3, 2024, the disclosure of which is incorporated herein by reference in its entirety.

[0055] 200 Mobile object remote operation support system, 201 Traveling operation estimation unit, 202 Mobile object position and orientation estimation unit, 203 Transport object position and orientation estimation unit, 204 Loading operation estimation unit, 205 Priority image capture means determination unit, 700 Mobile object remote operation support system, 701 Image acquisition unit, 702 Mobile object position and orientation estimation unit, 703 Transport object position and orientation estimation unit, 704 Control command acquisition unit, 705 Traveling operation estimation unit, 706 Loading operation estimation unit, 707 Priority image capture means determination unit

Claims

1. A mobile body remote control support system comprising: a mobile body position and attitude estimation means for estimating the position and attitude of a mobile body; a transport object position and attitude estimation means for estimating the position and attitude of a transport object; a loading and unloading operation estimation means for estimating whether the mobile body is engaged in loading and unloading operation based on the estimated position and attitude of the mobile body and the position and attitude of the transport object; and a priority imaging means determination means for determining an imaging means to be preferentially displayed on an operator's control screen based on the estimation that loading and unloading operation is in progress.

2. The mobile body remote control support system of claim 1 further comprises an image acquisition means for acquiring images using an imaging means installed on the mobile body or an imaging means for monitoring the periphery of the mobile body, wherein the mobile body position and attitude estimation means estimates the position and attitude of the mobile body using the acquired images, and the transported object position and attitude estimation means estimates the position and attitude of the transported object using the acquired images.

3. A mobile body remote control support system as described in claim 1, wherein the loading and unloading work estimation means estimates that the mobile body is engaged in loading and unloading work when the distance between the mobile body and the object to be transported is equal to or less than a distance threshold related to distance and the angle difference between the gripping part of the mobile body and the gripped part of the object to be transported is equal to or less than an angle threshold related to angle.

4. The mobile body remote control support system described in claim 1, wherein the loading and unloading work estimation means calculates the speed of the mobile body using control command information and the position and attitude of the mobile body, and estimates whether or not the mobile body is transitioning from moving work to loading and unloading work.

5. A mobile object remote control support system according to claim 2, wherein the imaging means displayed on the electronic map is illuminated to notify the operator of which position of the imaging means is being displayed with priority.

6. The mobile object remote operation support system according to claim 1, wherein the priority display can be canceled or the display can be fixed.

7. A mobile object remote operation support system according to claim 1, wherein said cargo handling operation estimation means learns and makes estimations according to said operator.

8. A method for supporting remote operation of a mobile body, in which an information processing device estimates the position and attitude of a mobile body, estimates the position and attitude of an object to be transported, estimates whether the mobile body is engaged in loading and unloading operations based on the estimated position and attitude of the mobile body and the position and attitude of the object to be transported, and determines an imaging means to be preferentially displayed on an operator's control screen based on the estimation that loading and unloading operations are being performed.

9. A method for assisting remote operation of a mobile body as described in claim 8, comprising: acquiring an image captured using an imaging means installed on the mobile body or an imaging means monitoring the periphery of the mobile body; estimating the position and orientation of the mobile body using the acquired image; and estimating the position and orientation of the object to be transported using the acquired image.

10. The method for remotely controlling a mobile body described in claim 8, wherein estimating whether or not loading and unloading work is being performed comprises estimating that the mobile body is performing loading and unloading work when the distance between the mobile body and the object to be transported is equal to or less than a distance threshold related to distance, and the angle difference between the gripping part of the mobile body and the gripped part of the object to be transported is equal to or less than an angle threshold related to angle.

11. A method for remotely controlling a mobile body as described in claim 8, wherein estimating whether or not loading and unloading work is in progress involves calculating the speed of the mobile body using control command information and the position and attitude of the mobile body, and estimating whether or not a transition from moving work to loading and unloading work is being made.

12. A method for assisting remote operation of a mobile object according to claim 9, wherein the imaging means displayed on the electronic map is illuminated to notify the operator of which position of the imaging means is being displayed with priority.

13. The mobile object remote operation assistance method according to claim 8, wherein the priority display can be canceled or the display can be fixed.

14. A method for remotely controlling a mobile object according to claim 8, wherein the estimation of whether or not loading and unloading work is being performed is learned and performed according to the operator.

15. A program that causes an information processing device to execute the following: estimating the position and attitude of a moving body; estimating the position and attitude of an object to be transported; estimating whether the moving body is engaged in loading and unloading operations based on the estimated position and attitude of the moving body and the position and attitude of the object to be transported; and determining an imaging means to be preferentially displayed on an operator's control screen based on the estimation that loading and unloading operations are in progress.

16. The program according to claim 15, which acquires images taken using an imaging means installed on the moving body or an imaging means monitoring the periphery of the moving body, estimates the position and orientation of the moving body using the acquired images, and estimates the position and orientation of the object to be transported using the acquired images.

17. The program described in claim 15, wherein estimating whether loading and unloading work is in progress comprises estimating that the moving body is engaged in loading and unloading work if the distance between the moving body and the object to be transported is equal to or less than a distance threshold related to distance, and the angle difference between the gripping part of the moving body and the gripped part of the object to be transported is equal to or less than an angle threshold related to angle.

18. The program described in claim 15, wherein estimating whether loading and unloading work is in progress calculates the speed of the moving body using control command information and the position and orientation of the moving body, and estimates whether a transition from moving work to loading and unloading work is being made.

19. The program according to claim 16, wherein the imaging means displayed on the electronic map is illuminated to notify the operator of the position of the imaging means being displayed with priority.

20. The program according to claim 15, wherein the priority display can be canceled or the display can be fixed.

Images