Information processing device, system, method, and program

Abstract

The present invention reduces the operation load of an autonomous transport device for transporting a to-be-transported object placed at an arbitrary place, and promotes practical use. This information processing device comprises: a first acquisition means for acquiring, from an image captured by an imaging terminal, a first position orientation of the imaging terminal estimated on the basis of an environment map including an imaging point; a reception means for receiving a designated position at which a portion of the image displayed on a screen is designated; a second acquisition means for acquiring a third position orientation pertaining to a transport start point by means of the autonomous transport device of the to-be-transported object calculated on the basis of the first position orientation and a second position orientation of the imaging terminal estimated on the basis of a position in the image of the to-be-transported object when the to-be-transported object is recognized in an area including the designated position in the image; and an output means for outputting the third position orientation.

Description

Information processing device, system, method, and program

[0001] The present disclosure relates to an information processing device, a system, a method, and a program.

[0002] In recent years, autonomous guided vehicles have been put to practical use. For example, autonomous forklifts are required to autonomously navigate to a pallet (loading platform) location at a construction site or warehouse, pick up the pallet, and transport it to another location in that state.

[0003] Patent Literature 1 discloses a technology related to a control system for a mobile object such as a forklift. The control system includes a mobile object equipped with a distance measuring device and a terminal equipped with a camera. The mobile object acquires point cloud data of an object to be transported measured by the distance measuring device, detects the object (e.g., a pallet) by analyzing the point cloud data, and transmits the detection result to the terminal. The terminal calculates the position and orientation of the mobile object by capturing an image of the object with the camera. The terminal then detects the object by analyzing an image of the object captured by the camera and the detection result received from the mobile object. The control system controls the mobile object to move toward an object selected on the terminal.

[0004] Japanese Patent Application Laid-Open No. 2023-059409

[0005] In the technology disclosed in Patent Document 1, a mobile object acquires point cloud data of an object (a transport target) using a ranging device. Therefore, the mobile object must be moved in advance to an initial position where the transport target can be measured by the ranging device. The control system disclosed in Patent Document 1 must also set the initial position in advance. At work sites where autonomous guided vehicles are used, the placement location of the transport target often changes depending on the work situation, etc. This makes it difficult to fix the placement location of the transport target. Therefore, in order to apply the technology disclosed in Patent Document 1 to a work site, the initial position must be reset each time the placement location of the transport target changes. However, resetting the initial position of the control system each time is problematic due to labor and time constraints.

[0006] In view of the above-mentioned problems, the purpose of the present disclosure is to provide an information processing device, system, method, and program that reduces the operational burden of an autonomous transport device that transports objects placed at any location and promotes practical application.

[0007] The information processing device according to the present disclosure comprises: a first acquisition means for acquiring a first position and attitude of an imaging terminal estimated from an image captured by the imaging terminal based on an environmental map including the imaging point; a receiving means for receiving a designated position that designates a part of the image displayed on a screen; a second acquisition means for acquiring a second position and attitude of the imaging terminal estimated based on the position of the object to be transported in the image when the object to be transported is recognized in an area of ​​the image including the designated position, and a third position and attitude related to a starting point of transport by an autonomous transport device of the object to be transported calculated based on the first position and attitude; and an output means for outputting the third position and attitude.

[0008] The information processing system according to the present disclosure includes an imaging terminal having a screen and an autonomous transport device, wherein the imaging terminal acquires a first position and attitude of the imaging terminal estimated from an image captured by the imaging terminal based on an environmental map including the imaging point, accepts a designated position specifying a part of the image displayed on the screen, acquires a second position and attitude of the imaging terminal estimated based on the position of the object to be transported in the image when the object to be transported is recognized in an area of ​​the image including the designated position, and acquires a third position and attitude related to the start point of transport by the autonomous transport device of the object to be transported calculated based on the first position and attitude, and outputs the third position and attitude.

[0009] The information processing method according to the present disclosure includes a computer: acquiring a first position and attitude of an imaging terminal estimated from an image captured by the imaging terminal based on an environmental map including the imaging point; accepting a designated position that specifies a portion of the image displayed on a screen; acquiring a second position and attitude of the imaging terminal estimated based on the position of the object to be transported in the image when the object to be transported is recognized in an area of ​​the image including the designated position; and acquiring a third position and attitude related to the starting point of transport by an autonomous transport device of the object to be transported, calculated based on the first position and attitude; and outputting the third position and attitude.

[0010] The information processing program according to the present disclosure causes a computer to execute the following steps: a first acquisition process for acquiring a first position and attitude of an imaging terminal estimated from an image captured by the imaging terminal based on an environmental map including the imaging point; a reception process for accepting a designated position where a part of the image displayed on the screen is designated; a second acquisition process for acquiring a second position and attitude of the imaging terminal estimated based on the position of the object to be transported in the image when the object to be transported is recognized in an area of ​​the image including the designated position, and a third position and attitude related to a start point of transport by an autonomous transport device of the object to be transported calculated based on the first position and attitude; and an output process for outputting the third position and attitude.

[0011] The present disclosure can reduce the operational burden of autonomous transport devices that transport objects placed at any location, and promote practical application.

[0012] FIG. 1 is a block diagram showing a configuration of an information processing device according to the present disclosure. FIG. 2 is a flowchart showing a flow of an information processing method according to the present disclosure. FIG. 3 is a block diagram showing an overall configuration of an information processing system according to the present disclosure. FIG. 4 is a diagram showing an example of an object to be transported according to the present disclosure. FIG. 5 is a block diagram showing a configuration of a mobile terminal according to the present disclosure. FIG. 6 is a block diagram showing a configuration of an autonomous transport device according to the present disclosure. FIG. 7 is a diagram for explaining the concept of each coordinate system according to the present disclosure. FIG. 8 is a sequence chart showing a flow of a transportation assistance process according to the present disclosure. FIG. 9 is a flowchart showing a flow of a position and orientation estimation process according to the present disclosure. FIG. 10 is a diagram for explaining the concept of selecting an object to be transported on a screen of a mobile terminal according to the present disclosure. FIG. 11 is a diagram for explaining the concept of a facing position for loading an object to be transported according to the present disclosure. FIG. 11 is a block diagram showing an overall configuration of an information processing system according to the present disclosure. FIG. 12 is a block diagram showing a configuration of a mobile terminal according to the present disclosure. FIG. 13 is a block diagram showing a configuration of an estimation server according to the present disclosure. FIG. 14 is a sequence chart showing a flow of a transportation assistance process according to the present disclosure. FIG. 15 is a flowchart showing a flow of a position and orientation estimation process according to the present disclosure. FIG. 16 is a block diagram showing a hardware configuration of a mobile terminal according to the present disclosure. FIG. 17 is a block diagram showing a hardware configuration of an estimation server according to the present disclosure.

[0013] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In each drawing, the same or corresponding elements are designated by the same reference numerals, and for clarity of explanation, duplicate explanations will be omitted as necessary.

[0014] 1 is a block diagram showing the configuration of an information processing device 1. The information processing device 1 is an information processing device that acquires and outputs information about a starting point of transportation by an autonomous transport device (not shown) from a captured image including an object to be transported, in order to support the operation of the autonomous transport device that transports the object to be transported (not shown) placed at an arbitrary location. The information processing device 1 includes a first acquisition unit 11, a reception unit 12, a second acquisition unit 13, and an output unit 14.

[0015] The first acquisition unit 11 acquires a first position and orientation of an imaging terminal (not shown) estimated from an image captured by the imaging terminal based on an environmental map including the imaging point. The imaging terminal may be a so-called mobile terminal. The imaging terminal may be either one mounted on the information processing device 1 or an information terminal external to the information processing device 1. Therefore, when the imaging terminal is mounted on the information processing device 1, the information processing device 1 may be a mobile terminal. On the other hand, the imaging terminal and the information processing device 1 are not mounted on an autonomous transport device.

[0016] The "environmental map" is map data that expresses the current position (photography point) of the photographing terminal and the shapes of objects and terrains existing in the surrounding environment, as well as the positional relationships of the objects, in three-dimensional information. The environmental map may be generated in advance or may be generated from an image photographed by the photographing terminal. The "first position and orientation" is information that estimates the position and orientation of the photographing terminal based on the environmental map from the image photographed by the photographing terminal.

[0017] The reception unit 12 receives a designated position where a part of an image displayed on the screen is designated. The "screen" is a display device built into the information processing device 1, a display device connected to the information processing device 1, or a display device mounted on the photographing terminal. Specifically, the reception unit 12 receives as the designated position a position on the screen designated by the user of the information processing device 1 using a finger or any input device. The "designated position" is a partial area (position) within the image photographed by the photographing terminal.

[0018] The second acquisition unit 13 acquires a third position and attitude calculated based on the second position and attitude and the first position and attitude. Here, the "second position and attitude" is information obtained by estimating the position and attitude of the imaging terminal based on the position of the object to be transported in the image when the object to be transported is recognized in an area including the designated position in the image captured by the imaging terminal. Furthermore, the "third position and attitude" is information related to the start point of transport of the object to be transported by the autonomous transport device. For example, the third position and attitude is information calculated based on the position and attitude of the autonomous transport device at the start point of transport, or the position and attitude of the object to be transported itself. Note that each of the "first position and attitude," "second position and attitude," and "third position and attitude" may be information corresponding to three-dimensional coordinates.

[0019] The output unit 14 outputs the third position and orientation. For example, the output unit 14 may output the third position and orientation to the autonomous transport device or an external device that controls the autonomous transport device.

[0020] 2 is a flowchart showing the flow of the information processing method. First, the first acquisition unit 11 acquires a first position and orientation of the photographing terminal estimated from an image photographed by the photographing terminal based on an environmental map including the photographing point (S1). Next, the acceptance unit 12 accepts a designated position that designates a part of the image displayed on the screen (S2). Note that steps S1 and S2 may be performed on the same image. Therefore, the processing order of steps S1 and S2 may be reversed or may be performed in parallel.

[0021] Next, the second acquisition unit 13 acquires a third position and orientation of the object to be transported, calculated based on the second position and orientation and the first position and orientation, relative to the start point of transport by the autonomous transport device (S3). Here, the second position and orientation is the position and orientation of the imaging device estimated based on the position of the object to be transported in the image displayed on the screen in step S2 when the object to be transported is recognized in an area including the specified position. Thereafter, the output unit 14 outputs the third position and orientation (S4).

[0022] In this way, the technology disclosed herein can output information about the starting point of transport by the autonomous transport device using an image including the transport object captured by a photographing device separate from the autonomous transport device. This does not require measurement data from a distance measuring device mounted on the autonomous transport device. Therefore, unlike Patent Document 1, for example, there is no need to set the initial position of the autonomous transport device in the control system. Therefore, the technology disclosed herein does not require a fixed location for the transport object, and there are no restrictions on the location of the transport object, making it applicable to transport objects placed in any location. In other words, it can also be applied to autonomous transport devices not equipped with a distance measuring device. Therefore, the technology disclosed herein can reduce the operational burden of autonomous transport devices that transport objects placed in any location and promote practical use.

[0023] The information processing device 1 includes a processor, a memory, and a storage device (not shown). The storage device stores a computer program that implements the process of the information processing method shown in FIG. 2 . The processor then loads the computer program from the storage device into the memory and executes the computer program. This allows the processor to implement the functions of a first acquisition unit 11, a reception unit 12, a second acquisition unit 13, and an output unit 14.

[0024] Alternatively, each component of the information processing device 1 may be realized by dedicated hardware. Furthermore, some or all of the components of each device may be realized by general-purpose or dedicated circuits, processors, etc., or a combination thereof. These may be configured by a single chip, or by multiple chips connected via a bus. Some or all of the components of each device may be realized by a combination of the above-mentioned circuits, etc., and a program. Furthermore, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an FPGA (Field-Programmable Gate Array), a quantum processor (quantum computer control chip), etc., may be used as the processor.

[0025] Furthermore, when some or all of the components of the information processing device 1 are realized by multiple information processing devices, circuits, etc., the multiple information processing devices, circuits, etc. may be centrally or distributed. For example, the information processing devices, circuits, etc. may be realized as a client-server system, a cloud computing system, or the like, in which they are connected via a communication network. Furthermore, the functions of the information processing device 1 may be provided in a SaaS (Software as a Service) format.

[0026] Second Embodiment Fig. 3 is a block diagram showing the overall configuration of an information processing system 1000. The information processing system 1000 is an information system for supporting the operation of an autonomous transport device 200 that transports an object 5 to be transported placed at an arbitrary location. The object 5 to be transported has cargo 52 loaded on a pallet 51. The pallet 51 is a loading platform on which the cargo 52 is placed. The pallet 51 also has insertion holes 511 and 512 into which the forks 201 of the autonomous transport device 200 are inserted. Fig. 4 is a diagram showing an example of the object 5 to be transported.

[0027] The information processing system 1000 includes a mobile terminal 100, an autonomous transport device 200, an environmental map DB (DataBase) 300, and a route planning device 400. The mobile terminal 100, the autonomous transport device 200, the environmental map DB 300, and the route planning device 400 are communicably connected to each other via a communication network N. Here, the communication network N is a communication network including a wireless communication network.

[0028] The portable terminal 100 is a wirelessly capable information processing terminal, a so-called mobile terminal, that is carried and operated by a user U. The portable terminal 100 may be, for example, a smartphone, a tablet terminal, or a notebook PC (Personal Computer). The portable terminal 100 is an example of the information processing device 1 equipped with the above-mentioned photographing terminal. The portable terminal 100 photographs, for example, an object 5 to be transported in response to an operation by the user U, and acquires an image including the object 5 to be transported. The portable terminal 100 also has a built-in screen and displays the acquired image on the screen. The portable terminal 100 accepts a designated position, in which the user U designates a part of the image displayed on the screen. The detailed configuration of the portable terminal 100 will be described later.

[0029] The autonomous transport device 200 is an autonomous transport vehicle that travels autonomously based on a position and attitude instructed by the mobile terminal 100 and the path planning device 400, and transports the transport object 5, etc. In the example of FIG. 3 , the autonomous transport device 200 is an autonomous forklift. Therefore, the autonomous transport device 200 is equipped with forks 201 that are inserted into insertion holes 511 and 512 of the pallet 51 to load the transport object 5. Note that the autonomous transport device 200 according to the present disclosure is not limited to an autonomous forklift. Therefore, the autonomous transport device 200 may be another automatic guided vehicle that travels autonomously based on a position and attitude instructed by an external device, and transports the transport object 5, etc. Note that the detailed configuration of the autonomous transport device 200 will be described later.

[0030] The environmental map DB 300 is a database server that manages the above-mentioned environmental map. The environmental map is a collection of three-dimensional coordinate data of the surrounding environment including at least the transport object 5, the mobile terminal 100, and the autonomous transport device 200, which is generated from image or video data captured by the mobile terminal 100 or the like using a VSLAM (Visual SLAM (Simultaneous Localization and Mapping)) function. The environmental map DB 300 can be referenced by the mobile terminal 100, the autonomous transport device 200, and the path planning device 400 via the communication network N. The environmental map DB 300 may also receive an environmental map generated by the VSLAM function in the mobile terminal 100, the autonomous transport device 200, the path planning device 400, or another computer, and register the received environmental map in the database.

[0031] The path planning device 400 is a computer server that plans a path for the autonomous transport device 200 to move from its current position to a position and attitude where the object 5 can be loaded, in accordance with a transport instruction received from the mobile terminal 100. The path planning device 400 may be realized as a computer system in which functions are distributed or made redundant using multiple computer devices. The path planning device 400 includes a transmission / reception unit 410 and a path planning unit 420.

[0032] The transmitter / receiver 410 receives the self-position and attitude of the autonomous transport device 200 from the autonomous transport device 200. The "self-position and attitude of the autonomous transport device 200" is information about the position and attitude at which the autonomous transport device 200 currently exists. The transmitter / receiver 410 also receives a transport instruction from the mobile terminal 100. The "transport instruction" includes a facing position and attitude at which the autonomous transport device 200 starts transporting the transport object 5. The "facing position and attitude" is an example of a third position and attitude related to the start point of transport of the transport object 5 by the autonomous transport device 200. The "facing position and attitude" is a position and attitude at which the autonomous transport device 200 can load the transport object 5. In other words, the "facing position and attitude" is a position and attitude in which the autonomous transport device 200 faces the insertion holes 511 and 512 of the transport object 5.

[0033] The route planning unit 420 refers to the environmental map DB 300, plans a route for the autonomous transport device 200 to move from its own position and posture to a facing position and posture, and generates route information. The route planning unit 420 may be realized using any route planning algorithm or trained model that receives an environmental map, a departure point, and an arrival point as input and outputs route information.

[0034] Furthermore, the transmitting / receiving unit 410 transmits to the autonomous transport device 200 a transport instruction including the facing position information received from the mobile terminal 100 and the route information generated by the route planning unit 420 .

[0035] 5 is a block diagram showing the configuration of the mobile terminal 100. The mobile terminal 100 includes a storage unit 110, a camera 121, a display device 122, a designated position acquisition device 123, a reception unit 131, a position and orientation estimation unit 132, a transport target recognition unit 133, a calculation unit 134, and a transport instruction unit 135. The position and orientation estimation unit 132 is an example of the first acquisition unit 11 described above. The reception unit 131 is an example of the reception unit 12 described above. The position and orientation estimation unit 132, the transport target recognition unit 133, and the calculation unit 134 are an example of the second acquisition unit 13 described above. The transport instruction unit 135 is an example of the output unit 14 described above.

[0036] The storage unit 110 includes, for example, a non-volatile storage device such as a hard disk or flash memory, and a memory such as a random access memory (RAM), i.e., a volatile storage device. The storage unit 110 stores a recognition model 111. The recognition model 111 is an AI (Artificial Intelligence) model for recognizing a conveyed object (particularly, a pallet) in an area of ​​an image that includes a specified position. The recognition model 111 is, for example, a regression model, but is not limited to this.

[0037] For example, the recognition model 111 may be a model that receives an image and front surface information (image coordinates of feature points) of the pallet 51 in the image as input, and outputs rear surface information (image coordinates of feature points) of the pallet 51. Here, "front surface information" refers to the image coordinates (two-dimensional coordinates in the image) of feature points on the front surface of the pallet 51 (the side facing the insertion holes 511). The feature points on the front surface of the pallet 51 may include, for example, each corner of the surface of the pallet 51 facing the insertion holes 511, and at least one corner of a rectangular area surrounding the pallet 51. Furthermore, the feature points on the front surface of the pallet 51 may include, for example, at least one corner of each of the insertion holes 511 and 512, and at least one corner of each of the rectangular areas surrounding each of the insertion holes 511 and 512. Furthermore, "rear surface information" refers to the image coordinates of feature points on the rear surface of the pallet 51 (the side behind the insertion holes 511). The feature points on the rear surface of the pallet 51 may include, for example, each corner on the back side of the insertion holes 511 of the pallet 51. The recognition model 111 is a trained model that has been machine-learned using training data that includes a photographed image of the transport object 5 and a pair of front surface information and rear surface information of the pallet 51 in the image.

[0038] Alternatively, the recognition model 111 may be a model that receives an image and an input position within the image, determines whether the object 5 (pallet 51) is displayed in an area of ​​the image that includes the input position, and outputs the determination result as the recognition result. In other words, the recognition model 111 may be a model that receives an image and an input position within the image, performs recognition processing on the pallet 51 for an area of ​​the image that includes the input position, and outputs the recognition result. Here, the recognition result may be the position and orientation of the pallet 51 based on an environmental map. In this case, the recognition model 111 may be a trained model that has been machine-learned using training data that includes a captured image of the object 5, image coordinates of an area in the image that includes the pallet 51, and the position and orientation of the pallet 51. Therefore, the recognition model 111 can also be said to be a model that estimates rear surface information that is difficult to discern from an image captured from the front of the pallet 51 (the side of the insertion hole 511).

[0039] The camera 121 corresponds to the photographing device mounted on the photographing terminal described above. The camera 121 photographs an image in response to an operation by the user U and outputs the photographed image to the reception unit 131. The display device 122 has a screen and displays the image photographed by the camera 121, etc. The designated position acquisition device 123 detects a position within the image designated by the user U's designation operation on the screen of the display device 122, acquires it as a designated position, and outputs the designated position to the reception unit 131. The user U can designate the designated position in various ways, such as by touching the screen with the user U's finger or a stylus pen, or by operating a mouse or keyboard connected to the mobile terminal 100.

[0040] The reception unit 131 receives an image captured by the camera 121. Then, the reception unit 131 outputs the received image to the display device 122, the position and orientation estimation unit 132, etc. The reception unit 131 also receives a designated position where a part of the image displayed on the screen of the display device 122 is designated. Specifically, the reception unit 131 receives a designated position acquired by the designated position acquisition device 123.

[0041] The position and orientation estimation unit 132 refers to the environmental map DB 300 and estimates the position and orientation (self-position and orientation) of the mobile device 100 (camera 121) with reference to the environmental map from the image accepted by the acceptance unit 131. Specifically, the position and orientation estimation unit 132 may use a VSLAM function to estimate the position and orientation of the mobile device 100 with reference to the environmental map from the image. In other words, the position and orientation estimation unit 132 acquires the first position and orientation by estimating the position and orientation of the mobile device 100 with reference to the environmental map from the image.

[0042] The transfer target recognition unit 133 recognizes the transfer target 5 from the image based on the specified position, and thereby estimates a coordinate group in the image of the transfer target 5. In response to this, the position and orientation estimation unit 132 estimates, as a second position and orientation, a relative position and orientation of the mobile terminal 100 based on the position of the transfer target 5 as a reference, based on the coordinate group estimated by the transfer target recognition unit 133.

[0043] Specifically, the transfer target recognition unit 133 estimates a group of coordinates in the image of the transfer target using the recognition model 111. For example, the transfer target recognition unit 133 estimates front surface information (image coordinates) of the pallet 51 from the image based on the specified position using a predetermined object detection algorithm or the like. The transfer target recognition unit 133 then inputs the image and front surface information to the recognition model 111 and obtains as output (recognition result) rear surface information (image coordinates) of the pallet 51 recognized by the recognition model 111. Note that the transfer target recognition unit 133 may calculate the reliability of each image coordinate included in the rear surface information and remove image coordinates with reliability below a threshold to obtain the rear surface information.

[0044] The position and orientation estimation unit 132 then estimates the position and orientation of the pallet 51 from the front surface information and the rear surface information using a calculation formula that takes into account the depth of each image coordinate from the camera 121. It is preferable to use a predefined calculation formula. The position and orientation estimation unit 132 then estimates the second position and orientation of the mobile terminal 100 by converting the first position and orientation of the mobile terminal 100 based on the environmental map (MAP coordinate system) into a PALLET coordinate system with the position and orientation of the pallet 51 as the origin. It is also possible for the position and orientation estimation unit 132 to estimate the second position and orientation by converting three-dimensional coordinates corresponding to the first position and orientation in the MAP coordinate system into three-dimensional coordinates in the PALLET coordinate system.

[0045] The calculation unit 134 acquires a third position and attitude calculated as a position and attitude at which the autonomous transport device 200 can load the transport object 5 based on the first position and attitude and the second position and attitude. Here, the "third position and attitude" is a position and attitude of the autonomous transport device 200 facing a direction in which the autonomous transport device 200 can load the transport object 5 and is at a predetermined distance from the transport object 5 (front-facing position and attitude). In other words, the calculation unit 134 acquires the third position and attitude by calculating a position and attitude at which the autonomous transport device 200 can load the transport object 5 based on the first position and attitude and the second position and attitude. At this time, the calculation unit 134 may acquire the third position and attitude by calculating a position and attitude taking into account an orientation at which the autonomous transport device 200 can load the transport object 5 based on the first position and attitude and the second position and attitude.

[0046] The transport instruction unit 135 outputs a transport instruction including the third position and attitude so as to move the autonomous transport device 200 to the position indicated by the third position and attitude (transport start point) and load the transport target object 5. Specifically, the transport instruction unit 135 generates a transport instruction for the autonomous transport device 200 including the facing position and attitude, and transmits the generated transport instruction to the route planning device 400.

[0047] 6 is a block diagram showing the configuration of the autonomous guided vehicle 200. The autonomous guided vehicle 200 includes a camera 221, a traveling device 222, a cargo handling device 223, a reception unit 231, a position and orientation estimation unit 232, a transmission / reception unit 233, a traveling control unit 234, and a cargo handling control unit 235. The camera 221 is an example of an imaging unit included in the autonomous guided vehicle 200.

[0048] The camera 221 captures images of the periphery of the autonomous guided vehicle 200 in accordance with instructions from the position and orientation estimation unit 232 and the like, and outputs the captured images to the position and orientation estimation unit 232. The traveling device 222 is a device for traveling the autonomous guided vehicle 200 in accordance with instructions from the traveling control unit 234. The traveling device 222 includes, for example, a steering wheel and a throttle. The loading device 223 is a device for loading cargo such as the object to be transported 5 in accordance with instructions from the loading control unit 235. The loading device 223 includes, for example, forks 201 and a belt.

[0049] The reception unit 231 receives an image captured by the camera 221 and outputs the received image to the position and orientation estimation unit 232. The position and orientation estimation unit 232 references the environmental map DB 300 and estimates the position and orientation (self-position and orientation) of the autonomous transport device 200 (camera 221) based on the environmental map from the image received by the reception unit 231. Specifically, the position and orientation estimation unit 232 may use a VSLAM function to estimate the self-position and orientation of the autonomous transport device 200 based on the environmental map from the image. In other words, the position and orientation estimation unit 232 acquires the self-position and orientation (fourth position and orientation) of the autonomous transport device 200 estimated based on the environmental map from the second image captured by the imaging unit.

[0050] The transmitting / receiving unit 233 transmits the self-position and attitude estimated by the position and attitude estimation unit 232 to the route planning device 400. The transmitting / receiving unit 233 also receives a transportation instruction from the route planning device 400. In other words, the transmitting / receiving unit 233 acquires the transportation instruction from the mobile terminal 100 via the route planning device 400.

[0051] The travel control unit 234 controls the travel device 222 so that the autonomous transport device 200 travels according to the route information included in the transport instruction, and moves the autonomous transport device 200 to the facing position and posture included in the transport instruction.

[0052] The cargo handling control unit 235 controls the cargo handling device 223 so that the autonomous transport device 200 loads the transport object 5 after moving from its own position and posture to the position indicated by the facing position and posture. In other words, the autonomous transport device 200 moves from the fourth position and posture to the transport start point indicated by the third position and posture in response to the transport instruction, and loads the transport object 5 after the movement.

[0053] FIG. 7 is a diagram illustrating the concept of each coordinate system. The MAP coordinate system C1 is a three-dimensional coordinate system of the surrounding environment including the transport object 5, the mobile terminal 100, and the autonomous transport device 200, based on the environmental map in the environmental map DB 300. The MAP coordinate system C1 has an origin at an arbitrary point on the environmental map. The PALLET coordinate system C2 is a three-dimensional coordinate system based on the position of the pallet 51. The PALLET coordinate system C2 has an origin at the center of the pallet 51. The PALLET coordinate system C2 shows an example in which the x-axis is in the direction of the front surface (insertion hole 511 side) of the pallet 51. Therefore, the facing position P1 is a position facing the insertion hole 511 side of the pallet 51 and at a predetermined distance from the pallet 51. The facing position P1 is a three-dimensional coordinate corresponding to a facing position and orientation in which the autonomous transport device 200 can load the transport object 5. The CAM coordinate system C3 is a three-dimensional coordinate system based on the mobile terminal 100 (i.e., the camera 121). The CAM coordinate system C3 has its origin at the center of the mobile terminal 100. The CAM coordinate system C3 shows an example in which the shooting direction (in this example, the direction toward the pallet 51) is the x-axis.

[0054] 8 is a sequence chart showing the flow of the transportation support process. First, the autonomous transport device 200 captures an image of the surroundings of the autonomous transport device 200 using the camera 221 (S111). Then, the autonomous transport device 200 references the environmental map DB 300 via the communication network N and estimates its own position and orientation based on the environmental map from the captured image (S112). That is, the autonomous transport device 200 estimates its own position and orientation in the MAP coordinate system C1. Here, the position of the autonomous transport device 200 in the MAP coordinate system C1 is expressed as t ｆ ＭＡＰ , posture R ｆ ＭＡＰ In addition, t ｘ Ｙ is a matrix with 3 rows and 1 column. ｘ Ｙ is a matrix with 3 rows and 3 columns. The same applies to the following. Then, the autonomous transport device 200 transmits its own position and orientation to the path planning device 400 via the communication network N (S113). In response to this, the path planning device 400 registers the current position and orientation of the autonomous transport device 200 received from the autonomous transport device 200 in a storage unit (not shown).

[0055] Furthermore, the mobile terminal 100 photographs the transport object 5 with the camera 121 in response to an operation by the user U (S101). Then, the mobile terminal 100 performs a position and orientation estimation process (S102).

[0056] 9 is a flowchart showing the flow of the position and orientation estimation process. First, the receiving unit 131 of the mobile terminal 100 receives an image of the transport object 5 (S201). Next, the position and orientation estimation unit 132 estimates the self-position and orientation (first position and orientation) of the mobile terminal 100 based on the image and the environmental map (S202). That is, the mobile terminal 100 estimates the self-position and orientation in the MAP coordinate system C1. Here, the position of the mobile terminal 100 in the MAP coordinate system C1 is expressed as t Ｃ ＭＡＰ , posture R Ｃ ＭＡＰ Let's say.

[0057] The display device 122 displays the image received in step S201 on the screen (S203). The receiving unit 131 then receives a specified position of a part of the image on the screen as a result of an operation by the user U (S204).

[0058] 10 is a diagram for explaining the concept of selecting a transport object 5 on the screen of the mobile terminal 100. Here, it is assumed that an image including the transport object 5 is displayed on the screen of the display device 122 of the mobile terminal 100. The user U performs a designation operation U1 on the screen of the display device 122 to designate the vicinity of the transport object 5. For example, it is assumed that the user U designates a designated position P0 of the transport object 5 in the image by a designation operation U2.

[0059] 9 , the transfer target recognition unit 133 performs a recognition process for the transfer target 5 in the area including the designated position P0 from the image (S205). Then, the transfer target recognition unit 133 determines whether the transfer target 5 has been recognized from the recognition result (S206). If the transfer target 5 has been recognized, and after step S202, the position and orientation estimation unit 132 estimates the relative position and orientation (second position and orientation) of the mobile terminal 100 with respect to the transfer target 5 based on the recognition result (S207). That is, the mobile terminal 100 estimates the position t Ｃ ＭＡＰ and posture R Ｃ ＭＡＰ is converted into the PALLET coordinate system C2, and the position t Ｃ ＰＡＬＬＥＴ and posture R Ｃ ＰＡＬＬＥＴ is estimated as the relative position and orientation.

[0060] Then, the calculation unit 134 calculates the facing position and orientation of the autonomous transport device 200 to face the transport target object 5 based on the relative position and orientation of the mobile terminal 100 and its own position and orientation (S208). Specifically, first, the center of the pallet 51 in the PALLET coordinate system C2 is set to the origin as shown in the following formula (1).

[0061] Next, the calculation unit 134 calculates the facing position t of the autonomous transport device 200 in the PALLET coordinate system C2 as shown in the following formulas (2) and (3). ａ ＰＡＬＬＥＴ and posture R ａ ＰＡＬＬＥＴ Here, L is the length of the fork 201, and m is the margin from the fork 201 to the pallet 51.

[0062] Then, the calculation unit 134 converts the facing position and orientation of the autonomous transport device 200 into the viewpoint of the mobile terminal 100 (camera 121). Specifically, the calculation unit 134 converts the facing position and orientation of the autonomous transport device 200 from the PALLET coordinate system C2 to the CAM coordinate system C3 as shown in the following formulas (4) and (5). In other words, the calculation unit 134 converts the position t Ｃ ＰＡＬＬＥＴ and posture R Ｃ ＰＡＬＬＥＴ , the facing position t of the autonomous transport device 200 in the PALLET coordinate system C2 is calculated. ａ ＰＡＬＬＥＴ and posture R ａ ＰＡＬＬＥＴ is the facing position t ａ ＣＡＭ and posture R ａ ＣＡＭ Convert to.

[0063] Then, the calculation unit 134 converts the facing position and orientation of the autonomous transport device 200 into a position and orientation based on the environmental drawing. Specifically, the calculation unit 134 converts the facing position and orientation of the autonomous transport device 200 from the CAM coordinate system C3 to the MAP coordinate system C1, as shown in the following formulas (6) and (7). In other words, the calculation unit 134 converts the position t Ｃ ＭＡＰ and posture R Ｃ ＭＡＰ , the facing position t of the CAM coordinate system C3 of the autonomous transport device 200 is calculated. ａ ＣＡＭ and posture R ａ ＣＡＭ t in the MAP coordinate system C1ａ ＭＡＰ and posture R ａ ＭＡＰ Convert to.

[0064] 11 is a diagram illustrating the concept of the facing position P1 for loading the transport object 5. The facing position P1 is located a predetermined distance, which is the sum of the margin m and the fork length L, from the front surface of the pallet 51 (the side of the insertion holes 511 and 512).

[0065] If the transport object 5 is not recognized in step S206, the mobile terminal 100 displays a message indicating that the object cannot be recognized on the screen (S209). In this case, the information processing system 1000 ends the position and orientation estimation process and the transport assistance process.

[0066] 9, the transport instruction unit 135 of the mobile terminal 100 transmits a transport instruction including the facing position and orientation calculated in step S208 (S103). Specifically, the transport instruction unit 135 transmits the transport instruction to the route planning device 400 via the communication network N.

[0067] In response to this, the path planning device 400 receives a transport instruction from the mobile terminal 100 via the communication network N. Then, the path planning device 400 performs a path planning process (S121). Specifically, the path planning unit 420 references the environmental map DB 300 and generates path information for moving from the current self-position and attitude of the autonomous transport device 200 received in step S113 to the facing position and attitude included in the transport instruction. Then, the path planning device 400 transmits a transport instruction including the facing position and attitude and the path information (S122). Specifically, the path planning device 400 transmits the transport instruction to the autonomous transport device 200 via the communication network N.

[0068] In response to this, the autonomous transport device 200 receives a transport instruction from the path planning device 400 via the communication network N. Then, the autonomous transport device 200 moves from its own position and posture estimated in step S112 to the facing position and posture included in the transport instruction in accordance with the path information included in the transport instruction (S131). Then, the autonomous transport device 200 loads the transport target object 5 with the fork 201 at the facing position and posture after the movement (S132).

[0069] As a result, the autonomous transport device 200 can autonomously transport the loaded transport object 5 to a predetermined position and unload it. Note that publicly known technology is used for the autonomous control after loading.

[0070] As described above, in this embodiment, the facing position and orientation of the autonomous transport device 200 is calculated by estimating the position and orientation of the transport object 5 (pallet 51) from an image captured by the mobile terminal 100. The facing position and orientation is a position and orientation at which the autonomous transport device 200 can load the transport object 5, and therefore the autonomous transport device 200 can move to a more accurate transport start point. Therefore, the autonomous transport device 200 can load the object at a practical position and orientation. Furthermore, by using the recognition result of the pallet 51 recognized from the image, the relative distance and direction between the mobile terminal 100 and the pallet 51 can be estimated. Therefore, the facing position and orientation of the autonomous transport device 200 can be calculated more accurately. Therefore, this embodiment also reduces the operational burden of an autonomous transport device that transports an object placed at an arbitrary location and promotes practical use.

[0071] In this third embodiment, an example will be described in which the mobile terminal acquires the facing position and orientation by having an external system execute the position and orientation estimation process that was previously performed internally in the mobile terminal. In the following, a description of the same functions as in the second embodiment will be omitted, and the differences from the second embodiment will be mainly described.

[0072] 12 is a block diagram showing the overall configuration of an information processing system 1000a. The information processing system 1000a includes a mobile terminal 100a, an autonomous transport device 200, an environmental map DB 300, a route planning device 400, and an estimation server 600. The mobile terminal 100a, the autonomous transport device 200, the environmental map DB 300, the route planning device 400, and the estimation server 600 are each connected to each other so as to be able to communicate with each other via a communication network N.

[0073] FIG. 13 is a block diagram showing the configuration of the mobile terminal 100a. The mobile terminal 100a is configured by deleting the (storage unit 110), the recognition model 111, the position and orientation estimation unit 132, the transport target recognition unit 133, and the calculation unit 134 from the above-described FIG. 5, and adding an estimation request unit 136. The estimation request unit 136 transmits an image and a designated position to the estimation server 600. Then, the estimation request unit 136 receives from the estimation server 600 a third position and orientation calculated based on a first position and orientation estimated from the image in the estimation server 600 and a second position and orientation estimated based on the image and the designated position. The estimation request unit 136 is an example of a second acquisition unit that acquires the third position and orientation by receiving the third position and orientation from the estimation server 600.

[0074] 14 is a block diagram showing the configuration of the estimation server 600. The estimation server 600 is a computer server that performs position and orientation estimation processing when an estimation request is received from an external device, and returns the facing position and orientation of the autonomous transport device 200. The estimation server 600 is an example of the information processing device 1 described above when the imaging terminal is the mobile terminal 100a. Note that the estimation server 600 may be realized as a computer system in which functions are distributed or made redundant using multiple computer devices. The estimation server 600 includes a storage unit 610, a reception unit 631, a position and orientation estimation unit 632, a transport target recognition unit 633, a calculation unit 634, and an output unit 635.

[0075] The storage unit 610 stores a recognition model 611. The storage unit 610 and the recognition model 611 are equivalent to the storage unit 110 and the recognition model 111 shown in FIG. 5 , respectively. The reception unit 631 receives an estimation request from the mobile terminal 100a. That is, the reception unit 631 receives an image and a specified position included in the estimation request. Therefore, the reception unit 631 is an example of the reception unit 12 described above. The position and orientation estimation unit 632 acquires a first position and orientation by estimating a first position and orientation of the mobile terminal 100a estimated from the image received by the reception unit 631 based on an environmental map including the shooting point. Therefore, the position and orientation estimation unit 632 is an example of the first acquisition unit 11 described above. In addition, the position and orientation estimation unit 632 has functions equivalent to those of the position and orientation estimation unit 132 described above. The transfer target recognition unit 633 recognizes the transfer target 5 using the recognition model 611 from the image and the specified position received by the reception unit 631. In addition, the transfer target recognition unit 633 has functions equivalent to those of the transfer target recognition unit 133 described above. The calculation unit 634 has functions equivalent to those of the calculation unit 134 described above. Therefore, the position and orientation estimation unit 632, the transfer target recognition unit 633, and the calculation unit 634 are an example of the second acquisition unit 13 described above. The output unit 635 outputs the third position and orientation (front facing position and orientation) calculated by the calculation unit 634 to the mobile terminal 100a that has issued the estimation request. Therefore, the output unit 635 is an example of the output unit 14 described above.

[0076] FIG. 15 is a sequence chart showing the flow of the transportation support process. Note that explanations of processes similar to those in FIG. 8 will be omitted where appropriate. The mobile terminal 100a photographs the transportation target 5 using the camera 121 in response to operation by the user U (S101). Then, the reception unit 131 of the mobile terminal 100a receives the image of the transportation target 5. The display device 122 displays the received image on the screen (S301). Then, the reception unit 131 receives a specified position of a part of the image on the screen by operation by the user U (S302). Then, the estimation request unit 136 transmits an estimation request including the image and the specified position to the estimation server 600 via the communication network N (S303).

[0077] 16 is a flowchart showing the flow of the position and orientation estimation process. In response to step S303, the reception unit 631 of the estimation server 600 receives an estimation request including an image and a specified position from the mobile terminal 100a via the communication network N (S311). Next, the position and orientation estimation unit 632 estimates the self-position and orientation (first position and orientation) of the mobile terminal 100a based on the image and the environmental map (S312).

[0078] The transfer target recognition unit 633 also performs a recognition process for the transfer target 5 in an area including the specified position P0 from the image included in the estimation request received in step S311 (S315). The transfer target recognition unit 633 then determines from the recognition result whether the transfer target 5 has been recognized (S316). If the transfer target 5 has been recognized, and after step S312, the position and orientation estimation unit 632 estimates a relative position and orientation (second position and orientation) of the mobile terminal 100 based on the recognition result and with respect to the transfer target 5 (S317). The calculation unit 634 then calculates a facing position and orientation of the autonomous transport device 200 for facing the transfer target 5 based on the relative position and orientation of the mobile terminal 100a and its own position and orientation (S318). Steps S312, S315, S316, S317, and S318 are the same processes as steps S202, S205, S206, S207, and S208 in FIG.

[0079] If the transport target object 5 is not recognized in step S316, the estimation server 600 returns a message indicating that the object 5 cannot be recognized to the requesting mobile terminal 100a (S319). In this case, the mobile terminal 100a may display the received message indicating that the object 5 cannot be recognized on its screen. Then, the information processing system 1000a ends the position and orientation estimation process and the transport assistance process.

[0080] 15, after step S318 in Fig. 16, the output unit 635 of the estimation server 600 transmits the facing position and orientation calculated in step S318 to the mobile terminal 100a via the communication network N (S305). In response to this, the mobile terminal 100a receives the facing position and orientation of the autonomous transport device 200 from the estimation server 600. Then, the mobile terminal 100a transmits a transport instruction including the facing position and orientation received in step S305 to the path planning device 400 via the communication network N (S103). Thereafter, processing is performed in the same manner as in Fig. 8 above.

[0081] As described above, in the third embodiment, the mobile terminal 100a has an external estimation server execute the position and orientation estimation process itself, and receives the facing position and orientation of the autonomous transport device 200. Therefore, the mobile terminal 100a can be applied to a mobile terminal with lower specifications than the mobile terminal 100 of the second embodiment. In other words, in the third embodiment, the processing cost of the mobile terminal 100a can be reduced compared to the mobile terminal 100 of the second embodiment. Furthermore, the third embodiment can also achieve the same effects as the second embodiment described above.

[0082] Other Embodiments Note that some of the processing of the estimation server 600 in the third embodiment may be executed on the mobile terminal 100a side. Furthermore, the output unit 635 of the estimation server 600 in the third embodiment may have the function of the transport instruction unit 135 in Fig. 5 above. In that case, the output unit 635 may perform processing equivalent to step S103 instead of step S305 in Fig. 15 above. In other words, the output unit 635 may transmit a transport instruction including the facing position and orientation calculated in step S318 to the path planning device 400.

[0083] Furthermore, the estimation server 600 in the third embodiment may be a desktop PC or the like. In this case, the mobile terminal 100a transmits the image captured in step S101 of FIG. 15 to the estimation server 600 via the communication network N. The reception unit 631 of the estimation server 600 then receives the image from the mobile terminal 100a. The estimation server 600 may then display the image and receive a designated position, similar to steps S203 and S204 of FIG. 9. Thereafter, the estimation server 600 may perform the same processes as steps S205 to S208 of FIG. 9.

[0084] The information processing device 1 (the mobile terminal 100 or the estimation server 600) may acquire, as the third position and orientation, the position and orientation of the object 5 to be transported calculated based on the environmental map. In this case, the path planning device 400 may calculate a facing position and orientation of the autonomous transport device 200 with respect to the object 5 to be transported, from the position and orientation of the object 5 to be transported included in the transport instruction received from the information processing device 1. Then, the path planning device 400 may generate route information using the calculated facing position and orientation. Alternatively, the path planning device 400 may generate route information from the current position of the autonomous transport device 200 to a position indicated by the position and orientation of the object 5 to be transported, without calculating the facing position and orientation. In this case, the autonomous transport device 200 may calculate a facing position and orientation with respect to the object 5 to be transported, from the position and orientation of the object 5 that is the arrival point in the route information included in the transport instruction received from the path planning device 400. Then, the autonomous transport device 200 may autonomously travel from the departure point in the route information to the position indicated by the facing position and orientation.

[0085] 17 is a block diagram showing the hardware configuration of the mobile terminals 100 and 100a. The mobile terminal 100 includes a memory 101, a processor 102, a network interface 103, a touch panel 104, and a camera 105.

[0086] The memory 101 is configured by a combination of volatile memory and non-volatile memory. The volatile memory is, for example, a volatile storage device such as RAM, and is a storage area for temporarily holding information while the processor 102 is operating. The non-volatile memory is, for example, a non-volatile storage device such as flash memory. The memory 101 stores at least a computer program that implements at least part of the processing of the information processing method of the mobile terminal 100, etc., according to the present disclosure.

[0087] The processor 102 is a control device that controls each component of the mobile terminal 100, etc. The processor 102 reads and executes software (computer programs) from the memory 101. As a result, the processor 102 realizes the functions of the reception unit 131, the position and orientation estimation unit 132, the transport target recognition unit 133, the calculation unit 134, and the transport instruction unit 135 (and the estimation request unit 136). In other words, the processor 102 performs at least a part of the processing of the information processing method in the mobile terminal 100, etc. according to the present disclosure. The processor 102 may be, for example, a microprocessor, an MPU (Multi Processing Unit), or a CPU (Central Processing Unit). The processor 102 may also include multiple processors.

[0088] The network interface 103 may be used to communicate with network nodes. The network interface 103 may include, for example, a network interface card (NIC) conforming to the IEEE 802.3 series. IEEE stands for Institute of Electrical and Electronics Engineers. The network interface 103 may also include a wireless local area network (LAN), a wired LAN, Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0089] The touch panel 104 is a display device that displays information instructed by the processor 102 and is an input device that accepts operations from the user. The touch panel 104 is, for example, a screen such as a liquid crystal display or an organic electroluminescence (EL) display. The touch panel 104 corresponds to the display device 122 and the designated position acquisition device 123 described above.

[0090] The camera 105 photographs the transport object 5 etc. in response to an instruction from the processor 102, and outputs the photographed image to the processor 102. The camera 105 is, for example, a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) sensor. The camera 105 corresponds to the camera 121 described above.

[0091] 18 is a block diagram showing the hardware configuration of the estimation server 600. The estimation server 600 includes a memory 601, a processor 602, and a network interface 603.

[0092] The memory 601 stores at least a computer program that implements at least a portion of the processing of an information processing method, including the position and orientation estimation processing of the estimation server 600 or the like according to the present disclosure. Furthermore, the memory 601 may store a recognition model 611. Other configurations of the memory 601 are similar to those of the memory 101. The processor 602 is a control device that controls each component of the estimation server 600. The processor 602 reads and executes software (computer programs) from the memory 601. As a result, the processor 602 realizes the functions of a reception unit 631, a position and orientation estimation unit 632, a transport target recognition unit 633, a calculation unit 634, and an output unit 635. That is, the processor 602 performs at least a portion of the processing of the information processing method in the estimation server 600 according to the present disclosure. Other configurations of the processor 602 are similar to those of the processor 102. The network interface 603 has a configuration similar to that of the network interface 103.

[0093] Although the present disclosure has been described above with reference to the embodiments, the present disclosure is not limited to the above-described embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure. Furthermore, each embodiment can be combined with other embodiments as appropriate.

[0094] Each drawing is merely an example for describing one or more embodiments. Each drawing may not relate to only one particular embodiment, but may also relate to one or more other embodiments. As will be understood by those skilled in the art, various features or steps described with reference to any one drawing can be combined with features or steps shown in one or more other drawings to create, for example, an embodiment not explicitly shown or described. Not all features or steps shown in any one drawing are necessary to describe an exemplary embodiment, and some features or steps may be omitted. The order of steps described in any drawing may be changed as appropriate.

[0095] Some or all of the above embodiments may be described as, but are not limited to, the following supplementary notes: (Supplementary Note A1) An information processing device comprising: a first acquisition means for acquiring a first position and orientation of an imaging terminal estimated from an image captured by the imaging terminal based on an environmental map including a shooting point; a receiving means for receiving a designated position specifying a part of the image displayed on a screen; a second acquisition means for acquiring a second position and orientation of the imaging terminal estimated based on a position of the image of the object to be transported when the object to be transported is recognized in an area of ​​the image including the designated position, and a third position and orientation related to a start point of transport by an autonomous transport device of the object to be transported calculated based on the first position and orientation; and an output means for outputting the third position and orientation. (Supplementary Note A2) The information processing device according to Supplementary Note A1, wherein the second acquisition means acquires the third position and attitude calculated as a position and attitude at which the autonomous transport device can load the transport object based on the first position and attitude and the second position and attitude, and the output means outputs a transport instruction including the third position and attitude to move the autonomous transport device to a position indicated by the third position and attitude to load the transport object. (Supplementary Note A3) The information processing device according to Supplementary Note A2, wherein the second acquisition means acquires the third position and attitude by calculating a position and attitude at which the autonomous transport device can load the transport object based on the first position and attitude and the second position and attitude. (Appendix A4) The information processing device described in any one of Appendices A1 to A3, wherein the second acquisition means estimates a group of coordinates in the image of the object to be transported by recognizing the object to be transported from the image based on the specified position, estimates a relative position and attitude of the imaging terminal based on the group of coordinates and using the position of the object to be transported as the second position and attitude, and acquires the third position and attitude by calculating a position and attitude taking into account an orientation in which the autonomous transport device can load the object to be transported based on the first position and attitude and the second position and attitude.(Supplementary Note A5) The information processing device according to any one of Supplements A1 to A3, wherein the information processing device is equipped with the photographing terminal, and the second acquisition means transmits the image and the specified position to an estimation server that estimates a position and attitude, and acquires the third position and attitude by receiving from the estimation server the third position and attitude calculated based on the first position and attitude estimated in the estimation server from the image and the second position and attitude estimated based on the image and the specified position. (Supplementary Note A6) The information processing device according to any one of Supplements A1 to A5, wherein the third position and attitude is a position and attitude of the autonomous transport device facing a direction in which the autonomous transport device can load the transport object and is at a predetermined distance from the transport object. (Supplementary Note A7) The information processing device according to any one of Supplements A1 to A6, wherein the first acquisition means acquires the first position and attitude by estimating from the image the position and attitude of the photographing terminal with respect to the environmental map. (Appendix B1) An information processing system comprising: an imaging terminal having a screen; and an autonomous transport device, wherein the imaging terminal: acquires a first position and attitude of the imaging terminal estimated from an image captured by the imaging terminal based on an environmental map including the imaging point; accepts a designated position specifying a part of the image displayed on the screen; acquires a second position and attitude of the imaging terminal estimated based on the position of the object to be transported in the image when the object to be transported is recognized in an area of ​​the image including the designated position; acquires a third position and attitude related to a start point of transport by the autonomous transport device of the object to be transported, calculated based on the first position and attitude; and outputs the third position and attitude.(Appendix B2) The information processing system described in Appendix B1, wherein the photographing terminal outputs a transport instruction including the third position and attitude so as to move the autonomous transport device to the transport start point and load the transport object, the autonomous transport device having a photographing unit, acquiring a fourth position and attitude of the autonomous transport device estimated based on the environmental map from a second image photographed by the photographing unit, acquiring the transport instruction from the photographing terminal, and moving from the fourth position and attitude to the transport start point indicated by the third position and attitude in accordance with the transport instruction, and loading the transport object after the movement. (Appendix C1) An information processing method in which a computer acquires a first position and attitude of an imaging terminal estimated from an image captured by the imaging terminal based on an environmental map including the imaging point, accepts a designated position specifying a part of the image displayed on a screen, acquires a second position and attitude of the imaging terminal estimated based on the position of the object to be transported in the image when the object to be transported is recognized in an area of ​​the image including the designated position, and acquires a third position and attitude related to the starting point of transport by an autonomous transport device of the object to be transported calculated based on the first position and attitude, and outputs the third position and attitude. (Appendix D1) An information processing program that causes a computer to execute the following steps: a first acquisition process that acquires a first position and attitude of an imaging terminal that is estimated from an image captured by the imaging terminal based on an environmental map that includes the imaging point; a reception process that accepts a designated position that specifies a part of the image displayed on a screen; a second acquisition process that acquires a second position and attitude of the imaging terminal that is estimated based on the position of the object to be transported in the image when the object to be transported is recognized in an area of ​​the image that includes the designated position, and a third position and attitude related to a start point of transport by an autonomous transport device of the object to be transported that is calculated based on the first position and attitude; and an output process that outputs the third position and attitude.

[0096] Some or all of the elements (e.g., configurations and functions) described in Appendix A2 to Appendix A7 that are dependent on Appendix A1 (e.g., device) may also be dependent on Appendix B1 (e.g., system), Appendix C1 (e.g., method), and Appendix D1 (e.g., program) in the same dependency relationship as Appendix A2 to Appendix A7. Some or all of the elements described in any appendix may be applied to various hardware, software, recording means for recording software, systems, and methods.

[0097] This application claims priority based on Japanese Patent Application No. 2024-129664, filed on August 6, 2024, the disclosure of which is incorporated herein in its entirety by reference.

[0098] 1 Information processing device, 11 First acquisition unit, 12 Reception unit, 13 Second acquisition unit, 14 Output unit, 1000 Information processing system, U User, N Communication network, 100 Portable terminal, 200 Autonomous transport device, 201 Fork, 300 Environmental map DB, 400 Path planning device, 410 Transmitting / receiving unit, 420 Path planning unit, 5 Transport object, 51 Pallet, 511 Insertion hole, 512 Insertion hole, 52 Baggage, 110 Memory unit, 111 Recognition model, 121 Camera, 122 Display device, 123 Designated position acquisition device, 131 Reception unit, 132 Position and orientation estimation unit, 133 Transport object recognition unit, 134 Calculation unit, 135 Transport instruction unit, 221 Camera, 222 Traveling device, 223 Loading device, 231 Reception unit, 232 Position and orientation estimation unit, 233 Transmitting and receiving unit, 234 Travel control unit, 235 Load handling control unit, P0 Designated position, U1 Designation operation, U2 Designation operation, C1 MAP coordinate system, C2 PALLET coordinate system, C3 CAM coordinate system, P1 Direct facing position, L Fork length, m Margin, 1000a Information processing system, 100a Portable terminal, 136 Estimation request unit, 600 Estimation server, 610 Storage unit, 611 Recognition model, 631 Reception unit, 632 Position and orientation estimation unit, 633 Conveyance object recognition unit, 634 Calculation unit, 635 Output unit, 101 Memory, 102 Processor, 103 Network interface, 104 Touch panel, 105 Camera, 601 Memory, 602 Processor, 603 Network interface

Claims

1. An information processing device comprising: a first acquisition means for acquiring a first position and attitude of an image captured by an image capturing device, estimated based on an environmental map including the image capturing location; a reception means for receiving a designated position that specifies a portion of the image displayed on a screen; a second acquisition means for acquiring a second position and attitude of the image capturing device, estimated based on the position of the object to be transported in the image when the object to be transported is recognized in an area of ​​the image including the designated position, and a third position and attitude related to the starting point of transport by an autonomous transport device of the object to be transported, calculated based on the first position and attitude; and an output means for outputting the third position and attitude.

2. The information processing device according to claim 1, wherein the second acquisition means acquires the third position and attitude calculated as a position and attitude at which the autonomous transport device can load the transport object based on the first position and attitude and the second position and attitude, and the output means outputs a transport instruction including the third position and attitude so as to move the autonomous transport device to a position indicated by the third position and attitude and load the transport object.

3. The information processing device according to claim 2, wherein the second acquisition means acquires the third position and attitude by calculating a position and attitude at which the autonomous transport device can load the transport object based on the first position and attitude and the second position and attitude.

4. The information processing device described in claim 1 or 2, wherein the second acquisition means estimates a group of coordinates in the image of the object to be transported by recognizing the object to be transported from the image based on the specified position, estimates the relative position and attitude of the photographing terminal based on the group of coordinates and using the position of the object to be transported as the second position and attitude, and acquires the third position and attitude by calculating a position and attitude taking into account an orientation in which the autonomous transport device can load the object to be transported based on the first position and attitude and the second position and attitude.

5. The information processing device according to claim 1 or 2, wherein the information processing device is equipped with the photographing terminal, and the second acquisition means acquires the third position and orientation by transmitting the image and the specified position to an estimation server that estimates the position and orientation, and receiving from the estimation server the third position and orientation calculated based on the first position and orientation estimated from the image in the estimation server and the second position and orientation estimated based on the image and the specified position.

6. An information processing device according to claim 1 or 2, wherein the third position and attitude is a position and attitude of the autonomous transport device facing a direction in which the autonomous transport device can load the transport object and being at a predetermined distance from the transport object.

7. An information processing device according to any one of claims 1 to 6, wherein the first acquisition means acquires the first position and orientation by estimating the position and orientation of the photographing terminal based on the environmental map from the image.

8. An information processing system comprising: an imaging terminal having a screen; and an autonomous transport device, wherein the imaging terminal acquires a first position and attitude of the imaging terminal estimated from an image captured by the imaging terminal based on an environmental map including the imaging point; accepts a designated position specifying a part of the image displayed on the screen; acquires a second position and attitude of the imaging terminal estimated based on the position of the object to be transported in the image when the object to be transported is recognized in an area of ​​the image including the designated position; and acquires a third position and attitude related to the start point of transport by the autonomous transport device of the object to be transported, calculated based on the first position and attitude; and outputs the third position and attitude.

9. The information processing system of claim 8, wherein the photographing terminal outputs a transport instruction including the third position and attitude so as to move the autonomous transport device to the transport start point and load the transport object, the autonomous transport device having a photographing unit, acquiring a fourth position and attitude of the autonomous transport device estimated based on the environmental map from a second image photographed by the photographing unit, acquiring the transport instruction from the photographing terminal, and moving from the fourth position and attitude to the transport start point indicated by the third position and attitude in accordance with the transport instruction, and loading the transport object after the movement.

10. The information processing system according to claim 8, wherein the photographing terminal acquires the third position and attitude calculated as a position and attitude at which the autonomous transport device can load the transport object based on the first position and attitude and the second position and attitude, and outputting the third position and attitude includes outputting a transport instruction including the third position and attitude so as to move the autonomous transport device to a position indicated by the third position and attitude and load the transport object.

11. The information processing system described in claim 8, wherein the photographing terminal acquires the third position and attitude by calculating a position and attitude at which the autonomous transport device can load the transport object based on the first position and attitude and the second position and attitude.

12. The information processing system described in claim 8, wherein the photographing terminal estimates a group of coordinates in the image of the object to be transported by recognizing the object to be transported from the image based on the specified position, estimates the relative position and attitude of the photographing terminal based on the group of coordinates and using the position of the object to be transported as the second position and attitude, and calculates a position and attitude taking into account an orientation in which the autonomous transport device can load the object to be transported based on the first position and attitude and the second position and attitude, thereby obtaining the third position and attitude.

13. The information processing system of claim 8, wherein the photographing terminal transmits the image and the specified position to an estimation server that estimates the position and orientation, and receives from the estimation server the third position and orientation calculated based on the first position and orientation estimated from the image in the estimation server and the second position and orientation estimated based on the image and the specified position, thereby acquiring the third position and orientation.

14. An information processing system as described in claim 8, wherein the third position and attitude is a position and attitude of the autonomous transport device facing a direction in which the transport object can be loaded and at a predetermined distance from the transport object.

15. An information processing system according to claim 8, wherein the photographing terminal acquires the first position and orientation by estimating the position and orientation of the photographing terminal based on the environmental map from the image.

16. An information processing method in which a computer acquires a first position and attitude of an imaging terminal estimated from an image captured by the imaging terminal based on an environmental map including the imaging point; accepts a designated position specifying a part of the image displayed on a screen; acquires a second position and attitude of the imaging terminal estimated based on the position of the object to be transported in the image when the object to be transported is recognized in an area of ​​the image including the designated position; and acquires a third position and attitude related to the starting point of transport by an autonomous transport device of the object to be transported calculated based on the first position and attitude; and outputs the third position and attitude.

17. The information processing method according to claim 16, wherein the computer acquires the third position and attitude calculated as a position and attitude at which the autonomous transport device can load the transport object based on the first position and attitude and the second position and attitude, and outputting the third position and attitude includes outputting a transport instruction including the third position and attitude so as to move the autonomous transport device to a position indicated by the third position and attitude and load the transport object.

18. The information processing method according to claim 16, wherein the computer acquires the third position and attitude by calculating a position and attitude at which the autonomous transport device can load the transport object based on the first position and attitude and the second position and attitude.

19. The information processing method described in claim 16, wherein the computer estimates a group of coordinates in the image of the object to be transported by recognizing the object to be transported from the image based on the specified position, estimates the relative position and attitude of the imaging terminal based on the group of coordinates and using the position of the object to be transported as the second position and attitude, and calculates a position and attitude that takes into account an orientation in which the autonomous transport device can load the object to be transported based on the first position and attitude and the second position and attitude, thereby obtaining the third position and attitude.

20. An information processing program that causes a computer to execute the following steps: a first acquisition process that acquires a first position and attitude of the photographing terminal estimated from an image photographed by the photographing terminal based on an environmental map including the photographing location; a reception process that accepts a designated position that specifies a part of the image displayed on the screen; a second acquisition process that acquires a second position and attitude of the photographing terminal estimated based on the position of the object to be transported in the image when the object to be transported is recognized in an area of ​​the image that includes the designated position, and a third position and attitude related to the starting point of transport by the autonomous transport device of the object to be transported, calculated based on the first position and attitude; and an output process that outputs the third position and attitude.

Images