Control system, information processing device, program and control method

Abstract

To provide a control system that can easily control autonomous operation of a mobility, an information processing device, a program, and a control method.SOLUTION: A control system comprises: a detection device that detects operation of a mobility as operation information; an information processing device that grasps a spatial area in which the mobility operates based on the operation information on the mobility detected by the detection device; and a control device that controls the autonomous operation of the mobility based on the spatial area grasped by the information processing device.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a control system, an information processing device, a program, and a control method, and particularly to a control system, an information processing device, a program, and a control method for controlling mobility. [Background technology]

[0002] In recent years, technology related to autonomously running or driving mobility has been advancing, and technologies have been proposed to control various types of mobility, such as large heavy machinery such as crane vehicles, flying objects such as drones, or robots that sort luggage in warehouses, to run or drive autonomously in any space or area.

[0003] Patent Document 1 discloses a technology for setting a departure point and an arrival point for an autonomous moving body, generating route information for the autonomous moving body from the departure point and the arrival point, and controlling the movement of the autonomous moving body based on the generated route information. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent Publication No. 2023-112671 Summary of the Invention [Problem to be solved by the invention]

[0005] Incidentally, when controlling the autonomous operation of mobility, it is necessary to set in advance information regarding the route along which the mobility will travel and the operating range within which the mobility will operate, based on, for example, map information and geographical information of the local area where the mobility will operate.

[0006] The task of setting up such information in advance may in some cases require the creation of local geographical information in advance, and in such cases it is expected that the number of steps required for setting up will increase, and it is expected that precise information will need to be set up, so there is a concern that the workload when executing control over the autonomous operation of mobility will increase.

[0007] The present invention has been made in consideration of the above circumstances, and its objective is to provide a control system, an information processing device, a program, and a control method that can easily control the autonomous operation of mobility. [Means for solving the problem]

[0008] In order to achieve the above-mentioned object, the control system of the present invention comprises a detection device that detects the movement of the mobility as movement information, an information processing device that grasps the spatial area in which the mobility operates based on the movement information of the mobility detected by the detection device, and a control device that controls the autonomous movement of the mobility based on the spatial area grasped by the information processing device.

[0009] This allows the spatial area in which the mobility operates to be grasped based on the operation information detected by the detection device when the mobility operates, and therefore the autonomous operation of the mobility can be easily controlled based on the grasped spatial area.

[0010] The information processing device of this control system executes a reception process that receives input of the initial position of the mobility relative to the spatial region, the starting position of the mobility's operation, and the ending position of the mobility's operation, and also executes a path calculation process that calculates the path along which the mobility will operate in the spatial region based on the initial position, starting position, and ending position received in the reception process.

[0011] The information processing device of this control system executes an object recognition process in which, when the detection device detects an object in the spatial domain while the mobility is operating, the detected object is recognized as object recognition information, and the control device controls the autonomous operation of the mobility based on the object recognition information recognized by the object recognition process.

[0012] In this control system, multiple detection devices are provided in the mobility, and the information processing device performs a synchronization process to synchronize the operation information detected by the multiple detection devices based on the time when the operation information was detected by the multiple detection devices.

[0013] In order to achieve the above-mentioned object, the information processing device of the present invention is an information processing device having a processor and a memory in which a program is stored, and by executing the program by the processor, it performs a spatial area recognition process that recognizes the spatial area in which the mobility operates based on mobility operation information detected by a detection device that detects the mobility operation as operation information, a reception process that accepts input of the initial position of the mobility with respect to the spatial area recognized in the spatial area recognition process, the starting position of the mobility operation, and the ending position of the mobility operation, and a path calculation process that calculates the path in which the mobility operates in the spatial area based on the initial position, starting position, and ending position received in the reception process.

[0014] To achieve the above-mentioned object, the program of the present invention causes an information processing device implemented by a computer to execute a spatial area recognition process that recognizes the spatial area in which the mobility operates based on mobility operation information detected by a detection device that detects the mobility operation as operation information, a reception process that receives input of the initial position of the mobility with respect to the spatial area recognized in the spatial area recognition process, the starting position of the mobility operation, and the ending position of the mobility operation, and a path calculation process that calculates the path in which the mobility operates in the spatial area based on the initial position, starting position, and ending position received in the reception process.

[0015] To achieve the above-mentioned object, the control method of the present invention involves an information processing device implemented by a computer executing a spatial area recognition process that recognizes the spatial area in which the mobility operates based on movement information of the mobility detected by a detection device that detects the movement of the mobility as movement information, a reception process that receives input of the initial position of the mobility with respect to the spatial area recognized in the spatial area recognition process, the starting position of the mobility's movement, and the ending position of the mobility's movement, and a route calculation process that calculates the route along which the mobility operates in the spatial area based on the initial position, starting position, and ending position received in the reception process, and a control device implemented by the computer controls the autonomous operation of the mobility based on the route calculated in the route calculation process. [Effects of the Invention]

[0016] According to the present invention, the autonomous operation of the mobility can be easily controlled. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a block diagram illustrating an outline of the configuration of a control system according to an embodiment of the present invention. [Figure 2] FIG. 10 is a diagram for explaining an outline of a camera of the control system according to the present embodiment. [Figure 3] 1 is a block diagram illustrating an outline of the configuration of a computer that implements an information processing device and a control device of the control system according to the present embodiment. [Figure 4] 1 is a block diagram for explaining an outline of the functions of an information processing device of the control system according to the present embodiment. FIG. [Figure 5] FIG. 10 is a diagram for explaining an outline of three-dimensional map information processed by the control system according to the present embodiment. [Figure 6] FIG. 10 is a diagram for explaining an outline of the processing of the control system according to the present embodiment. [Figure 7] 1 is a block diagram for explaining an outline of the functions of a control device of a control system according to the present embodiment. FIG. [Figure 8] 10 is a flowchart outlining the processing of the control system according to the present embodiment. [Figure 9] FIG. 2 is a diagram for explaining the outline of the configuration of a crane vehicle that is the control target of the control system according to the present embodiment. [Figure 10] FIG. 2 is a block diagram illustrating an outline of the configuration of a crane vehicle that is the object of control by the control system according to the present embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0018] Next, a control system according to an embodiment of the present invention will be described with reference to FIGS.

[0019] First, before describing the control system, an outline of the configuration of the mobility that is the control target of the control system of this embodiment will be described.

[0020] <Mobility configuration> 9 and 10 are diagrams for explaining an outline of a mobility that is the control target of the control system of this embodiment. As shown in the figures, the mobility is a crane vehicle 200 in this embodiment.

[0021] This crane vehicle 200 is equipped with a crane device 210, which includes a hydraulic supply device 211, a group of hydraulic actuators 212, a post 213, a boom 214, and a winch 215 as its main components.

[0022] The hydraulic supply device 211 is a device that supplies hydraulic oil to the hydraulic actuator group 212, and includes a hydraulic pump driven by an engine (not shown) of the crane vehicle 200, piping that transports the hydraulic oil when driven by the hydraulic pump, and an electromagnetic valve installed in the piping.

[0023] The hydraulic actuator group 212 is made up of a plurality of devices that operate the post 213, the boom 214, and the winch 215, and includes, for example, a swing motor, a boom cylinder, a telescopic cylinder, a hydraulic motor, and the like.

[0024] The swing motor swings the boom 214 via the post 213 when hydraulic oil is supplied to it, and the boom raising and lowering cylinder extends and contracts when hydraulic oil is supplied to it, causing the boom 214 to rise and lower.

[0025] The telescopic cylinder extends and retracts when hydraulic oil is supplied to it, thereby extending and retracting the boom 214, and the hydraulic motor rotates when hydraulic oil is supplied to it, thereby rotating the drum of the winch 215.

[0026] Next, the configuration of the control system according to this embodiment will be described.

[0027] <Overall system configuration> 1 is a block diagram illustrating an outline of the configuration of a control system according to this embodiment. As shown in the figure, a control system 10 mainly comprises a plurality of cameras 20 serving as detection devices, an information processing device 30, and a control device 40, which are connected to each other via a network N such as the Internet.

[0028] In this embodiment, the camera 20 is provided on a mobility vehicle, in this embodiment a crane vehicle 200, the information processing device 30 is managed by a business operator 1 that provides services using the control system 10, and the control device 40 is arranged in the crane vehicle 200, for example, in the passenger seat.

[0029] In this embodiment, the service using the control system 10 is a service that provides any business operator that operates the crane vehicle 200 with a process for controlling the crane vehicle 200 using the information processing device 30.

[0030] Next, the specific configuration of each part of the control system 10 will be described.

[0031] <Camera configuration> The camera 20 captures an image of any object as a subject, and in this embodiment, detects and acquires the operation of the crane vehicle 200 as operation information.

[0032] In this embodiment, this camera 20 detects and acquires the operation of the crane vehicle 200 within the work area where the crane vehicle 200 performs any work as operation information, and in this embodiment, the operation information is image information that includes both moving images and still images.

[0033] 2 is a diagram illustrating an example in which multiple cameras 20 are provided on a mobile crane 200. As shown in the figure, the cameras 20 are attached at positions that allow an overall view of the operation of the mobile crane 200, such as two cameras at both ends of the front of the mobile crane 200, two cameras at both ends of the rear of the mobile crane 200, one camera at the tip of the boom 214, and one camera between the tip and base of the boom 214.

[0034] For example, cameras 20 attached to both the front and rear ends of the crane vehicle 200 can detect the movement of the crane vehicle 200, and cameras 20 attached to the boom 214 can detect the movement of the boom 214, so it is expected that the overall movement of the crane vehicle 200 can be grasped.

[0035] <Computer configuration> In this embodiment, the information processing device 30 and the control device 40 shown in FIG. 1 are implemented by a computer having substantially the same hardware configuration, such as a desktop or notebook computer.

[0036] 3 is a block diagram illustrating the general configuration of a computer. As shown in the figure, the computer mainly comprises a processor 101, a memory 102, a storage 103, a transmitting / receiving unit 104, and an input / output unit 105, which are electrically connected to each other via a bus 106.

[0037] The processor 101 is an arithmetic unit that controls the operation of the computer, controls the transmission and reception of data between elements, and performs processing necessary for executing application programs.

[0038] In this embodiment, the processor 101 is, for example, a CPU (Central Processing Unit), and executes programs loaded in a memory 102 (described below) to perform various processes.

[0039] The memory 102 is implemented by a main storage device that is configured as a volatile storage device such as a DRAM (Dynamic Random Access Memory).

[0040] The memory 102 is used as a working area for the processor 101, and also stores a BIOS (Basic Input / Output System) that is executed when the information processing device 30 is started up, various setting information, and the like.

[0041] The storage 103 stores data and the like used for various processes by application programs and the like.

[0042] The transmitting / receiving unit 104 connects the computer to the network N. This transmitting / receiving unit 104 may be compatible with a wireless communication standard such as Wi-Fi, or may be equipped with a short-range communication interface such as Bluetooth (registered trademark) or BLE (Bluetooth Low Energy).

[0043] If necessary, information input devices such as a keyboard and a mouse and output devices such as a display are connected to the input / output unit 105. In this embodiment, a keyboard, a mouse, and a display are all connected.

[0044] The bus 106 transmits, for example, address signals, data signals, and various control signals between the processor 101, memory 102, storage 103, transmission / reception unit 104, and input / output unit 105 that are connected to it.

[0045] <Functions of information processing device> 4 is a block diagram illustrating an outline of the functions of the information processing device 30 of the control system 10. As shown in the figure, the information processing device 30 includes a motion information receiving unit 31, a motion information synchronization unit 32, a motion information storage unit 33, a spatial region recognition unit 34, a spatial region information storage unit 35, a position input receiving unit 36, a path calculation unit 37, a matching unit 38, and an object recognition unit 39.

[0046] The motion information storage unit 33 and the spatial region information storage unit 35 are realized by partitioning the storage region of the storage 103 .

[0047] On the other hand, the motion information receiving unit 31, the motion information synchronization unit 32, the spatial area recognition unit 34, the position input receiving unit 36, the path calculation unit 37, the matching unit 38 and the object recognition unit 39 are realized by the processor 101 executing a program stored in the memory 102.

[0048] The operation information receiving unit 31 receives input of operation information acquired by the plurality of cameras 20 attached to the crane vehicle 200.

[0049] In this embodiment, the operation information synchronization unit 32 performs synchronization processing of multiple pieces of operation information by matching the time codes between multiple pieces of operation information, for example, based on the time codes that are constantly recorded in units of [hours / minutes / seconds / frames] in each of the multiple cameras 20 (synchronization processing).

[0050] In this embodiment, the motion information storage section 33 stores the motion information for which the motion information synchronization section 32 has executed the synchronization process.

[0051] In this embodiment, the spatial area grasping unit 34 grasps the spatial area in which the crane vehicle 200 has operated based on the operation information stored in the operation information storage unit 33, grasps the grasped spatial area as a point cloud, and executes a process to generate three-dimensional map information (spatial area grasping process).

[0052] 5 is a diagram showing an example of a spatial area (three-dimensional map information) grasped by the spatial area grasping unit 34. As shown in the figure, the spatial area R is a three-dimensional, stereoscopic area having an x ​​direction, a y direction, and a z direction, and in this embodiment, this spatial area R is grasped as a working area of ​​the crane vehicle 200.

[0053] In this embodiment, the spatial region information storage unit 35 shown in FIG. 4 stores three-dimensional map information generated from the spatial region R grasped by the spatial region grasping unit 34 as spatial region information.

[0054] In this embodiment, the position input receiving unit 36 ​​executes a process of receiving input of the initial position of the crane vehicle 200, the starting position of the operation of the crane vehicle 200, and the ending position of the operation of the crane vehicle 200 (reception process).

[0055] 6 is a diagram illustrating an outline of the processing of the position input receiving unit 36. As shown in the figure, in this embodiment, the position input receiving unit 36 ​​receives the position where the crane vehicle 200 stops prior to the start of work as the initial position (x0, y0, z0).

[0056] On the other hand, the position input receiving unit 36 ​​receives the position where the crane vehicle 200 starts operating, in this embodiment the sling position where the material S is engaged with the hook of the crane device 210, as the start position (x1, y1, z1), and receives the position where the crane vehicle 200 ends operating, in this embodiment the position where the material S is placed in the spatial region R, as the end position (x2, y2, z2).

[0057] In this embodiment, the path calculation unit 37 shown in Figure 4 executes a process of calculating a path along which the crane vehicle 200 will operate in the spatial region R based on the initial position, start position, and end position received by the position input receiving unit 36 ​​(path calculation process).

[0058] For example, the route along which the crane vehicle 200 (crane device 210) operates is calculated from the coordinates of the initial position (x0, y0, z0), via the coordinates of the start position (x1, y1, z1), to the coordinates of the end position (x2, y2, z2).

[0059] In this embodiment, the calculated route is stored in the spatial region information storage unit 35 in a manner superimposed on the spatial region information stored therein.

[0060] In this embodiment, the comparison unit 38 compares the spatial region information stored in the spatial region information storage unit 35 with the operation information of the crane vehicle 200 acquired in real time by the multiple cameras 20 attached to the crane vehicle 200, and performs a process of generating control information for autonomously operating the crane vehicle 200 according to the path calculated by the path calculation unit 37.

[0061] In this embodiment, the control signal generated by the collating unit 38 is transmitted to a control device 40, which will be described later.

[0062] In this embodiment, when the camera 20 detects any object in the spatial region R while the crane vehicle 200 is operating, the object grasping unit 39 executes a process of grasping the detected object as object confirmation information (object grasping process).

[0063] If the object grasping unit 39 detects an object while the comparison unit 38 is comparing the spatial domain information with the operation information acquired in real time by the camera 20, the object grasping unit 39 generates correction information for correcting the operation of the crane vehicle 200 so as to avoid the detected object and transmits it to the control device 40, and also corrects the spatial domain information stored in the spatial domain information storage unit 35 based on the correction information.

[0064] <Controller function> 7 is a block diagram illustrating an outline of the functions of the control device 40. As shown in the figure, the control device 40 includes a control processing unit 41, which is realized by the processor 101 executing a program stored in a memory 102.

[0065] In this embodiment, the control processing unit 41 generates a control signal that controls the autonomous operation of the crane vehicle 200 based on the control information and correction information transmitted from the information processing device 30, and controls the crane vehicle 200.

[0066] Next, an outline of the processing of the control system 10 according to this embodiment will be described.

[0067] <System processing flow> 8 is a flowchart outlining the processing of the control system. First, prior to carrying out actual work, when the crane vehicle 200 is operated arbitrarily in the work area (for example, swinging, raising or lowering or extending the boom 214), as shown in the figure, in step S1, multiple pieces of operation information of the crane vehicle 200 acquired by the multiple cameras 20 are accepted (initial operation setting).

[0068] In step S2, a synchronization process is executed on the received plurality of pieces of motion information (synchronization process), and the synchronized motion information is stored in the motion information storage unit 33.

[0069] Next, in step S3, the spatial area in which the crane vehicle 200 has operated is determined based on the operation information, and the determined spatial area is determined as a point cloud to generate three-dimensional map information (spatial area determination process). The spatial area determined as this spatial area information is determined as the working area of ​​the crane vehicle 200.

[0070] Meanwhile, in step S4, the initial position of the crane vehicle 200 (the position where the crane vehicle 200 stops prior to the start of work), the starting position (rigging position) of the crane vehicle 200, and the ending position of the crane vehicle 200 (the position where the material S is placed in the spatial region R) are accepted, for example, as coordinate information (acceptance process).

[0071] In step S5, a process is executed to calculate a path along which the crane vehicle 200 operates in the spatial region R based on the initial position, start point position, and end point position received as coordinate information (path calculation process).

[0072] After the route calculation is performed, the process moves to actual work using the crane vehicle 200. In this embodiment, it is assumed that the crane vehicle 200 transports materials from a starting point position to an arbitrary installation position in a work area grasped as a spatial region R for the purpose of constructing an arbitrary structure.

[0073] In actual operation, first, in step S6, the spatial domain information is compared with the operation information of the crane vehicle 200 acquired in real time by the multiple cameras 20 attached to the crane vehicle 200, and control information is generated to autonomously operate the crane vehicle 200 according to the route calculated by the route calculation unit 37.

[0074] Based on this control information, the control device 40 generates a control signal for controlling the autonomous operation of the crane vehicle 200, thereby controlling the autonomous operation of the crane vehicle 200. In this way, the material S is transported from the start position to the end position.

[0075] When the camera 20 detects any object in the spatial region while the crane vehicle 200 is performing actual work, the detected object is recognized as object confirmation information in step S7 (object recognition process).

[0076] At this time, in step S8, correction information for correcting the operation of the crane vehicle 200 so as to avoid the detected object is generated and transmitted to the control device 40, and the spatial region information stored in the spatial region information storage unit 35 is corrected based on the correction information.

[0077] The control device 40 controls the operation of the crane vehicle 200 based on the correction information so as to avoid the detected object.

[0078] <Action and effect> In this way, the spatial area R in which the crane vehicle 200 operates can be grasped based on the operation information detected by the camera 20 when the crane vehicle 200 operates, and the autonomous operation of the crane vehicle 200 can be easily controlled based on the grasped spatial area R.

[0079] <Other embodiments> The present invention is not limited to the above-described embodiment, and various modifications are possible without departing from the spirit of the invention.

[0080] In the above embodiment, a case has been described in which the crane vehicle 200 is controlled based on three-dimensional map information in which the spatial region R grasped from the operation information is grasped as a point cloud. However, for example, the crane vehicle 200 may be controlled by reading a pre-created spatial region into markers placed in the working area, or the crane vehicle 200 may be controlled based on a three-dimensional model of the working area.

[0081] In the above embodiment, the detection device is described as a camera 20, but the detection device may also be various sensors such as a GPS sensor or distance sensor that detects the position of mobility such as a crane vehicle 200 or a crane device 210, or similar detection devices.

[0082] In the above embodiment, the mobility is described as a crane vehicle 200, but the mobility may be various other types of mobility, such as large heavy machinery other than the crane vehicle 200 (power shovel, bulldozer, etc.), an aircraft such as a drone, or a robot that sorts luggage in a warehouse, etc.

[0083] In the above embodiment, the case where the camera 20 and the information processing device 30 are connected via the network N has been described, but for example, the camera 20 and the information processing device 30 may be connected via a relay device between them.

[0084] In the above embodiment, the case where the information processing device 30 is implemented on a computer managed by the business operator 1 has been described, but for example, the information processing device 30 may be a computer implemented in a cloud environment. [Explanation of symbols]

[0085] 1 business operator 10. Control System 20 Camera (detection device) 30 Information processing equipment 40 Control device 200 Crane Vehicles (Mobility) 210 Crane equipment

Claims

1. a detection device that detects the movement of the mobility as movement information; an information processing device that grasps a spatial region in which the mobility is operating based on the motion information of the mobility detected by the detection device; a control device that controls the autonomous operation of the mobility based on the spatial region grasped by the information processing device; A control system comprising:

2. The information processing device includes: executing a reception process for receiving input of an initial position of the mobility with respect to the spatial region, a start position of a motion of the mobility, and an end position of the motion of the mobility; The control system of claim 1 .

3. The information processing device includes: a reception process for receiving input of an initial position of the mobility with respect to the spatial region, a start position of a motion of the mobility, and an end position of the motion of the mobility; a route calculation process for calculating a route along which the mobility will operate in the spatial domain based on the initial position, the start position, and the end position received in the reception process; 3. A control system according to claim 1 or 2.

4. The information processing device includes: executing an object recognition process for recognizing an object as object confirmation information when the detection device detects an object in the spatial region while the mobility is operating; The control device controlling an autonomous operation of the mobility based on the object confirmation information grasped in the object grasping process; 3. A control system according to claim 1 or 2.

5. A plurality of the detection devices are provided on the mobility; The information processing device includes: executing a synchronization process for synchronizing the motion information detected by the plurality of detection devices based on the time when the motion information was detected by the plurality of detection devices; 3. A control system according to claim 1 or 2.

6. An information processing device including a processor and a memory in which a program is stored, When the program is executed by the processor, a spatial area recognition process for recognizing a spatial area in which the mobility is operating based on the motion information of the mobility detected by a detection device that detects the motion of the mobility as motion information; a receiving process for receiving input of an initial position of the mobility, a start position of a motion of the mobility, and an end position of a motion of the mobility with respect to the spatial region grasped in the spatial region grasping process; a route calculation process for calculating a route along which the mobility will operate in the spatial region based on the initial position, the start position, and the end position received in the reception process; An information processing device that executes the above.

7. An information processing device implemented by a computer, a spatial area recognition process for recognizing a spatial area in which the mobility is operating based on the motion information of the mobility detected by a detection device that detects the motion of the mobility as motion information; a receiving process for receiving input of an initial position of the mobility, a start position of a motion of the mobility, and an end position of a motion of the mobility with respect to the spatial region grasped in the spatial region grasping process; a route calculation process for calculating a route along which the mobility will operate in the spatial region based on the initial position, the start position, and the end position received in the reception process; A program that executes the following.

8. An information processing device implemented by a computer, a spatial area recognition process for recognizing a spatial area in which the mobility is operating based on the motion information of the mobility detected by a detection device that detects the motion of the mobility as motion information; a receiving process for receiving input of an initial position of the mobility, a start position of a motion of the mobility, and an end position of a motion of the mobility with respect to the spatial region grasped in the spatial region grasping process; a route calculation process for calculating a route along which the mobility will operate in the spatial domain based on the initial position, the start position, and the end position received in the reception process; a computer-implemented control device, controlling an autonomous operation of the mobility based on the route calculated in the route calculation process; Control method.

Images