Control device, control method, and computer-readable recording medium

Abstract

This control device comprises: a generation unit that uses state information indicating the movement of a work device and environmental information indicating a work environment including a part or all of the work device observed by an observation device, and generates controlled unit information indicating the current state of a controlled unit of the work device and obstacle information indicating obstacles present in the work environment; a detection unit that generates detection information indicating the distance between the controlled unit and the obstacles on the basis of the controlled unit information and the obstacle information; a determination unit that uses the distance between the controlled unit and an object among the obstacles that the controlled unit should not approach to determine whether the controlled unit and the object are approaching each other; and a planning unit that generates plan information for controlling the controlled unit so as not to approach the object when it is determined that the controlled unit is approaching the object.

Description

Control device, control method, and computer-readable recording medium

[0001] The present disclosure relates to a control device, a control method, and a computer-readable recording medium.

[0002] There are known techniques for improving the efficiency of work using work devices such as construction machines with movable mechanisms, arm robots, etc. There are also known techniques for preventing the work device from coming into contact with obstacles that may impede the work, such as structures, workers, or other work devices.

[0003] As a related technique, Patent Document 1 discloses a maneuvering assistance system that reduces the burden on the operator of a work implement. The maneuvering assistance system in Patent Document 1 selects from a number of avoidance actions based on the relative distance to an obstacle and an estimated separation distance based on an estimated movement course.

[0004] Japanese Patent Application Laid-Open No. 2020-193503

[0005] However, in the obstacle determination in Patent Document 1, an avoidance operation is selected based on the relative distance between a certain determination point on the working device and the obstacle, and therefore, it is not possible to consider the possibility that a point on the working device other than the initial determination point may approach the obstacle. Furthermore, determination based on the relative distance to the obstacle, i.e., size, does not take into account direction (orientation) or speed. Furthermore, the accuracy of determination based on the estimated separation distance is based on the estimated movement course of the working device and the obstacle, and therefore depends on the accuracy of that estimation. In particular, hydraulically controlled construction machinery (e.g., excavators, backhoes, cranes, etc.) is subject to overshoot and undershoot due to control errors and inertia, and therefore may not always be able to be controlled according to the estimated movement course.

[0006] In other words, with the technology disclosed in Patent Document 1, if the accuracy of the judgment is insufficient, braking may not be applied in time, safety may not be ensured, or work efficiency may decrease due to false detection. Also, the avoidance action is a stop and a predetermined action, making it difficult to continue work, and therefore work efficiency decreases.

[0007] As described above, with the technology disclosed in Patent Document 1, it is difficult to achieve both safety and work efficiency in unexpected situations where control errors and inertia have an effect on a work device having a moving part. In other words, it is difficult to ensure safety by detecting an obstacle early, ensuring a braking distance, avoiding the obstacle without stopping work as much as possible, and suppressing a decrease in work efficiency.

[0008] An object of the present invention is to ensure safety and prevent a decrease in work efficiency when working with a work device having a movable part.

[0009] In order to achieve the above object, a control device in one aspect of the present disclosure is characterized by having: a generation unit that generates controlled unit information that represents the current state of a controlled unit possessed by the work device and obstacle information that represents obstacles present in the work environment, using status information that represents the movement of the work device and environmental information that represents a work environment including all or part of the work device observed by an observation device; a detection unit that generates detection information that represents the distance between the controlled unit and the obstacle based on the controlled unit information and the obstacle information; a determination unit that determines whether the controlled unit and the object are approaching each other, using an object that the controlled unit must not approach among the obstacles and the distance between the controlled unit and the object; and a planning unit that generates planning information to control the controlled unit not to approach the object when it is determined that the controlled unit is approaching the object.

[0010] Furthermore, in order to achieve the above object, a control method in one aspect of the present disclosure is characterized in that an information processing device uses status information representing the movement of a work device and environmental information representing a work environment including part or all of the work device observed by an observation device to generate controlled unit information representing the current state of a controlled unit possessed by the work device and obstacle information representing obstacles present in the work environment; generates detection information representing the distance between the controlled unit and the obstacle based on the controlled unit information and the obstacle information; determines whether the controlled unit and an object that the controlled unit must not approach are approaching each other using the distance between the controlled unit and an object among the obstacles; and if it is determined that the controlled unit is approaching the object, generates plan information to control the controlled unit not to approach the object.

[0011] Furthermore, in order to achieve the above object, a computer-readable recording medium according to one aspect of the present disclosure is characterized in that it causes a computer to: use status information representing the movement of a work device and environmental information representing a work environment including all or part of the work device observed by an observation device to generate controlled unit information representing the current state of a controlled unit possessed by the work device and obstacle information representing obstacles present in the work environment; generate detection information representing the distance between the controlled unit and the obstacle based on the controlled unit information and the obstacle information; determine whether the controlled unit and the object are approaching each other using the distance between the controlled unit and an object that the controlled unit must not approach among the obstacles; and if it is determined that the controlled unit is approaching the object, generate plan information for controlling the controlled unit not to approach the object.

[0012] As described above, according to the present disclosure, safety can be ensured and a decrease in work efficiency can be suppressed in work using a work device having a movable part.

[0013] FIG. 1 is a diagram for explaining an example of the configuration of a control system according to the first embodiment. FIG. 2 is a diagram for explaining an example of the operation of a control device according to the first embodiment. FIG. 3 is a diagram for explaining an example of detection information in the control device according to the first embodiment. FIG. 4 is a diagram for explaining an example of a determination process and an operation restriction of the control device according to the first embodiment. FIG. 5 is a diagram for explaining an example of a determination process and a control correction of the control device according to the first embodiment. FIG. 6 is a diagram for explaining an example of the operation of a control device according to the second embodiment. FIG. 7 is a diagram for explaining an example of a determination process and a trajectory correction of the control device according to the second embodiment. FIG. 8 is a diagram for explaining an example of the configuration of a control system according to the third embodiment. FIG. 9 is a diagram for explaining an example of the operation of the control device according to the third embodiment. FIG. 10 is a diagram for explaining an example of a computer that realizes the control devices according to the first, second, and third embodiments.

[0014] Hereinafter, embodiments will be described with reference to the drawings. In the drawings described below, elements having the same or corresponding functions are denoted by the same reference numerals, and repeated description thereof may be omitted.

[0015] (First embodiment) [System configuration] A control system will be described. Fig. 1 is a diagram illustrating an example of the configuration of a control system according to the first embodiment. The control system 100 shown in Fig. 1 includes a work device 1, a control device 2, an observation device 3, and a data storage unit D1. The work device 1 includes a controlled unit 101 and a control unit 102. The control device 2 includes a generation unit 201, a detection unit 202, a determination unit 203, and a planner 204.

[0016] The control system 100 is a system that controls the controlled unit 101 of the operating apparatus 1 based on the processing of a generating unit 201, a detecting unit 202, a determining unit 203, and a planning unit 204 included in a control device 2. Note that the configuration of the first embodiment shown in FIG. 1 does not limit the physical layout. For example, the control device 2 may be provided within the operating apparatus 1 or in an independent location. Furthermore, although connections between the components are omitted, the components are capable of transmitting necessary information.

[0017] The working device 1 is, for example, a construction machine or heavy machine (hereinafter referred to as construction machine), a robot, or a transport vehicle that has a controlled unit 101 that is a movable mechanism. However, the working device 1 is not limited to the above-mentioned construction machine, robot, or transport vehicle.

[0018] The controlled unit 101 is, for example, a movable mechanism provided in a construction machine such as a power shovel, a backhoe, a crane, a forklift, etc. The controlled unit 101 is, for example, an arm, a bucket, a shovel, etc. However, the controlled unit 101 is not limited to the movable mechanisms described above.

[0019] The operation of the controlled unit 101 is controlled by the control unit 102 of the maintenance device 1. However, the control method and control period of the controlled unit 101 differ for each maintenance device 1.

[0020] The control unit 102 performs feedback control based on the planning information generated by the planning unit 204 (target trajectory obtained by time-series control target values) and one or more of the real-time position, posture, velocity, and acceleration of the controlled unit 101, thereby guiding the controlled unit 101 to a position and posture corresponding to a specified control target value.

[0021] Specifically, in order to control the position and attitude of the controlled unit 101, the control unit 102 generates a control signal for moving a movable unit (actuator) of the controlled unit 101, and outputs the generated control signal to the working device 1. Note that if the actuator is controlled by an electrical control signal, the control signal may be output directly to the actuator.

[0022] Furthermore, if the actuator cannot be electrically controlled, for example, if it is controlled hydraulically, it may be controlled by outputting a predetermined signal to a control unit (not shown) that controls the hydraulic pressure, or by outputting a predetermined signal to a remote control device (not shown) such as a surrogate attached to an operating lever that controls the operation of the controlled unit 101.

[0023] The observation device 3 will now be described. The observation device 3 observes the space in which the maneuvering apparatus 1 operates and generates environmental information. Hereinafter, the space will be simply referred to as the working environment of the maneuvering apparatus 1. Specifically, the observation device 3 acquires three-dimensional data within the range of movement of the controlled unit 101.

[0024] The observation device 3 (imaging sensor) is, for example, a device such as a camera (RGB-D camera) that combines a monocular, compound eye, monochrome, or RGB camera with a depth sensor, or a ToF (Time of Flight) camera. The observation device 3 is also a device that optically measures the distance to an object in two or three dimensions, horizontally or vertically to the distance direction. The observation device 3 may be, for example, a lidar (Light Detection And Ranging: LiDAR). The observation device 3 may also be, for example, a radar (Radio Detection and Ranging: Radar).

[0025] The observation device 3 may be a single device as described above, or a combination of multiple devices. Furthermore, the observation device 3 may include the controlled unit 101 in its field of view.

[0026] The environmental information includes a part of the housing of the controlled unit 101. The environmental information may also include information about the surrounding environment, such as obstacles, and information about the controlled unit 101 of the working apparatus 1 that is the control target.

[0027] In addition, the environmental information is determined using, for example, conditions such as the installation position and installation direction (angle) when the observation device 3 is installed, the unique specifications and performance of the observation device 3, and parameters (for example, field of view angle, measurable distance, etc.).

[0028] The installation of the observation device 3 can be determined appropriately based on the type and performance of the observation device 3, the specifications of the work device 1 (e.g., type, size, range of motion, etc.), the work content, and the surrounding environment.

[0029] It is assumed that the installation position and installation direction of the observation device 3 are known in the coordinate system defined by the work device 1. In other words, the observation device 3 is ideally calibrated. Examples of the specifications and performance of the observation device 3 include the field of view (FOV), the longest measurement distance, and the resolution. It is also possible to mount the observation device 3 on the work device 1.

[0030] The format of the environmental information may be at least three-dimensional information of the space in which the working device 1 operates, such as a combination of RGB pixel data and depth and distance information, or set information of three-dimensional positions. Set information of three-dimensional positions may be, for example, point cloud data. In addition, information such as calibration information (installation position and installation direction), specific performance, and parameters of the observation device 3 may be stored in the data storage unit D1.

[0031] The physical layout and connection of the processing units constituting the control unit 2 are not limited to those shown in Fig. 1. That is, the processing units may be integrated into an information processing device, or each processing unit may be an independent information processing device.

[0032] The generation unit 201 uses a virtual environment to simulate the movement (dynamics) of the controlled unit 101. Specifically, the generation unit 201 reproduces (models) the actual moving state of the controlled unit 101, i.e., the movement (dynamics), in real time.

[0033] Specifically, the generation unit 201 uses state information representing the movement of the controlled unit 101 provided on the work apparatus 1 and environmental information representing the work environment including part or all of the work apparatus 1 observed by the observation device 3 to generate (mathematically reproduce: model) controlled unit information representing the current state of the controlled unit 101 possessed by the work apparatus 1 and obstacle information representing obstacles present in the work environment.

[0034] In addition, the controlled unit information and obstacle information may be generated by adding external shape information representing the external shape of the controlled unit 101 to the state information and environmental information representing the actual movement of the controlled unit 101.

[0035] The virtual environment of the generation unit 201 may be, for example, a simulator, a digital twin, or the like.

[0036] Here, reproducing (modeling) the movement means reproducing the actual position and angular displacement of the controlled unit 101 through calculation, and does not mean reproducing the mechanism or internal structure of the controlled unit 101. Therefore, it is sufficient that the model of the controlled unit 101 matches the actual controlled unit 101 within a predetermined error range at least in terms of the geometric configuration, dimensions, and range of motion.

[0037] The external shape information is information that represents, for example, the configuration, dimensions, and movable range of the movable part of the controlled unit 101. Note that the external shape information may also be information that represents the size and three-dimensional shape (surface shape). However, it is not necessary to reproduce the material, etc.

[0038] The status information is, for example, information about the installation position and direction of the working device 1, and position and orientation data of each part or movable part (actuator) that constitutes the controlled device 101. The position and orientation data is acquired from the movable parts that constitute the controlled device 101, or from sensors attached to the controlled device 101, or from the control device 102. For example, if the working device 1 is a hydraulically controlled construction machine such as a backhoe, the position and orientation data is acquired from each movable part of the controlled device 101, or from sensors attached to the housing of the controlled device 101.

[0039] Examples of the sensor include sensors that are installed outside the controlled device 101, such as tilt sensors, gyro sensors, acceleration sensors, and encoders, as well as hydraulic sensors. The installation positions and number of sensors can be designed and installed as appropriate for each task performed by the target working device 1.

[0040] For example, if the working device 1 is a robot equipped with a movable arm, a so-called multi-joint robot arm, angle data of each joint constituting the arm is acquired as position and orientation data. The angle data is typically acquired from an electrical signal output from a sensor (e.g., a rotary encoder) attached to an actuator that drives each joint.

[0041] The position and orientation data described above is information acquired within a certain range of error and delay time in response to the actual movement of the controlled unit 101. The temporal frequency (sampling rate) and spatial resolution (accuracy) of the electrical signal are determined appropriately depending on, for example, the size, characteristics, and work content of the working device 1.

[0042] Furthermore, external shape information (information indicating the configuration, dimensions, range of movement of the movable parts, etc.) required for reproducing (modeling) the controlled part 101 may be stored in the data storage unit D1.

[0043] Here, the environmental information observed by the observation device 3 may include information about the controlled unit 101 in addition to information about the surrounding environment such as obstacles. In this case, processing is required to remove the information about the controlled unit 101 from the environmental information. This processing may be performed by the generation unit 201.

[0044] The state information representing the actual movement of the controlled unit 101 and the environmental information observed by the observation device 3 are each mathematically expressed. Furthermore, the generation unit 201 can generate obstacle information representing obstacles (for example, structures, workers, other work devices, etc.) by excluding controlled unit information that models the controlled unit 101 from the environmental information that includes information about the controlled unit 101. A specific processing method will be described later.

[0045] Therefore, the generating unit 201 can generate controlled unit information and obstacle information using the state information and environmental information of the actual controlled unit 101 .

[0046] Regarding the detection unit The detection unit 202 first acquires the controlled unit information and obstacle information generated by the generation unit 201. Next, the detection unit 202 generates information (hereinafter referred to as detection information) that indicates the relationship between the controlled unit 101 and the obstacle. Note that specific processing (detection processing) by the detection unit 202 will be described later.

[0047] The controlled unit information is information (three-dimensional position information) that indicates the current three-dimensional position of the controlled unit 101. The controlled unit information is also information that is generated in time series according to the movement of the controlled unit 101.

[0048] The obstacle information is information (three-dimensional position information) that represents the three-dimensional position of an obstacle in the work environment. The obstacle information is also information (aggregate information) that is expressed as point cloud data, for example. The obstacle information is information that is generated over time in response to changes in the work environment.

[0049] The detection information is information that is generated based on the controlled unit information and the obstacle information, and that indicates the distance, direction, relative speed, etc. between the controlled unit 101 and the obstacle.

[0050] Specifically, the detection information includes information representing the nearest point of the obstacle to a specific location (judgment point) of the controlled unit 101, information representing the distance between the judgment point and the nearest point (nearest distance), information representing a vector connecting the judgment point and the nearest point (collision vector), information representing a velocity vector representing the moving direction of the judgment point, information representing a component of the velocity vector projected onto the collision vector, etc. However, the detection information is not limited to the above-mentioned information.

[0051] The detection information may include one or more of the above-described pieces of information. Furthermore, the detection information may be expressed using, for example, a quantity (vector) including a magnitude (scalar) and a direction (orientation).

[0052] Regarding the Determination Unit The determination unit 203 first acquires the detection information output by the detection unit 202. Next, the determination unit 203 determines whether the Controlled Unit 101 is approaching an object, among obstacles, that the Controlled Unit 101 must not approach, using the distance between the Controlled Unit 101 and the object. Note that the determination of whether the Controlled Unit 101 is approaching an object may be made by determining an approaching motion, by determining that the Controlled Unit 101 is in an approaching state, or by determining both. Next, the determination unit 203 generates a determination value based on the determination result. Note that the specific processing (determination processing) of the determination unit 203 will be described later.

[0053] The object (prohibited object) is an obstacle that the controlled unit 101 is not permitted to approach. The judgment value is information indicating whether the detection information satisfies the judgment criterion. For example, it is binary information (true / false information). The judgment criterion is, for example, a preset threshold value for the distance between the controlled unit 101 and the object, which is included in the detection information.

[0054] The determination unit 203 outputs the determination value to the planning unit 204 of the control device 2 , or to the control unit 102 of the working device 1 and the planning unit 204 of the control device 2 .

[0055] Furthermore, the determination unit 203 may generate a plurality of determination values ​​based on a plurality of determination criteria, for example.

[0056] Specifically, by setting multiple criteria, the determining unit 203 compares the distance between the controlled unit 101 and the approach-prohibited obstacle with the multiple criteria, and generates a graded determination value for each criterion. That is, by using the graded determination value, the degree of approach to the object is determined.

[0057] Furthermore, as another variation of the detection information, information on the direction or relative speed of the controlled unit 101 to the object (approach prohibition obstacle), or both, may be used to predict future approach, collision, or contact with the object. That is, the determination unit 203 uses the direction or relative speed, or both, which represent the relationship between the controlled unit 101 and the object, and sets a determination criterion for each of the information on the direction and relative speed, and compares it with the determination criterion set for each of the direction and relative speed to generate a determination value that predicts future approach, collision, or contact with the object for each of the direction and relative speed.

[0058] The information regarding the above-mentioned criteria is stored in advance in the data storage unit D1 or the like.

[0059] Regarding the Planning Unit When it is determined that the controlled unit 101 is approaching an object during a preset period, the planning unit 204 generates planning information for controlling the controlled unit 101 so that the controlled unit 101 does not approach the object. The planning information is information (target trajectory) in which control target values ​​are generated in a time series. The period may be divided into sections for each operation of the controlled unit 101.

[0060] Thereafter, when the plan information is input to the control unit 102 of the maintenance apparatus 1, the control unit 102 controls the operation of the controlled unit 101 based on the plan information. Note that the specifications of the plan information and the control method differ depending on the type and operation of the maintenance apparatus 1 or the controlled unit 101, and are therefore determined appropriately.

[0061] Specifically, the planner 204 first acquires the judgment value output by the judger 203. Next, the planner 204 generates plan information based on the acquired judgment value. Next, the plan information is transmitted to the operation device 1 via a communication unit (not shown) of the control device 2. Then, when the communication unit (not shown) of the operation device 1 receives the plan information, the control unit 102 controls the operation of the controlled unit 101 based on the plan information.

[0062] In the example of normal operation described above, an example was described in which the control unit 102 controls the controlled unit 101 based on the planning information output by the planning unit 204, but the control unit 102 may also obtain control instructions in addition to the planning information, and change the control of the operation of the controlled unit 101 based on the planning information based on the control instructions.

[0063] Specifically, based on a judgment value indicating the degree of approach to the object, the control unit 102 controls the controlled unit 101 to slow down the movement speed or stop the movement. Hereinafter, this judgment value will also be referred to as a judgment value J1 of the movement restriction.

[0064] Furthermore, based on a judgment value that predicts future approach, collision, or contact with an object, the planning unit 204 performs control to slow down the motion speed of the controlled unit 101, or change the motion of the controlled unit 101, or the trajectory of the controlled unit 101. Hereinafter, this judgment value will also be referred to as a control correction judgment value J2.

[0065] However, the relationship between the judgment value and the control is not limited to this. Note that the specific processing (control planning processing) of the planning unit 204 will be described later.

[0066] The data storage unit will now be described. The data storage unit D1 stores calibration information (installation position and direction) for the observation device 3, information such as specific performance and parameters, and information necessary for the generation unit 201 to reproduce the controlled unit 101. For example, information about the configuration and dimensions of the controlled unit 101, its range of motion, and information about the judgment criteria for the judgment unit 203 is stored. The above-mentioned information may be stored in advance before the working device 1 starts work, or may be added or updated after work has started.

[0067] The physical location and connection of the data storage unit D1 are not limited. The data storage unit D1 may be provided in the work device 1 or the control device 2, for example.

[0068] [Device Operation] Fig. 2 is a diagram for explaining an example of the operation of the control device according to embodiment 1. The processing of the control device 2 (steps S101 to S108) will be described below with reference to Fig. 2 .

[0069] The generation unit 201 uses status information representing the movement of the controlled unit 101 provided on the work device 1 and environmental information representing the work environment including part or all of the work device 1 observed by the observation device 3 to generate controlled unit information representing the current state of the controlled unit 101 and obstacle supplement information representing obstacles present in the work environment (step S101).

[0070] In step S101, the controlled unit information and obstacle information may be generated by adding shape information representing the outer shape of the controlled unit 101 to the state information and environmental information representing the actual movement of the controlled unit 101.

[0071] Specifically, in step S101, the generation unit 201 first acquires state information representing the movement of the working device 1, environmental information representing the working environment observed by the observation device 3, and external shape information acquired from the data storage unit D1. Next, the generation unit 201 sets the actual environmental state for generation based on the acquired state information, environmental information, and external shape information of the working device 1, and generates controlled unit information for the controlled unit 101. Furthermore, if the environmental information includes information about the controlled unit 101, the generation unit 201 executes processing to exclude the information about the controlled unit 101 from the environmental information, and generates obstacle information.

[0072] The process of excluding information about the controlled unit 101 from the environmental information will be described. In the generation unit 201, the model simulating the controlled unit 101 can be synchronized with the movement of the real controlled unit 101. That is, when the observation device 3 acquires environmental information, the relationship between the position and orientation of the observation device 3 and the controlled unit 101 can be made to match in reality and in the virtual environment. Even when the controlled unit 101 is moving, synchronization can be achieved by reflecting the time-series state information of the controlled unit 101 and the time-series environmental information acquired by the observation device 3. Therefore, the three-dimensional area occupied (occluded) by the controlled unit 101 in the environmental information observed by the observation device 3 can be calculated within the virtual environment set by the generation unit 201.

[0073] Here, the three-dimensional area occupied (blocked) by the controllable unit 101 can be calculated based on the surface shape of the controllable unit 101. When the generation unit 201 reproduces the geometric configuration of the actual controllable unit 101, the surface shape of the controllable unit 101 is represented by shape information that approximates the geometric configuration and the elements that make up the controllable unit 101 with a basic shape, for example, by length, width, and height when approximated by a rectangular parallelepiped. Alternatively, when approximated by a cylinder, by diameter and height, etc. This information may be stored in advance in the data storage unit D1. Furthermore, when the generation unit 201 reproduces the outer shape (surface shape) of the actual controllable unit 101, that information can also be used.

[0074] From the above, it is possible to calculate the three-dimensional area of ​​the environmental information occupied by the controlled unit 101, and by subtracting this area from the environmental information, it is possible to obtain obstacle information.

[0075] The process of excluding specific regions from such three-dimensional information (filtering, masking) may use an existing method as appropriate. As described above, the three-dimensional region occupied by the controlled unit 101 may be approximated by a basic shape and then subtracted, or other methods may be used. For example, one method represents the three-dimensional region as a regular lattice (voxel) and expresses the presence or absence of an object at each lattice point using 0 / 1 information. In this method, the environmental information and the three-dimensional region occupied by the controlled unit 101 are each represented by separate voxels, and overlapping regions can be removed using a process that detects overlap between these voxels, such as a logical operation called XOR (Exclusive OR). This method can be processed quickly and with a low computational load. However, the exclusion process is not limited to these.

[0076] Next, the detection unit 202 uses the controlled unit information and the obstacle information to output detection information that indicates the relationship between the controlled unit 101 and the obstacle (step S102).

[0077] The detection process of the detection unit 202 will be described. The obstacle information is represented, for example, by point cloud data as a set of information on three-dimensional positions. The controlled unit information is information on the geometric configuration of the controlled unit 101, or external shape information. For example, if the work device 1 is a backhoe and the controlled unit 101 is its moving parts (boom and arm) and bucket, the position information of the joint points of each moving part and the bucket can be calculated.

[0078] In general, this can be calculated using forward kinematics based on the geometric configuration information of the controlled unit 101 and the angle information of each joint. When the controlled unit information is information on the geometric configuration of the controlled unit 101, a specific point (judgment point) on the controlled unit 101 is used as the object that represents the relationship with the obstacle. The position of the judgment point is associated with the joint point of the movable unit.

[0079] On the other hand, when the controlled unit information is the external shape information of the controlled unit 101, any position (at least one or more points) on the surface of the controlled unit 101 can be set as the judgment point. Note that the number and positions of the judgment points are not specified and can be set depending on the operating device 1 or the controlled unit 101, the work content, etc., and this information may be stored in the data storage unit D1. Furthermore, the position information of the judgment point can be dynamically calculated based on the time-series controlled unit information even when the controlled unit 101 is moving.

[0080] 3 is a diagram illustrating an example of detection information in the control device according to the first embodiment. Specific examples of detection processing and detection information will be described with reference to FIG. 3. As described above, the detection information includes, for example, information representing the closest point between a specific location (judgment point) of the controlled unit and an obstacle, information representing the distance (nearest distance) between the judgment point and the closest point, information representing a vector (collision vector) connecting the judgment point and the closest point, information representing a velocity vector representing the moving direction of the judgment point, and information representing a component of the velocity vector projected onto the collision vector.

[0081] 3 illustrates the working device 1, its controlled device 101, and a dump truck as an example of obstacle information. The controlled device 101 is represented as geometric configuration or external shape information, and the obstacle information is represented as a set of information (point cloud data) of three-dimensional positions. In FIG. 3, only one judgment point is shown at the bottom of the bucket of the controlled device 101, but this is merely an example and is not intended to be limiting. Furthermore, the detection information shown below is also merely an example and is not intended to be limiting.

[0082] The nearest point of the judgment point and the obstacle information will now be described. The nearest point can be determined based on the distance (nearest distance) between the judgment point and the point (nearest point) that is closest to the judgment point among the set of position information representing the obstacle information. The nearest point can be found using an algorithm such as nearest neighbor search or k-nearest neighbor search, for example. However, the method is not limited to the above-mentioned algorithm.

[0083] In FIG. 3, an example of a nearest point is shown at the rear of the dump truck bed. In this case, the vector connecting the judgment point and the nearest point is referred to as a collision vector C. This collision vector C includes information on the magnitude and direction, which represent the distance between the judgment point and the nearest point. Note that when there are multiple judgment points, calculation of the nearest distance between each judgment point and a set representing obstacle information may be performed in parallel.

[0084] The velocity vector representing the direction of movement of the decision point will now be described. FIG. 3 illustrates an example of a target trajectory for the decision point. This target trajectory is generated by the planning unit 204, and preferably, the controlled unit 101 is controlled to follow this target trajectory. At this time, a velocity vector can be calculated from the positional displacement per unit time of the decision unit, i.e., the differential value, and the direction of this velocity vector is the tangent direction of the target trajectory at that time. An example of this velocity vector V is shown in FIG. 3. In this case, the relationship between the collision vector and the velocity vector can be calculated by the inner product relationship: C·V = |C||V| cos(θ), where θ is the angle they form. Furthermore, the component of the velocity vector V projected onto the collision vector C can be calculated from |V| cos(θ).

[0085] Next, the determination unit 203 outputs a determination value based on the detection information (steps S103 and S106).

[0086] First, steps S103 to S105 show a process of determining and controlling the degree of current approach to a prohibited object as one of the processes of the determining unit 203.

[0087] The judgment unit 203 outputs a judgment value (judgment value J1 for operation restriction) based on detection information indicating the current degree of approach to the object, for example, the nearest distance between the judgment point and the nearest point, i.e., the magnitude of the collision vector (step S103).

[0088] At this time, if this judgment value is greater than a certain set value (threshold value Tha) (if the criterion is met: step S104: YES), the judgment unit 203 can determine that there is no risk of collision or contact, and the working device 1 continues to operate.

[0089] On the other hand, if the judgment value J1 is smaller than the threshold value Tha (if the criterion is not met: step S104: NO), the judgment unit 203 determines that there is a risk of collision or contact, and outputs a warning (alert) and a control instruction to the control unit 102 (step S105).

[0090] The control instruction to the control unit 102 is, for example, to slow down or stop the operation of the controlled unit 101. Note that by setting a plurality of criteria for the determination value J1, multi-stage determination is also possible.

[0091] Next, steps S106 to S108 show other processing by the determination unit 203, in which a control process is performed by predicting future approach to or collision with an object.

[0092] The judgment unit 203 outputs a judgment value (control correction judgment value J2) based on detection information for predicting future approach to or collision with an object, such as the angle between the collision vector and the velocity vector, and the detection information of the component of the velocity vector projected onto the collision vector (step S106).

[0093] At this time, if the judgment value J2 satisfies a certain set criterion (step S107: YES), the judgment unit 203 can determine that there is no risk of collision or contact, and the working device 1 can continue operating.

[0094] On the other hand, if the judgment value J2 does not satisfy the criteria (step S107: NO), the judgment unit 203 determines that there is a risk of collision or contact, and outputs a warning (alert) and a control instruction to the planning unit 204 (step S108).

[0095] The control instruction to the planning unit 204 is, for example, a control to decelerate the operation of the controlled unit 101 or a control to change the operation or trajectory. The operation and trajectory can be changed, for example, by changing a control parameter such as a control gain. Note that information required for the change may be output as a control instruction from the determining unit 203 to the planning unit 204. Similarly, by setting multiple determination criteria, multi-stage determination is also possible.

[0096] The above-described operations of outputting the determination value by the determination unit 203 and issuing control instructions to the control (steps S103 to S105 and steps S106 to S108) are preferably executed in parallel. This can be realized when the detection information output by the detection unit 202 can be calculated in parallel, as exemplified in the first embodiment, and when the control instructions to the control unit based on each determination are different between the control unit 102 and the planning unit 204. However, there are no restrictions on such an apparatus configuration and whether parallel processing is possible or not.

[0097] Here, among the examples of the determination process described above, a method of determining the current degree of approach to an object and restricting operation by issuing a control instruction to the control unit 102 will be described using an example in which the working device 1 is a backhoe in Fig. 4. Fig. 4 is a diagram for explaining an example of the determination process and operation restriction of the control device according to the first embodiment.

[0098] Figures 4A and 4B show a two-dimensional simulation of the arrangement shown in Figure 3 as viewed from a cross section perpendicular to the plane of the page, illustrating the controlled unit 101 and obstacle information. Note that the controlled unit 101 in Figure 4 simulates only the work device 1 in Figure 3, i.e., the bucket part of the backhoe, and the obstacle information only simulates the rear part of the dump bed, with other parts not shown.

[0099] In FIG. 4A, it is assumed that the controlled unit 101 starts operating at time step t1. The planning information output by the planning unit 204 at this time, i.e., the target position and target trajectory, is shown. The controlled unit continues operating at time steps t2 and t3, but an example is shown in which the controlled unit deviates from the target trajectory. This occurs, for example, when the bucket load is heavier than expected, or when control parameters such as control gains are inappropriate, resulting in insufficient control by the control unit 102 to track the target trajectory. However, the causes are not limited to these.

[0100] 4A illustrates the judgment point of the controlled unit 101, the nearest point of the obstacle information output as detection information, the collision vector, and the reference value for judgment processing by the judgment unit 203 in this situation at time step t3. As the judgment reference value is set to a certain constant value as the magnitude of the collision vector from the judgment point, a circle centered on the judgment point is depicted in FIG. 4. In reality, it is three-dimensional, so it is a sphere.

[0101] At this time step t3, the magnitude of the collision vector is equal to or smaller than the reference value. Therefore, in FIG. 3, it is determined that the reference value is not met, i.e., there is a risk of collision or contact, and a warning (alert) and a control instruction are output to the control unit 102 (step S105).

[0102] An example of the operation restriction of the controlled device 101 by the control unit 102 at this time is shown in Fig. 4B. Fig. 4B illustrates a case where a control instruction to stop is given as an operation restriction to the control unit 102, and shows a state in which the operation has stopped at time step t3. Note that even after the control instruction to stop is given, there is a braking distance until the controlled device 101 completely stops, so the actual position of the controlled device 101 has changed from the position at time step t3, but in Fig. 4B, this position difference is ignored.

[0103] Next, the other example of the determination process described above, that is, a control correction method for predicting future approach to or collision with an object and changing the operation and trajectory by issuing a control instruction to the planning unit 204, will be described using an example in which the working device 1 is a backhoe, as in Fig. 4. Fig. 5 is a diagram for explaining an example of the determination process and control correction of the control device according to the first embodiment.

[0104] Figure 5A shows the case where the actual trajectory of the controlled unit 101 deviates from the target trajectory, as in Figure 4A, but shows the velocity vector, which is the controlled unit information at time step t2, the collision vector, which is the detection information, and the collision vector component of the velocity.

[0105] Here, an example will be described in which the determination process by the determination unit 203 is YES, i.e., the condition for continuing the operation is set as the product (AND) of two conditions: the angle between the velocity vector and the collision vector is equal to or greater than a reference value, and the collision vector component of the velocity is equal to or less than a reference value. This condition predicts that the angle between the velocity vector and the collision vector is smaller than the target trajectory at time step t1, i.e., that the vehicle is moving in a collision direction. Furthermore, the collision vector component of the velocity can be used to determine the possibility of a collision based on the current velocity.

[0106] Therefore, if the angle between the velocity vector and the collision vector falls below a reference value and the collision vector component of the velocity exceeds the reference value, it is determined based on the prediction that there is a possibility of a collision, and a process (step S108) is performed to output a warning (alert) and a control instruction to the planning unit 204. Note that the above conditions and product set settings are merely examples and are not limited to these.

[0107] An example of the operation correction of the controlled unit 101 by the planner 204 at this time is shown in Fig. 5B. Fig. 5B illustrates a case where a control instruction to change the control gain is given as the operation correction to the planner 204. As the control gain change process, for example, an algorithm is assumed that reduces the component of the control gain in the direction of the collision vector and increases the component perpendicular to the collision vector.

[0108] FIG. 5B shows a case where the control gain in the rightward direction, which corresponds to the collision vector in the rightward direction at time step t2, is decreased, and the control gain in the perpendicular upward direction is increased. By setting such control gains, the velocity vector is corrected upward compared to the direction at time step t2, as shown in the figure, and the position of the controlled unit changes upward. As a result, the condition for determining the possibility of a collision is resolved at time step t3, and operation continues. At this time, the magnitude of the collision vector described above is larger than in the case of FIG. 4A, so the determination satisfies the criterion value (step S105), and operation continues.

[0109] The above describes the difference between the control based on the two determination processes. In the example of control correction based on prediction shown in FIG. 5, prediction and determination are made at time step t2, which is earlier than the determination time step t3 shown in FIG. 4, and thus operation can be continued without being stopped. In other words, work efficiency can be improved. On the other hand, if the prediction shown in FIG. 5 is inappropriate, or the control correction due to the change in control gain is inappropriate, and the planned operation is not achieved, the determination shown in FIG. 4 continues at subsequent time steps, so that operation can be stopped if there is a possibility of a collision. In other words, safety can be ensured.

[0110] Here, a supplementary explanation will be given regarding the generating unit 201. As described above, the generating unit 201 simulates the controlled unit 101 through calculations. The degree to which the controlled unit 101 is simulated is determined by simulating at least the geometric configuration and dimensions of the controlled unit 101 and the dynamics (movement) that reproduces the range of motion, but whether or not to simulate the external shape information of the controlled unit 101 may be determined depending on the type of working apparatus 1, the work content, etc.

[0111] The following describes specific examples of simulating external shape information and its advantages. The simulated controlled unit 101 may be a model that reproduces the external shape, i.e., size and three-dimensional shape, of the controlled unit 101 at the same size as the real thing, or within a certain error range or scale. This model of the controlled unit 101 can be constructed, for example, using polygons or a collection of polygons (mesh) based on design drawings, CAD data, images, etc. of the controlled unit 101. In this case, the model is approximated according to the shape, size, density, etc. of the polygons, and the degree of approximation can be determined appropriately depending on the size, etc., of the controlled unit 101. Furthermore, the method of constructing the model is not limited to the method described above. The information necessary to reproduce the controlled unit 101 may be stored in the data storage unit D1.

[0112] The advantages of modeling the controlled unit 101 including its external shape information are as follows. The first advantage is that any position on the surface of the controlled unit 101 can be set as a judgment point. In modeling only the geometric configuration, the judgment point is geometrically associated with the joint position in the simulated geometric configuration. In other words, the position of the judgment point needs to be uniquely calculated from the position of the joint position.

[0113] The judgment points serve as reference points for the working device 1 when determining approach and collision, and can be set appropriately depending on the type of working device 1, the work content, the surrounding environment, the level of safety that must be met, and other factors. However, it may be difficult to set the judgment points appropriately on a geometric model. In such cases, by adopting a model that includes the above-mentioned external shape information, the model can visually simulate the actual working device 1, making setting and confirmation easier. Furthermore, judgment points that are difficult to directly associate with joint positions can be set appropriately. This improves the accuracy of determining approach and collision, thereby contributing to improved safety.

[0114] [Effects of First Embodiment] First, a control system to be compared with the control system 100 will be described. In the control system to be compared, which has a control function for detecting obstacles and preventing collisions, detection accuracy and control performance determine safety and work efficiency. In the example described in Patent Document 1, detection is based on the relative distance between the obstacle and the position of a specific point of the work device based on the estimated movement course, and therefore, it is assumed that the actual work device will be controlled according to the estimated movement course. Therefore, if the actual work device deviates from the estimated movement course, safety issues may arise due to overlooking the detection, and work efficiency issues may arise due to unnecessary detection, which may lead to work interruptions such as stoppages. In other words, there is a challenge in achieving both improved safety and improved work efficiency.

[0115] The following describes the differences and advantages of the control system 100 compared to the above-mentioned systems to be compared. The first difference is that it assumes a case where the controlled unit 101 of the actual working device 1 cannot be controlled according to the control plan, i.e., deviates from the target trajectory. In an ideal control state, the difference between the target trajectory and the current state is fed back and control is performed so that the difference is equal to or less than a certain value. On the other hand, if the difference exceeds a certain value, it is considered an abnormality and an alert can be issued or control can be stopped. In other words, it is possible to detect non-ideal situations without using detection, but in reality, the following issues arise.

[0116] First, there is the setting of the difference between the target trajectory and the current state when it is considered to be non-ideal. In reality, there are errors and delays in the current state information of the controlled unit 101. Furthermore, if the working device 1 is hydraulically controlled or if the operating lever is configured to be operated by a remote control device, errors in three-dimensional position and time may occur in the control of the controlled unit 101. In other words, the difference between the target trajectory and the current state is not always sufficiently small. In such cases, if the difference value considered to be non-ideal is fixed to a small value, abnormalities will be detected frequently, resulting in reduced work efficiency. On the other hand, if the difference value is set to a large value, abnormalities will be more difficult to detect, resulting in safety issues. In other words, safety and work efficiency are heavily dependent on the setting of the judgment value, and the problem cannot be solved.

[0117] Second, even if the above-mentioned judgment value can be appropriately set, it may be inappropriate depending on the work content of the working device 1. When the working device 1 is a construction machine such as a shovel, backhoe, or crane, a device for performing the work, specifically a bucket for excavating soil and gravel, is attached to the tip of the controlled device 101. The load amount of the bucket may vary depending on the environment and the work content. For example, when excavating soil and gravel with a backhoe, if more soil than planned is excavated or if the soil density is higher than expected, the weight of the bucket may become heavier than expected. If this weight exceeds the control assumption, i.e., if ideal control cannot be performed under the set parameters such as the control gain, an error occurs with respect to the target trajectory, resulting in a so-called undershoot state. Because it is not easy to set the above-mentioned judgment value taking such conditions into consideration, even a slight undershoot may result in excessive detection, or a collision may occur if no detection is performed. Furthermore, if the weight of such a bucket is heavy, it is significantly affected by inertia. Construction machinery such as backhoes involves swinging movements, but the center of rotation and the bucket performing the work are spaced apart. Generally, the moment of inertia is proportional to the distance from the axis of rotation and the mass. Therefore, bucket movement away from the center of rotation is significantly affected by inertia. Therefore, a so-called overshoot, in which the bucket exceeds the target position after control, is likely to occur. In the case of overshooting, the target position has already been exceeded by the time an error between the target trajectory and the current state is detected, increasing the likelihood of a collision. Even if a stop command is issued after detection, the error from the target position increases due to the influence of inertia even before the machine is stopped. Therefore, detecting the error based on the current state is likely to be too late. While the above example illustrates a construction machinery such as a backhoe as the working device 1, similar issues also exist in robotic arms and other similar devices. Robotic arms also have a robotic hand or the like attached to the tip of the controlled unit 101 for work, and the robotic hand is used to grasp and carry objects. This type of work is similar to that of construction machinery in that it depends on the weight of the object and is affected by inertia depending on the structure of the robotic hand and the weight of the object.

[0118] The control system 100 differs from the above-described comparative systems in that it detects deviation from the target trajectory. In other words, if the control system is controlled to follow the target trajectory, no detection occurs. This assumes that the target trajectory is appropriately planned. However, in the first embodiment, the method for generating the target trajectory, i.e., the control planning method, is not important, and therefore is considered appropriate. By detecting deviation from the target trajectory, detection can be performed that does not rely on detection based on the difference between the target trajectory and the current state, as described above. In other words, even if the difference between the target trajectory and the current state is set to a large value to prevent excessive detection, the method of the first embodiment can still detect the possibility of a collision. Therefore, issues that are heavily dependent on the settings can be resolved, and a decrease in work efficiency due to excessive detection can be suppressed.

[0119] The second differentiation and advantage of the control system 100 is its ability to execute both processes contributing to safety and processes contributing to work efficiency in parallel. The control system 100 can execute processes outputting a judgment value and issuing control instructions to the control unit of the work device in parallel in steps S103 to S105 and steps S106 to S108. The judgment and control correction based on predictions of approach and collision (steps S106 to S108) contribute to the effect of not stopping operation as much as possible, i.e., not reducing work efficiency, by predicting and responding in advance before the possibility of approach or collision actually increases. On the other hand, the judgment and operation restriction based on the current state (steps S103 to S105) contribute to the effect of improving safety by reliably responding when there is an actual risk of approach or collision. By executing processes that achieve these different effects in parallel, the trade-off between safety and work efficiency can be eliminated.

[0120] [Program] The program in the first embodiment may be any program that causes a computer to execute steps S101 to S108 shown in Fig. 2. By installing and executing this program on a computer, the control device and control method in the first embodiment can be realized. In this case, the processor of the computer functions as a generation unit 201, a detection unit 202, a determination unit 203, and a planning unit 204 and performs processing.

[0121] The program in the first embodiment may be executed by a computer system constructed by a plurality of computers. In this case, for example, each computer may function as one of the generating unit 201, the detecting unit 202, the determining unit 203, and the planning unit 204.

[0122] (Embodiment 2) [Device Configuration] Since the configuration of the control system 100 of embodiment 2 is the same as that of embodiment 1, diagrams showing the configuration and explanations thereof will be omitted. However, the flow of information differs from embodiment 1 in that the planning information output by the planning unit 204 is input to the generating unit 201.

[0123] The planning information input to the generation unit 201 is time series information of the control target value separated by a certain predetermined time or operation, and is at least either information on the final target value of that separation (target position) or time series information of the intermediate period (target trajectory).

[0124] The generating unit 201 simulates the dynamics (movement) of the controlled unit 101, but the simulation is not limited to the movement based on the current state information of the controlled unit 101 shown in the first embodiment. In other words, it may have a plurality of models of the controlled unit 101 and reproduce movements based on other information. The generating unit 201 of the second embodiment receives information on the control plan output by the planning unit 204 and reproduces the future movement of the controlled unit 101 based on the control plan, i.e., the movement of the target trajectory.

[0125] [Device Operation] Fig. 6 is a diagram for explaining an example of the operation of the control device according to embodiment 2. Of the processes shown in Fig. 6, steps S103 to S108 are the same as those in embodiment 1, and therefore description thereof will be omitted.

[0126] In step S201, the generation unit 201 first acquires state information of the actual controlled unit 101, environmental information about the work environment observed by the observation device 3, shape information acquired from the storage device D1, and plan information generated by the planner 204. Next, the generation unit 201 sets the state of the actual environment in the virtual environment based on the acquired state information of the actual controlled unit 101, the environmental information, and the shape information of the data storage unit D1, and generates controlled unit information for the controlled unit 101. Furthermore, based on the acquired plan information, the generation unit 201 sets a planned state of the controlled unit 101 in the virtual environment and generates controlled unit information. If the environmental information includes information about the controlled unit 101, the generation unit 201 generates obstacle information by excluding the information about the controlled unit 101 from the environmental information.

[0127] Next, the detection unit 202 generates detection information representing the relationship between the current controlled unit 101, the planned controlled unit 101, and the obstacle, using the controlled unit information and the obstacle information (step S202).

[0128] The detection process performed by the detection unit 202 for the current controlled unit 101 is the same as in the first embodiment, and therefore will not be described here. The second embodiment is characterized in that the detection process is also performed for the planned controlled unit 101. The detection process will be described below.

[0129] As described above, in the second embodiment, the generation unit 201 is configured to independently set at least a model that simulates the current controllable unit 101 and a model that simulates the planned controllable unit 101. Then, using each model, first controllable unit information about the current controllable unit 101 and second controllable unit information about the planned controllable unit 101 are generated. As in the first embodiment, each model may simulate the geometric configuration of the controllable unit 101 or may simulate the external shape information as well. Furthermore, there is no restriction that the degree of simulation of each model must be consistent, and therefore the degree of simulation can be determined appropriately depending on the type of operating apparatus 1, the operation content, and the like.

[0130] Here, the second controlled unit information about the planned controlled unit 101 differs from the first controlled unit information about the current controlled unit 101 in the following respects: The first controlled unit information is based on state information of the controlled unit 101 at the current time, whereas the second controlled unit information is time-series information of control target values ​​separated by a certain predetermined time or operation, and includes state information of the controlled unit 101 at multiple time steps. Therefore, the controlled unit information can also be information for multiple time steps.

[0131] However, as described above, the planned information may be the final time step of a predetermined time or a time interval separated by an operation, or any single or multiple time steps leading up to that step. In other words, the planned controlled unit information for the controlled unit 101 is state information for the controlled unit 101 on the model at any single or multiple time steps in the target position or target trajectory. The variation in this time step may be determined appropriately depending on the working device 1 or the work content.

[0132] Furthermore, different settings may be made depending on the number of time steps in the plan information, the type of movement, the distance from the current position to the target position, the movement speed of the controllable unit 101, etc. For example, if the number of time steps in the plan information for a certain movement is long and the distance from the current position to the target position is long, the information may be set as controllable unit information at a time step along the target trajectory, and updated to controllable unit information at a previous time step in accordance with the actual movement of the controllable unit 101, i.e., the progress of the time steps of control. On the other hand, if the number of time steps in the plan information is short and the distance from the current position to the target position is short, the information may be set as controllable unit information at the final target position.

[0133] The detection information for the planned controllable unit 101 is defined in the same way as the detection information for the current controllable unit 101 shown in FIG. 3. When the controllable unit information for the planned controllable unit 101 is information for multiple time steps, it is generated for each time step based on the current obstacle information. Specifically, it is the nearest point in the obstacle information for each judgment point of the controllable unit 101 at any single or multiple time steps in the target position or target trajectory, and the collision vector. Therefore, the detection information for the planned controllable unit 101 is a set of information for the number of time steps contained in the controllable unit information. The detection process for generating specific detection information is the same as in embodiment 1, so a description thereof will be omitted. Furthermore, since the detection information for the current and planned controllable units 101 is detected independently, it may be processed in parallel.

[0134] The determination process (steps S103 to S108) of the determination unit 203 based on the detection information for the current controllable unit 101 is the same as in embodiment 1, and therefore will not be described again. Detection information for the planned controllable unit 101 is input to the determination unit 203, which outputs a determination value indicating whether the planned controllable unit 101 is approaching an object. Hereinafter, the determination process for the planned controllable unit 101 will also be referred to as trajectory determination, and the determination value based on this trajectory determination will also be referred to as trajectory correction determination value.

[0135] Hereinafter, as the processing of the determination unit 203, processing of correcting the trajectory based on the processing of determining the trajectory of the controlled unit 101 planned (steps S203 to S206) will be described.

[0136] As the detection information for the trajectory determination, for example, a determination value (trajectory correction determination value J3) is output based on the nearest distance between the determination point and the nearest point, that is, the magnitude of the collision vector (step S203).

[0137] At this time, if this judgment value J3 is greater than a certain set value (threshold value Thb) (if the criteria are met: step S204: YES), it can be determined that there is no risk of collision or contact, and the working device 1 can start or continue operation (step S204).

[0138] On the other hand, if the judgment value J3 is smaller than the threshold value Thb (if the criterion is not met: step S204: NO), it is determined that there is a risk of collision or contact, and a warning (alert) and a control instruction are output to the planning unit 204 (step S205).

[0139] The control instruction to the planner 204 is, for example, an instruction to change the target position (target position correction information). The determiner 203 generates a correction vector as the target position correction information. The planner 204 receives the control instruction, changes the target position based on the target position correction information, and regenerates the target trajectory (step S206).

[0140] The changed target position and regenerated target trajectory are input to the detection unit 202 and reflected in the detection process (step S202), and the detection process and the determination process of the determination unit 203 are executed again. Note that at the time of determination, it does not matter whether the controlled unit 101 has started operating, i.e., is moving, or has not yet started operating, i.e., is stopped, but the trajectory determination process targets a point on the target trajectory that is a time step ahead of the current position of the controlled unit 101. Furthermore, the method of regenerating the target trajectory from the target position can be the same as the method of generating the initial target trajectory, and is not limited in the second embodiment.

[0141] A trajectory correction method for determining whether the planned controlled unit 101 is approaching or colliding with an object and changing the trajectory by issuing a control command to the planning unit 204 will be described using an example in which the working device 1 is a backhoe, as in Fig. 4. Fig. 7 is a diagram for explaining an example of the determination process and trajectory correction of the control device according to the second embodiment.

[0142] 7A, the controlled unit 101 is in a state of time step t1 at the start of operation, which is the time when the trajectory determination process is executed. The obstacle information at this time is illustrated as obstacle information t1. On the other hand, the planning information output by the planning unit 204, i.e., the target position and target trajectory, is generated before the start of operation, and therefore is assumed to have been generated at time step t0, which is earlier than time step t1, and the obstacle information at this time is assumed to be obstacle information t0, and the target trajectory is assumed to be target trajectory t0.

[0143] 7 illustrates a situation in which the position of obstacle information t0 at time step t0 when plan information was generated differs from the position of obstacle information t1 at time step t1 when the determination process was executed at the start of the operation. This can occur, for example, when an obstacle moves between time steps t0 and t1, or when another obstacle appears at the position of time step t1. In other words, this is an example of a case in which obstacle information is time-series information in an environment in which obstacles change over time (dynamically).

[0144] In the second embodiment, the observation frequency by the observation device 3 and the frequency of obstacle information generation by the generation unit 201 are not limited, so that it is possible to deal with cases where the obstacle information changes over time, as in this example. Note that the case where the obstacle information changes over time is not limited to the above-described example. Furthermore, FIG. 7A illustrates, as controlled unit information based on the target trajectory t0, a time step tm during the target trajectory t0, the planned position of the controlled unit 101 at the time step tE of the target position, and a judgment point. The nearest point and collision vector are respectively illustrated as detection information for this controlled unit information. Thus, FIG. 7A illustrates an example in which the controlled unit information and detection information based on the planning information are each a pair of two sets of information with different time steps, but these are merely examples.

[0145] The trajectory determination process at time step t1 is performed based on, for example, the magnitude of the collision vector, i.e., the distance between the determination point and the nearest point. Fig. 7A illustrates a time step tm along the target trajectory t0, a collision vector at time step tE of the target position, and a determination criterion for a certain distance from the determination point. Since the magnitude of the collision vector at time step tE is equal to or smaller than the reference value, it is determined that the criterion is not met in the operation flowchart, i.e., that there is a risk of collision or contact, and the determination unit 203 outputs a warning (alert) and a control instruction to the planning unit 204.

[0146] On the other hand, the criterion is met at time step tm. In this way, when there are multiple pieces of detection information, if it is determined that at least one piece of detection information does not meet the criterion, the determination unit 203 outputs a warning and a control instruction to the planning unit 204. However, these determinations and the output of the warning and control instruction may be performed in parallel or separately at different times. For example, first, a trajectory determination process is performed at time step tm close to time step t1 at the start of the operation, and it is determined that the criterion is met, and it is determined that the operation can be started. Next, before or after the controlled unit 101 starts its operation, a trajectory determination process is performed at time step tE, and it is determined that the criterion is not met, and a control instruction for trajectory correction is output. When the control instruction for trajectory correction is output, the planning unit 204 regenerates the target trajectory, for example, by changing the target position.

[0147] An example of changing the target position and regenerating the target trajectory is shown in FIG. 7B. FIG. 7B illustrates the changed target position and the regenerated target trajectory t1 based on the obstacle information t1 and the determination at time step t1. In this example, a correction vector, which is target value correction information, is generated in a direction opposing the collision vector, which is the detection information at time step t1, i.e., in a direction avoiding the obstacle information t1. The target position is changed based on this correction vector, and the planner 204 regenerates the target trajectory t1. Note that the above-mentioned changes in the correction vector and target position, and the regenerated target trajectory are merely examples and are not limited to these.

[0148] [Effects of Second Embodiment] By changing the target trajectory as in the above example, it is possible to start the operation of the controlled unit 101 based on a target trajectory that does not have the possibility of approaching or colliding with an object. Note that if the trajectory determination process is performed sequentially for different time steps, and the controlled unit 101 starts an operation, and it is determined that the criterion is not satisfied and the target trajectory is regenerated afterwards, the target trajectory for that time step is updated to the regenerated target trajectory and control is continued, thereby preventing operations that would reduce work efficiency, such as deceleration or stopping.

[0149] As described above, the second embodiment is characterized in that the operation of outputting a determination value by the determination unit 203 and issuing a control instruction to the control is a process in which steps S103 to S105 and steps S106 to S108, similar to those in the first embodiment, are further processed by adding steps S203 to S206. These processes are preferably executed in parallel.

[0150] The advantages of the control system 100 of the second embodiment, which differ from the first embodiment, will be described. The second embodiment is characterized in that the generation unit 201 first sets models of multiple controlled units 101, outputs controlled unit information and detection information corresponding to each model, and preferably outputs judgment values ​​in parallel and different control instructions. This operation is due to the feature that the configuration of the control system 100 is not premised on single, serial processing. The first embodiment demonstrated the effect of improving both safety and work efficiency by executing judgments based on the current state of the controlled units and control instructions for operation restrictions, and judgments based on predictions of approach and collision and control instructions for control adjustments in parallel.

[0151] In the second embodiment, the following effects are further achieved by executing the determination based on the target trajectory and the control instruction for trajectory correction in parallel.

[0152] The first point is how to deal with cases where the target trajectory is inappropriate. In the first embodiment, it is assumed that the target position and target trajectory are correct, and the target trajectory cannot be changed even by performing control correction. In other words, since an inappropriate target trajectory cannot be dealt with, there is a possibility that operation will ultimately be stopped due to a determination based on the current state of the controlled unit, resulting in a decrease in work efficiency. Examples of cases where the target trajectory is inappropriate include a case where there is an error or inaccuracy in the target trajectory generation by the planning unit 204, or a case where the planning unit 204 was unable to take obstacles into account when generating the target trajectory. Even in such cases, by applying the second embodiment, the target trajectory can be corrected to one that takes obstacles into account. In other words, even if the planning unit 204 does not have the function of generating a target trajectory that takes obstacles into account, the second embodiment can achieve control that takes obstacles into account.

[0153] The second advantage is that it can respond to dynamic changes in the work environment. Specifically, even if the type, number, or location of obstacles changes between the time when the planning unit 204 generates the target trajectory and the time when the controlled unit 101 actually starts operating, the second embodiment can correct the target trajectory by taking into account the state of the obstacles at each moment.

[0154] [Program] The program in embodiment 2 may be any program that causes a computer to execute steps S201, S202, S103 to S108, and S203 to S206 shown in Fig. 6. By installing and executing this program on a computer, the control device and control method in embodiment 2 can be realized. In this case, the processor of the computer functions as a generation unit 201, a detection unit 202, a determination unit 203, and a planning unit 204 and performs processing.

[0155] The program in embodiment 2 may be executed by a computer system constructed by a plurality of computers. In this case, for example, each computer may function as one of the generating unit 201, the detecting unit 202, the determining unit 203, and the planning unit 204.

[0156] (Embodiment 3) [Device Configuration] Fig. 8 is a diagram for explaining an example of the configuration of a control system according to embodiment 3. The configuration shown in Fig. 8 differs from the control system 100 of embodiments 1 and 2 in that the determination unit 203 of the control device 2 is replaced with a determination inference unit 303, and a determination learning unit (learning unit) 4 is newly added. The other configuration is the same as that of the control system 100, and therefore description thereof will be omitted. Furthermore, the example of the configuration of embodiment 3 shown in Fig. 8 does not specify a physical layout, and connections between components are also omitted, but it is assumed that each component is capable of transmitting necessary information.

[0157] The judgment learning unit 4 performs a learning process so that when the controlled unit information output by the generation unit 201 and the detection information output by the detection unit 202 are input, the judgment unit 203 outputs a judgment value and a control instruction to the control unit 102 or the control unit 102 and the planning unit 204.

[0158] As described in the first and second embodiments, the judgment value may be a plurality of sets of information, but in the third embodiment, it is also simply referred to as the judgment value. Also, there are a plurality of control instructions to the control unit 102, such as operation restriction, control correction, and trajectory correction, but in the third embodiment, these are also collectively referred to as the control instruction. Note that, once learning by the judgment learning unit 4 has been completed, the control system 300 does not need to include the judgment unit 203 of the control system 100, as in the example of FIG. 8.

[0159] The judgment values ​​and control instruction data required for learning by the judgment learning unit 4 are stored in advance. These data may be stored in the data storage unit D1. The learned information (learned information) may also be stored in the data storage unit D1.

[0160] The judgment and inference unit 303 (judgment unit 203) includes an inference device that outputs a judgment value and a control instruction when it receives controlled device information output by the generation unit 201 and detection information output by the detection unit 202, based on information learned by the judgment and learning unit 4 (hereinafter referred to as learning information). Information necessary for setting the inference device includes information learned by the judgment and learning unit 4 (for example, parameters, etc.), and is preferably acquired from the data storage unit D1.

[0161] The judgment value and control instruction output by the judgment inference unit 303 correspond to the information output by the judgment unit 203 of the control system 100, and therefore the other configurations can be the same as those of the control system.

[0162] [Device Operation] Fig. 9 is a diagram for explaining an example of the operation of the control device according to embodiment 3. Of the processes shown in Fig. 9, steps S101 and S102 are the same as those in embodiment 1, and therefore description thereof will be omitted.

[0163] First, it is confirmed whether there is learning information necessary for setting the inference unit of the judgment and inference unit 303. If there is no learning information (step S301: NO), information on judgment values ​​and control instructions corresponding to the controlled unit information and detection information is acquired (step S302).

[0164] Next, using the acquired information, the judgment learning unit 4 performs learning to output a judgment value and a control instruction from the controlled unit information and the detection information (step S303). If the learning is complete (step S304: YES), the learning information is stored in the data storage unit D1. If the learning is not complete (step S304: NO), the necessary information is increased (step S302), or the learning is continued.

[0165] Once the learning information necessary to set the inference unit of the judgment and inference unit 303 has been acquired (step S301: YES), the learning information is set in the inference unit of the judgment and inference unit 303 of the control device 2 (step S306). In this state, when controlled unit information and detection information are input to the judgment and inference unit 303, a judgment value and a control instruction are output (step S307). At this time, if the judgment value meets the criterion (YES in step S308), the operation continues. If the criterion is not met (step S308: NO), an alert and a control instruction are output to the control unit 102 or the planning unit 204 (step S309).

[0166] In the third embodiment, preferably, the generation of learning information through learning by the judgment learning unit 4 (steps S302 to S306) is completed before the operating device 1 starts operating. Therefore, when the operating device 1 starts operating, for example, processing equivalent to the processing of the control system 100 of the first embodiment (steps S101 to S108) becomes processing of the control system 300 of the third embodiment (steps S101, S102, and steps S307 to S309). Note that the judgment processing and output of control instructions of the first or second control system 100 are parallel processing based on different information, but the judgment inference unit 303 of the third embodiment may also realize similar functions.

[0167] Here, a supplementary explanation will be given regarding the process of acquiring judgment values ​​and control instruction information corresponding to the controlled unit information and detection information (step S302). The controlled unit information is information output by the generation unit 201 and is based on the state information of the controlled unit. The detection information is information output by the detection unit 202 and is based on the controlled unit information and obstacle information. The obstacle information is based on the state information of the controlled unit and environmental information acquired by the observation device 3. Therefore, instead of acquiring state information and environmental information from the actual working device 1 and observation device 3 in the actual working environment, the controlled unit information and detection information may be generated from virtually generated state information and environmental information. The means for generating virtual information is not limited in the present invention, but virtual information can be generated appropriately based on information such as the type of working device, the working environment, the target work, and expected obstacles. For example, a simulator or digital twin may be used for generation, similar to the generation unit 201, or the generation unit 201 may include a generation function. The generated controlled unit information and detection information can be input to the determination unit 203 of the control system 100 to obtain judgment values ​​and control instruction information.

[0168] [Effects of Embodiment 3] Advantages unique to Embodiment 3 will be described. The judgment process of the judgment unit 203 in the control system 100 in Embodiments 1 and 2 was based on judgment criteria, such as preset thresholds. Therefore, appropriate judgment criteria must be set in advance depending on the target operating device 1, the work environment, and the work content. In contrast, in Embodiment 3, the judgment learning unit 4 learns the relationship between the controlled device information and detection information used as input for the judgment process, and the output, i.e., the judgment value representing the judgment result and the control instruction, and the judgment inference unit 303 performs inference using that information. Therefore, because appropriate judgment criteria are also learned, the effort of setting them in advance and the risk of inadequate settings can be eliminated. Furthermore, by learning from data related to the input and output of various judgment processes, it is expected that the generalizability and accuracy of the judgment process of the judgment unit 203 in the control system 100 can be improved compared to the artificially set judgment process.

[0169] Another advantage of the third embodiment is as follows. In this embodiment, the learning information used for learning is not limited to data acquired in the actual work environment, but generated data can be used. Therefore, if learning information is not available before the system is applied, or if there is insufficient learning information available in the actual environment, the system can be operated by generating the necessary data. This feature is expected to have the effect of expanding the scope of application of the system and shortening the time required for application.

[0170] [Program] The program in embodiment 3 may be any program that causes a computer to execute steps S101, S102, and S301 to S309 shown in Fig. 9. By installing and executing this program on a computer, the control device and control method in embodiment 2 can be realized. In this case, the processor of the computer functions as the generation unit 201, the detection unit 202, the judgment and inference unit 303, and the planner 204 and performs processing.

[0171] The program in the third embodiment may be executed by a computer system constructed by a plurality of computers. In this case, for example, each computer may function as one of the generating unit 201, the detecting unit 202, the judgment and inference unit 303, and the planning unit 204.

[0172] The present invention has been described above using the above-mentioned embodiment as an example. However, the present invention is not limited to the above-mentioned content. The present invention can be applied to various forms within the scope of the gist of the present invention.

[0173] [Physical Configuration] Here, a computer that realizes the control device, the operation device, and the judgment learning unit by executing the programs in the first, second, and third embodiments will be described with reference to Fig. 10. Fig. 10 is a diagram for explaining an example of a computer that realizes the control device in the first, second, and third embodiments.

[0174] 10 , the computer 110 includes a CPU (Central Processing Unit) 111, a main memory 112, a storage device 113, an input interface 114, a display controller 115, a data reader / writer 116, and a communication interface 117. These components are connected to each other via a bus 121 so as to be able to communicate data with each other. Note that the computer 110 may include a GPU (Graphics Processing Unit) or an FPGA (Field-Programmable Gate Array) in addition to or instead of the CPU 111.

[0175] The CPU 111 loads a program in the embodiment, which is composed of a group of codes and stored in the storage device 113, into the main memory 112 and executes each code in a predetermined order to perform various calculations. The main memory 112 is typically a volatile storage device such as a DRAM (Dynamic Random Access Memory).

[0176] The program in the embodiment is provided in a state stored in a computer-readable recording medium 120. The program in the embodiment may be distributed over the Internet connected via the communication interface 117.

[0177] Specific examples of the storage device 113 include a hard disk drive and a semiconductor storage device such as a flash memory. The input interface 114 mediates data transmission between the CPU 111 and input devices 118 such as a keyboard and a mouse. The display controller 115 is connected to a display device 119 and controls the display on the display device 119.

[0178] The data reader / writer 116 mediates data transmission between the CPU 111 and the recording medium 120, reads programs from the recording medium 120, and writes processing results from the computer 110 to the recording medium 120. The communication interface 117 mediates data transmission between the CPU 111 and other computers.

[0179] Specific examples of the recording medium 120 include general-purpose semiconductor storage devices such as CF (Compact Flash (registered trademark)) and SD (Secure Digital), magnetic recording media such as flexible disks, or optical recording media such as CD-ROMs (Compact Disk Read Only Memory).

[0180] The control device in the first, second, and third embodiments can be realized not by a computer with a program installed, but by hardware corresponding to each unit, such as an electronic circuit. Furthermore, the control device may be partially realized by a program and the remaining part by hardware. In the first, second, and third embodiments, the computer is not limited to the computer shown in FIG. 10.

[0181] [Supplementary Note] The following supplementary note is further disclosed regarding the above-described embodiment. Some or all of the above-described embodiment can be expressed by (Supplementary Note 1) to (Supplementary Note 24) described below, but is not limited to the following descriptions.

[0182] (Supplementary Note 1) A control device having: a generation unit that generates controlled unit information that represents a current state of a controlled unit possessed by a work device and obstacle information that represents obstacles present in the work environment, using status information that represents the movement of the work device and environmental information that represents a work environment including all or part of the work device observed by an observation device; a detection unit that generates detection information that represents a distance between the controlled unit and the obstacle based on the controlled unit information and the obstacle information; a determination unit that determines whether the controlled unit and the object are approaching each other, using an object that the controlled unit must not approach among the obstacles and the distance between the controlled unit and the object; and a planning unit that generates plan information to control the controlled unit not to approach the object when it is determined that the controlled unit is approaching the object.

[0183] (Supplementary Note 2) The control device according to Supplementary Note 1, wherein the determination unit compares the distance between the controlled unit and the object with a plurality of determination criteria to generate a determination value.

[0184] (Supplementary Note 3) The control device according to Supplementary Note 1, wherein the determination unit further uses a direction or a relative speed, or both, that represents a relationship between the controlled unit and the object, to compare the direction and the relative speed with determination criteria set for each of the direction and the relative speed, and generates a determination value for each of the direction and the relative speed.

[0185] (Supplementary Note 4) The control device according to Supplementary Note 1, wherein the determination unit further generates a determination value based on a relationship between at least one determination point on a surface of the controlled unit and the object.

[0186] (Supplementary Note 5) The control device according to Supplementary Note 2, wherein the control device causes the working device to change control of the operation of the controlled unit based on the plan information, based on the determination value.

[0187] (Supplementary Note 6) The control device according to Supplementary Note 3, wherein the planner generates the controlled unit information and the obstacle information using the plan information in addition to the state information representing a movement of the work device and the environmental information.

[0188] (Supplementary Note 7) The control device according to Supplementary Note 1, wherein the determination unit inputs the controlled unit information and the detection information to a trained inference unit, and outputs a determination value.

[0189] (Supplementary Note 8) The control device according to Supplementary Note 7, further comprising a learning unit that inputs the controlled device information and the detection information generated based on the state information of the controlled device and the environmental information, and learns to output the judgment value.

[0190] (Supplementary Note 9) A control method in which an information processing device uses status information representing the movement of a work device and environmental information representing a work environment including all or part of the work device observed by an observation device to generate controlled unit information representing the current state of a controlled unit possessed by the work device and obstacle information representing obstacles present in the work environment; generates detection information representing the distance between the controlled unit and the obstacle based on the controlled unit information and the obstacle information; determines whether the controlled unit and an object that the controlled unit must not approach are approaching each other using the distance between the controlled unit and an object among the obstacles; and generates plan information to control the controlled unit to prevent the controlled unit from approaching the object if it is determined that the controlled unit is approaching the object.

[0191] (Supplementary Note 10) The control method according to Supplementary Note 9, wherein the determination comprises comparing the distance between the controlled unit and the object with a plurality of determination criteria to generate a determination value.

[0192] (Supplementary Note 11) The control method according to Supplementary Note 9, further comprising the steps of: in the determination, using a direction or a relative velocity, or both, representing a relationship between the controlled unit and the object, comparing the direction and the relative velocity with a determination standard set for each of the direction and the relative velocity, and generating a determination value for each of the direction and the relative velocity.

[0193] (Supplementary Note 12) The control method according to Supplementary Note 9, wherein the determination further comprises generating a determination value based on a relationship between at least one determination point on the surface of the controlled unit and the object.

[0194] (Supplementary Note 13) The control method according to Supplementary Note 10, further comprising causing the work device to change control of the operation of the controlled unit based on the plan information, based on the determination value.

[0195] (Supplementary Note 14) The control method according to Supplementary Note 11, wherein in the plan, the controlled unit information and the obstacle information are generated using the plan information in addition to the state information representing a movement of the work device and the environmental information.

[0196] (Supplementary Note 15) The control method according to Supplementary Note 9, wherein in the determination, the information on the controlled unit and the detection information are input to a trained inference device, and a determination value is output.

[0197] (Supplementary Note 16) The control method according to Supplementary Note 15, further comprising inputting the controlled unit information and the detection information generated based on the state information of the controlled unit and the environmental information, and learning to output the judgment value.

[0198] (Supplementary Note 17) A computer-readable recording medium having recorded thereon a program including instructions to: cause a computer to: use status information representing the movement of a work device and environmental information representing a work environment including all or part of the work device observed by an observation device to generate controlled unit information representing the current state of a controlled unit possessed by the work device and obstacle information representing obstacles present in the work environment; generate detection information representing the distance between the controlled unit and the obstacle based on the controlled unit information and the obstacle information; determine whether the controlled unit and an object that the controlled unit must not approach are approaching each other using the distance between the controlled unit and an object among the obstacles; and generate plan information for controlling the controlled unit to prevent the controlled unit from approaching the object if it is determined that the controlled unit is approaching the object.

[0199] (Supplementary Note 18) The computer-readable recording medium according to Supplementary Note 17, wherein the determination includes comparing the distance between the controlled unit and the object with a plurality of determination criteria to generate a determination value.

[0200] (Supplementary Note 19) The computer-readable recording medium according to Supplementary Note 17, further comprising, in the judgment, using a direction or a relative velocity, or both, representing a relationship between the controlled unit and the object, comparing the direction and the relative velocity with judgment criteria set for each of the direction and the relative velocity, and generating judgment values ​​for each of the direction and the relative velocity.

[0201] (Supplementary Note 20) The computer-readable recording medium according to Supplementary Note 17, wherein the determination further generates a determination value based on a relationship between at least one determination point on the surface of the controlled unit and the object.

[0202] (Supplementary Note 21) The computer-readable recording medium according to Supplementary Note 18, wherein the task device is caused to change control of the operation of the controlled unit based on the plan information, based on the determination value.

[0203] (Supplementary Note 22) The computer-readable recording medium according to Supplementary Note 19, wherein in the plan, the controlled unit information and the obstacle information are generated using the plan information in addition to the state information representing the movement of the work device and the environmental information.

[0204] (Supplementary Note 23) The computer-readable recording medium according to Supplementary Note 17, wherein in the judgment, the information on the controlled unit and the detection information are input to a trained inference device, and the trained inference device outputs a judgment value.

[0205] (Supplementary Note 24) The computer-readable recording medium according to Supplementary Note 23, further comprising: inputting the controlled unit information and the detection information generated based on the state information of the controlled unit and the environmental information, and learning to output the judgment value.

[0206] Although the present invention has been described above with reference to the embodiments, the present invention is not limited to the above-described embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention.

[0207] According to the above description, it is possible to realize control of a working device having a moving part that achieves both safety and work efficiency, and is useful in fields where control of a working device having a moving part is required.

[0208] 1 Working device 2 Control device 3 Observation device 4 Decision learning unit 100, 300 Control system 101 Controlled unit 102 Control unit 201 Generation unit 202 Detection unit 203 Decision unit 204 Planning unit 303 Decision inference unit D1 Data storage unit 110 Computer 111 CPU 112 Main memory 113 Storage device 114 Input interface 115 Display controller 116 Data reader / writer 117 Communication interface 118 Input device 119 Display device 120 Recording medium 121 Bus

Claims

1. (Embodiment 1) A control device includes: a generation means for generating controlled part information representing the current state of a controlled part of the working device and obstacle information representing obstacles existing in the working environment, using state information representing the movement of the working device and environment information representing a part or all of the working device observed by an observation device; a detection means for generating detection information representing the distance between the controlled part and the obstacle based on the controlled part information and the obstacle information; a determination means for determining whether the controlled part and the object are approaching each other, using the distance between the controlled part and the object which is an object that the controlled part should not approach; and a planning means for generating planning information for controlling the controlled part so as not to approach the object when it is determined that the controlled part is approaching the object.

2. The control device according to claim 1, wherein the determination means generates a determination value by comparing the distance between the controlled part and the object with a plurality of determination criteria.

3. The control device according to claim 1, wherein the determination means further generates a determination value for each of the direction and the relative speed by comparing the direction or the relative speed or both representing the relationship between the controlled part and the object with determination criteria set for each of the direction and the relative speed.

4. The control device according to claim 1, wherein the determination means further generates a determination value based on the relationship between at least one or more determination points on the surface of the controlled part and the object.

5. The control device according to claim 2, wherein based on the determination value, the operation of the controlled part of the working device is changed to be controlled according to the planning information.

6. The control device according to claim 3, wherein the planning means generates the controlled part information and the obstacle information using the planning information in addition to the state information representing the movement of the working device and the environment information.

7. The control device according to claim 1, wherein the determination means inputs the controlled part information and the detection information into a learned inference device and outputs a determination value.

8. Further, a learning unit that learns to input the state information of the controlled unit, the controlled unit information and the detection information generated based on the environmental information, and output the determination value, the control device according to claim 7.

9. An information processing device uses state information representing the movement of a working device and environmental information representing a working environment including part or all of the working device observed by an observation device to generate controlled unit information representing the current state of a controlled unit of the working device and obstacle information representing obstacles existing in the working environment, generates detection information representing the distance between the controlled unit and the obstacle based on the controlled unit information and the obstacle information, determines whether the controlled unit and a target object that the controlled unit should not approach are approaching each other using the distance between the target object and the controlled unit among the obstacles, and generates planning information for controlling the controlled unit not to approach the target object when it is determined that the controlled unit is approaching the target object.

10. A computer-readable recording medium recording a program including instructions to cause a computer to generate controlled unit information representing the current state of a controlled unit of a working device and obstacle information representing obstacles existing in a working environment using state information representing the movement of the working device and environmental information representing a working environment including part or all of the working device observed by an observation device, cause the computer to generate detection information representing the distance between the controlled unit and the obstacle based on the controlled unit information and the obstacle information, cause the computer to determine whether the controlled unit and a target object that the controlled unit should not approach are approaching each other using the distance between the target object and the controlled unit among the obstacles, and cause the computer to generate planning information for controlling the controlled unit not to approach the target object when it is determined that the controlled unit is approaching the target object.

Images