Control device, control method, and storage medium

Abstract

The present application provides a control device, a control method and a storage medium capable of appropriately evaluating the risk of a travel track while reducing the processing load. The control device includes: a surrounding image acquisition unit that acquires an image captured by a fisheye camera mounted on a mobile body, i.e., a surrounding image of the mobile body; a base track calculation unit that calculates an indication related to future travel of the mobile body as a base track in an orthogonal coordinate system; a coordinate transformation unit that transforms the acquired base track coordinates in the orthogonal coordinate system into a base track in a fisheye camera coordinate system; a risk calculation unit that calculates the risk of the base track in the fisheye camera coordinate system based on the surrounding image and the base track in the fisheye camera coordinate system; and a travel track calculation unit that corrects the base track in the orthogonal coordinate system based on the risk of the base track in the fisheye camera coordinate system, thereby calculating a travel track.

Description

Technical Field

[0001] The present invention relates to a control device, a control method and a storage medium. Background Art

[0002] Technologies for mounting a fisheye camera on an autonomously moving robot are known. For example, Japanese Patent Application Laid-Open No. 2004-303137 discloses a technology for mounting a fisheye camera on an autonomously moving robot and calculating the robot's trajectory based on images captured by the fisheye camera. Summary of the Invention

[0003] The technology described in Japanese Patent Application Laid-Open No. 2004-303137 is as follows: a marker is set to indicate a specific location, and the robot is driven using the relative distance and direction between the marker and the robot. However, setting the marker to drive the robot is time-consuming.

[0004] Furthermore, in the prior art, all points in an image captured by a fisheye camera are transformed from a fisheye camera coordinate system into an orthogonal coordinate system, and the risk of the driving track is evaluated based on the transformed coordinates to correct the driving track. Therefore, it is assumed that the processing load is large when transforming from a fisheye camera coordinate system into an orthogonal coordinate system.

[0005] The present invention has been made in consideration of such circumstances, and one object of the present invention is to provide a control device, a control method, and a storage medium capable of appropriately evaluating the risk of a travel trajectory while reducing the processing load.

[0006] The control device, control method and storage medium of the present invention adopt the following structure. (1): The control device of one embodiment of the present invention comprises: a peripheral image acquisition unit, which acquires an image captured by a fisheye camera mounted on a mobile body, i.e., a peripheral image of the mobile body; a basic track calculation unit, which calculates an indication related to the future travel of the mobile body as a basic track in an orthogonal coordinate system; a coordinate transformation unit, which transforms the acquired basic track coordinates in the orthogonal coordinate system into a basic track in a fisheye camera coordinate system; a risk calculation unit, which calculates the risk of the basic track in the fisheye camera coordinate system based on the peripheral image and the basic track in the fisheye camera coordinate system; and a driving track calculation unit, which corrects the basic track in the orthogonal coordinate system based on the risk of the basic track in the fisheye camera coordinate system, thereby calculating a driving track.

[0007] (2): Based on the solution of (1) above, the coordinate transformation unit transforms the coordinates of the travel track in the orthogonal coordinate system into the travel track in the fisheye camera coordinate system.

[0008] The risk calculation section calculates a risk of the travel track in the fisheye camera coordinate system based on the peripheral image and the travel track in the fisheye camera coordinate system,

[0009] The travel track calculation section corrects the travel track in the orthogonal coordinate system based on the risk of the travel track in the fisheye camera coordinate system, and thereby calculates the travel track again.

[0010] (3) : The above (1) or (2) is based on the scheme, further provided with travel control section, the travel control section makes the mobile body along the basic track or travel track,

[0011] In the case where the risk calculation section determines that the risk of the basic track or travel track is below the threshold value, the travel control section makes the mobile body along the basic track or travel track.

[0012] (4) : In any of the above (1) to (3) is based on the scheme, further provided with gesture detection section, the gesture detection section detects the body movement based on the user's representation and the future travel of the mobile body related to the instruction,

[0013] The basic track calculation section calculates the basic track in the orthogonal coordinate system based on the body movement detected by the gesture detection section.

[0014] (5) : The control method of other schemes of the present application makes the computer mounted on the mobile body to perform the following processing:

[0015] Obtain the image photographed by the fisheye camera mounted on the mobile body, that is, the peripheral image of the mobile body;

[0016] Obtain the instruction related to the future travel of the mobile body as the basic track in the orthogonal coordinate system;

[0017] Transform the obtained basic track coordinates in the orthogonal coordinate system into the basic track in the fisheye camera coordinate system;

[0018] Calculate the risk of the basic track in the fisheye camera coordinate system based on the peripheral image and the basic track in the fisheye camera coordinate system; and

[0019] Correct the basic track in the orthogonal coordinate system based on the risk of the basic track in the fisheye camera coordinate system, and thereby calculate the travel track.

[0020] (6) : The storage medium of other schemes of the present application stores a program, wherein the program makes the computer mounted on the mobile body to perform the following processing:

[0021] acquire an image captured by a fisheye camera mounted on a mobile body, i.e., a surrounding image of the mobile body;

[0022] acquire an indication related to future travel of the mobile body as a base track in an orthogonal coordinate system;

[0023] transform the acquired base track coordinates in the orthogonal coordinate system into a base track in a fisheye camera coordinate system;

[0024] calculate a risk of the base track in the fisheye camera coordinate system based on the surrounding image and the base track in the fisheye camera coordinate system; and

[0025] correct the base track in the orthogonal coordinate system based on the risk of the base track in the fisheye camera coordinate system, thereby calculating a travel track.

[0026] According to the schemes of (1) to (6), it is possible to appropriately evaluate the risk of the travel track while reducing the processing load. BRIEF DESCRIPTION OF DRAWINGS

[0027] Figure 1 is a diagram showing an example of a scenario in which a mobile body equipped with a control device of an embodiment is used.

[0028] Figure 2 is a diagram for explaining an example of the overall structure of a mobile body.

[0029] Figure 3 is a diagram showing an example of a gesture shown by a user.

[0030] Figure 4 is a diagram showing an example of coordinate transformation of a base track performed by a coordinate transformation section.

[0031] Figure 5 is a diagram showing an example of a scenario in which a risk calculation section calculates a risk of a base track in a fisheye camera coordinate system.

[0032] Figure 6 is a diagram showing an example of detection processing of a drivable space performed by a drivable space detection section.

[0033] Figure 7 is a diagram showing an example of a scenario in which a travel track calculation section calculates a travel track in an orthogonal coordinate system.

[0034] Figure 8 is a diagram showing an example of calculation processing of a travel track performed by a travel track calculation section and coordinate transformation of the travel track performed by a coordinate transformation section.

[0035] Figure 9 is a flowchart showing an example of a flow of processing performed by a control device. DETAILED DESCRIPTION

[0036] <Embodiment>

[0037] Hereinafter, a control device, a control method, and a storage medium of an embodiment of the present application will be described with reference to the drawings.

[0038] [Overall Structure]

[0039] Figure 1 is a diagram showing an example of a scenario in which a mobile body 10 provided with a control device of an embodiment is used. The mobile body 10 is an autonomous mobile robot provided with a fisheye camera 20, a receptacle 30, and a wheel 40. As a usage form of the mobile body 10, there are the following forms. A user U holds baggage B, and the mobile body 10 moves in the direction of the user U to accommodate the baggage B in accordance with a gesture of the user. Since there is an obstacle OB between the mobile body 10 and the user U, the mobile body 10 autonomously moves to the vicinity of the user U while avoiding the obstacle OB. After the user U places the baggage B in the receptacle 30, the mobile body 10 moves following the user U.

[0040] The fisheye camera 20 is, for example, a camera including a fisheye lens and capable of capturing the periphery of the mobile body 10 in a wide angle (for example, 360 degrees). The fisheye camera 20 captures the periphery of the mobile body 10 in a wide angle in the horizontal direction, for example, by being installed on the upper portion of the mobile body 10. The fisheye camera 20 can also be implemented by combining a plurality of 120-degree cameras or 60-degree cameras.

[0041] The receptacle 30 is a container for accommodating arbitrary articles or baggage, and is fixed to the main body portion of the mobile body 10.

[0042] The wheel 40 is driven by a plurality of motors 50 mounted inside the mobile body 10, and realizes movement based on the mobile body 10. The wheel 40 includes, for example, a drive wheel driven in the rotation direction by the motor 50, and a non-drive wheel, i.e., a steering wheel, driven in the deflection direction. By adjusting the angle of the steering wheel, the mobile body 10 is able to change the advancing path.

[0043] Note that in the present application, the mobile body 10 is not necessarily provided with the receptacle 30. Furthermore, in the present embodiment, the mobile body 10 is provided with the wheel 40 as a mechanism for realizing movement, but the present application is not limited to this structure, and for example, the mobile body 10 can also be a multi-legged walking robot.

[0044] Figure 2is a diagram for explaining an example of the overall structure of the mobile body 10. The mobile body 10 has, in addition to the fisheye camera 20, the housing 30, the wheels 40, and the motor 50 described above, a control device 100. The control device 100 has, for example, a surrounding image acquisition section 110, a gesture detection section 120, a base track calculation section 130, a coordinate conversion section 140, a risk calculation section 150, a drivable space detection section 160, a travel track calculation section 170, and a travel control section 180. Each of the sections of the control device 100 is realized, for example, by a hardware processor such as a CPU (Central Processing Unit) executing a program (software). Part or all of the sections of the control device 100 can be realized by a hardware (including circuitry) such as an LSI (Large Scale Integration), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), a GPU (Graphics Processing Unit), and the like, and can be realized by a cooperation of software and hardware. The program can be stored in advance in a storage device (a storage device having a non-transitory storage medium) such as an HDD (Hard Disk Drive), a flash memory, and the like, or in a removable storage medium (a non-transitory storage medium) such as a DVD, a CD-ROM, and the like, and installed by mounting the storage medium in a drive device.

[0045] The surrounding image acquisition section 110 acquires an image (hereinafter referred to as a "surrounding image") captured by the fisheye camera 20. The surrounding image acquired at this time is a 360-degree image, and thus the surrounding image acquisition section 110 holds the acquired surrounding image as pixel data in the fisheye camera coordinate system.

[0046] The gesture detection section 120 detects a body movement (hereinafter referred to as a "gesture") of the user U on the basis of one or more surrounding images. In the present embodiment, the gesture is set in advance as a content indicating an instruction related to the future travel of the mobile body 10. The gesture detection section 120 holds data (template data) related to a plurality of gestures in advance, and detects a gesture in a surrounding image by matching feature data acquired from the surrounding image with the template data. The feature data and the template data are, for example, data representing feature points of a skeleton such as a fingertip, a joint of a finger, a wrist, an elbow, and a link (1ink) connecting them, by abstract data.

[0047] Figure 3 is a diagram representing an example of a gesture shown by the user U. In Figure 3In this case, the user U extends the right hand to the front. In the present embodiment, the gesture of extending the right hand to the front has the meaning of "please come to the user U". In addition to this, the gesture detection section 120 detects various gestures having the meanings of self-rotation, stop, backward movement, and the like, but the details thereof are omitted.

[0048] Note that in the present embodiment, the mobile body 10 is provided with the gesture detection section 120 that detects the gesture of the user U, but as long as an instruction related to the future movement of the mobile body 10 can be recognized, another structure can be used. For example, the mobile body 10 can be provided with a sound recognition section that recognizes a sound emitted by the user U, and the instruction related to the future movement of the mobile body 10 (for example, "come" and the like) can be recognized by recognizing the case where the user U transmits the instruction by sound. Further, as another example, the user U can use a dedicated application to transmit information indicating an instruction related to the future movement of the mobile body 10 from an information processing terminal such as a smartphone held by the user U to the mobile body 10, and the mobile body 10 can recognize the instruction by receiving the information.

[0049] The base track calculation section 130 calculates a base track BL that is initial information of a track in which the mobile body 10 should move in the future, on the basis of the gesture detected by the gesture detection section 120. At this time, the base track calculation section 130 calculates the base track BL in an orthogonal coordinate system. The orthogonal coordinate system is a coordinate system that takes the position of the mobile body 10 as the origin and takes as axes arbitrary directions that are orthogonal to each other and are fixedly set to the main body of the mobile body 10. For example, in the case where the user U shows the gesture of extending the right hand to the front, the base track calculation section 130 calculates a base track BL-1 that takes the position of the mobile body 10 as a starting point, an initial movement vector coincides with the orientation of the mobile body 10, and takes the position immediately in front of the user U as an end point. The base track BL-1 can also be calculated so that the front surface of the mobile body 10 (the surface on the side on which the housing 30 is present) faces the user U at the position immediately in front of the user U. The base track BL-1 is calculated, for example, by fitting the state to a geometric model such as a Bezier curve. The base track is calculated, for example, without taking into account the presence of the obstacle OB. The base track BL is actually generated as a set of a finite number of track points.

[0050] The coordinate conversion section 140 performs coordinate conversion between the orthogonal coordinate system and the fisheye camera coordinate system. Between the orthogonal coordinate system and the fisheye camera coordinate system, a one-to-one relationship between the coordinates is established, which is held as correspondence information in the coordinate conversion section 140. Hereinafter, the case where the information is in the orthogonal coordinate system is indicated by the symbol "-1", and the case where the information is in the fisheye camera coordinate system is indicated by the symbol "-2". In the case where the two are not distinguished, "-1", "-2" are omitted. The coordinate conversion section 140 converts the base track BL-1 in the orthogonal coordinate system, which is calculated by the base track calculation section 130, into the base track BL-2 in the fisheye camera coordinate system. The coordinate conversion section 140 also converts the travel track TL-1 in the orthogonal coordinate system, which is calculated by the travel track calculation section 170, into the travel track TL-2 in the fisheye camera coordinate system, as described later.

[0051] Here, the travel track TL refers to a track obtained by modifying the base track BL, which is calculated, for example, in the case where it is determined that the risk of traveling on the base track BL is high due to the presence of the obstacle OB, as described later.

[0052] Figure 4 is a diagram indicating an example of coordinate conversion of the base track BL performed by the coordinate conversion section 140. Figure 4 The upper left portion of indicates the base track BL-1 calculated by the base track calculation section 130, Figure 4 The lower left portion of indicates the surrounding image of the mobile body 10 acquired by the surrounding image acquisition section 110. The coordinate conversion section 140 can acquire the base track BL-2 in the fisheye camera coordinate system by coordinate-converting the base track BL-1 into the fisheye camera coordinate system, and calculate the risk by superimposing the base track BL-2 on the surrounding image.

[0053] The risk calculation section 150 calculates the risk of each coordinate in the fisheye camera coordinate system, and finds the distribution of the risk in the target region of the fisheye camera coordinate system. The risk is an index value indicating the level of the possibility of collision of the mobile body 10 with other obstacles. The risk calculation section 150 calculates the risk of the base track BL-2 or the travel track TL-2 using the distribution of the risk (risk function). The risk calculation section 150 calculates the risk J of the base track BL-2 or the travel track TL-2, for example, by calculating the risk Ji at each track point on the base track BL or the travel track TL of the mobile body 10, and finding the sum ∑Ji of the risks Ji. That is, J = ∑Ji holds.

[0054] The risk at a certain track point on the base track BL-2 or the travel track TL-2 can be calculated based on the distance between the track point and the obstacle, the speed of the obstacle. For example, the smaller the distance between each track point on the base track BL-2 or the travel track TL-2 and the obstacle, the higher the risk at the track point calculated by the risk calculating section 150, the larger the distance between a certain track point on the base track BL-2 or the travel track TL-2 and the obstacle, the lower the risk at the track point calculated by the risk calculating section 150. More specifically, the risk calculating section 150 sets the risk in the region containing the minimum circumscribed circle of the obstacle OB to a prescribed value (for example, 1) and calculates the risk Ji in such a manner that it becomes smaller the farther from the region and becomes zero at a certain place. Further, with respect to the moving obstacle OB, since the risk at each time in the future is different, the data of the risk distribution is prepared for each arrival time of the moving body 10 in the travel track TL.

[0055] Figure 5 is a graph showing an example of a scenario in which the risk calculating section 150 calculates the risk of the base track BL-2 in the fisheye camera coordinate system. Figure 5 is a graph showing the risk of the base track BL-2 in the fisheye camera coordinate system, which is obtained by the coordinate converting section 140 in Figure 4 Figure 5 In the graph of, the hatched region RR represents a risk region in which the risk is positive. In the graph, P1 to P4 are track points constituting the base track BL-2. The risk calculating section 150 substitutes the positions of these track points P1 to P4 into the risk function for calculating the risk of each track point to calculate the risk values Ji P1 , Ji P2 , Ji P3 , and Ji P4 at each track point. Next, the risk calculating section 150 obtains the sum of the risk values at each track point to calculate the risk ∑J = Ji P1 + Ji P2 + Ji P3 + Ji P4 of the base track BL. As a result, the risk calculating section 150 can calculate the risk of the base track BL-2. Note that the method by which the risk calculating section 150 calculates the risk of the travel track TL-2 is the same.

[0056] After calculating the risk ΣJ of the base track BL-2 in the fisheye camera coordinate system, the risk calculation unit 150 compares the calculated risk ΣJ with a threshold value Th (e.g., 1) to determine whether to cause the moving object 10 to travel along the base track BL. If the risk calculation unit 150 determines that the risk ΣJ is within the threshold value Th, the moving object 10 is caused to travel along the base track BL. On the other hand, if the risk calculation unit 150 determines that the risk ΣJ is greater than the threshold value Th, the moving object 10 is not caused to travel along the base track BL.

[0057] It should be noted that, in this embodiment, the risk calculation unit 150 compares the risk ΣJ with the threshold value Th. However, the method of comparing the risk ΣJ with the threshold value Th is not limited to this structure. For example, the risk calculation unit 150 may also compare the risk ΣJ with the threshold value Th. Figure 5 Extract the risk values ​​J on the basic track BL-2 P1 、J P2 、J P3 and J P4 The maximum value in Figure 5 In the case of J P3 ), and compares the maximum value with the threshold value Th. The method of comparing the risk ΣJ with the threshold value Th is advantageous in that the moving body 10 can travel more safely, and the method of comparing the maximum value with the threshold value Th is advantageous in that the moving body 10 can travel more efficiently.

[0058] When the risk calculation unit 150 determines not to cause the vehicle 10 to travel along the base track BL, the traversable space detection unit 160 detects a traversable space FS-2 in the fisheye camera coordinate system based on the surrounding image.

[0059] Figure 6 1 is a diagram showing an example of a detection process of the drivable space FS by the drivable space detection unit 160. Figure 6 As shown, since the risk calculation unit 150 determines that the moving body 10 should not move along the basic track BL, the drivable space detection unit 160 detects the space represented by the diagonal portion, that is, the space extended from the starting point of the basic track BL-2 (the position of the moving body 10) to the end point (the position in front of the user U) and the obstacles (obstacle OB and user U) are removed as the drivable space FS-2.

[0060] The travel trajectory calculating section 170 calculates a correction amount of each trajectory point in the orthogonal coordinate system based on the risk of each trajectory point calculated by the risk calculating section 150, as a correction amount in the orthogonal coordinate system, of each trajectory point in the range of the travelable space FS-2 detected by the travelable space detecting section 160. Here, the larger the risk value Ji at the corresponding trajectory point in the fisheye camera coordinate system, the larger the correction width Δ in the direction orthogonal to the tangent line of the trajectory point is made to execute the correction amount of each trajectory point in the orthogonal coordinate system. That is, the travel trajectory calculating section 170 calculates the travel trajectory TL-1 by changing the trajectory points of the base trajectory BL-1 by the correction width Δ corresponding to each risk value Ji. In this way, the travel trajectory calculating section 170 does not transform the image coordinates captured by the fisheye camera 20 into the orthogonal coordinate system, but calculates the correction amount of the corresponding trajectory point in the orthogonal coordinate system based on the risk of each trajectory point in the fisheye camera coordinate system, so it is possible to reduce the amount of calculation required for the correction of the trajectory.

[0061] Figure 7 is an example of a scene in which the travel trajectory calculating section 170 calculates the travel trajectory TL-1 in the orthogonal coordinate system. In Figure 7 , the trajectory points Q1, Q2, Q3, and Q4 are trajectory points obtained by changing each trajectory point P1, P2, P3, and P4 of the base trajectory BL-1 by the trajectory correction parameter Δ corresponding to each risk value Ji. In Figure 7 , the risk values Ji of the trajectory points P1, P2, P3, and P4 become larger in the order of J P3 , J P2 , J P4 , and J P1 , so the correction width Δ in the direction orthogonal to the tangent line of each trajectory point also becomes larger in the order of P3, P2, P4, and P1.

[0062] Next, the coordinate transforming section 140 transforms the travel trajectory TL-1 in the orthogonal coordinate system into the travel trajectory TL-2 in the fisheye camera coordinate system, and the risk calculating section 150 calculates the risk values Ji Q1 , J Q2 , J Q3 , and J Q4 at each trajectory point Q1 to Q4 on the travel trajectory TL-2 by substituting the positions of the trajectory points into the risk function. At this time, the risk distribution (risk function) used for the calculation is the same as the risk distribution used when the risk of the base trajectory BL-2 is calculated. Next, the risk calculating section 150 calculates the risk ∑J = Ji Q1 + Ji Q2 + Ji Q3 + Ji Q4 of the travel trajectory TL-1 by obtaining the sum of the risk values at each trajectory point. In Figure 7In this case, the track points Q1 to Q4 are located outside the risk region RR of the obstacle OB, and thus the risk calculating section 150 obtains ∑J = J Q1 + J Q2 + J Q3 + J Q4 = 0.

[0063] The risk calculating section 150, when calculating the risk ∑J of the travel track TL-1 in the fisheye camera coordinate system, next compares the calculated risk ∑J with the threshold value Th, and determines whether or not to cause the mobile body 10 to travel along the travel track TL. The risk calculating section 150, in the case where it is determined that the risk ∑J is within the threshold value Th, determines to cause the mobile body 10 to travel along the travel track TL. On the other hand, the risk calculating section 150, in the case where it is determined that the risk ∑J is greater than the threshold value Th, determines not to cause the mobile body 10 to travel along the travel track TL, and repeatedly performs the same correction until the risk ∑J becomes below the threshold value Th, and calculates the travel track TL again. In Figure 7 In this case, the risk ∑J = 0 becomes smaller than the threshold value Th, and thus the risk calculating section 150 determines to cause the mobile body 10 to travel along the travel track TL.

[0064] Note that, in the above description, the correction width Δ of the track points is a value corresponding to the respective risk values Ji, but at this time, the travel track calculating section 170 can also multiply the track correction parameter Δ by a random number in order to avoid taking a local solution of the risk ∑J by the travel track TL-1.

[0065] Figure 8 is a diagram showing an example of the calculation processing of the travel track TL by the travel track calculating section 170 and the coordinate transformation of the travel track TL by the coordinate transformation section 140. The travel track calculating section 170, as described above, uses the correction amount in the orthogonal coordinate system calculated based on the risk of each track point in the fisheye camera coordinate system, and changes the corresponding track point of the base track BL-1 by the track correction parameter Δ, to calculate the travel track TL-1. Next, the coordinate transformation section 140 coordinate-transforms the travel track TL-1 in the orthogonal coordinate system into the travel track TL-2 in the fisheye camera coordinate system, and the risk calculating section 150 calculates the risk of the travel track TL-2 based on the surrounding image and the travel track TL-2 in the fisheye camera coordinate system. In this way, by calculating the correction amount of the corresponding track point in the orthogonal coordinate system based on the risk of each track point in the fisheye camera coordinate system, it is possible to appropriately evaluate the risk of the travel track while reducing the processing load required for the correction.

[0066] In a case where the risk calculating section 150 determines that the risk value of the base track BL or the travel track TL is below the threshold value Th, the travel control section 180 causes the mobile body 10 to travel along the base track BL or the travel track TL. Specifically, the travel control section 180 outputs an instruction value for causing the mobile body 10 to travel along the base track BL or the travel track TL to the motor 50, and the motor 50 rotates the wheel 40 in accordance with the instruction value.

[0067] [Flow of processing]

[0068] Next, the flow of processing by the control device 100 according to the present embodiment will be described with reference to Figure 9 The flow of processing by the control device 100 according to the present embodiment will be described. Figure 9 is a flowchart showing an example of the flow of processing performed by the control device 100. The processing of the present flowchart is performed for each predetermined control cycle.

[0069] First, the surrounding image acquiring section 110 acquires the surrounding image of the mobile body 10 captured by the fisheye camera 20 (step S100). Next, the gesture detecting section 120 detects a gesture indicating an instruction related to the future travel of the mobile body 10 on the basis of the surrounding image acquired by the surrounding image acquiring section 110 (step S101). Next, the base track calculating section 130 calculates the base track BL-1 in the orthogonal coordinate system on the basis of the gesture detected by the gesture detecting section 120 (step S102). Next, the coordinate transforming section 140 coordinates transforms the base track BL-1 in the orthogonal coordinate system calculated by the base track calculating section 130 into the base track BL-2 in the fisheye camera coordinate system (step S103). Next, the risk calculating section 150 calculates the risk of the base track BL-2 in the fisheye camera coordinate system on the basis of the surrounding image and the base track BL-2 in the fisheye camera coordinate system (step S104).

[0070] Next, the risk calculating section 150 determines whether the calculated risk of the basic track BL-2 is below the threshold value Th (step S105). In the case where the risk calculating section 150 determines that the calculated risk of the basic track BL-2 is below the threshold value Th, the travel control section 180 causes the mobile body 10 to travel along the basic track BL (step S106). On the other hand, in the case where the risk calculating section 150 determines that the calculated risk of the basic track BL-2 is greater than the threshold value Th, the travelable space detecting section 160 detects a travelable space FS-2 in the fisheye camera coordinate system based on the surrounding image (step S107). Next, the travel track calculating section 170 calculates, as the correction amount in the orthogonal coordinate system, a correction amount of each track point that is within the range of the travelable space FS-2 detected by the travelable space detecting section 160, based on the risk of each track point of the basic track BL-2 calculated by the risk calculating section 150 (step S108). Next, the travel track calculating section 170 calculates a travel track TL-1 by correcting the basic track BL-1 by the correction amount in the orthogonal coordinate system (step S109). Next, the coordinate converting section 140 converts the travel track TL-1 in the orthogonal coordinate system into a travel track TL-2 in the fisheye camera coordinate system (step S110). Next, the risk calculating section 150 calculates the risk of the travel track TL-2 in the fisheye camera coordinate system based on the surrounding image and the travel track TL-2 (step S111).

[0071] Next, the risk calculating section 150 determines whether the calculated risk of the travel track TL-2 is below the threshold value Th (step S112). In the case where the risk calculating section 150 determines that the calculated risk of the travel track TL-2 is below the threshold value Th, the travel control section 180 causes the mobile body 10 to travel along the travel track TL (step S113). On the other hand, in the case where the risk calculating section 150 determines that the calculated risk of the travel track TL-2 is greater than the threshold value Th, the process is returned to step S109 again, and the travel track TL-1 is calculated again.

[0072] As described above, according to the embodiment of the present application, the control device does not convert the image captured by the fisheye camera into the orthogonal coordinate system, but calculates a correction amount of the basic track in the orthogonal coordinate system based on the risk of the basic track in the fisheye camera coordinate system, corrects the basic track in the orthogonal coordinate system by the correction amount, and again converts the travel track obtained thereby into the fisheye camera coordinate system, and evaluates the risk of the travel track in the fisheye camera coordinate system. Thereby, it is possible to appropriately evaluate the risk of the travel track while reducing the processing load.

[0073] The above-described embodiment can be expressed as follows.

[0074] A control device, wherein

[0075] The control device is configured to have:

[0076] a storage device storing a program; and

[0077] a hardware processor,

[0078] performing the following processing by executing the program stored in the storage device by the hardware processor:

[0079] acquiring an image captured by a fisheye camera mounted on a mobile body, i.e., a surrounding image of the mobile body;

[0080] calculating an indication related to future travel of the mobile body as a base track in an orthogonal coordinate system;

[0081] transforming the acquired base track coordinates in the orthogonal coordinate system into a base track in a fisheye camera coordinate system;

[0082] calculating a risk of the base track in the fisheye camera coordinate system based on the surrounding image and the base track in the fisheye camera coordinate system; and

[0083] correcting the base track in the orthogonal coordinate system based on the risk of the base track in the fisheye camera coordinate system, thereby calculating a travel track.

[0084] The above describes the specific embodiments of the present application using the embodiments, but the present application is not limited at all by such embodiments, and various modifications and substitutions can be applied within the scope of the gist of the present application.

Claims

1. A control device, wherein the control device is provided with: a surrounding image acquisition section that acquires an image captured by a fisheye camera mounted on a mobile body, i.e., a surrounding image of the mobile body; a base track calculation section that calculates an indication related to future travel of the mobile body as a base track in an orthogonal coordinate system; a coordinate conversion section that converts the acquired base track coordinates in the orthogonal coordinate system into a base track in a fisheye camera coordinate system; a risk calculation section that calculates a risk of the base track in the fisheye camera coordinate system based on the surrounding image and the base track in the fisheye camera coordinate system; and a travel track calculation section that corrects the base track in the orthogonal coordinate system based on the risk of the base track in the fisheye camera coordinate system, thereby calculating a travel track, the coordinate conversion section converts the travel track in the orthogonal coordinate system into a travel track in the fisheye camera coordinate system, the risk calculation section calculates a risk of the travel track in the fisheye camera coordinate system based on the surrounding image and the travel track in the fisheye camera coordinate system, the travel track calculation section corrects the travel track in the orthogonal coordinate system based on the risk of the travel track in the fisheye camera coordinate system without correcting the travel track in the fisheye camera coordinate system, thereby recalculating the travel track, the coordinate conversion section converts the travel track recalculated in the orthogonal coordinate system into a travel track in the fisheye camera coordinate system, the control device is further provided with a travel control section that causes the mobile body to travel along the base track or travel track.

2. The control device according to claim 1, wherein in a case where the risk calculation section determines that the risk of the base track or travel track is below a threshold value, the travel control section causes the mobile body to travel along the base track or travel track.

3. The control device according to claim 1 or 2, wherein the control device is further provided with a gesture detection section that detects a body motion based on a user's expression of an indication related to future travel of the mobile body, the base track calculation section calculates the base track in the orthogonal coordinate system based on the body motion detected by the gesture detection section.

4. A control method, wherein the control method causes a computer mounted on a mobile body to perform the following processes: acquire an image captured by a fisheye camera mounted on a mobile body, i.e., a surrounding image of the mobile body; acquire an indication related to future travel of the mobile body as a base track in an orthogonal coordinate system; convert the acquired base track in the orthogonal coordinate system into a base track in a fisheye camera coordinate system; calculate a risk of the base track in the fisheye camera coordinate system based on the surrounding image and the base track in the fisheye camera coordinate system; correct the base track in the orthogonal coordinate system based on the risk of the base track in the fisheye camera coordinate system, thereby calculating a travel track; and correct the travel track in the orthogonal coordinate system based on the risk of the travel track in the fisheye camera coordinate system without correcting the travel track in the fisheye camera coordinate system, thereby recalculating the travel track. transforming the travel track coordinates in the orthogonal coordinate system into a travel track in the fisheye camera coordinate system; calculating a risk of the travel track in the fisheye camera coordinate system based on the surrounding image and the travel track in the fisheye camera coordinate system; correcting the travel track in the orthogonal coordinate system based on the risk of the travel track in the fisheye camera coordinate system without correcting the travel track in the fisheye camera coordinate system, thereby recalculating the travel track; transforming the travel track coordinates recalculated in the orthogonal coordinate system into a travel track in the fisheye camera coordinate system; and causing the mobile body to travel along the base track or the travel track.

5. A storage medium storing a program, wherein, The program causes a computer mounted on a mobile body to perform the following processing: acquiring an image captured by a fisheye camera mounted on a mobile body, i.e., a surrounding image of the mobile body; acquiring an indication related to future travel of the mobile body as a base track in an orthogonal coordinate system; transforming the acquired base track coordinates in the orthogonal coordinate system into a base track in a fisheye camera coordinate system; calculating a risk of the base track in the fisheye camera coordinate system based on the surrounding image and the base track in the fisheye camera coordinate system; correcting the base track in the orthogonal coordinate system based on the risk of the base track in the fisheye camera coordinate system, thereby calculating a travel track; transforming the travel track coordinates in the orthogonal coordinate system into a travel track in the fisheye camera coordinate system; calculating a risk of the travel track in the fisheye camera coordinate system based on the surrounding image and the travel track in the fisheye camera coordinate system; correcting the travel track in the orthogonal coordinate system based on the risk of the travel track in the fisheye camera coordinate system without correcting the travel track in the fisheye camera coordinate system, thereby recalculating the travel track; transforming the travel track coordinates recalculated in the orthogonal coordinate system into a travel track in the fisheye camera coordinate system; and causing the mobile body to travel along the base track or the travel track.

Images