Mobile body control device, mobile body control method, and program

Abstract

To provide a movable body control device, a movable body control method, and a program capable of executing more appropriate control of a movable body.SOLUTION: A movable body control device according to an embodiment includes: a recognition section 130 for recognizing a situation around a movable body on the basis of a first image captured by an image capturing section; an action plan generating section 140 for generating a future action plan of the movable body on the basis of a recognition result obtained by the recognition section 130; and a driving control section 160 for controlling at least one of steering and a speed of the movable body on the basis of the action plan generated by the action plan generating section 140. The recognition section 130 recognizes an object around the movable body in a first recognition mode in which recognition is performed by using a second image obtained by reducing the amount of data in the first image, and assumes that another object having a larger risk area than the recognized object has been recognized, when accuracy of the recognized object is less than a threshold value.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a mobile object control device, a mobile object control method, and a program. [Background technology]

[0002] Conventionally, in a vehicle control device that plans a target trajectory of a vehicle based on recognition information from an external sensor such as a camera, a technology has been known that, when an object around the vehicle is recognized by the external sensor, plans a target trajectory that widens the actual detection range of the external sensor (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-100827 Summary of the Invention [Problem to be solved by the invention]

[0004] Generally, the higher the camera resolution, the better the recognition performance. However, the larger the data volume, the more burden there is on image processing, resulting in longer processing times. Conversely, lowering the resolution to shorten processing time can lead to the possibility of misrecognition of objects. However, conventional technology does not take into account how to process objects that are likely to be misrecognized. As a result, it has sometimes been impossible to generate a target trajectory that corresponds to the surrounding conditions of the moving object, making it impossible to perform appropriate driving control.

[0005] The aspects of the present invention have been made in consideration of these circumstances, and one of their objectives is to provide a mobile body control device, a mobile body control method, and a program that can perform more appropriate mobile body control. [Means for solving the problem]

[0006] A mobile object control device, a mobile object control method, and a program according to the present invention employ the following configuration. (1): A mobile body control device according to one embodiment of the present invention includes a recognition unit that recognizes the surrounding conditions of the mobile body based on a first image captured by an imaging unit, a behavior plan generation unit that generates a future behavior plan for the mobile body based on the recognition result by the recognition unit, and a driving control unit that controls at least one of the steering and speed of the mobile body based on the behavior plan generated by the behavior plan generation unit, wherein the recognition unit recognizes objects around the mobile body in a first recognition mode that performs recognition using a second image that has a reduced amount of data from the first image, and when the accuracy of the recognized object is less than a threshold, it is determined that another object with a larger risk area than the recognized object has been recognized.

[0007] (2): In the above aspect (1), when the accuracy of the object recognized by the first recognition mode is less than a threshold, the recognition unit extracts a first partial image area including the object from the first image, and recognizes the object by a second recognition mode in which recognition is performed on the extracted first partial image area using a third image having a larger amount of data than the second image, and when the object recognized by the first recognition mode and the object recognized by the second recognition mode are different, the behavior plan generation unit generates the behavior plan based on information about the object recognized by the second recognition mode.

[0008] (3) In the above aspect (1) or (2), the data amount includes at least one of the resolution and the frame rate.

[0009] (4): In any one of the above aspects (1) to (3), the recognition result by the recognition unit includes at least the position, size, and type of the object.

[0010] (5): In any one of the above aspects (1) to (4), when the objects recognized in the first recognition mode include other moving bodies and a dividing line dividing the area in which the moving body moves, the recognition unit extracts a second partial image area including the dividing line from the first image, recognizes the dividing line in a second recognition mode in which recognition is performed on the extracted second partial image area using a third image having a larger amount of data than the second image, and recognizes the position of the other moving body based on the positional relationship between the recognized dividing line and the other moving body.

[0011] (6): In the above aspect (5), the recognition unit recognizes the position of the other moving body based on the positional relationship between a virtual line extending from a dividing line that divides the area in which the moving body moves and a dividing line that exists within a predetermined distance from the other moving body.

[0012] (7) In the above aspect (2), the other object having a large risk area is an object that is predicted to move a greater amount in a predetermined time than the object recognized from the second image.

[0013] (8): A mobile body control method according to one embodiment of the present invention is a mobile body control method in which a computer recognizes the surrounding situation of the mobile body based on a first image captured by an imaging unit, generates a future action plan for the mobile body based on the recognition result, controls at least one of the steering and speed of the mobile body based on the generated action plan, recognizes objects around the mobile body in a first recognition mode in which recognition is performed using a second image that has a reduced amount of data from the first image, and if the accuracy of the recognized object is less than a threshold, considers that another object with a larger risk area than the recognized object has been recognized.

[0014] (9): A program according to one embodiment of the present invention causes a computer to recognize the surrounding situation of a moving body based on a first image captured by an imaging unit, generate a future action plan for the moving body based on the recognition results, control at least one of the steering and speed of the moving body based on the generated action plan, recognize objects around the moving body in a first recognition mode that performs recognition using a second image that has a reduced amount of data from the first image, and, if the accuracy of the recognized object is less than a threshold, determine that another object with a larger risk area than the recognized object has been recognized. [Effects of the Invention]

[0015] According to the above aspects (1) to (9), more appropriate moving body control can be performed. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a configuration diagram of a vehicle system 1 including a mobile object control device according to a first embodiment. [Figure 2] FIG. 2 is a functional configuration diagram of a first control unit 120 and a second control unit 160 according to the first embodiment. [Figure 3] 10 is a diagram for explaining generation of an action plan based on a recognition result by a recognition unit 130. FIG. [Figure 4] FIG. 10 is a diagram for explaining a risk area for each object. [Figure 5] 10 is a diagram for explaining a risk area when the recognition accuracy of the object OB1 is less than a threshold value. FIG. [Figure 6] 3 is a flowchart showing an example of the flow of processing executed by the automatic driving control device 100 in the first embodiment. [Figure 7] 10 is a flowchart showing an example of the flow of processing executed by the automatic driving control device 100 in the second embodiment. [Figure 8] FIG. 10 is a diagram for explaining recognition of a partial image area including a lane in a second recognition mode. [Figure 9]FIG. 10 is a diagram for explaining recognition of the positional relationship of lane markings. [Figure 10] 10 is a flowchart showing an example of the flow of processing executed by the automatic driving control device 100 in the third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0017] Hereinafter, with reference to the drawings, embodiments of a mobile body control device, a mobile body control method, and a program according to the present invention will be described. In the following description, an example in which the mobile body control device is mounted on a mobile body will be described. A mobile body is a structure that can move using its own drive mechanism, such as a vehicle, micromobility, an autonomous mobile robot, a ship, or a drone. Control of a mobile body includes, for example, driving control that at least temporarily controls one or both of the steering and speed of the mobile body to move the mobile body autonomously. Furthermore, control of a mobile body may include mainly manual driving, providing advice on driving operations via voice or display, or performing some degree of interference control, and may also include controlling the activation of a protection device to protect the occupants of the mobile body. In the following description, it is assumed that the mobile body is a vehicle that moves on the ground, and the configuration and functions for moving the vehicle on the ground will be described. Vehicle driving control may include various driving controls, such as automatic driving controls such as LKAS (Lane Keeping Assistance System), ALC (Auto Lane Changing), and ACC (Adaptive Cruise Control), as well as driving assistance controls such as contact avoidance control, emergency stop control, and lane departure avoidance control during manual driving.

[0018] (First embodiment) [Overall configuration] 1 is a configuration diagram of a vehicle system 1 including a mobile object control device according to a first embodiment. The vehicle (hereinafter referred to as the subject vehicle M) on which the vehicle system 1 is mounted is, for example, a two-wheeled, three-wheeled, or four-wheeled vehicle, and its drive source is an internal combustion engine such as a diesel engine or a gasoline engine, an electric motor, or a combination of these. The electric motor operates using power generated by a generator connected to the internal combustion engine, or discharged power from a secondary battery or a fuel cell.

[0019] The vehicle system 1 includes, for example, a camera 10, a radar device 12, a LIDAR (Light Detection and Ranging) 14, a communication device 20, an HMI (Human Machine Interface) 30, vehicle sensors 40, a navigation device 50, an MPU (Map Positioning Unit) 60, a driving operator 80, an automatic driving control device 100, a driving force output device 200, a braking device 210, and a steering device 220. These devices and equipment are connected to each other via multiplex communication lines such as a CAN (Controller Area Network) communication line, serial communication lines, a wireless communication network, etc. Note that the configuration shown in FIG. 1 is merely an example, and some of the configuration may be omitted, or other configurations may be added. The camera 10 is an example of an "imaging unit." The automatic driving control device 100 is an example of a "mobile body control device."

[0020] The camera 10 is, for example, a digital camera using a solid-state imaging element such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor). The camera 10 is attached to any location of the host vehicle M on which the vehicle system 1 is installed. When capturing an image of the front, the camera 10 is attached to the top of the front windshield, the back of the rearview mirror, or the like. The camera 10, for example, periodically captures images of the surroundings of the host vehicle M. The camera 10 may be a stereo camera. The camera 10 may also be a camera capable of capturing images of the surroundings of the host vehicle M at a wide angle (for example, 360 degrees). The camera 10 may also be realized by combining multiple cameras. When the camera 10 has multiple cameras, each camera may capture images at a different resolution or frame rate (FPS; Frames Per Second).

[0021] The radar device 12 emits radio waves such as millimeter waves around the vehicle M and detects radio waves reflected by an object (reflected waves) to detect at least the position (distance and direction) of the object. The radar device 12 is attached to any location on the vehicle M. The radar device 12 may detect the position and speed of an object using an FM-CW (Frequency Modulated Continuous Wave) method.

[0022] The LIDAR 14 irradiates the surroundings of the vehicle M with light (or electromagnetic waves with wavelengths similar to light) and measures the scattered light. The LIDAR 14 detects the distance to the target based on the time between light emission and light reception. The irradiated light is, for example, pulsed laser light. The LIDAR 14 is attached to any location on the vehicle M.

[0023] The communication device 20 communicates with other vehicles in the vicinity of the vehicle M, for example, using a cellular network, a Wi-Fi network, Bluetooth (registered trademark), DSRC (Dedicated Short Range Communication), etc., or communicates with various server devices via a wireless base station.

[0024] The HMI 30 outputs various information to the occupants of the vehicle M under the control of the HMI control unit 170. The HMI 30 may also function as a reception unit that receives input operations by the occupants. The HMI 30 includes, for example, a display device, a speaker, a microphone, a buzzer, keys, an indicator lamp, etc. The display device is, for example, an LCD (Liquid Crystal Display) or an organic EL (Electro Luminescence) display device, etc.

[0025] The vehicle sensors 40 include a vehicle speed sensor that detects the speed of the host vehicle M, an acceleration sensor that detects acceleration, a yaw rate sensor that detects the angular velocity around a vertical axis, and a direction sensor that detects the orientation of the host vehicle M. The vehicle sensors 40 may also include a position sensor that acquires the position of the vehicle M. The position sensor is, for example, a sensor that acquires position information (longitude and latitude information) from a GPS (Global Positioning System) device. The position sensor may also be a sensor that acquires position information using a GNSS (Global Navigation Satellite System) receiver 51 of the navigation device 50.

[0026] The navigation device 50 includes, for example, a GNSS receiver 51, a navigation HMI 52, and a route determination unit 53. The navigation device 50 stores first map information 54 in a storage device such as a hard disk drive (HDD) or flash memory. The GNSS receiver 51 identifies the position of the vehicle M based on signals received from GNSS satellites. The position of the vehicle M may be identified or supplemented by an inertial navigation system (INS) that uses the output of the vehicle sensors 40. The navigation HMI 52 includes a display device, a speaker, a touch panel, keys, etc. The navigation HMI 52 may share some or all of the components with the HMI 30 described above. The route determination unit 53 determines, for example, a route (hereinafter, a route on a map) from the position of the vehicle M identified by the GNSS receiver 51 (or any input position) to a destination input by the occupant using the navigation HMI 52, with reference to the first map information 54. The first map information 54 is information that represents road shapes using, for example, links indicating roads and nodes connected by the links. The first map information 54 may also include information such as road curvature and POI (Point of Interest) information. The route on the map is output to the MPU 60. The navigation device 50 may provide route guidance using the navigation HMI 52 based on the route on the map. The navigation device 50 may be realized, for example, by the functions of a terminal device such as a smartphone or tablet device carried by the occupant. The navigation device 50 may transmit the current position and destination to a navigation server via the communication device 20 and obtain a route equivalent to the route on the map from the navigation server.

[0027] The MPU 60 includes, for example, a recommended lane determination unit 61, and stores second map information 62 in a storage device such as an HDD or flash memory. The recommended lane determination unit 61 divides the route on the map provided by the navigation device 50 into a plurality of blocks (for example, by dividing it into 100 m intervals in the vehicle travel direction), and determines a recommended lane for each block by referring to the second map information 62. The recommended lane determination unit 61 determines, for example, which lane from the left the vehicle should travel in. When there is a branch point on the route on the map, the recommended lane determination unit 61 determines a recommended lane so that the vehicle M can travel on a reasonable route to the branch point.

[0028] The second map information 62 is map information with higher accuracy than the first map information 54. The second map information 62 includes, for example, information on the center of lanes or information on lane boundaries. The second map information 62 may also include road information, traffic regulation information, address information (address and postal code), facility information, telephone number information, etc. The second map information 62 may be updated as needed by the communication device 20 communicating with other devices.

[0029] The driving operators 80 include, for example, a steering wheel, an accelerator pedal, a brake pedal, a shift lever, and other operators. The driving operators 80 are fitted with sensors that detect the amount of operation or the presence or absence of operation, and the detection results are output to the automatic driving control device 100 or some or all of the driving force output device 200, the braking device 210, and the steering device 220.

[0030] Next, prior to describing the automatic driving control device 100, the driving force output device 200, the brake device 210, and the steering device 220 will be described. The driving force output device 200 outputs a driving force (torque) to the drive wheels for driving the host vehicle M. The driving force output device 200 includes, for example, a combination of an internal combustion engine, an electric motor, a transmission, etc., and an ECU (Electronic Control Unit) that controls these. The ECU controls the above components according to information input from the automatic driving control device 100 (specifically, the second control unit 160 described later) or information input from the driving operator 80.

[0031] Braking device 210 may include, for example, a brake caliper, a cylinder that transmits hydraulic pressure to the brake caliper, an electric motor that generates hydraulic pressure in the cylinder, and a brake ECU. The brake ECU controls the electric motor according to information input from second control unit 160 or information input from driving operation device 80, so that a brake torque corresponding to the braking operation is output to each wheel. Braking device 210 may include a backup mechanism that transmits hydraulic pressure generated by operation of a brake pedal included in driving operation device 80 to the cylinder via a master cylinder. Note that braking device 210 is not limited to the configuration described above, and may also be an electronically controlled hydraulic brake device that controls an actuator according to information input from second control unit 160 to transmit hydraulic pressure from a master cylinder to the cylinder.

[0032] The steering device 220 includes, for example, a steering ECU and an electric motor. The electric motor applies a force to a rack and pinion mechanism to change the direction of the steered wheels, for example. The steering ECU drives the electric motor to change the direction of the steered wheels in accordance with information input from the second control unit 160 or information input from the steering wheel 82 of the driving operator 80.

[0033] Next, the automatic driving control device 100 will be described. The automatic driving control device 100 includes, for example, a first control unit 120, a second control unit 160, an HMI control unit 170, and a storage unit 180. The first control unit 120 and the second control unit 160 are each realized by, for example, a hardware processor such as a CPU (Central Processing Unit) executing a program (software). Furthermore, some or all of these components may be realized by hardware (including circuitry) such as an LSI (Large Scale Integration), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a GPU (Graphics Processing Unit), or may be realized by a combination of software and hardware. The program may be stored in advance in a storage device (a storage device with a non-transitory storage medium) such as the HDD or flash memory of the automatic driving control device 100, or may be stored in a removable storage medium such as a DVD or CD-ROM, and installed in the HDD or flash memory of the automatic driving control device 100 by inserting the storage medium (non-transitory storage medium) into a drive device. The second control unit 160 is an example of a "driving control unit."

[0034] The storage unit 180 may be realized by the various storage devices described above, or a solid-state drive (SSD), an electrically erasable programmable read-only memory (EEPROM), a read-only memory (ROM), or a random-access memory (RAM). The storage unit 180 stores, for example, programs and various other information. The map information (first map information 54, second map information 62) described above may be stored in the storage unit 180.

[0035] 2 is a functional configuration diagram of the first control unit 120 and the second control unit 160 according to the first embodiment. The first control unit 120 includes, for example, a recognition unit 130 and an action plan generation unit 140. The first control unit 120, for example, implements a function based on AI (Artificial Intelligence) and a function based on a pre-given model in parallel. For example, the function of "recognizing an intersection" may be implemented by executing in parallel recognition of the intersection using deep learning or the like and recognition based on pre-given conditions (such as the presence of traffic lights and road signs that can be pattern-matched), and by assigning scores to both and comprehensively evaluating them.

[0036] The recognition unit 130 recognizes the surrounding conditions of the host vehicle M. The recognition unit 130 includes, for example, a first acquisition unit 132, a recognition mode setting unit 134, an object recognition unit 136, and a risk area derivation unit 138.

[0037] The first acquisition unit 132 acquires, for example, data of the detection results from some or all of the camera 10, the radar device 12, and the LIDAR 14. The first acquisition unit 132 may also perform processing to convert the coordinate system (for example, a camera coordinate system with a forward viewing angle) of the image captured by the camera 10 (hereinafter referred to as a camera image) into a coordinate system (vehicle coordinate system, bird's-eye view coordinate system) based on the position of the host vehicle M when viewed from above. The camera image is an example of a "first image," and includes not only still images but also moving images.

[0038] The recognition mode setting unit 134 sets a recognition mode when the object recognition unit 136 performs object recognition on the data acquired by the first acquisition unit 132. The recognition modes include, for example, a first recognition mode and a second recognition mode in which a larger amount of data is used for object recognition than in the first recognition mode, but may also include other recognition modes with different amounts of data. For example, if the data acquired by the first acquisition unit 132 is image data captured by the camera 10 (hereinafter referred to as a camera image), the amount of data includes at least one of the image resolution (number of pixels) and frame rate. The amount of data may also include whether or not there is a detection result by the radar device 12 or the LIDAR 14, and the amount of data for the detection result (for example, a detection range or a detection period).

[0039] The recognition mode setting unit 134 may set a predetermined recognition mode from among a plurality of recognition modes, or may set the recognition mode based on the recognition result by the object recognition unit 136. For example, the recognition mode setting unit 134 sets the first recognition mode when a predetermined object is not recognized by the object recognition unit 136 or when the accuracy of the recognized object is equal to or greater than a threshold. The predetermined object is an object that affects the driving control of the host vehicle M, and includes, for example, other vehicles (including two-wheeled vehicles), traffic participants such as pedestrians and bicycles, and road structures such as lane lines, curbs, and median strips that demarcate the area in which the host vehicle M travels (the area in which the moving object moves). The accuracy is an index value indicating the likelihood of the object. The accuracy may also be referred to as reliability or likelihood. Furthermore, the recognition mode setting unit 134 sets the second recognition mode, for example, when the accuracy of the recognized object is less than a threshold.

[0040] Furthermore, the recognition mode setting unit 134 may generate data according to the set recognition mode from the data acquired by the first acquisition unit 132. For example, when the first recognition mode is set, the recognition mode setting unit 134 generates a second image having a lower resolution (smaller numerical value of resolution) than the resolution of the camera image (reference resolution). When the second recognition mode is set, the recognition mode setting unit 134 generates a third image having a higher resolution than the second image. The resolution of the third image is equal to or lower than the resolution of the camera image. Therefore, the recognition mode setting unit 134 may use the camera image as the third image. Furthermore, instead of (or in addition to) adjusting the resolution, the recognition mode setting unit 134 may generate the second image or the third image by adjusting the frame rate of the camera image. When adjusting the frame rate, the recognition mode setting unit 134 reduces the frame rate, for example, by thinning out image frames at a predetermined interval from the frame rate of the camera image (reference frame rate).

[0041] Furthermore, when images are captured by the camera 10 at different resolutions or frame rates, the recognition mode setting unit 134 may select a camera image that corresponds to the set recognition mode.

[0042] The object recognition unit 136 performs sensor fusion processing on the data of the detection results obtained by some or all of the camera 10, radar device 12, and LIDAR 14, acquired by the first acquisition unit 132, using the recognition mode set by the recognition mode setting unit 134, to recognize objects that exist within a predetermined distance from the vehicle M. The objects include, for example, traffic participants such as other vehicles, pedestrians, and bicycles, as well as road structures such as dividing lines, curbs, medians, road signs, traffic signals, railroad crossings, and crosswalks.

[0043] For example, the object recognition unit 136 performs a predetermined image analysis process using an image with a data amount associated with the recognition mode set by the recognition mode setting unit 134, and recognizes an object contained in the image through a matching process by referring to a predefined pattern matching model or the like based on the image information of the analysis result. The image analysis process includes, for example, an edge extraction process in which edge points with a large difference in brightness from adjacent pixels are extracted and the extracted edge points are connected to acquire the contour of an object, and a process in which feature quantities are extracted from the color, shape, size, etc. within the contour. The model is, for example, a trained model such as a deep neural network (DNN) that receives the analysis result or data acquired by the first acquisition unit 132 as input and is trained to output the type (category) of an object contained in the image, but is not limited thereto. The model may be stored in the storage unit 180, for example, or may be acquired from an external device via the communication device 20. The model may also be updated as appropriate through feedback control of the recognition result, update data from an external device, or the like.

[0044] Furthermore, the object recognition unit 136 may perform a matching process using the analysis result and the model, and output a probability that the object in the image is an object (type of object) defined in the model. The probability may be defined in the model, or may be set based on the degree of match (similarity) in the matching process, for example.

[0045] Furthermore, when an object is recognized, the object recognition unit 136 recognizes the object's position, speed (absolute speed, relative speed), acceleration (absolute acceleration, relative acceleration), direction of travel (other direction of movement), etc. The object's position is recognized as a position on an absolute coordinate system with a representative point of the host vehicle M (such as the center of gravity or the center of the drive shaft) as the origin, and is used for control. The object's position may be represented by a representative point such as the center of gravity or a corner of the object, or may be represented by an area. For example, when the object is another moving body such as another vehicle, the "state" of the object may include the acceleration or jerk of the other moving body, or the "behavioral state" (for example, whether or not the other moving body is changing lanes or is about to change lanes).

[0046] Furthermore, for example, when a dividing line is recognized, the object recognition unit 136 may recognize the lane (driving lane) in which the host vehicle M is traveling. For example, the object recognition unit 136 recognizes the driving lane by comparing the pattern of road dividing lines (e.g., an arrangement of solid lines and dashed lines) obtained from the second map information 62 with the pattern of dividing lines (road dividing lines) around the host vehicle M recognized from the image captured by the camera 10. Note that the object recognition unit 136 may recognize road boundaries (road boundaries) including curbs, medians, etc., in addition to dividing lines. In this recognition, the position of the host vehicle M obtained from the navigation device 50 and the processing results by the INS may also be taken into consideration.

[0047] Furthermore, when recognizing the driving lane of the host vehicle M, the object recognition unit 136 may recognize the position and orientation of the host vehicle M with respect to the driving lane. In this case, the object recognition unit 136 recognizes the deviation of the reference point of the host vehicle M from the center of the lane and the angle it forms with a line connecting the centers of the lanes in the traveling direction of the host vehicle M as the relative position and orientation of the host vehicle M with respect to the driving lane. Alternatively, the object recognition unit 136 may recognize the position of the reference point of the host vehicle M with respect to either side edge (road boundary) of the driving lane as the relative position of the host vehicle M with respect to the driving lane.

[0048] The risk area derivation unit 138 derives an area of risk (hereinafter referred to as a risk area) potentially distributed or potentially present around an object recognized by the object recognition unit 136. The risk is, for example, the risk that the object poses to the host vehicle M. More specifically, the risk is, for example, the risk that a sudden movement of the object will force the host vehicle M to suddenly brake or steer. The risk may also be the risk that the host vehicle M poses to the object. Hereinafter, the level of such risk will be treated as a quantitative index value, and this index value will be referred to as a "risk potential" in the following description. An area where the risk potential is equal to or greater than a threshold is derived as a risk area. The risk area derivation unit 138 may also adjust the risk area based on the accuracy of the object recognized by the object recognition unit 136. The object recognition unit 136 outputs the recognition result to the action plan generation unit 140.

[0049] The behavior plan generation unit 140 generates a future behavior plan for the host vehicle M based on the recognition result by the recognition unit 130. For example, the behavior plan generation unit 140 generates a target trajectory along which the host vehicle M will automatically travel in the future (without relying on the driver's operation) so that the host vehicle M will travel, in principle, along the recommended lane determined by the recommended lane determination unit 61 and can respond to the surrounding conditions of the host vehicle M. The target trajectory includes, for example, a speed element. For example, the target trajectory is expressed as a sequential arrangement of points (trajectory points) to be reached by the host vehicle M. The trajectory points are points to be reached by the host vehicle M at predetermined travel distances (e.g., on the order of several meters) along a road. Separately, target speeds and target accelerations are generated as part of the target trajectory for each predetermined sampling time (e.g., on the order of a few tenths of a second). Alternatively, the trajectory points may be positions to be reached by the host vehicle M at each sampling time for each predetermined sampling time. In this case, information on the target speed and target acceleration is expressed as the interval between trajectory points.

[0050] The behavior plan generation unit 140 may set an autonomous driving event when generating the target trajectory. The autonomous driving events include, for example, a constant speed driving event, a low-speed following driving event, a lane change event, a branching event, a merging event, a contact avoidance event, an emergency stop event, and a takeover event. The behavior plan generation unit 140 generates a target trajectory according to the activated event. Furthermore, when performing driving control of the vehicle M or a predetermined event, the behavior plan generation unit 140 may suggest (recommend) the driving control or the execution of the event to the occupant, and generate a corresponding target trajectory if the suggestion is accepted.

[0051] The second control unit 160 controls the traveling driving force output device 200, the braking device 210, and the steering device 220 so that the host vehicle M passes through the target trajectory generated by the action plan generation unit 140 at the scheduled time.

[0052] The second control unit 160 includes, for example, a second acquisition unit 162, a speed control unit 164, and a steering control unit 166. The second acquisition unit 162 acquires information on the target trajectory (trajectory points) generated by the action plan generation unit 140 and stores it in a memory (not shown). The speed control unit 164 controls the driving force output device 200 or the brake device 210 based on a speed element associated with the target trajectory stored in the memory. The steering control unit 166 controls the steering device 220 according to the curvature of the target trajectory stored in the memory. The processing of the speed control unit 164 and the steering control unit 166 is realized by, for example, a combination of feedforward control and feedback control. As an example, the steering control unit 166 executes a combination of feedforward control according to the curvature of the road ahead of the host vehicle M and feedback control based on the deviation from the target trajectory.

[0053] The HMI control unit 170 notifies the occupant (driver) of the host vehicle M of predetermined information via the HMI 30. The predetermined information includes, for example, information related to driving control of the host vehicle M or a predetermined event. For example, when performing contact avoidance control, emergency stop control, or the like, the HMI control unit 170 causes the HMI 30 to output warning information or the like. The HMI control unit 170 may generate an image including the above-mentioned predetermined information and display the generated image on the display device of the HMI 30, or may generate sound indicating the predetermined information and output the generated sound from a speaker of the HMI 30. Furthermore, the HMI control unit 170 may output the information received by the HMI 30 to, for example, the communication device 20, the navigation device 50, the first control unit 120, or the like.

[0054] [Generating action plans based on recognition results] Next, a specific example of generating an action plan based on the recognition result by the recognition unit 130 will be described. Note that, although the following description mainly focuses on object recognition based on camera images, object recognition based on the detection results of the radar device 12 and the LIDAR 14 is also performed.

[0055] Fig. 3 is a diagram for explaining generation of an action plan based on the recognition result by the recognition unit 130. The example of Fig. 3 shows two lanes L1 and L2 that can be traveled in the same direction (X-axis direction in the figure). Lane L1 is divided by marking lines LM1 and LM2, and lane L2 is divided by marking lines LM2 and LM3. The example of Fig. 3 also shows a host vehicle M traveling at a speed VM in the direction of extension of lane L1 (X-axis direction in the figure), and an object OB1 moving ahead of the host vehicle M on lane L1 at a speed V1. It is assumed that object OB1 is a pedestrian.

[0056] For example, when an instruction to start autonomous driving control is received from an occupant of the host vehicle M via the HMI 30, the recognition mode setting unit 134 sets the first recognition mode as the initial recognition mode. The object recognition unit 136 recognizes the surrounding situation of the host vehicle M using the second image associated with the first recognition mode, and recognizes the position, size, type, etc. of the object OB1. The object recognition unit 136 also acquires the accuracy of the recognized object OB1, and determines the speed V1 and moving direction of the object OB1. The object recognition unit 136 also recognizes the positions, sizes, and types of the lane markings LM1 to LM3.

[0057] The risk area derivation unit 138 derives a risk area of the object OB1 that may come into contact with the host vehicle M based on the recognition result by the object recognition unit 136.

[0058] Fig. 4 is a diagram for explaining risk areas for each object. In the example of Fig. 4, a pedestrian, a two-wheeled vehicle (motorcycle), and a four-wheeled vehicle are shown as examples of objects. Hereinafter, for convenience of explanation, they may be referred to as pedestrian OBa, two-wheeled vehicle OBa, and four-wheeled vehicle OBC.

[0059] The risk area derivation unit 138 derives a risk area such that the closer the vehicle is to the object, the higher the risk potential, and the farther away the object, the lower the risk potential. The risk area derivation unit 138 may also adjust the risk potential so that the closer the distance between the vehicle M and the object, the higher the risk potential (in other words, the farther the distance, the lower the risk potential).

[0060] Furthermore, the risk area derivation unit 138 may adjust the risk area according to the type of object. For example, even if the position, movement direction, and speed (speeds Va and Vb shown in FIG. 4) at the time of recognition are the same between a pedestrian OBa and a two-wheeled vehicle OBb, the amount of movement thereafter over a predetermined time period may be significantly different. Furthermore, the size (dimension) of the object itself differs between a pedestrian OBa and a two-wheeled vehicle OBb and a four-wheeled vehicle OBc. Therefore, the risk area derivation unit 138 derives the risk area according to the type of object. Specifically, the risk area derivation unit 138 adjusts the risk area so that the larger the predicted value of the movement amount over a predetermined time period (hereinafter, predicted movement amount) is, the larger the risk area becomes. Instead of (or in addition to) the above adjustment, the risk area derivation unit 138 adjusts the risk area so that the larger the size of the object is, the larger the risk area becomes.

[0061] In the example of FIG. 4, two-wheeled vehicle OBb can move at a higher speed than pedestrian OBa and has a larger predicted movement amount. Therefore, the risk area RAb of two-wheeled vehicle OBb is larger than the risk area RAa of pedestrian OBa. Furthermore, even though four-wheeled vehicle OBc has the same predicted movement amount as two-wheeled vehicle OBb, it is larger in size than two-wheeled vehicle OB2. Therefore, the risk area RAc of four-wheeled vehicle OBc is larger than the risk area RAb. In this way, by deriving the risk area according to the type of object, it is possible to derive the risk area more quickly.

[0062] The risk area derivation unit 138 may also adjust the risk area depending on the speed and direction of movement of the object. For example, the risk area derivation unit 138 may derive a risk area based on the risk potential by increasing the risk potential as the absolute speed or absolute acceleration of the object increases. The risk potential may also be determined appropriately depending on the relative speed or relative acceleration between the host vehicle M and the object, TTC (Time to Collision), predicted collision position, etc., instead of (or in addition to) the absolute speed or absolute acceleration of the object. The risk potential may also be adjusted depending on surrounding conditions such as road shape, congestion level, weather, time of day, etc.

[0063] Here, the object recognition unit 136 determines whether the accuracy of the recognized object is equal to or greater than a threshold, and if it is determined that the accuracy is equal to or greater than the threshold, causes the risk area derivation unit 138 to derive a risk area for the recognized object. Conversely, if it is determined that the accuracy is not equal to or greater than the threshold (if it is determined that the accuracy is less than the threshold), the object recognition unit 136 determines that another object with a larger risk area than the recognized object has been recognized, and causes the risk area derivation unit 138 to derive a risk area. An example of an another object with a larger risk area is an object with a larger predicted movement amount than the recognized object (for example, an object that can move faster than the recognized object). Furthermore, the example of an another object is an object of a different type.

[0064] For example, if the type of object whose accuracy is less than the threshold is a pedestrian, the object recognition unit 136 may determine that the object is a two-wheeled vehicle, and if the type is a two-wheeled vehicle, the object recognition unit 136 may determine that the object is a four-wheeled vehicle, and so on. The object recognition unit 136 may also select other objects based on the accuracy, the speed of the object, and so on. In this case, the object recognition unit 136 selects an object whose risk area becomes larger as the accuracy decreases and / or the speed increases. For example, if the accuracy when recognizing the object as a pedestrian is less than a first predetermined value, the object recognition unit 136 determines that the object is a two-wheeled vehicle, and if the accuracy is less than a second predetermined value that is smaller than the first predetermined value, the object recognition unit 136 determines that the object is a four-wheeled vehicle.

[0065] In the example of Fig. 3, the accuracy of recognizing the object OB1 as a pedestrian is equal to or greater than a threshold, so a risk area RAa associated with the pedestrian is set based on the position of the object OB1. Fig. 5 is a diagram for explaining a risk area when the recognition accuracy of the object OB1 is less than a threshold. In the example of Fig. 5, the accuracy of recognizing the object OB1 as a pedestrian is less than the threshold, so the object OB1 is regarded as a two-wheeled vehicle, and a risk area RAb associated with the two-wheeled vehicle is set based on the position of the object OB1.

[0066] The behavior plan generation unit 140 generates a target trajectory for the host vehicle M to travel at a position a predetermined distance away from the risk area so that the future trajectory of the host vehicle M does not come into contact with the risk area. Therefore, in the first embodiment, when the recognition accuracy of the object OB1 is equal to or greater than a threshold, a target trajectory K1 based on the risk area RAa is generated as shown in FIG. 3, and when the recognition accuracy is less than the threshold, a target trajectory K2 based on the risk area RAb is generated as shown in FIG. 5. As a result, when the presence of an object is recognized but the type of object cannot be fully identified, a larger risk area can be set and a safer target trajectory can be generated. Furthermore, generating the above-described behavior plan enables high-speed object recognition using images with a small amount of data, thereby realizing more appropriate driving control of the host vehicle M according to the surrounding conditions.

[0067] [Processing flow of the first embodiment] Next, the flow of processing executed by the autonomous driving control device 100 of the first embodiment will be described. Note that the processing of the flowchart below will be described mainly focusing on the point of generating an action plan based on a risk area, which is based mainly on the object recognition results, among the processing executed by the autonomous driving control device 100. The processing shown below may be executed repeatedly at a predetermined timing.

[0068] FIG. 6 is a flowchart showing an example of the flow of processing executed by the autonomous driving control device 100 in the first embodiment. In the example of FIG. 6, the first acquisition unit 132 acquires a camera image (first image) or the like (step S100). Next, the object recognition unit 136 recognizes objects around the host vehicle M in a first recognition mode using a second image generated from the camera image (step S102). Next, when the object recognition unit 136 recognizes an object, it determines whether the accuracy of the recognized object is equal to or greater than a threshold (step S104). When it is determined that the accuracy of the recognized object is equal to or greater than the threshold, the object recognition unit 136 determines that the object has been recognized as it was recognized. When it is determined that the accuracy of the recognized object is less than the threshold, the object recognition unit 136 determines that a different object having a larger risk area than the recognized object has been recognized (step S108).

[0069] Next, the risk area derivation unit 138 derives a risk area for the object recognized by the object recognition unit 136 (step S110). Next, the behavior plan generation unit 140 generates a target trajectory based on the derived risk area, etc. (step S112). Next, the second control unit 160 executes driving control to control at least one of the steering and speed of the host vehicle M so that the host vehicle M travels along the generated target trajectory, and causes the host vehicle M to travel (step S114). This ends the processing of this flowchart.

[0070] According to the first embodiment described above, the automatic driving control device (an example of a mobile body control device) 100 includes a recognition unit 130 that recognizes the surrounding situation of the host vehicle (an example of a mobile body) M based on a first image captured by a camera (an example of an imaging unit) 10, a behavior plan generation unit 140 that generates a future behavior plan for the host vehicle M based on the recognition result by the recognition unit 130, and a driving control unit (second control unit 160) that controls at least one of the steering and speed of the host vehicle M based on the behavior plan generated by the behavior plan generation unit 140.The recognition unit 130 recognizes objects around the host vehicle M in a first recognition mode that performs recognition using a second image that has a reduced amount of data from the first image, and if the accuracy of the recognized object is less than a threshold, it determines that another object with a larger risk area than the recognized object has been recognized, thereby enabling more appropriate mobile body control to be performed.

[0071] Furthermore, according to the first embodiment, for example, by performing recognition in the first recognition mode using an image with low resolution, it is possible to obtain recognition results more quickly, and a target trajectory is generated based on a risk area according to the accuracy of the recognition result, so that even if there is an object that is likely to have been erroneously recognized, it is possible to execute more appropriate driving control according to the surrounding situation of the vehicle M. Furthermore, according to the first embodiment, since recognition results can be obtained in more real time, for example, when the vehicle M and an object are approaching each other, it is possible to speed up the initial response of driving control such as contact avoidance and emergency stop, thereby realizing safer vehicle control.

[0072] (Second embodiment) Next, a second embodiment will be described. The second embodiment differs from the first embodiment in that, when the accuracy of the recognition result is less than a threshold, a partial image area (an example of a first partial image area) including an object is extracted from the camera image, and object recognition is performed in a second recognition mode using an image of the extracted area. Therefore, the following description will mainly focus on the above-mentioned differences. Note that, since the second embodiment can be configured similarly to the first embodiment, the following description will be given using the configuration of the automatic driving control device 100 shown in the first embodiment. The same applies to a third embodiment described later.

[0073] [Processing flow of the second embodiment] Fig. 7 is a flowchart showing an example of the flow of processing executed by the automatic driving control device 100 in the second embodiment. The processing shown in Fig. 7 differs from that shown in Fig. 6 in that processing of steps S200 to S212 is added after steps S100 to S114. Therefore, the following description will mainly focus on the processing of steps S200 to S212.

[0074] In the example of FIG. 7, after processing in step S114, the object recognition unit 136 determines whether the accuracy of the recognized object is equal to or greater than a threshold (step S200). If it is determined that the accuracy is not equal to or greater than the threshold (is less than the threshold), the object recognition unit 136 performs a cropping process to extract a partial image area including the object from the camera image based on the position information of the recognized object (step S202). The partial image area is, for example, a bounding box that surrounds the outline (periphery) of the recognized object. Furthermore, the bounding box area of the partial image area may be adjusted depending on the speed and direction of movement of the object and its relative position from the host vehicle M, and may be adjusted so that the partial image area becomes larger as the accuracy decreases.

[0075] Next, the object recognition unit 136 generates a third image having a larger amount of data than the second image for the extracted partial image area, and performs object recognition in the second recognition mode using the generated third image (step S204). Note that in the process of step S04, the object recognition unit 136 may perform object recognition by increasing the amount of data from the radar device 12 or the LIDAR 14.

[0076] Next, the object recognition unit 136 determines whether the recognition results differ between the first recognition mode and the second recognition mode (step S206). The recognition results differ when, for example, at least one of the position, size, and type of the recognized object differs beyond an allowable range. In addition to the above, this may also include when the movement direction differs by more than a predetermined angle or when the movement amount differs by more than a predetermined amount. If it is determined that the recognition results differ between the first recognition mode and the second recognition mode, the risk area derivation unit 138 derives a risk area for the object recognized in the second recognition mode (step S208).

[0077] Next, the behavior plan generation unit 140 adjusts the existing target trajectory based on the risk area, etc. derived by the processing of step S208 (step S210). In the processing of step S210, for example, when the risk area recognized in the second recognition mode (hereinafter referred to as the second risk area) is larger than the risk area for the object recognized in the first recognition mode (hereinafter referred to as the first risk area), the behavior plan generation unit 140 generates the target trajectory based on the second risk area as a reference. Furthermore, the behavior plan generation unit 140 may control the adjustment amount of the already generated target trajectory based on, for example, the expansion ratio of the second risk area to the first risk area. Furthermore, when the second risk area is smaller than the first risk area, the behavior plan generation unit 140 may regenerate the target trajectory based on the second risk area as a reference, or may leave the current target trajectory as it is. When the second risk area is smaller, the possibility of the host vehicle M coming into contact with an object is low even if the current target trajectory is left as it is. Therefore, by maintaining the current target trajectory, it is possible to prevent the behavior of the host vehicle M from temporarily becoming erratic due to a change in the target trajectory while traveling.

[0078] After the process of step S210, the second control unit 160 executes driving control so that the host vehicle M travels along the generated target trajectory (step S212). This ends the process of this flowchart. Also, if it is determined in the process of step S200 that the accuracy of the recognized object is equal to or greater than a threshold, the process of this flowchart ends.

[0079] According to the second embodiment described above, in addition to achieving the same effects as the first embodiment, when the accuracy is below a threshold, the amount of data for the partial image area including the object is increased for object recognition, thereby improving the accuracy of the object's position, size, and type, and realizing more accurate object recognition. Therefore, the target trajectory can be appropriately corrected based on the recognition result, and more appropriate driving control can be performed.

[0080] (Third embodiment) Next, a third embodiment will be described. The third embodiment differs from the first embodiment in that, when a marking line that marks the lane (area) in which the host vehicle M is traveling is recognized together with an object, a partial image area (an example of a second partial image area) including the marking line is extracted from the camera image, and the position of the object is recognized in the second recognition mode using an image of the extracted area. Therefore, the following description will mainly focus on the above-mentioned differences.

[0081] Fig. 8 is a diagram for explaining recognition of a partial image area including a lane in the second recognition mode. In the example of Fig. 8, in two lanes that can be traveled in the same direction, a host vehicle M is traveling in a traveling direction (X-axis direction in the figure) on lane L1 at a speed VM, and an object OB2 is traveling ahead of the host vehicle M on lane L2 at a speed V2 in the traveling direction. The object OB2 is an example of an "other moving body," and specifically, is another vehicle (four-wheeled vehicle).

[0082] In the example of FIG. 8, the object recognition unit 136 recognizes objects around the host vehicle M in a first recognition mode using the second image, and recognizes object OB2 and lane markings LM1, LM2a, LM2b, and LM3. Here, the object recognition unit 136 determines whether the recognition accuracy of object OB2 is equal to or greater than a threshold. If it determines that the recognition accuracy is not equal to or greater than the threshold (is less than the threshold), it extracts a partial image area including the lane markings from the camera image by cropping or the like, and performs object recognition on the extracted partial image area in a second recognition mode using the third image. The partial image area may be, for example, a bounding box that surrounds the outline of the lane markings, and the size of the area may be adjusted depending on the surrounding conditions such as the road shape and the accuracy. The partial image area may also be an area that includes multiple lane markings.

[0083] For example, the object recognition unit 136 may extract a partial image area that includes all of the marking lines recognized in the first recognition mode, or may extract a partial image area that includes some of the marking lines. Some of the marking lines are, for example, marking lines whose length is less than a predetermined length. If the line is less than the predetermined length, it is highly likely that the marking line is broken. Therefore, by targeting the area including the line in the third image (high-precision image), the marking line can be recognized more accurately.

[0084] Furthermore, the "some of the marking lines" may be marking lines that are referenced to identify the position of object OB2 as seen from host vehicle M, and more specifically, may be marking lines that exist between host vehicle M and object OB2. For example, if object OB2 is present ahead of host vehicle M on the right side, a partial image area is extracted that includes the marking line on the right side of host vehicle M and the marking line on the left side of object OB2. The "some of the marking lines" may also include multiple marking lines. In the example of FIG. 8, a partial image area PA that includes parts of marking lines LM2a and LM2b is extracted as the partial image area.

[0085] The object recognition unit 136 recognizes the lane markings in the second recognition mode using the third image corresponding to the extracted partial image area PA, and recognizes the positional relationship between the lane markings near the vehicle M and the lane markings near the object OB2 based on the recognition results. Instead of (or in addition to) recognizing the lane markings near the vehicle M from the third image, the lane markings near the vehicle M may be obtained by referring to map information based on the position information of the vehicle M obtained from the vehicle sensor 40.

[0086] The object recognition unit 136 recognizes the positional relationship between the marking line LM2a (the marking line that marks the vehicle's lane) located on the right side closest to the vehicle M and the marking line LM2b (the marking line that marks the object's lane) located on the left side closest to the object OB2. FIG. 9 is a diagram for explaining the recognition of the positional relationship of marking lines. In the example of FIG. 9, the object recognition unit 136 sets a virtual line VL by extending the marking line LM2a located on the right side of the vehicle M along the extension direction of the lanes L1 and L2 (the X-axis direction in the figure), and determines whether the marking lines LM2a and LM2b are the same marking line based on the positional relationship between the set virtual line VL and the marking line LM2b on the left side of the object OB2. If the lateral distance D (in the road width direction, Y-axis direction) between the virtual line VL and the marking line LM2b is less than a threshold, the object recognition unit 136 recognizes that the marking lines LM2a and LM2b are the same marking line. Furthermore, if the distance D is equal to or greater than the threshold, the object recognition unit 136 recognizes that the marking lines LM2a and LM2b are not the same marking line (they are different marking lines). Note that if there are multiple marking lines on the right side of the vehicle M or on the left side of the object OB2, the above determination may be made for each marking line. After recognizing the marking lines in this manner, the object recognition unit 136 recognizes the position of the object OB2 based on the recognition results of the marking lines in the second recognition mode. This allows the position of the object OB2 to be recognized more accurately, making it possible to assign the risk area of the object OB2 to an appropriate position and generate a more appropriate target trajectory.

[0087] [Processing flow of the third embodiment] Fig. 10 is a flowchart showing an example of the flow of processing executed by the automatic driving control device 100 in the third embodiment. The processing shown in Fig. 10 differs from steps S100 to S114 shown in Fig. 6 in that steps S300 to S304 have been added between steps S108 and S110. Therefore, the following description will mainly focus on the processing of steps S300 to S304.

[0088] 10, after processing step S108, object recognition unit 136 determines whether the object in the recognition result includes a lane marking (step S300). If it is determined that a lane marking is included, object recognition unit 136 extracts a partial image area including the lane marking, recognizes the lane marking in the extracted partial image area in the second recognition mode using the third image (step S302), and recognizes the position of the object based on the recognized lane position (step S304). After processing step S106 or S304 is completed, processing from step S110 onwards is executed.

[0089] The third embodiment described above not only achieves the same effects as the first embodiment, but also recognizes the lane markings with high accuracy in the second recognition mode when the recognition results in the first recognition mode include lane markings, and can recognize the position of the object as seen from the vehicle M with high accuracy based on the positions of the recognized lane markings. Furthermore, the third embodiment recognizes only lane markings, which are somewhat more limited in shape, size, etc. than traffic participants such as pedestrians and vehicles. Therefore, lane markings can be recognized quickly even when recognition processing is performed in the second recognition mode, which has a higher resolution than the first recognition mode. Therefore, a more appropriate target trajectory can be generated depending on the surrounding conditions.

[0090] In the third embodiment, road structures other than the dividing lines (for example, curbs, median strips, etc.) may be recognized instead of (or in addition to) the dividing lines.

[0091] Each of the first to third embodiments described above may be combined with part or all of the other embodiments. For example, by combining the lane marking recognition of the third embodiment with the second embodiment, it is possible to set a more appropriate risk area at a more appropriate position.

[0092] The above-described embodiment can be expressed as follows. a storage medium storing computer readable instructions; a processor connected to the storage medium; The processor executes the computer-readable instructions to: Recognizing a surrounding situation of the moving object based on a first image captured by the imaging unit; generating a future action plan for the moving object based on the recognition result; Controlling at least one of the steering and the speed of the moving body based on the generated action plan; Recognizing objects around the moving object in a first recognition mode using a second image obtained by reducing the amount of data of the first image; If the accuracy of the recognized object is less than a threshold, it is determined that another object having a larger risk area than the recognized object has been recognized. Mobile control device.

[0093] The above describes the form for carrying out the present invention using an embodiment, but the present invention is not limited to such an embodiment, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention. [Explanation of symbols]

[0094] 1...vehicle system, 10...camera, 12...radar device, 14...LIDAR, 20...communication device, 30...HMI, 40...vehicle sensor, 50...navigation device, 60...MPU, 80...driving operator, 100...automatic driving control device, 120...first control unit, 130...recognition unit, 132...first acquisition unit, 134...recognition mode setting unit, 136...object recognition unit, 138...risk area derivation unit, 140...action plan generation unit, 160...second control unit, 162...second acquisition unit, 164...speed control unit, 166...steering control unit, 170...HMI control unit, 180...memory unit, 200...driving force output device, 210...brake device, 220...steering device, M...host vehicle

Claims

1. a recognition unit that recognizes a surrounding situation of the moving object based on a first image captured by the imaging unit; a behavior plan generation unit that generates a future behavior plan of the moving object based on the recognition result by the recognition unit; a driving control unit that controls at least one of steering and speed of the moving body based on the action plan generated by the action plan generation unit, the recognition unit recognizes objects around the moving body in a first recognition mode in which recognition is performed using a second image obtained by reducing the amount of data of the first image, and when the accuracy of the recognized object is less than a threshold, it determines that another object having a larger risk area than the recognized object has been recognized. Mobile control device.

2. the recognition unit extracts a first partial image area including the object from the first image when the accuracy of the object recognized in the first recognition mode is less than a threshold, and recognizes the object in a second recognition mode in which recognition is performed on the extracted first partial image area using a third image having a larger amount of data than the second image; the behavior plan generation unit generates the behavior plan based on information of the object recognized in the second recognition mode when the object recognized in the first recognition mode is different from the object recognized in the second recognition mode. The mobile object control device according to claim 1 .

3. The data amount includes at least one of a resolution and a frame rate. The mobile object control device according to claim 1 or 2.

4. The recognition result by the recognition unit includes at least the position, size, and type of the object. The mobile object control device according to any one of claims 1 to 3.

5. When the objects recognized in the first recognition mode include another moving body and a demarcation line that demarcates an area in which the moving body moves, the recognition unit extracts a second partial image area including the demarcation line from the first image, recognizes the demarcation line in a second recognition mode that performs recognition on the extracted second partial image area using a third image having a larger amount of data than the second image, and recognizes the position of the other moving body based on the positional relationship between the recognized demarcation line and the other moving body. The mobile object control device according to any one of claims 1 to 4.

6. the recognition unit recognizes the position of the other moving body based on a positional relationship between a virtual line obtained by extending a demarcation line that demarcates an area in which the moving body moves and a demarcation line that exists within a predetermined distance from the other moving body; The mobile object control device according to claim 5 .

7. The other object having a large risk area is an object that is predicted to move by a larger amount in a predetermined time period than the object recognized from the second image. The mobile object control device according to any one of claims 1 to 6.

8. The computer Recognizing a surrounding situation of the moving object based on a first image captured by the imaging unit; generating a future action plan for the moving object based on the recognition result; Controlling at least one of the steering and the speed of the moving body based on the generated action plan; Recognizing objects around the moving object in a first recognition mode using a second image obtained by reducing the amount of data of the first image; If the accuracy of the recognized object is less than a threshold, it is determined that another object having a larger risk area than the recognized object has been recognized. A mobile object control method.

9. On the computer, Recognizing a surrounding situation of the moving object based on a first image captured by the imaging unit; generating a future action plan for the moving object based on the recognition result; controlling at least one of the steering and the speed of the moving body based on the generated action plan; Recognizing objects around the moving object in a first recognition mode in which recognition is performed using a second image obtained by reducing the amount of data of the first image; If the accuracy of the recognized object is less than a threshold, another object having a risk area larger than that of the recognized object is determined to have been recognized. program.

Images