Mobile device control device, mobile device control method, and program

Abstract

The objective is to enable automated driving without deviating from the intended path in areas where it is not easy to recognize the path. [Solution] The mobile body control device of the embodiment comprises a recognition unit that recognizes objects present around the mobile body using at least a camera, and a driving control unit that performs automatic driving by automatically controlling at least one of the speed or steering of the mobile body based on the recognized objects, wherein the recognition unit recognizes two-dimensional or three-dimensional markers and the boundary of the path on which the mobile body travels, and the driving control unit switches the automatic driving performed between a first automatic driving, which is the automatic driving based on the markers, and a second automatic driving, which is the automatic driving based on the boundary.

Description

Technical Field

[0001] The present invention relates to a movement control device, a movement control method, and a program.

Background Art

[0002] There are two types of modes for automatic driving of vehicles. For example, the first automatic driving mode is a mode in which a road is recognized based on an image captured by an external camera, and the vehicle is automatically driven along the recognized road. The second automatic driving mode is a mode in which the vehicle is automatically driven based on pre-prepared map information. When the road cannot be recognized using the external camera, the automatic driving switches to the automatic driving using the map information (see, for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the conventional technology, the situation where the road cannot be recognized using the external camera may occur due to various influences such as weather, thin white lines (road markings), insufficient ambient brightness, or may also occur due to conditions such as the existence of multiple driving routes and road markings (road paint) indicating them.

[0005] In the latter case, the road changes greatly at intersections such as right and left turns, U-turn roads, and branch roads. Road markings may be drawn at places where the vehicle needs to move significantly in the right or left turn direction or the turning direction, and there may be cases where the road cannot be recognized using the external camera.

[0006] Conventional technology matches the vehicle's position with a map before the vehicle enters an area where the road is unrecognizable. After the vehicle enters this area, it controls its trajectory based on the pre-matched map's route information and the estimated vehicle movement. However, in areas with large movements in the direction of right or left turns or rotations, where the latter condition occurs, the estimated vehicle position may deviate significantly from the actual position due to the nonlinear characteristics (slip) of tire rolling, potentially causing the vehicle to deviate from the road.

[0007] One aspect of the present invention has been made in consideration of these circumstances, and aims to provide a mobile object control device, a mobile object control method, and a program that enable a mobile object, such as a vehicle, to automatically drive without deviating from the intended path, even when the mobile object has to move significantly in the direction of turning left or right or in the direction of rotation in an area where it is not easy to recognize the path. [Means for solving the problem]

[0008] The mobile device control device, mobile device control method, and program according to the present invention employ the following configuration.

[0009] A first aspect of the present invention is a mobile body control device comprising: a recognition unit that recognizes objects present around a mobile body using at least a camera; and a driving control unit that performs automatic driving by automatically controlling at least one of the speed or steering of the mobile body based on the recognized objects, wherein the recognition unit recognizes two-dimensional or three-dimensional markers and the boundary of the path on which the mobile body travels, and the driving control unit switches the automatic driving performed between a first automatic driving, which is automatic driving based on the markers, and a second automatic driving, which is automatic driving based on the boundary.

[0010] In the second embodiment, the operation control unit switches to the first automatic operation when the marker is recognized, and switches to the second automatic operation when the marker is not recognized, or when the marker is recognized again after switching to the first automatic operation.

[0011] In a third embodiment, in the first or second embodiment, the recognition unit recognizes a first marker and a second marker different from the first marker, and the operation control unit switches the automatic operation to be performed between a first automatic operation based on the first marker and a first automatic operation based on the second marker.

[0012] A fourth embodiment further comprises a generation unit that generates a target trajectory on which the mobile body should travel based on the markers in the third embodiment, wherein if the route to the destination is a first route, the generation unit generates a first target trajectory, which is the target trajectory when traveling the first route, based on the result of comparing the recognized first marker with the first marker included in the map information, and if the route to the destination is a second route, the generation unit generates a second target trajectory, which is the target trajectory when traveling the second route, based on the result of comparing the recognized second marker with the second marker included in the map information, and the driving control unit causes the mobile body to travel along the first route by performing the first automatic driving based on the first target trajectory, and causes the mobile body to travel along the second route by performing the first automatic driving based on the second target trajectory.

[0013] A fifth embodiment is the fourth embodiment, wherein the first marker and the second marker are installed within a predetermined area where the first route and the second route coexist.

[0014] The sixth embodiment is the fifth embodiment, wherein a third marker is further installed within the predetermined area, which is used for generating both the first target trajectory and the second target trajectory, and the generation unit generates the first target trajectory based on the comparison result between the recognized first and third markers and the first and third markers included in the map information when the route to the destination is the first route, and generates the second target trajectory based on the comparison result between the recognized second and third markers and the second and third markers included in the map information when the route to the destination is the second route.

[0015] A seventh aspect is that, in the fifth aspect, a fourth marker, which is a two-dimensional or three-dimensional marker, is installed at the entrance to the predetermined area, the recognition unit further recognizes the fourth marker, and the generation unit starts generating the first target trajectory or the second target trajectory when the fourth marker is recognized.

[0016] The eighth aspect is that, in the seventh aspect, the fourth marker is installed at the exit of the predetermined area, and when the recognition unit recognizes the fourth marker again, it recognizes the boundary of the track beyond the position where the fourth marker was re-recognized, the generation unit generates a third target track, which is the target track when traveling along the track, based on the recognized boundary, and the driving control unit causes the moving body to travel along the track by performing the second automatic driving based on the third target track.

[0017] The ninth aspect is, in the eighth aspect, the generation unit generates a fourth target trajectory for slowing down and stopping the moving body if the boundary of the track beyond the position where the fourth marker is re-recognized is not recognized, and the driving control unit stops the moving body by automatically controlling at least one of the speed or steering of the moving body based on the fourth target trajectory.

[0018] A tenth embodiment is a third embodiment in which at least one of the first marker and the second marker is painted on the road surface using a trompe-l'oeil effect as a two-dimensional road marking.

[0019] The eleventh aspect is a mobile body control method using a computer mounted on a mobile body equipped with at least a camera, the method comprising: recognizing objects present around the mobile body using the camera; performing automatic driving that automatically controls at least one of the speed or steering of the mobile body based on the recognized objects; recognizing two-dimensional or three-dimensional markers and the boundary of the path on which the mobile body travels; and switching the performed automatic driving between a first automatic driving, which is automatic driving based on the markers, and a second automatic driving, which is automatic driving based on the boundary.

[0020] A twelfth aspect is a program to be executed by a computer mounted on a mobile body on which at least a camera is installed, the program including: using the camera to recognize objects present around the mobile body; performing automatic driving that automatically controls at least one of the speed or steering of the mobile body based on the recognized objects; recognizing two-dimensional or three-dimensional markers and the boundary of the path on which the mobile body travels; and switching the performed automatic driving between a first automatic driving, which is automatic driving based on the markers, and a second automatic driving, which is automatic driving based on the boundary. [Effects of the Invention]

[0021] According to any of the above embodiments, even in situations where a moving object such as a vehicle must move significantly in the direction of a right or left turn or a rotation in an area where it is not easy to recognize the road, it is possible to drive automatically without deviating from the intended road. [Brief explanation of the drawing]

[0022] [Figure 1]It is a configuration diagram of the movement control system 1 using the automatic driving control device 100 according to this embodiment. [Figure 2] It is a diagram showing an example of a predetermined area. [Figure 3] It is a diagram showing an example of a predetermined area. [Figure 4] It is a diagram showing an example of a marker installed in the map matching area. [Figure 5] It is a diagram showing an example of the dedicated marker MK3 for route A. [Figure 6] It is a diagram showing an example of an image in front of the host vehicle M captured by the camera 10. [Figure 7] It is a diagram showing another example of the dedicated marker MK3 for route A. [Figure 8] It is a flowchart showing an example of a series of processing flows by the automatic driving control device 100 according to this embodiment.

Embodiments for Carrying Out the Invention

[0023] Hereinafter, embodiments of the movement control device, movement control method, and program of the present invention will be described with reference to the drawings. The movement control device of the embodiment is applied to, for example, an autonomous vehicle. Autonomous driving is, for example, controlling the driving of a vehicle by controlling one or both of the vehicle speed and steering. The above-described vehicle driving control includes various driving controls such as ACC (Adaptive Cruise Control System), TJP (Traffic Jam Pilot), ALC (Auto Lane Changing), CMBS (Collision Mitigation Brake System), and LKAS (Lane Keeping Assistance System). The autonomous vehicle may be controlled by manual driving of an occupant (driver).

[0024] [Overall Configuration] Figure 1 is a diagram showing the configuration of a mobile vehicle control system 1 using the automatic driving control device 100 according to this embodiment. The mobile vehicle on which the mobile vehicle control system 1 is installed is typically an automobile.

[0025] Automobiles are vehicles such as two-wheeled, three-wheeled, or four-wheeled vehicles, and their power sources are internal combustion engines such as diesel or gasoline engines, electric motors, or a combination of these. Electric motors operate using electricity generated by a generator connected to an internal combustion engine, or using discharged electricity from secondary batteries or fuel cells.

[0026] The mobile body on which the mobile body control system 1 is installed is not limited to automobiles, but may be any vehicle-type mobile body that runs on an electric motor powered by electricity supplied from a battery, such as an electric-assist bicycle. Furthermore, the mobile body on which the mobile body control system 1 is installed is not limited to a vehicle-type mobile body, but may be other electric mobility devices such as mobile robots. Hereinafter, as an example, the mobile body on which the mobile body control system 1 is installed will be described as a four-wheeled automobile, and the automobile on which the mobile body control system 1 is installed will be referred to as "the vehicle M".

[0027] The mobile vehicle control system 1 includes, for example, a camera 10, a radar device 12, a LiDAR (Light Detection and Ranging) 14, an object recognition device 16, a communication device 20, an HMI (Human Machine Interface) 30, a vehicle sensor 40, a navigation device 50, an MPU (Map Positioning Unit) 60, a driver control device 80, a driver monitor camera 90, an automatic driving control device 100, a driving force output device 200, a brake device 210, and a steering device 220. These devices and equipment are connected to each other by multiplex communication lines such as CAN (Controller Area Network) communication lines, serial communication lines, wireless communication networks, etc. The configuration shown in Figure 1 is merely an example, and some of the configuration may be omitted, or other configurations may be added. The automatic driving control device 100 is an example of a "mobile vehicle control device".

[0028] Camera 10 is a digital camera that uses a solid-state image sensor such as a CCD (Charge Coupled Device) or CMOS (Complementary Metal Oxide Semiconductor). Camera 10 can be mounted at any location on the vehicle M. For example, when imaging the area in front of the vehicle M, camera 10 can be mounted on the top of the front windshield or behind the rearview mirror. When imaging the area behind the vehicle M, camera 10 can be mounted on the top of the rear windshield. When imaging the right or left side of the vehicle M, camera 10 can be mounted on the right or left side of the vehicle body or door mirror. Camera 10, for example, periodically and repeatedly images the area around the vehicle M. Camera 10 may also be a stereo camera.

[0029] The radar device 12 emits radio waves such as millimeter waves around the vehicle M and detects radio waves reflected by objects (reflected waves) to determine at least the position (distance and bearing) of an object. The radar device 12 can be mounted at any location on the vehicle M. The radar device 12 may also detect the position and velocity of an object using the FM-CW (Frequency Modulated Continuous Wave) method.

[0030] The LIDAR 14 illuminates the area around the vehicle M with light and measures the scattered light from that illumination. Based on the time from emission to reception, the LIDAR 14 detects the distance to the target. The illuminated light may be, for example, pulsed laser light. The LIDAR 14 can be attached to any location on the vehicle M.

[0031] The object recognition device 16 performs sensor fusion processing on the detection results from some or all of the camera 10, radar device 12, and LIDAR 14 to recognize the position, type, speed, etc., of an object. The object recognition device 16 outputs the recognition results to the automatic driving control device 100. Alternatively, the object recognition device 16 may output the detection results from the camera 10, radar device 12, and LIDAR 14 directly to the automatic driving control device 100. In this case, the object recognition device 16 may be omitted from the mobile vehicle control system 1.

[0032] The communication device 20 communicates with other vehicles in the vicinity of its own vehicle M, or with various server devices via a wireless base station, for example, by using a cellular network, Wi-Fi network, Bluetooth®, DSRC (Dedicated Short Range Communication), etc.

[0033] The HMI30 presents various information to the occupants of the vehicle M (including the driver) and accepts input operations from the occupants. The HMI30 may be equipped with, for example, a display, speaker, buzzer, touch panel, microphone, switch, key, etc.

[0034] The vehicle sensor 40 includes a vehicle speed sensor for detecting the speed of the vehicle M, an acceleration sensor for detecting acceleration, a yaw rate sensor for detecting angular velocity around the vertical axis, and an orientation sensor for detecting the orientation of the vehicle M.

[0035] The navigation device 50 includes, for example, a GNSS (Global Navigation Satellite System) receiver 51, a navigation HMI 52, and a route determination unit 53. The navigation device 50 stores first map information 54 in a storage device such as an HDD (Hard Disk Drive) or flash memory.

[0036] The GNSS receiver 51 determines the position of the vehicle M based on signals received from GNSS satellites. The position of the vehicle M may also be determined or supplemented by an INS (Inertial Navigation System) that utilizes the output of the vehicle sensor 40.

[0037] The navigation HMI 52 includes a display device, speaker, touch panel, keys, etc. The navigation HMI 52 may be partially or completely shared with the HMI 30 described above. For example, instead of, or in addition to, entering the destination of their vehicle M into the HMI 30, the occupant may enter the destination of their vehicle M into the navigation HMI 52.

[0038] The route determination unit 53 determines, for example, a route (hereinafter referred to as the route on the map) from the position of the vehicle M identified by the GNSS receiver 51 (or any inputted position) to the destination input by the occupant using the HM30 or navigation HMI 52, by referring to the first map information 54.

[0039] The first map information 54 is, for example, information in which the road shape is represented by links indicating roads and nodes connected by those links. The first map information 54 may also include information such as road curvature and POI (Point of Interest) information. The route on the map is output to the MPU 60.

[0040] The navigation device 50 may provide route guidance using the navigation HMI 52 based on the route on the map. The navigation device 50 may be implemented, for example, by the functions of a terminal device such as a smartphone or tablet held by an occupant. The navigation device 50 may transmit the current location and destination to the navigation server via the communication device 20 and obtain a route equivalent to the route on the map from the navigation server.

[0041] The MPU 60 includes, for example, a recommended lane determination unit 61 and stores second map information 62 in a storage device such as an HDD or flash memory. The recommended lane determination unit 61 divides the map route provided by the navigation device 50 into multiple blocks (for example, every 100m with respect to the vehicle's direction of travel) and determines a recommended lane for each block by referring to the second map information 62. The recommended lane determination unit 61 makes decisions such as which lane from the left the vehicle should travel in. If there is a branching point on the map route, the recommended lane determination unit 61 determines a recommended lane so that the vehicle M can travel along a reasonable route to proceed to the branching point.

[0042] The second map information 62 is map information with higher accuracy than the first map information 54. The second map information 62 includes, for example, information on the center of lanes or information on lane boundaries. The second map information 62 may also include road information, traffic regulation information, address information (address and postal code), facility information, telephone number information, etc. The second map information 62 may be updated as needed by the communication device 20 communicating with other devices.

[0043] The driver control elements 80 include, for example, an accelerator pedal, a brake pedal, a shift lever, a steering wheel, a modified steering wheel, a joystick, and other controls. The driver control elements 80 are equipped with sensors that detect the amount of operation or whether or not an operation is being performed, and the detection results are output to the automatic driving control device 100, or to some or all of the driving force output device 200, the brake device 210, and the steering device 220.

[0044] For example, a sensor attached to the steering wheel (hereinafter referred to as the steering sensor) detects a weak electric current generated when an occupant touches the steering wheel. The steering sensor may also detect the steering torque generated around the rotation axis (shaft) of the steering wheel. When the steering sensor detects an electric current or steering torque, it outputs a signal indicating the detection result to the automatic driving control device 100.

[0045] The driver monitoring camera 90 is a camera that captures images of the interior of the vehicle M. The driver monitoring camera 90 is a digital camera that uses a solid-state image sensor such as a CCD or CMOS. When the driver monitoring camera 90 captures images of the interior of the vehicle M, it outputs the image data to the automatic driving control device 100.

[0046] The automatic driving control device 100 includes, for example, a processing unit 110 and a storage unit 130. The processing unit 110 includes, for example, a recognition unit 112, a generation unit 114, and a driving control unit 116. These components of the processing unit 110 are realized by a hardware processor such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit) executing a program (software). Some or all of these components may be realized by hardware (including circuitry) such as an LSI (Large Scale Integration), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), or SOC (System On Chip), or by the cooperation of software and hardware. The program may be stored in advance in a storage device such as the HDD or flash memory of the automatic driving control device 100 (a storage device equipped with a non-transient storage medium), or it may be stored in a removable storage medium such as a DVD or CD-ROM, and installed in the HDD or flash memory of the automatic driving control device 100 when the storage medium (non-transient storage medium) is inserted into a drive device.

[0047] The memory unit 130 is implemented using the various storage devices described above. For example, the memory unit 130 can be implemented using an HDD, flash memory, EEPROM (Electrically Erasable Programmable Read Only Memory), ROM (Read Only Memory), or RAM (Random Access Memory). The memory unit 130 stores, for example, programs that are read and executed by the processor, as well as AI (Artificial Intelligence) models and matching map information 132 used to recognize the surrounding conditions and environment (especially the boundaries of the road) of the vehicle M. Details of the AI ​​model and matching map information 132 will be described later.

[0048] The recognition unit 112 recognizes the situation or environment around the vehicle M. An AI (Artificial Intelligence) model may be used for this recognition. As the AI ​​model, for example, a deep neural network such as an encoder, decoder, and transformer may be employed.

[0049] For example, the recognition unit 112 recognizes objects present around the vehicle M based on information input from the camera 10, radar device 12, and LIDAR 14 via the object recognition device 16. Objects recognized by the recognition unit 112 include, for example, bicycles, motorcycles, four-wheeled vehicles, pedestrians, road signs, road markings, lane markings, utility poles, guardrails, and fallen objects.

[0050] Furthermore, the recognition unit 112 recognizes the position, velocity, acceleration, and other states of the object. The position of the object is recognized, for example, as a position on a relative coordinate system with the origin being a representative point of the vehicle M (such as the center of gravity or the center of the drive axis) (i.e., the position relative to the vehicle M), and is used for control. The position of the object may be represented by a representative point such as the center of gravity or a corner of the object, or by a represented region. The "state" of the object may include the object's acceleration, jerk, or "action state" (for example, whether or not it is changing lanes or is about to change lanes).

[0051] Furthermore, the recognition unit 112 recognizes, for example, the lane in which the vehicle M is traveling (hereinafter referred to as the vehicle's lane) and adjacent lanes adjacent to the vehicle's lane. For example, the recognition unit 112 recognizes the vehicle's lane and adjacent lanes by comparing the road marking pattern obtained from the second map information 62 (for example, an arrangement of solid and dashed lines) with the road marking pattern around the vehicle M recognized from the image captured by the camera 10.

[0052] Furthermore, the recognition unit 112 may recognize lanes such as the current lane and adjacent lanes by recognizing not only road markings but also road boundaries. Road boundaries include road markings, shoulders, curbs, median strips, and guardrails. In addition, road boundaries may include markers or objects that indicate the boundary between the roadway and the sidewalk. In this recognition, the position of the current vehicle M obtained from the navigation device 50 and the processing results from the INS may be taken into consideration. The recognition unit 112 may also recognize stop lines, obstacles, red lights, toll booths, and other road phenomena.

[0053] When the recognition unit 112 recognizes its own lane, it recognizes the relative position and orientation of the vehicle M with respect to the lane. For example, the recognition unit 112 may recognize the deviation of the vehicle M's reference point from the center of the lane, and the angle it makes with a line connecting the centerlines of the lanes in the direction of travel of the vehicle M, as the relative position and orientation of the vehicle M with respect to the lane. Alternatively, the recognition unit 112 may recognize the position of the vehicle M's reference point relative to any side edge of the lane (road marking or road boundary), as the relative position of the vehicle M with respect to the lane.

[0054] Furthermore, the objects recognized by the recognition unit 112 include various markers present in a predetermined area. In this predetermined area, multiple routes of different types, such as U-turn paths and straight paths, coexist.

[0055] Figures 2 and 3 illustrate an example of a designated area. As shown in the figures, a U-turn path is formed in the middle of a straight road within the designated area. A vehicle that reaches the designated area can either make a U-turn within the area or continue straight ahead.

[0056] In such designated areas, with permission from the road management authorities, such as the police or local government, the surface of one or more of the lanes may be painted in a designated color (e.g., blue or red). In the example shown, the surface of the U-turn lane is colored by painting.

[0057] When road surfaces are painted and colored, it can become difficult to recognize the boundaries of the road, or the accuracy of recognizing the boundaries may decrease. In other words, the designated area can be described as an area where it is difficult to recognize the boundaries of the road, or where the accuracy of recognizing the boundaries of the road is reduced.

[0058] Furthermore, in designated areas where multiple lanes coexist, as mentioned above, movement in the direction of right or left turns or rotation tends to be large, and due to the nonlinear characteristics (slip) of tire rolling, the estimated position of the vehicle may deviate significantly from its actual position, potentially causing it to deviate from the lane.

[0059] Therefore, in this embodiment, a map matching method is used to enable automatic driving within a predetermined area without deviating from the intended route. For this reason, the predetermined area will be specifically referred to as the "map matching area" in the following description.

[0060] In the map matching area (i.e., a designated area), several markers are installed for map matching. The markers may be painted on the road surface as two-dimensional road markings, or they may be three-dimensional road signs or facilities such as signs, delineators, or lane dividers.

[0061] Figure 4 shows an example of markers placed in a map matching area. For example, the map matching area marker MK1, the shared marker MK2, the Route A-specific marker MK3, and the Route B-specific marker MK4 are placed in the map matching area.

[0062] Map matching area markers MK1 are installed at the entrance and exit of map matching areas. Map matching area markers MK1 are used to switch between automated driving based on map matching and automated driving based on road recognition, and vice versa. Map matching area markers MK1 are an example of a "fourth marker".

[0063] The shared marker MK2 is used when the vehicle is automatically driving along both Route A and Route B. For example, Route A may be a straight road, and Route B may be a U-turn road. The shared marker MK2 is placed at the point where the boundaries of Route A and Route B overlap (i.e., where the boundary of Route A and the boundary of Route B coincide). The shared marker MK2 is an example of a "third marker".

[0064] Route A-specific marker MK3 is a marker used only when automatically navigating Route A. Route A-specific marker MK3 is installed at the boundary of the Route A path. Route A-specific marker MK3 is an example of a "first marker".

[0065] The Route B-specific marker MK4 is a marker used only when automatically navigating Route B. The Route B-specific marker MK4 is placed at the boundary of the Route B route. The Route B-specific marker MK4 is an example of a "secondary marker".

[0066] The recognition unit 112 recognizes the map matching area marker MK1, the shared marker MK2, the route A-specific marker MK3, and the route B-specific marker MK4 as objects present around the vehicle M.

[0067] Below, we will explain the Route A-dedicated marker MK3 as a representative example among the map matching area marker MK1, shared marker MK2, Route A-dedicated marker MK3, and Route B-dedicated marker MK4. Note that the information applicable to the Route A-dedicated marker MK3 in the following explanation can also be applied to all other markers.

[0068] Figure 5 shows an example of a Route A-specific marker MK3. In the following explanation, the X direction in three-dimensional space is defined as the direction of road surface extension (direction of vehicle travel), the Y direction as the width direction of the road surface, and the Z direction as the perpendicular direction to the road surface.

[0069] The Route A-specific marker MK3 may, for example, be painted on the road surface as a two-dimensional road marking. In this case, the Route A-specific marker MK3 may be painted on the road surface stretched in the direction of extension of the road surface (the X direction in the diagram) compared to the width direction of the road surface (the Y direction). More specifically, if the Route A-specific marker MK3 is a circular mark, it will be painted on the road surface as an elliptical mark with its minor axis in the Y direction and its major axis in the X direction.

[0070] Figure 6 shows an example of an image of the front of the vehicle M captured by camera 10. As described above, when the Route A dedicated marker MK3 is painted on the road surface stretched in the direction of extension of the road surface (X direction in the figure) compared to the width direction of the road surface (Y direction), the camera 10 can recognize the Route A dedicated marker MK3 as having approximately the same shape in terms of scale in the width direction (Y direction) and the direction of extension of the road surface (X direction in the figure) due to the optical illusion effect. In other words, the Route A dedicated marker MK3 painted on the road surface can be recognized with high accuracy.

[0071] Figure 7 shows another example of the Route A-specific marker MK3. The Route A-specific marker MK3 may be a three-dimensional road sign or facility, for example, instead of a two-dimensional road marking.

[0072] Returning to the explanation of Figure 1, the generation unit 114 generates a future target trajectory TR for the vehicle M to move automatically (without driver intervention). The target trajectory TR includes, for example, a position element that defines the future position of the vehicle M and a speed element that defines the future speed of the vehicle M.

[0073] For example, the generation unit 114 determines a number of points (track points) that the vehicle M should sequentially reach as position elements of the target track TR. Track points are points that the vehicle M should reach at predetermined travel distances (e.g., a few meters). The predetermined travel distance may be calculated, for example, by the distance traveled along the route.

[0074] Furthermore, the generation unit 114 determines the target velocity ν and target acceleration α for each predetermined sampling time (for example, around a fraction of a second [s]) as velocity elements of the target trajectory TR. The trajectory points may also be the positions that the vehicle M should reach at each predetermined sampling time. In this case, the target velocity ν and target acceleration α are determined by the sampling time and the interval between trajectory points.

[0075] For example, when a road boundary is recognized by the recognition unit 112, the generation unit 114 generates a target trajectory TR within the recognized road boundary, which in principle allows the vehicle M to automatically travel in the recommended lane determined by the recommended lane determination unit 61. In this case, the generation unit 114 may generate the target trajectory TR so that the vehicle M can respond to the surrounding conditions when automatically traveling in the recommended lane. Specifically, the generation unit 114 may generate a target trajectory TR that allows the vehicle M to take various actions such as constant speed driving, following, lane changes, branching, merging, overtaking, and avoidance. Hereinafter, the target trajectory TR generated when a road boundary is recognized will be referred to as the "third target trajectory TR3" and described accordingly.

[0076] If the recognition unit 112 does not recognize the track boundary, but various markers are recognized, the generation unit 114 performs map matching based on the recognized markers and the matching map information 132, and generates a target trajectory TR suitable for either route A or route B.

[0077] The matching map information 132 includes at least Route A and Route B, which are located within the map matching area, and also includes the locations where markers are placed and the types of those markers.

[0078] For example, when the generation unit 114 automatically drives along route A (e.g., a straight road) using map matching, it compares (matches) the shared marker MK2 included in the matching map information 132 with the recognized shared marker MK2, and also compares (matches) the route A-specific marker MK3 included in the matching map information 132 with the recognized route A-specific marker MK3.

[0079] Various methods can be used for map matching, such as VSLAM (Visual Simultaneous Localization and Mapping), empirical map matching, sequential least squares, and extreme value searching.

[0080] When the generation unit 114 matches the shared marker MK2 and the Route A-dedicated marker MK3, it reads from the map a pre-set target trajectory TR1 for the shared marker MK2 and the Route A-dedicated marker MK3 on the map. The target trajectory TR1 may be set at a distance of several meters from the shared marker MK2 and the Route A-dedicated marker MK3. The generation unit 114 outputs the target trajectory TR1 read from the matching map information 132 to the operation control unit 116. Hereinafter, the target trajectory TR1 read from the matching map information 132 will be referred to as the "first target trajectory TR1".

[0081] Similarly, for example, when the generation unit 114 automatically drives along Route B (e.g., a U-turn route) using map matching, it compares (matches) the shared marker MK2 included in the matching map information 132 with the recognized shared marker MK2, and also compares (matches) the Route B-specific marker MK4 included in the matching map information 132 with the recognized Route B-specific marker MK4.

[0082] When the generation unit 114 matches the shared marker MK2 and the Route B-dedicated marker MK4, it reads from the map the target trajectory TR2 that has been pre-set for the shared marker MK2 and the Route B-dedicated marker MK4 on the map. The target trajectory TR2, like the target trajectory TR1, may be set at a distance of several meters from the shared marker MK2 and the Route B-dedicated marker MK4. The generation unit 114 outputs the target trajectory TR2 read from the matching map information 132 to the operation control unit 116. Hereinafter, the target trajectory TR2 read from the matching map information 132 will be referred to as the "second target trajectory TR2".

[0083] The driving control unit 116 controls the driving force output device 200, the braking device 210, and the steering device 220 so that the vehicle M passes through the target trajectory TR generated by the generation unit 114 at the scheduled time.

[0084] The operation control unit 116 obtains the target trajectory TR from the generation unit 114 and stores it in the memory of the storage unit 130.

[0085] The driving control unit 116 controls one or both of the driving force output device 200 and the braking device 210 based on the velocity elements (e.g., target velocity ν and target acceleration α) included in the target trajectory TR stored in memory.

[0086] The driving control unit 116 controls the steering device 220 according to position elements included in the target trajectory stored in memory (for example, the curvature κ of the target trajectory, the amount of steering displacement u according to the position of the trajectory point, etc.).

[0087] Speed ​​control and steering control are achieved, for example, by a combination of feedforward control and feedback control. As an example, the driving control unit 116 performs a combination of feedforward control according to the curvature of the road in front of the vehicle M and feedback control based on the deviation from the target trajectory TR.

[0088] The driving force output device 200 outputs driving force (torque) to the drive wheels for the vehicle M to move. The driving force output device 200 includes, for example, a combination of an internal combustion engine, an electric motor, and a transmission, and a power ECU (Electronic Control Unit) that controls them. The power ECU controls the above configuration according to information input from the driving control unit 116 or information input from the driving control device 80.

[0089] The brake system 210 includes, for example, a brake caliper, a cylinder that transmits hydraulic pressure to the brake caliper, an electric motor that generates hydraulic pressure in the cylinder, and a brake ECU. The brake ECU controls the electric motor according to information input from the driving control unit 116 or from the driving control element 80, so that brake torque corresponding to the braking operation is output to each wheel. The brake system 210 may also include a backup mechanism that transmits hydraulic pressure generated by the operation of the brake pedal included in the driving control element 80 to the cylinder via a master cylinder. The brake system 210 is not limited to the configuration described above, and may also be an electronically controlled hydraulic brake system that controls an actuator according to information input from the driving control unit 116 to transmit hydraulic pressure from the master cylinder to the cylinder.

[0090] The steering device 220 includes, for example, a steering ECU and an electric motor. The electric motor, for example, applies force to a rack and pinion mechanism to change the direction of the steering wheels. The steering ECU drives the electric motor to change the direction of the steering wheels according to information input from the driving control unit 116 or from the driving control element 80.

[0091] [Processing flow] The following describes a series of processes performed by the automatic driving control device 100 according to this embodiment, using a flowchart. Figure 8 is a flowchart showing an example of a series of processes performed by the automatic driving control device 100 according to this embodiment. The processes in this flowchart may be executed repeatedly, for example, at a predetermined cycle.

[0092] First, the recognition unit 112 starts recognizing the various markers mentioned above as objects present on the road on which the vehicle M is traveling (step S100).

[0093] Next, the generation unit 114 determines whether or not the map matching area marker MK1 has been recognized by the recognition unit 112 (step S102).

[0094] For example, if a vehicle has never entered a map matching area, and the map matching area marker MK1 is recognized for the first time, it is determined that the vehicle M has reached the entrance to or near the entrance of the map matching area.

[0095] When the recognition unit 112 recognizes the map matching area marker MK1 (i.e., when the vehicle M reaches the entrance of the map matching area or its vicinity), the generation unit 114 further determines whether or not the route that matches the destination is route A (step S102).

[0096] The destination referred to here may be, for example, a destination entered by the user into the navigation device 50. For example, in the situations illustrated in Figures 2 and 3, if the destination is in front of the vehicle M, Route A, which is a straight road, is selected as the route that matches the destination, and if the destination is behind the vehicle M, Route B, which is a U-turn, is selected as the route that matches the destination.

[0097] If the route that matches the destination is Route A, the generation unit 114 generates a first target trajectory TR1, which is a target trajectory specific to Route A, based on the Route A-specific marker MK3 (step S106).

[0098] Specifically, the generation unit 114 compares (matches) the shared marker MK2 and the Route A-specific marker MK3 included in the matching map information 132 with the recognized shared marker MK2 and the Route A-specific marker MK3, and reads out the target trajectory TR1 that has been pre-set for the shared marker MK2 and the Route A-specific marker MK3 on the matching map information 132. Then, the generation unit 114 sets the target trajectory TR1 read out from the matching map information 132 as the first target trajectory TR1.

[0099] Next, the driving control unit 116 controls the driving force output device 200, the braking device 210, and the steering device 220 based on the first target trajectory TR1 (step S108). In other words, the driving control unit 116 performs automatic driving based on the first target trajectory TR1. As a result, the vehicle M automatically travels along route A. Automatic driving based on the first target trajectory TR1 is an example of "first automatic driving".

[0100] On the other hand, in the determination process of S104, if it is determined that the route that matches the destination is Route B, the generation unit 114 generates a second target trajectory TR2, which is a target trajectory specifically for Route B, based on the Route B-specific marker MK4 (step S110).

[0101] Specifically, the generation unit 114 compares (matches) the shared marker MK2 and the Route B-specific marker MK4 included in the matching map information 132 with the recognized shared marker MK2 and the Route B-specific marker MK4, and reads out the target trajectory TR2 that has been pre-set for the shared marker MK2 and the Route B-specific marker MK4 on the matching map information 132. Then, the generation unit 114 sets the target trajectory TR2 read out from the matching map information 132 as the second target trajectory TR2.

[0102] Next, the driving control unit 116 controls the driving force output device 200, the braking device 210, and the steering device 220 based on the second target trajectory TR2 (step S112). In other words, the driving control unit 116 performs automatic driving based on the second target trajectory TR2. As a result, the vehicle M automatically travels along route B. Automatic driving based on the second target trajectory TR2 is another example of "first automatic driving".

[0103] Next, the generation unit 114 determines whether the map matching area marker MK1 has been recognized again by the recognition unit 112 (step S114).

[0104] For example, if the map matching area marker MK1 is recognized again after entering the map matching area, it is determined that the vehicle M has reached the exit of the map matching area or its vicinity.

[0105] If the recognition unit 112 does not re-recognize the map matching area marker MK1 (i.e., if the vehicle M has not reached the exit of the map matching area or its vicinity), the generation unit 114 returns to the process of S104. As a result, automatic driving based on map matching continues until the vehicle M reaches the exit of the map matching area.

[0106] On the other hand, the generation unit 114 determines whether the road boundary has been recognized by the recognition unit 112 if (i) in the determination process of S102, the map matching area marker MK1 is not recognized and the vehicle M has not reached the entrance or vicinity of the map matching area, or (ii) in the determination process of S114, the map matching area marker MK1 is re-recognized and the vehicle M has reached the exit or vicinity of the map matching area (step S116). In other words, the generation unit 114 determines whether the road boundary has been recognized by the recognition unit 112 if the vehicle M is outside the map matching area.

[0107] If a track boundary is recognized outside the map matching area, the generation unit 114 generates a third target trajectory TR3 for the vehicle M to automatically drive within the recognized track boundary (step S118).

[0108] For example, when the vehicle M reaches the exit of the map matching area, if the recognition unit 112 recognizes the track boundary beyond the exit of the map matching area, the generation unit 114 may generate a third target trajectory TR3 based on the track boundary beyond the exit of the map matching area.

[0109] Next, the driving control unit 116 controls the driving force output device 200, the braking device 210, and the steering device 220 based on the third target trajectory TR3 (step S120). In other words, the driving control unit 116 performs automatic driving based on the third target trajectory TR3. As a result, the vehicle M automatically travels on a road outside the map matching area. Automatic driving based on the third target trajectory TR3 is an example of "second automatic driving".

[0110] On the other hand, if a track boundary is not recognized outside the map matching area, the generation unit 114 may generate a third target trajectory TR3 using information other than the track boundary, or generate a fourth target trajectory TR4 to slow down and stop the vehicle M (step S122).

[0111] For example, when the vehicle M reaches the exit of the map matching area, the recognition unit 112 is expected to recognize the road boundary beyond the exit of the map matching area. However, due to various circumstances, road markings, shoulders, curbs, median strips, guardrails, etc., may not be recognized, resulting in the road boundary not being recognized or the recognition accuracy being reduced.

[0112] If such a track boundary is not recognized, or if the accuracy of track boundary recognition decreases, the generation unit 114 may generate a third target trajectory TR3 for automatically driving the vehicle M by using the position (GNSS coordinates) of the vehicle M identified by the GNSS receiver 51 or by using the processing results of the INS. If neither of the above information is available, the generation unit 114 may generate a fourth target trajectory TR4 for decelerating and stopping the vehicle M. This completes the processing of this flowchart.

[0113] According to the embodiments described above, the automatic driving control device 100 includes a recognition unit 112 that recognizes objects present around the vehicle M using at least a camera 10, a generation unit 114 that generates a target trajectory TR on which the vehicle M should travel based on the recognized objects, and a driving control unit 116 that performs automatic driving based on the target trajectory TR. As described above, automatic driving means automatically controlling at least one or both of the speed and / or steering of the vehicle M.

[0114] The recognition unit 112 recognizes two-dimensional or three-dimensional markers, including a map matching area marker MK1 (an example of a "fourth marker"), a shared marker MK2 (an example of a "third marker"), a Route A-specific marker MK3 (an example of a "first marker"), and a Route B-specific marker MK4 (an example of a "second marker").

[0115] Furthermore, the recognition unit 112 recognizes the road boundary. The road boundary includes road markings, shoulders, curbs, median strips, and guardrails. In addition, the road boundary may include markers or objects that indicate the boundary between the roadway and the sidewalk.

[0116] The driving control unit 116 switches between the automated driving to be performed based on various markers (an example of "first automated driving") and automated driving based on track boundaries (an example of "second automated driving").

[0117] The following processes are performed for autonomous driving based on various markers. First, if the route to the destination is Route A (an example of "First Route"), the generation unit 114 generates the first target trajectory TR1, which is the target trajectory TR when traveling along Route A, based on the comparison result (matching result) between the recognized Route A-specific marker MK3 and the Route A-specific marker MK3 included in the matching map information 132.

[0118] Similarly, if the route to the destination is Route B (an example of the "second route"), the generation unit 114 generates a second target trajectory TR2, which is the target trajectory TR when traveling along Route B, based on the comparison result (matching result) between the recognized Route B-specific marker MK4 and the Route B-specific marker MK4 included in the matching map information 132.

[0119] When the first target trajectory TR1 is generated, the operation control unit 116 performs automatic operation based on the first target trajectory TR1, thereby causing the vehicle M to automatically travel along route A.

[0120] When the second target trajectory TR2 is generated, the operation control unit 116 performs automatic operation based on the second target trajectory TR2, thereby causing the vehicle M to automatically travel along route B.

[0121] The following processes are performed for automated driving based on the track boundary. First, the generation unit 114 generates a third target trajectory TR3 for the vehicle M to automatically drive within the recognized track boundary.

[0122] When the third target trajectory TR3 is generated, the operation control unit 116 performs automatic operation based on the third target trajectory TR3, thereby causing the vehicle M to automatically travel along the track.

[0123] With this configuration, even in map matching areas where it is difficult to recognize the road path, the vehicle M can automatically drive without deviating from the intended path, even if it has to move significantly in the direction of a right or left turn or a rotation.

[0124] Although embodiments for carrying out the present invention have been described above using examples, the present invention is not limited in any way to these embodiments, and various modifications and substitutions can be made without departing from the spirit of the present invention. [Explanation of symbols]

[0125] 1...Mobile control system, 10...Camera, 12...Radar device, 14...Finder, 16...Object recognition device, 20...Communication device, 30...HMI, 40...Vehicle sensor, 50...Navigation device, 60...MPU, 80...Driver control unit, 90...Driver monitor camera, 100...Automatic driving control device, 110...Processing unit, 112...Recognition unit, 114...Generation unit, 116...Driving control unit, 130...Storage unit, M...Own vehicle

Claims

1. A recognition unit that recognizes objects present around a moving object using at least a camera, The system includes a driving control unit that performs automatic driving by automatically controlling at least one of the speed or steering of the moving body based on the recognized object, The recognition unit recognizes a two-dimensional or three-dimensional marker and the boundary of the path on which the moving object travels. The operation control unit switches the automatic operation being performed between a first automatic operation, which is an automatic operation based on the marker, and a second automatic operation, which is an automatic operation based on the boundary. Mobile device control system.

2. The operation control unit switches to the first automatic operation mode when the marker is recognized, and switches to the second automatic operation mode when the marker is not recognized, or when the marker is recognized again after switching to the first automatic operation mode. The mobile device control device according to claim 1.

3. The recognition unit recognizes a first marker and a second marker that is different from the first marker. The operation control unit switches the automatic operation being performed between the first automatic operation based on the first marker and the first automatic operation based on the second marker. A mobile device control device according to claim 1 or 2.

4. The system further comprises a generation unit that generates a target trajectory on which the moving object should travel, based on the aforementioned markers. The generating unit is If the route to the destination is the first route, a first target trajectory is generated, which is the target trajectory when traveling along the first route, based on the comparison result between the recognized first marker and the first marker included in the map information. If the route to the destination is the second route, a second target trajectory, which is the target trajectory when traveling along the second route, is generated based on the comparison result between the recognized second marker and the second marker included in the map information. The aforementioned operation control unit, By performing the first automated driving based on the first target trajectory, the moving body is made to travel along the first route. By performing the first automated driving based on the second target trajectory, the moving body is made to travel along the second route. The mobile device control device according to claim 3.

5. The first marker and the second marker are installed within a predetermined area where the first route and the second route coexist. The mobile device control device according to claim 4.

6. Within the aforementioned predetermined area, a third marker is installed, which is used for generating both the first target trajectory and the second target trajectory. The generating unit is If the route to the destination is the first route, the first target trajectory is generated based on the comparison result between the recognized first marker and the third marker and the first marker and the third marker included in the map information. If the route to the destination is the second route, the second target trajectory is generated based on the comparison result between the recognized second and third markers and the second and third markers included in the map information. The mobile device control device according to claim 5.

7. A fourth marker, which is a two-dimensional or three-dimensional marker, is installed at the entrance to the predetermined area. The recognition unit further recognizes the fourth marker, When the fourth marker is recognized, the generation unit starts generating the first target trajectory or the second target trajectory. The mobile device control device according to claim 5.

8. The fourth marker is installed at the exit of the predetermined area. When the recognition unit recognizes the fourth marker again, it recognizes the boundary of the track beyond the position where the fourth marker was re-recognized. The generation unit generates a third target trajectory, which is the target trajectory when traveling along the track, based on the recognized boundary. The operation control unit causes the moving body to travel along the track by performing the second automatic operation based on the third target trajectory. The mobile device control device according to claim 7.

9. If the boundary of the track beyond the position where the fourth marker is re-recognized is not recognized, the generation unit generates a fourth target trajectory to decelerate and stop the moving object. The operation control unit stops the moving body by automatically controlling at least one of the speed or steering of the moving body based on the fourth target trajectory. The mobile device control device according to claim 8.

10. At least one of the first marker and the second marker is painted on the road surface using an optical illusion effect as a two-dimensional road marking. The mobile device control device according to claim 3.

11. A method for controlling a mobile object using a computer mounted on a mobile object equipped with at least one camera, To recognize objects present in the vicinity of the moving object using the aforementioned camera, Based on the recognized object, perform automated driving that automatically controls at least one of the speed or steering of the moving body. Recognizing two-dimensional or three-dimensional markers and the boundaries of the path along which the moving object travels, The automated driving to be performed is to switch between a first automated driving, which is based on the marker, and a second automated driving, which is based on the boundary. A method for controlling a mobile object, including the method described above.

12. A program to be executed on a computer mounted on a mobile device that has a camera installed, To recognize objects present in the vicinity of the moving object using the aforementioned camera, Based on the recognized object, perform automated driving that automatically controls at least one of the speed or steering of the moving body. Recognizing two-dimensional or three-dimensional markers and the boundaries of the path along which the moving object travels, The automated driving to be performed is to switch between a first automated driving, which is based on the marker, and a second automated driving, which is based on the boundary. A program that includes this.

Images