Movement control system, movement control method, and movement control device

Abstract

A movement control system (1) according to the present disclosure is provided with a first processing unit (121) and a second processing unit (220) that communicate with one another. The first processing unit generates, on the basis of non-real-time processing that has no constraint in the response time that processing takes, path information for controlling a movement path of a mobile device by using sensor information acquired from the second processing unit. The second processing unit controls, on the basis of real-time processing that has a constraint in the response time that processing takes, movement of the mobile device in accordance with the path information generated by the first processing unit .

Description

Technical Field

[0001] The present disclosure relates to a movement control system and a movement control method, and more particularly to a movement control technology used for automatic driving or ADAS (Advanced Driver-Assistance Systems) of a moving device. Law and transfer Specifically, it relates to a movement control technology used for automatic driving or ADAS (Advanced Driver-Assistance Systems) of a moving device.

Background Art

[0002] As a technology related to a moving device such as an automobile, there is known a technology for autonomously driving a moving device by generating a route of the moving device by a processor or the like and performing control so as to travel following the route.

[0003] Regarding such an automatic driving technology, there is known a technology for controlling an accelerator, a brake, a steering wheel, etc. based on a driving route determined using map information acquired from a database and information acquired from sensors mounted on a vehicle (for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] Conventional technologies allow for highly accurate control of the movement of mobile devices, including path generation and path-following control. From a safety perspective, however, in addition to highly accurate movement control, highly responsive movement control is also required. However, performing such highly accurate movement control generally increases the processing load and reduces responsiveness. On the other hand, if the processing load is kept low and responsiveness is prioritized, the accuracy of movement control decreases, resulting in a compromise in safety. Thus, conventional technologies face the challenge of achieving both the high accuracy and responsiveness of movement control required for safe movement control.

[0006] Therefore, this disclosure proposes a movement control system, a movement control method, a movement control device, and an information processing device that realize high-precision movement control with high responsiveness.

[0007] To solve the above problems, a motion control system according to this disclosure comprises a first processing unit and a second processing unit that communicate with each other, wherein the first processing unit generates path information for controlling the movement path of a mobile device using sensor information acquired from the second processing unit based on non-real-time processing that has no constraints on the response time for processing, and the second processing unit controls the movement of the mobile device along the path information generated by the first processing unit based on real-time processing that has constraints on the response time for processing, wherein the second processing unit estimates the position of the mobile device based on internal sensor information which is sensor information related to the behavior of the mobile device among the sensor information, controls the mobile device to follow the movement path based on the result of the position estimation and the path information, and further detects objects on or around the movement path based on the sensor information, and calculates the travel distance when emergency braking control is performed on the mobile device and the travel distance when emergency braking control is not performed on the mobile device based on the behavior of the mobile device and the distance from the mobile device to the object. If the distance traveled without emergency braking control on the moving device is shorter than the distance to the object, the system instructs the first processing unit to replan the travel route without emergency braking control on the moving device. If the distance traveled without emergency braking control on the moving device is longer than the distance to the object, and the distance traveled with emergency braking control on the moving device is shorter than the distance to the object, the system instructs the first processing unit to replan the travel route while emergency braking control is applied to the moving device. If the distance traveled with emergency braking control on the moving device is longer than the distance to the object, the system instructs the moving device to stop by applying emergency braking control. [Brief explanation of the drawing]

[0008] [Figure 1] This is a diagram illustrating the overview of the motion control system according to the embodiment. [Figure 2] This is a diagram illustrating the movement control according to the embodiment. [Figure 3] This figure shows an example configuration of a real-time processing device according to the embodiment. [Figure 4] This figure shows an example of a database for a real-time processing device according to the embodiment. [Figure 5] This figure shows an example of a database for a real-time processing device according to the embodiment. [Figure 6] This figure shows an example configuration of a non-real-time processing device according to the embodiment. [Figure 7] This figure shows an example of a database for a non-real-time processing device according to the embodiment. [Figure 8] This figure shows an example of a database for a non-real-time processing device according to the embodiment. [Figure 9] Figure (1) illustrates the motion control according to the embodiment. [Figure 10] Figure (2) illustrates the motion control according to the embodiment. [Figure 11] Figure (3) illustrates the motion control according to the embodiment. [Figure 12] Figure (4) illustrates the motion control according to the embodiment. [Figure 13] Figure (5) illustrates the motion control according to the embodiment. [Figure 14] Figure (6) illustrates the motion control according to the embodiment. [Figure 15] Figure (7) illustrates the motion control according to the embodiment. [Figure 16] Figure (8) illustrates the motion control according to the embodiment. [Figure 17] Figure (9) illustrates the motion control according to the embodiment. [Figure 18] This flowchart shows the processing flow according to the embodiment. [Figure 19]It is a sequence diagram showing the flow of processing according to an embodiment. [Figure 20] It is a diagram showing an overview of a movement control system according to a modification example. [Figure 21] It is a block diagram showing a configuration example of a movement control system. [Figure 22] It is a diagram showing an example of a sensing area according to an embodiment. [Figure 23] It is a hardware configuration diagram showing an example of a computer that realizes the functions of a movement control system according to an embodiment.

Mode for Carrying Out the Invention

[0009] Hereinafter, embodiments of the present disclosure will be described in detail based on the drawings. In each of the following embodiments, the same parts are denoted by the same reference numerals, and redundant descriptions are omitted.

[0010] The present disclosure will be described according to the item order shown below. 1. Embodiment 1-1. Configuration of Movement Control System According to Embodiment 1-2. An Example of Movement Control According to Embodiment 1-3. Procedure of Movement Control According to Embodiment 1-4. Modification Example According to Embodiment 2. Other Embodiments 2-1. Configuration of Moving Device 2-2. Others 3. Effects of Movement Control System According to the Present Disclosure 4. Hardware Configuration

[0011] (1. Embodiment) (1-1. Configuration of Movement Control System According to Embodiment) FIG.1 is a diagram showing the configuration of a movement control system 1 according to an embodiment of the present disclosure. Specifically, FIG.1 shows a diagram showing an overview of a real-time processing device 100 and a non-real-time processing device that constitute the movement control system according to the present disclosure.

[0012] The motion control according to the embodiments of this disclosure applies when a predetermined mobile device performing autonomous driving determines a global route based on a destination set by the user of the mobile device and map information obtained from a database, and performs motion control that follows the local movement path of the mobile device based on the global route and information about the surroundings of the mobile device obtained from sensors, etc. Here, the global route refers to the general route from the starting point of the mobile device's journey to the destination. The local movement path refers to the specific movement path of the global route, such as the selection of specific roads and lanes, and whether to travel in the center of the lane or close to the edge. Hereafter, "movement path" refers to the local movement path, and "route information" refers to information indicating the movement path. In the embodiments, the mobile device includes automobiles, bicycles, motorcycles, cargo transport carts, robots, etc., but in the embodiments, an automobile is given as an example of a mobile device to explain the motion control of the mobile device. Also in the embodiments, the real-time processing unit 100 refers to the mobile device itself or a motion control device provided in the mobile device. Also in the embodiments, the non-real-time processing unit is described using a cloud server 200 located outside the real-time processing unit 100 as an example. The movement control according to this embodiment is performed by a real-time processing unit 100 and a non-real-time processing unit.

[0013] Generally, when an automobile performs autonomous mobility control, it determines a route based on a set destination and map information, and then performs mobility control to follow the determined route while recognizing surrounding objects and road shapes. Mobility control that follows a determined route is achieved, for example, by performing the following control: Information such as accelerator opening, transmission speed, and steering angle, determined by a control unit such as a processor, is transmitted to each ECU (Electronic Control Unit), such as the engine / transmission control ECU, brake control ECU, and power steering control ECU, via in-vehicle communication such as CAN (Controller Area Network) communication. Then, each ECU that controls the engine / transmission, brakes, and power steering controls the engine / transmission, brakes, and steering. The processing performed by each ECU that controls the engine / transmission, brakes, and power steering is called real-time processing or real-time processing, and it is a process with guaranteed response time in which the correctness of the processing result depends not only on the correctness of the output result value but also on the time the result is produced. In other words, real-time processing or real-time processing refers to a processing method designed to ensure that the delay time from the input of a processing command to the output of the processing result is not exceeded, by setting a deadline or tolerance range for that delay. In this way, real-time processing or real-time processing allows for highly responsive processing by imposing constraints on the time required to complete the processing and output the result. Automobiles can perform controls such as braking without delay by using real-time processing, which has the constraint of response time. In mobile devices such as automobiles, control delays can lead to serious accidents, so controls such as brakes and accelerators must be performed by real-time processing, which is processing with guaranteed response time. In particular, in situations where an automobile needs to stop suddenly, a series of controls are performed from the time the brake is applied until the automobile actually stops, including determining the situation requiring brake control, sending a control command to the brakes, and starting the brake process. At any of these points, delays can significantly impair safety.

[0014] However, even if the response time is guaranteed for each ECU that controls the engine / transmission, brakes, and power steering, it does not necessarily mean that the processing can be performed in an ideally safe manner. For example, engine / transmission control, brake control, and power steering control may be performed by real-time processing with guaranteed response times, while the determination and transmission of information such as accelerator opening, transmission speed, and steering angle may be performed by control without guaranteed response times. In this case, the determination and transmission of information that is a prerequisite for controlling the engine / transmission, brakes, and power steering can be processed with high precision by not imposing the constraint of guaranteed response times, but delays may occur in the series of controls, and as a result, safety may not be guaranteed.

[0015] To avoid the decrease in safety due to such control delays, the entire motion control system may be constructed using real-time processing with the constraint of guaranteed response time. However, when the entire motion control system is constructed using real-time processing with the constraint of guaranteed response time, the accuracy of the processing may decrease as a result of requiring guaranteed response time throughout the entire series of controls, which may ultimately compromise safety.

[0016] Therefore, the movement control system 1 relating to this disclosure solves the above problem by the processing described below. The movement control system 1 consists of a real-time processing device 100 that performs real-time processing with constraints on the response time for processing, and a cloud server 200 that performs non-real-time processing without constraints on the response time for processing. The cloud server 200 is an example of an information processing device and generates route information for controlling the movement path of a mobile device. The real-time processing device 100 is an example of a movement control device and controls the movement of the mobile device in accordance with the generated route information.

[0017] The cloud server 200 can generate highly accurate travel routes by generating route information through non-real-time processing that is not constrained by response time. On the other hand, the real-time processing unit 100 can perform highly responsive route-following control through real-time processing that is constrained by response time. In other words, the travel control system 1 imposes a constraint of guaranteed response time on the control of brakes and accelerators, and the determination and transmission of the information that is a prerequisite for these actions, but does not impose a constraint of guaranteed response time on the generation of route information that is a prerequisite for these actions. With this configuration, the travel control system 1 can perform travel control with guaranteed safety. To add to this, regarding the generation of route information, the cloud server 200 generates the next route information a predetermined time or distance before the real-time processing unit 100 reaches the end point of the travel route being driven. Therefore, even if there is some delay in the generation of route information due to not imposing a constraint of guaranteed response time, it will not cause any delay in the series of travel control actions.

[0018] The following describes an overview of the motion control system 1 according to an embodiment and an example of motion control, using Figures 1 and 2. In the following description, it is assumed that the real-time processing unit 100 constituting the motion control system 1 is mounted on an automobile, which is a mobile device, or that the automobile itself functions as the real-time processing unit 100. In other words, in the following description, the real-time processing unit 100 can be read as an automobile (mobile device).

[0019] In the example shown in Figure 1, the motion control system 1 includes a real-time processing unit 100 and a cloud server 200. The real-time processing unit 100 includes a first sensor 110, a control unit 120, a second sensor 130, an internal sensor 140, and an accelerator / brake / steering unit 150.

[0020] The first sensor 110 is a sensor that acquires information about the mobile device or its surroundings, and is sometimes referred to as a High-End Sensor. The information acquired by the first sensor is used for processing such as generating route information performed by the cloud server 200. The information acquired by the first sensor may also be transmitted to the cloud server 200 via communication such as Ethernet or WiFi. Specific examples of the first sensor 110 include cameras, LiDAR (Light Detection and Ranging), millimeter-wave radar, ultrasonic sensors, and GPS (Global Positioning System). Specific examples of information acquired by the first sensor 110 are described in detail in Figure 7.

[0021] The second sensor 130 is a sensor that acquires information about the moving device or its surroundings, and acquires second sensor information used in processes such as object detection and emergency braking performed by the safety MCU 122. It is sometimes referred to as a Safety Sensor. The second sensor 130 transmits the acquired information to the safety MCU 122 via CAN communication or the like. Specific examples of the second sensor 130 include cameras, LiDAR (Light Detection and Ranging), millimeter-wave radar, ultrasonic sensors, etc.

[0022] The internal sensor 140 is a sensor that acquires internal sensor information that provides information about the behavior of the mobile device itself, and is used for self-position estimation (dead reckoning) by the real-time processing unit 121. Specific examples of the internal sensor 140 include an IMU (Inertial Measurement Unit) (accelerometer, angular velocity sensor), and a vehicle speed (wheel) encoder.

[0023] The control unit 120 is a processor that performs real-time processing such as movement control that follows path information and emergency braking. The control unit 120 is realized by executing a program stored inside the control unit 120 (for example, the movement control program according to this disclosure) using RAM (Random Access Memory) as a working area, for example, by a CPU (Central Processing Unit), MPU (Micro Processing Unit), GPU (Graphics Processing Unit), etc. Furthermore, the control unit 120 is a controller and may be realized by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or FPGA (Field Programmable Gate Array).

[0024] The control unit 120 further includes a real-time processing unit 121 and a safety MCU 122, and realizes or executes the motion control functions and operations described below. Note that the internal configuration of the control unit 120 is not limited to the configuration shown in Figure 1, and other configurations are also acceptable as long as they perform the motion control described later.

[0025] The real-time processing unit 121 receives sensor information from the first sensor 110 and the internal sensor 140, adds a timestamp to the first sensor information and the internal sensor information, and transmits it to the cloud server 200. The real-time processing unit 121 is sometimes referred to as a Realtime Unit. It also performs position estimation of the real-time processing unit 100 based on the internal sensor information. Furthermore, it receives the position recognition result and route information of the real-time processing unit 100 from the cloud server 200, corrects the position estimation of the real-time processing unit 100, and performs driving control that follows the route information.

[0026] The safety MCU122 receives second sensor information from the second sensor 130 and performs actions such as object detection along the path and emergency control based on the second sensor information.

[0027] The accelerator / brake / steering unit 150 controls the real-time processing unit 100, including the accelerator, brakes, and steering, based on control information such as accelerator opening and steering angle received from the safety MCU 122 and the real-time processing unit 121. The accelerator / brake / steering unit 150 includes the hardware modules for the accelerator, brakes, and steering, as well as an accelerator control ECU, a brake control ECU, and a steering control ECU that actually control the hardware modules for the accelerator, brakes, and steering. Alternatively, instead of each of the individual ECUs, there may be a single ECU that controls all of the hardware modules for the accelerator, brakes, and steering.

[0028] The cloud server 200 is a device that performs non-real-time processing without the constraint of guaranteed response time. The cloud server 200 generates path information for the movement path of the real-time processing unit 100, recognizes the position of the real-time processing unit 100, and recognizes objects along the movement path of the real-time processing unit 100. In addition, all processing related to movement control other than that specifically performed by the real-time processing unit 100 may be performed by the cloud server 200. In the movement control system 1, the components other than the cloud server 200 perform processing with the constraint of guaranteed response time, while the cloud server 200 performs processing without the constraint of guaranteed response time.

[0029] Next, we will explain the overview of the movement control processing performed by the movement control system 1 using Figure 2. Figure 2 is a block diagram showing the movement control processing of the movement control system 1 performed by the real-time processing unit 100 and the cloud server 200. Figure 2 shows a schematic block diagram of the flow of the movement control processing performed by the movement control system 1.

[0030] In the example shown in Figure 2, first, the real-time processing unit 100 acquires second sensor information from the second sensor 130 (step S10). The real-time processing unit 100 outputs the second sensor information from the second sensor 130 (step S11). Next, the real-time processing unit 100 detects obstacles on the path of the real-time processing unit 100's movement based on the second sensor information from the second sensor 130 (step S12). Then, the real-time processing unit 100 outputs information about the detected obstacles to cause the real-time processing unit 100 to move along the movement path (step S13). Examples of obstacles detected by the real-time processing unit 100 in step S12 include other mobile devices, pedestrians, animals, static objects such as garbage and waste on the path, and traffic signs such as signals and signs. Furthermore, detection includes not only detecting the presence of an object, but also recognition processing of semantic information, such as detecting that it is a pedestrian crossing a crosswalk or a sign indicating a speed limit of 50 km / h.

[0031] Furthermore, the real-time processing unit 100 acquires sensor information from the internal sensor information (step S20). The real-time processing unit 100 outputs the acquired internal sensor information for use in self-position estimation (step S21). Next, the real-time processing unit 100 estimates its own position using the acquired internal sensor information (step S22). Then, the real-time processing unit 100 outputs the estimated position of the real-time processing unit 100 to drive the real-time processing unit 100 along the movement path (step S23). The real-time processing unit 100 also outputs the estimated position of the real-time processing unit 100 to correct it based on the position recognition of the real-time processing unit 100 by the cloud server 200, which will be described later (step S24). Furthermore, the real-time processing unit 100 outputs the acquired internal sensor information to add a timestamp (step S25).

[0032] Furthermore, the real-time processing unit 100 acquires first sensor information from the first sensor (step S30). The real-time processing unit 100 outputs the acquired first sensor information to add a timestamp (step S31).

[0033] Next, the real-time processing unit 100 assigns a timestamp to the internal sensor information and the first sensor information (step S32). Here, the timestamp is information indicating the time when the sensor information was acquired. The real-time processing unit 100 transmits the first sensor information, to the cloud server 200, for object recognition by the cloud server 200 (step S33). The real-time processing unit 100 also transmits the internal sensor information, to the cloud server 200, for position recognition of the real-time processing unit 100 (step S34).

[0034] Next, the cloud server 200 performs object recognition (step S35). Then, it outputs the results of the object recognition for path generation (step S36).

[0035] Next, the cloud server 200 performs position recognition of the real-time processing unit 100 based on the timestamped first sensor information and the internal sensor information (step S37). Here, position recognition means that the cloud server 200 recognizes the position of the mobile device. Unlike self-position estimation, in which the mobile device estimates the relative position change of the mobile device itself, position recognition recognizes the absolute position of the mobile device using the timestamped first sensor information. Note that the recognized position of the mobile device is the position of the mobile device at the time of the assigned timestamp, so at the time position recognition is completed, it will be the past position of the mobile device.

[0036] Next, the cloud server 200 transmits the location recognition results to the real-time processing unit 100 for correction of the location estimation performed by the real-time processing unit 100 (step S38). The cloud server 200 also outputs the location recognition results for route generation (step S39).

[0037] Next, the cloud server 200 generates route information for the movement path of the real-time processing unit 100 based on the object recognition results and the position recognition results of the real-time processing unit 100 (step S40). Then, the cloud server 200 transmits the route information for the movement path to the real-time processing unit 100 (step S41). Here, generating route information means generating route information that connects to the path that the mobile device is currently traveling on. Furthermore, the route generation in step S35 is not limited to generating a route that connects to the path that the mobile device is currently traveling on, but may also be limited to creating the next path in advance. The route information includes information such as the target position, attitude, speed, acceleration, angular velocity, and angular acceleration of the mobile device at a given point in time.

[0038] Next, the real-time processing unit 100 corrects the estimated position based on the position estimation result from the real-time processing unit 100 and the position recognition result from the cloud server 200 (step S42). Then, the real-time processing unit 100 outputs the corrected position estimation result (step S43). Since the position information of the mobile device recognized by the cloud server 200 is the position of the mobile device at the time of the assigned timestamp, the correction of the position estimation is a correction of the position estimation result at the time the said timestamp was assigned. The position estimation based on the sensor information of the internal sensor 140 in step S22 is a position estimation based on relative position changes, and therefore, by its nature, errors will accumulate. For example, if the self-position estimation of the internal sensor results in an error of 10 cm in the direction of travel of the mobile device for every 1 km traveled, then an error of 1 m will occur for every 10 km traveled. Therefore, the real-time processing unit 100 can correct the accumulated errors and perform more accurate position estimation by correcting the past position estimation results using the absolute position information recognized by the cloud server 200. To add to this, for such self-position correction, it is desirable that the position recognition of the mobile device by the cloud server 200 is more accurate than the position estimation performed by the mobile device itself. Therefore, in this embodiment, such position recognition is performed by the cloud server 200, which can perform highly accurate processing without the constraint of guaranteeing response time.

[0039] Next, the real-time processing unit 100 performs driving control that follows the moving path, based on the results of obstacle detection along the moving path by the real-time processing unit 100, the corrected position estimation results, and the path information of the moving path from the cloud server 200 (step S50).

[0040] As described above, in the mobile control system 1, the real-time processing unit 100 performs processing and control that requires guaranteed response time, such as position estimation, obstacle detection on the path, correction of position estimation results, and path-following driving along the movement path. On the other hand, the cloud server 200 performs processing that requires high accuracy but does not require immediate response, such as generating path information for the movement path and recognizing the position of the mobile device at a past point in time. In this way, the mobile control system 1 can achieve both high accuracy and high responsiveness in the processing and control of the mobile device by dividing the processing and control of the mobile device among devices with different characteristics. Therefore, the mobile control system 1 can perform highly safe mobile control.

[0041] Next, the internal configuration of the real-time processing unit 100 will be described using Figure 3. Figure 3 is a diagram showing an example configuration of the real-time processing unit 100 according to the embodiment. In Figure 3, a block diagram of the real-time processing unit 100 is shown. The real-time processing unit 100 includes the first sensor 110, control unit 120, second sensor 130, internal sensor 140, accelerator / brake / steering 150 shown in Figure 2, as well as a communication unit 160 and a storage unit 170. The processing performed by each part of the real-time processing unit 100 is real-time processing with the constraint of guaranteeing response time. However, some of the processing performed by each part of the real-time processing unit 100 may be non-real-time processing without the constraint of guaranteeing response time.

[0042] The communication unit 160 communicates with the real-time processing unit 100 and the network to send and receive predetermined information.

[0043] The storage unit 170 stores data such as sensor information with timestamps. The data stored in the storage unit 170 includes sensor information supplied from sensors provided in the real-time processing unit 100, such as the first sensor 110, and information supplied from the cloud server 200 via the communication unit 160. In addition, the data stored in the storage unit 170 is output to the control unit 120 and other units as appropriate for use, and is also output to the cloud server 200 via the communication unit 160 for use.

[0044] The control unit 120 includes a real-time processing unit 121 and a safety MCU 122. The real-time processing unit 121 includes a path tracking unit 1211, a self-position estimation unit 1212, a self-position correction unit 1213, and a timestamp assignment unit 1214.

[0045] The route following unit 1211 is responsible for processing the route information generated by the cloud server 200 so that the real-time processing unit 100 can perform route following driving.

[0046] The self-position estimation unit 1212 performs self-position estimation based on internal sensor information, which is sensor information related to the behavior of the real-time processing unit 100 acquired from the internal sensor 140. The behavior of the real-time processing unit 100 included in the internal sensor information is information such as the distance traveled and the speed of travel acquired by the wheel encoder, and acceleration information based on the IMU. The self-position estimation unit 1212 measures the relative speed and direction of travel of the real-time processing unit 100 using the internal sensor information and estimates the position of the real-time processing unit 100.

[0047] The self-position correction unit 1213 corrects the position of the real-time processing unit 100 estimated by the self-position estimation unit 1212 using the position recognition result of the real-time processing unit 100 obtained from the cloud server 200 via the communication unit. The cloud server 200 recognizes the position of the real-time processing unit 100 based on the first sensor information, the internal sensor information, and the timestamps of those sensor information. Therefore, the position recognition result of the real-time processing unit 100 by the cloud server 200 is the result at the time the timestamp was assigned. Thus, the position of the real-time processing unit 100 corrected by the self-position correction unit 1213 is also the position of the real-time processing unit 100 at the time the timestamp was assigned. This correction of the position of the real-time processing unit 100 performed by the self-position correction unit 1213 may also be called re-estimation of the position of the real-time processing unit 100. Furthermore, the path following unit 1211 may use the result of the re-estimation of the position to perform processing for path following driving.

[0048] The timestamp assignment unit 1214 assigns a timestamp to the first sensor information obtained from the first sensor 110 and the internal sensor information obtained from the internal sensor, indicating the time when each sensor information was acquired. By assigning a timestamp to the sensor information, the timestamp assignment unit 1214 enables the cloud server 200, for example, to perform more accurate processing by referring to the assigned timestamp when executing processing using the sensor information. Note that a timestamp may also be assigned to the second sensor information.

[0049] The safety MCU 122 includes an object detection unit 1221 and a safety mechanism 1222. The safety MCU 122 performs processes that require particular safety and urgency among the processes performed by the real-time processing unit 100.

[0050] The object detection unit 1221 uses the second sensor information acquired from the second sensor to detect objects that exist within the detection range of the second sensor, such as on or near the path the mobile device is traveling. The object detection performed by the object detection unit 1221 includes not only the perception of the presence of an object, but also recognition processing such as understanding the type of object, context, and predicting its behavior. For example, the object detection unit 1221 perceives the presence of a person and predicts their subsequent actions based on their movement history (direction of movement, acceleration) over the past few seconds, such as determining that the person is a pedestrian and is about to cross a crosswalk. Other functions include recognition and prediction of traffic light colors, road sign recognition, white line recognition, animal recognition such as bicycles, and behavior prediction.

[0051] The safety mechanism 1222 performs emergency control in emergencies, such as emergency control of the moving device, emergency avoidance, and instructions to the cloud server 200 to replan the travel route. The emergency control performed by the safety mechanism 1222 is based on information such as the travel speed of the real-time processing unit 100, the type of object detected by the object detection unit 1221 using second sensor information, and the distance from the real-time processing unit 100 to the object. The safety mechanism 1222 performs emergency control of the moving device when the route generation by the cloud server 200 is delayed, or when an object is present on the travel route of the real-time processing unit 100. As an example of emergency control, the safety mechanism 1222 may perform emergency control to stop the real-time processing unit 100 on or near the travel route.

[0052] An example of the database held by the memory unit 170 is shown in Figures 4 and 5. The database 171 in Figure 4 is a database that shows the correspondence between timestamps indicating the time when sensor information was acquired, each sensor information with a timestamp, and the results of self-position estimation based on each sensor information. Note that since the second sensor information is mainly used by the real-time processing unit 100, it is not essential that a timestamp is assigned to it. Also, for the sake of explanation, in Figure 4, the timing of sensor information acquisition for the first sensor, second sensor, and internal sensor is assumed to be unified, but the timing of sensor information acquisition for each sensor will differ unless they are synchronized. Therefore, different timestamps are usually assigned. Furthermore, as mentioned above, the first sensor, second sensor, and internal sensor include multiple sensors such as cameras and millimeter-wave radar, but here again for the sake of explanation, they will be described collectively as the first sensor, second sensor, and internal sensor. Furthermore, if, for example, the first sensor includes a camera and a millimeter-wave radar, then unless the timing of acquiring sensor information from the camera and the millimeter-wave radar is synchronized, the sensor information from the camera and the sensor information from the millimeter-wave radar will be assigned different timestamps.

[0053] The real-time processing unit 100 adds a timestamp indicating the time when the sensor information was assigned to each sensor, and then sends it to the cloud server 200. The cloud server 200 refers to the timestamp and performs various processing, such as position recognition of the mobile device based on the sensor information, at the time the sensor information was acquired. When the cloud server 200 performs various processing based on the sensor information, if a timestamp is not assigned to the sensor information, a time error will occur in the processing, such as recognizing the current self-position based on past sensor information. If a timestamp is assigned, it is possible to recognize the self-position at the past time when the timestamp was assigned, so no time error occurs in the processing.

[0054] The database 172 in Figure 5 stores route information generated by the cloud server 200 in the storage unit 170. For each route, the route information includes a target time and some or all of the information such as the target position, attitude, velocity, acceleration, angular velocity, and angular acceleration at that target time. If the route information includes all of this information, precise route tracking becomes possible, but on the other hand, it becomes necessary to refer to a lot of information. As mentioned above, this route tracking is performed by the real-time processing unit 100, which is constrained by the requirement of guaranteeing response time, so it may be difficult to control while referring to a lot of information within that constraint. In such cases, it is acceptable to refer to only the information included in the route information, for example, the target position, time, attitude, and velocity. Alternatively, all information may be referred to, but each piece of information may be weighted accordingly.

[0055] Furthermore, in the database 172 shown in Figure 5, for one route information R100, five target times T100 to T108 and target positions OP100 to 108 corresponding to each target time are set. Note that the cloud server 200 sets multiple target times and corresponding target positions for a single route information, but the number of target times set for a single route and the corresponding target positions are not limited to the example shown in Figure 5. The processing on the cloud server 200 side that generates the route information will be described later. The storage unit 170 stores the route information generated by the cloud server 200 in the database 172.

[0056] Next, the internal configuration of the cloud server 200 will be described using Figure 6. Figure 6 is a diagram showing an example configuration of a non-real-time processing unit (cloud server 200) according to the embodiment. In Figure 6, a block diagram of the cloud server 200 is shown. The cloud server 200 has a communication unit 210, a non-real-time processing unit 220, and a storage unit 230.

[0057] The communications unit communicates with the cloud server 200 and the network, sending and receiving various types of information between them.

[0058] The non-real-time processing unit 220 executes processing on the cloud server 200 and includes an acquisition unit 221, a position recognition unit 222, an object detection unit 223, and a path generation unit 224. The processing performed by the non-real-time processing unit is non-real-time processing without the constraint of guaranteeing response time, and processes based on various sensor information acquired from the real-time processing unit 100 and timestamps attached to the sensor information.

[0059] The memory unit 230 stores sensor information acquired from the real-time processing unit 100 and generates route information.

[0060] The acquisition unit 221 acquires various sensor information from the real-time processing unit 100 via the communication unit 210.

[0061] The position recognition unit 222 recognizes the position of the real-time processing unit 100 based on the sensor information of the first sensor 110 and the sensor information of the internal sensor 140 acquired by the acquisition unit 210, as well as the timestamps assigned to each sensor information. Therefore, the position recognition result of the position recognition unit 222 of the real-time processing unit 100 is more accurate than the position estimation result of the real-time processing unit 100 estimated by real-time processing based only on the internal sensor information. Furthermore, the position recognition result of the position recognition unit 222 of the real-time processing unit 100 is the past position of the real-time processing unit 100 at the time the timestamp was assigned. This refers to the sensor information of the first sensor 110 and the sensor information of the internal sensor 140.

[0062] The object detection unit 223 detects obstacles that exist within the detectable range of the first sensor 110, such as on or near the path the mobile device travels, based on the first sensor information acquired by the acquisition unit 210.

[0063] Examples of obstacles include other moving devices, pedestrians, animals, static objects such as trash and waste on the path, and traffic signs such as signals and signs. Furthermore, detection by the object detection unit 223 includes not only detecting the presence of an object, but also recognition processing such as understanding the type of object, context, and predicting its behavior. For example, it includes recognition processing of semantic information, such as detecting that the detected object is a pedestrian crossing a crosswalk or a sign indicating a speed limit of 50 km / h. Object detection also includes processing to predict subsequent actions based on the fact that the person is a pedestrian and is about to cross a crosswalk, and their movement history (direction of movement, acceleration) over the past few seconds. In addition, object detection also includes recognition and prediction of the color of traffic lights, road sign recognition, white line recognition, animal recognition such as bicycles, and behavior prediction. Furthermore, since object detection by the object detection unit 223 is performed by non-real-time processing without constraints on guaranteed response time, based on information from the first sensor 110, it can perform more accurate object detection compared to the processing performed by the object detection unit 1221, which is performed by real-time processing.

[0064] The route generation unit 224 generates route information based on the position of the real-time processing unit 100 recognized by the position recognition unit 222 and the object information detected by the object detection unit 223. The route information generated by the route generation unit 224 sets multiple target times and corresponding target locations for each route, but the number of target times and corresponding target locations set for a single route can be set as appropriate. For example, the route generation unit 224 may set one target time and corresponding target location for each route, or it may set 100 target times and corresponding target locations. For example, if the cloud server 200 is generating route information on a highway, since the road is wide and there are few curves, the length of one route may be set to 1 km, and five target times and corresponding target locations may be set for that route. On the other hand, when generating route information that passes through toll booths on an expressway, the route generation unit 224 may set the length of one route information to 100m and define 20 target times and corresponding target locations for that route. Furthermore, the route generation unit 224 may generate multiple route information at once, or if the real-time processing device 100 has already generated the next route information for the route it is currently traveling on, it may generate the next route after that.

[0065] Figure 7 shows database 231, an example of a database held by the storage unit 230, which shows information of the first sensor acquired from the first sensor 110. The information of the first sensor held in database 231 is accompanied by a timestamp corresponding to the time when the sensor information was acquired. For example, the position recognition unit 222 recognizes the position of the mobile device by referring to the sensor information of the first sensor 110 and the sensor information of the internal sensor 140 (not shown) stored in the storage unit 230, as well as the timestamps assigned to each sensor.

[0066] Figure 8 shows database 232, an example of a database held by the storage unit 230, which shows a route generated by the route generation unit 224. After the route generation unit 224 generates route information, the storage unit 230 stores the route information in the database. As described above, for each route, a target time is defined, and the route information includes some or all of the information such as the target position, attitude, speed, acceleration, angular velocity, and angular acceleration at that target time.

[0067] (1-2. An example of movement control according to the embodiment) Next, using Figure 9, we will explain the route-following control of the real-time processing unit 100 based on route information generated by the route generation unit 224 of the cloud server 200, along with the continuous time change from time t01 to time t02. Figure 9 is a diagram showing the control by which the real-time processing unit 100 receives route information from the cloud server 200 and travels in accordance with the route information. Route information R10 is route information generated by the cloud server 200, which the real-time processing unit 100 follows at time t01. Route information Rf is route information that the cloud server 200 plans to generate a predetermined time or distance before the real-time processing unit 100 reaches the end of route information R10, and is connected to the end of route information R10.

[0068] At time t01, the real-time processing unit 100 performs movement control following the route information R10 generated by the cloud server 200. At time t02, the cloud server 200 generates route information R11 corresponding to the route information Rf at time t01 and transmits it to the real-time processing unit 100. The real-time processing unit 100 receives the route information R11, and upon reaching the end of route information R10, performs movement control to follow the route information R11. In this way, the real-time processing unit 100 performs movement control that follows the route information while receiving route information generated by the cloud server 200 in a timely manner until it reaches its destination.

[0069] Next, using Figure 10, we will explain the process by which the real-time processing unit 100 corrects its position based on the position of the real-time processing unit 100 recognized by the cloud server 200. In Figure 10, times t10, t20, and t30 show continuous time changes. At time t10, the real-time processing unit 100 is traveling in accordance with the route information R20 generated by the cloud server 200. Based on the sensor information from the internal sensor 140, the real-time processing unit 100 estimates its own position to be position SP10. Position P10 is the actual position of the real-time processing unit 100, and as shown in Figure 10, there is a discrepancy between the actual position P10 of the real-time processing unit 100 and the position PS10 estimated by the self-position estimation unit 1212.

[0070] The cloud server 200 obtains sensor information from the real-time processing unit 100, specifically from the first sensor 110 and the internal sensor 140, which were acquired by the real-time processing unit 100 at time t10, and recognizes the position of the real-time processing unit 100 based on this sensor information. The position of the real-time processing unit 100 recognized by the cloud server 200 is the position of the real-time processing unit 100 corresponding to the timestamp attached to the sensor information of the first sensor 110 and the internal sensor 140. In other words, the position of the real-time processing unit 100 recognized by the cloud server 200 is the position at time t10, and at the time the recognition process is completed, it represents the past position of the real-time processing unit 100. The cloud server 200 transmits the recognized position of the real-time processing unit 100 to the real-time processing unit 100.

[0071] Next, at time t20, the real-time processing unit 100 corrects SP10, which is the estimated self-position result at time t10, based on the position of the real-time processing unit 100 recognized by the cloud server 200, and sets it to AP10. As mentioned above, the position of the real-time processing unit 100 recognized by the cloud server 200 is the position of the real-time processing unit 100 at time t10, so the corrected self-position estimation result is also the self-position estimation result at time t10. Here, P20 is the actual position of the real-time processing unit 100 at time t20, and SP20 is the estimated self-position result of the real-time processing unit 100 at time t20. At time t20, the real-time processing unit 100 recognizes that there was an error in the self-position estimated at time t10, but at time t20, it continues to drive following the route information R20 with that error accumulated.

[0072] Next, at time t30, the real-time processing unit 100 corrects the self-position SP20 estimated at time t20 to AP20 based on the corrected self-position AP10 at time t10. Since the real-time processing unit 100 estimates its own position using the internal sensor 140, any errors that occur will accumulate during travel. However, as shown in Figure 10, by using past positions recognized by the cloud server 200, this error can be reduced and an accurate self-position can be estimated.

[0073] Next, using Figures 11 to 17, the control by which the real-time processing unit 100 avoids collisions with objects on the path will be described. In the description of Figures 11 to 17, the detection of objects on the path may be performed by the object detection unit 1221 of the safety MCU 122 of the real-time processing unit 100, or by the object detection unit 223 of the non-real-time processing unit 220 of the cloud server 200. Similarly, the detection of the distance between the real-time processing unit 100 and objects on the path may also be performed by the object detection unit 1221 of the safety MCU 122 of the real-time processing unit 100, or by the object detection unit 223 of the non-real-time processing unit 220 of the cloud server 200. However, it is desirable that both the detection of objects on the path and the detection of the distance between the real-time processing unit 100 and objects on the path be performed by real-time processing.

[0074] First, using Figures 11 and 12, we will explain the case where the distance between the real-time processing unit 100 and the object on the path is sufficiently large. Here, "sufficiently large" means that there is enough time between the time the real-time processing unit 100 detects the object on the path and the time of collision for the real-time processing unit 100 to start driving based on the replanned path information. Replanning of the path information is done by the real-time processing unit 100 instructing the cloud server 200 to replan the path information, the cloud server 200 performing the replanning, and transmitting it to the real-time processing unit 100. The real-time processing unit 100 then starts driving following the replanned path information and drives in a manner that avoids the object on the path. The real-time processing unit 100 may also store data such as the average time required for the cloud server 200 to replan the path information. By comparing the average time required for the cloud server 200 to replan the path information with the time until collision calculated from the distance between the real-time processing unit 100 and the object on the path, it is possible to determine whether the distance is sufficient or not.

[0075] Figure 11 shows a situation where object H1 is located at a distance d1 from the real-time processing unit 100 on the path of route information R30, which the real-time processing unit 100 is following. In this case, if the real-time processing unit 100 continues to follow route information R30, it will collide with object H1 on the path. Therefore, the real-time processing unit 100 instructs the cloud server 200 to replan the route information, and the cloud server 200 replans route information R30, generates route information R31 that avoids object H1 on the path, and transmits it to the real-time processing unit 100. The real-time processing unit 100 can avoid object H1 on the path by following the received route information R31.

[0076] Figure 12 provides an overview of the situation in Figure 11. The horizontal axis in Figure 12 represents distance. Explanations that overlap with Figure 11 are omitted here. In Figure 12, distance d10 is the distance traveled by the real-time processing unit 100 until it comes to a stop when emergency braking is performed. Distance d20 is the distance traveled by the real-time processing unit 100 until the route information replanning is completed when the real-time processing unit 100 is decelerating due to emergency braking and the cloud server 200 is replanning the route information. Distance d30 is the distance traveled by the real-time processing unit 100 until it starts traveling in accordance with the replanned route information when the real-time processing unit 100 is decelerating due to emergency braking and the cloud server 200 is replanning the route information. Distance d40 indicates the distance traveled by the real-time processing unit 100 until the cloud server 200 completes replanning the route information, provided that the real-time processing unit 100 does not perform emergency braking. Distance d50 indicates the distance traveled by the real-time processing unit 100 until it begins traveling in accordance with the replanned route, provided that the cloud server 200 completes replanning the route information, provided that the real-time processing unit 100 does not perform emergency braking. Note that distances d10 to d50 will change depending on the speed of the real-time processing unit 100 and the road surface conditions of the route, so distances d10 to d50 only indicate the relative distance relationship with the position of objects on the route.

[0077] As described above, the distance d1 between the real-time processing unit 100 and the object H1 on the path, as shown in Figures 11 and 12, is greater than the distance d50. Therefore, even without the real-time processing unit 100 performing emergency braking, it can follow the replanned path information from the cloud server 200 and avoid a collision with the object H1 on the path.

[0078] Next, using Figures 13 and 14, we will explain the case where the distance between the real-time processing unit 100 and the object H2 on the path is greater than distance d30 and less than distance d40.

[0079] Figure 13 shows a situation where object H2 is located at a distance d2 from the real-time processing unit 100, on the path of route information R40, which the real-time processing unit 100 is following. In Figure 13, object H2 is located at a distance d2 from the real-time processing unit 100, on the path of route information R40, which the real-time processing unit 100 is following. In this case, if the real-time processing unit 100 continues to follow route information R40, it will collide with object H2 on the path. Therefore, the cloud server 200 replans route information R40, generates route information R41 that avoids object H2 on the path, and transmits it to the real-time processing unit 100. The real-time processing unit 100 can avoid object H2 on the path by following the received route information R41. However, in this case, if the real-time processing unit 100 does not decelerate by emergency braking, the real-time processing unit 100 will reach the distance d2 before the cloud server 200 finishes replanning the route information, and will collide with object H2. Therefore, in this case, the real-time processing unit 100 decelerates by emergency braking. By decelerating with emergency braking, the cloud server 200 completes the replanning of the route information before colliding with object H2 on the route, and the real-time processing unit 100 can avoid object H2 on the route. In this case, the real-time processing unit 100 could also avoid colliding with object H2 by stopping with emergency braking, but it is more preferable to decelerate with emergency braking and follow the route information R41, as this allows the unit to continue driving without stopping.

[0080] Figure 14 shows a situation where object H2 is located at a distance d2 from the real-time processing unit 100, on the path of the route information R40 that the real-time processing unit 100 is following. This figure provides an overview of the situation in Figure 13. The horizontal axis in Figure 14 represents distance. Explanations that overlap with Figure 13 are omitted here. As shown in Figure 14, the distance d2 from the real-time processing unit 100 to object H2 on the path is greater than the distance d30 and closer than the distance d40. In other words, even if the cloud server 200 generates route information R41 that avoids object H2, the real-time processing unit 100 cannot avoid a collision with object H2 unless it performs emergency braking. Therefore, the real-time processing unit 100 performs deceleration by emergency braking and follows the replanned route information R41. This allows the real-time processing unit 100 to avoid object H2 on the path and to continue driving without stopping.

[0081] Next, using Figures 15 and 16, we will explain the case where the distance between the real-time processing unit 100 and the object on the path is greater than distance d10 and less than distance d20.

[0082] In Figure 15, an object H3 exists on the path of the route information R50 that the real-time processing unit 100 is following, at a distance d3 from the real-time processing unit 100. In this case, if the real-time processing unit 100 continues to follow the route information R50, it will collide with object H3. Furthermore, at this distance, even if the real-time processing unit 100 performs emergency braking to decelerate, the cloud server 200 may not be able to complete the replanning of the route information before the real-time processing unit 100 collides with object H3. Therefore, in such cases, the real-time processing unit 100 performs emergency braking to stop. This makes it possible to avoid a collision with object H3 on the path.

[0083] Figure 16 is a bird's-eye view of the situation in Figure 15. Explanations that overlap with Figure 15 are omitted here. As shown in Figure 16, the distance d3 from the real-time processing unit 100 to object H3 on the path is greater than the distance d10 and closer than the distance d20. Given the distance to object H3, the real-time processing unit 100 will perform an emergency braking stop process because the cloud server 200 will not have enough time to replan the path information. This allows the real-time processing unit 100 to avoid a collision with object H3 on the path.

[0084] Next, using Figure 17, we will explain the case where the cloud server 200 does not respond, that is, the cloud server 200 does not plan the route information for some reason. The real-time processing unit 100 is traveling in accordance with the route information R60. The real-time processing unit 100 receives the next route information R61 from the cloud server 200 a predetermined time or distance before reaching the end of the route information R60, which is a distance d50 away, and continues traveling. However, if the cloud server 200 does not respond, the real-time processing unit 100 may not be able to receive the next route information R61 from the cloud server 200 a predetermined time or distance before reaching the end of the route information R60. In this case, if the real-time processing unit 100 reaches the end of the route information R60 while maintaining its speed, there is no further route information, and therefore the real-time processing unit 100 may become uncontrollable. Therefore, if the real-time processing unit 100 cannot receive the next route information R61 from the cloud server 200 before reaching the end of route information R60 at a predetermined time or distance, it controls the unit to stop at the end of route information R60. In this case, if the end of route information R60 is a dangerous location, such as a roadway, the safety mechanism 1222 of the real-time processing unit 100 may control the movement of the real-time processing unit 100 so that it can stop at a safe location.

[0085] (1-3. Procedure for movement control according to the embodiment) Next, the procedure for movement control according to the embodiment will be described using Figure 18. Figure 18 is a flowchart showing the processing flow according to the embodiment.

[0086] As shown in Figure 18, the real-time processing unit 100 acquires internal sensor information from the internal sensor 140 (step S60). Next, the real-time processing unit 100 acquires first sensor information from the first sensor 110 (step S61). Note that the acquisition of internal sensor information and the acquisition of first sensor information may be performed simultaneously, or the acquisition of first sensor information may be performed first.

[0087] Next, the real-time processing unit 100 assigns a timestamp to the first sensor information and the internal sensor information (step S62). Next, the real-time processing unit 100 estimates its own position using the internal sensor information (step S63).

[0088] The real-time processing unit 100 transmits the first sensor information and the internal sensor information, which have been time-stamped, to the cloud server 200 (step S64).

[0089] The cloud server 200 acquires map information (step S65). Next, the cloud server 200 receives time-stamped first sensor information and ambient sensor information from the real-time processing unit 100 (step S66). Note that the steps of the cloud server 200 acquiring map information and receiving time-stamped first sensor information and ambient sensor information are not limited to this order; they may be performed in reverse order or in parallel.

[0090] Next, the cloud server 200 recognizes the position of the real-time processing unit 100 based on the acquired first sensor information and internal sensor information (step S67). Furthermore, the cloud server 200 recognizes objects along the route information of the route the real-time processing unit 100 is traveling along, based on the acquired first sensor information (step S68). The order of steps S67 and S68 may also be reversed, or they may be performed in parallel.

[0091] Next, the cloud server 200 generates path information based on the results of object recognition along the path by the real-time processing unit 100 and the results of location recognition by the real-time processing unit 100 (step S69).

[0092] Then, the cloud server 200 sends the generated route information and the location recognition results from the real-time processing unit 100 to the real-time processing unit 100 (step 70).

[0093] Next, the real-time processing unit 100 receives route information and location recognition results from the cloud server 200 (step S71). Then, the real-time processing unit 100 corrects the location estimation result in step S63 based on the location recognition results received from the cloud server 200 (step S72). Here, the location estimation of the real-time processing unit 100 using the internal sensor information in step S63 can be performed at any timing as long as it is performed before step S72.

[0094] Then, the real-time processing unit 100 performs driving control that follows the route information based on the route information acquired in step S71 (step S73).

[0095] The above series of processes from step S60 to step S73 are repeatedly executed from the start of the driving control until the end of the driving control.

[0096] Next, a series of movement control procedures according to the embodiment will be described using Figure 19. Figure 19 is a sequence diagram showing the flow of a series of movement control procedures according to the embodiment.

[0097] First, the cloud server 200 transmits route information to the real-time processing unit 121 (step S80). Based on the received route information, the real-time processing unit 121 transmits information such as accelerator opening and steering angle to the accelerator / brake / steering unit 150 to follow the route (step S81). The accelerator / brake / steering unit 150 controls the accelerator and steering based on the received information such as accelerator opening and steering angle.

[0098] Next, the second sensor 130 transmits the second sensor information to the safety MCU 122 (step S82). Based on the received second sensor information, the safety MCU 122 detects obstacles in the path (step S83). If the safety MCU 122 detects an obstacle, it transmits information such as the accelerator opening and steering angle to the accelerator / brake / steering 150 to perform movement control to avoid a collision with the obstacle (step S84).

[0099] Next, the internal sensor 140 transmits the internal sensor information to the real-time processing unit 121 (step S85). Also, the first sensor 110 transmits the first sensor information to the real-time processing unit 121 (step S88).

[0100] The real-time processing unit 121 assigns a timestamp to the received first sensor information and internal sensor information (step S87). Furthermore, the real-time processing unit 121 transmits the timestamped first sensor information and internal sensor information to the cloud server 200 (step S88).

[0101] Next, the real-time processing unit 121 performs position estimation of the real-time processing unit 100 based on the internal sensor information (step S89). Note that the position estimation based on the internal sensor information performed by the real-time processing unit 121 can be performed even if a timestamp is not assigned, so it may be performed, for example, after step S85 and before step S87.

[0102] The cloud server 200 performs position recognition of the real-time processing unit 100 based on the received first sensor information and internal sensor information, as well as the timestamps assigned to the first sensor information and internal sensor information (step S90).

[0103] The cloud server 200 transmits the location recognition result to the real-time processing unit 121 (step S91). The real-time processing unit 121 corrects the location estimation result based on the received location recognition result (step S92).

[0104] Furthermore, the cloud server 200 recognizes objects necessary for controlling the movement of the real-time processing unit 100 (such as the color of a traffic light, road signs, white line recognition, and animal identification) based on the first sensor information (step S93). At this time, the cloud server 200 may also make predictions about the behavior of objects (such as the position of the object and the color of the traffic light after a predetermined time) based on the object's past time-series data (direction of movement, speed, acceleration, and signal lighting time).

[0105] Then, the cloud server 200 generates route information from the location recognition result in step S90 and the object recognition result in step S93 (step S94). At this time, in addition to the location recognition result and the object recognition result, the cloud server also stores map information and the route information, and the cloud server 200 sends this route information to the real-time processing unit 121 (step S95).

[0106] Based on the received route information, the real-time processing unit 121 transmits information such as accelerator opening and steering angle to the accelerator / brake / steering unit 150 to follow the route (step S96), and performs movement control according to the route information. Through this series of processes, the real-time processing unit 100 performs movement control.

[0107] Furthermore, the real-time processing unit 100 and the cloud server 200 appropriately execute steps S82 to S93 after steps S80 and S81, and before steps S95 and S96. That is, after acquiring the route information related to step S80, the real-time processing unit 100 performs each step from S82 to S92 at least once before acquiring the route information related to step S95. Similarly, after transmitting the route information related to step S80, the cloud server 200 performs each step from S90 to S94 at least once before generating the next route information related to step S94. Furthermore, the real-time processing unit 100 and the cloud server 200 do not need to perform each step from S82 to S93 the same number of times; for example, the detection of obstacles on the route related to steps S82 and S83 may be performed more times than the other steps.

[0108] (1-4. Modified examples according to the embodiment) The movement control according to this embodiment may be implemented in various other forms besides the embodiment described above. Therefore, other embodiments of the real-time processing device 100 will be described below.

[0109] In the embodiment, the non-real-time processing device was described as a cloud server 200. However, the non-real-time processing device is not limited to a cloud server 200, but may be a processor provided in a mobile device.

[0110] The following describes another embodiment of the real-time processing unit 100 with reference to Figure 20. In the example shown in Figure 20, the motion control system 1A has a real-time processing unit 100A, which comprises a non-real-time processing unit 200A, a first sensor 110A, a control unit 120A, a second sensor 130A, an internal sensor 140A, and an accelerator / brake / steering unit 150A. The control unit 120A also includes a real-time processing unit 121A and a safety MCU 122A.

[0111] The non-real-time processing unit 200A performs the same processing as the cloud server 200 shown in Figure 1. The first sensor 110A corresponds to the first sensor 110 shown in Figure 1. The control unit 120A corresponds to the control unit 120 shown in Figure 1. The second sensor 130A corresponds to the second sensor 130 shown in Figure 1. The internal sensor 140A corresponds to the internal sensor 140 shown in Figure 1. The accelerator / brake / steering unit 150A corresponds to the accelerator / brake / steering unit 150 shown in Figure 1. The real-time processing unit 121A corresponds to the real-time processing unit 121 shown in Figure 1. The safety MCU 122A corresponds to the safety MCU 122 shown in Figure 1.

[0112] As described above, when the non-real-time processing unit 200A is configured as a device provided in the processing unit 100A, the communication of data related to a series of movement control operations is completed within the processing unit. This has the effect of making the real-time processing unit 100A less susceptible to the effects of communication delays and the like.

[0113] (2. Other Embodiments) (2-1. Configuration of the mobile device) Figure 21 is a block diagram showing an example configuration of a vehicle control system 11, which is an example of a mobile device control system to which this technology is applied.

[0114] The vehicle control system 11 is installed in the vehicle 10 and performs processing related to driving assistance and autonomous driving of the vehicle 10.

[0115] The vehicle control system 11 includes a vehicle control ECU (Electronic Control Unit) 21, a communication unit 22, a map information storage unit 23, a GNSS (Global Navigation Satellite System) receiver unit 24, an external recognition sensor 25, an in-vehicle sensor 26, a vehicle sensor 27, a recording unit 28, a driving assistance / automatic driving control unit 29, a DMS (Driver Monitoring System) 30, an HMI (Human Machine Interface) 31, and a vehicle control unit 32.

[0116] The vehicle control ECU 21, communication unit 22, map information storage unit 23, GNSS receiver unit 24, external recognition sensor 25, in-vehicle sensor 26, vehicle sensor 27, recording unit 28, driving support / autonomous driving control unit 29, driver monitoring system (DMS) 30, human-machine interface (HMI) 31, and vehicle control unit 32 are interconnected and can communicate with each other via a communication network 41. The communication network 41 consists of an in-vehicle communication network or bus that conforms to digital bidirectional communication standards such as CAN (Controller Area Network), LIN (Local Interconnect Network), LAN (Local Area Network), FlexRay (registered trademark), and Ethernet (registered trademark). The communication network 41 may be used depending on the type of data being communicated; for example, CAN may be used for data related to vehicle control, and Ethernet may be used for large-capacity data. In addition, the various components of the vehicle control system 11 may be directly connected using wireless communication technologies intended for relatively short-range communication, such as Near Field Communication (NFC) or Bluetooth®, without going through the communication network 41.

[0117] In the following, when each part of the vehicle control system 11 communicates via the communication network 41, the description of the communication network 41 will be omitted. For example, when the vehicle control ECU 21 and the communication unit 22 communicate via the communication network 41, it will simply be described as the vehicle control ECU 1 and the communication unit 22 communicating.

[0118] The vehicle control ECU 21 is composed of various processors, such as a CPU (Central Processing Unit) and an MPU (Micro Processing Unit). The vehicle control ECU 21 controls the functions of the entire vehicle control system 11 or a part of it.

[0119] The communication unit 22 communicates with various devices inside and outside the vehicle, other vehicles, servers, base stations, etc., and transmits and receives various types of data. At this time, the communication unit 22 can communicate using multiple communication methods.

[0120] A brief explanation will be given regarding the external communication capabilities of the communication unit 22. The communication unit 22 communicates with servers (hereinafter referred to as "external servers") located on an external network via a base station or access point using wireless communication methods such as 5G (fifth-generation mobile communication system), LTE (Long Term Evolution), and DSRC (Dedicated Short Range Communications). The external network with which the communication unit 22 communicates is, for example, the internet, a cloud network, or a network specific to a carrier. The communication method used by the communication unit 22 to communicate with the external network is not particularly limited, as long as it is a wireless communication method capable of digital two-way communication at a predetermined communication speed and over a predetermined distance.

[0121] Furthermore, for example, the communication unit 22 can communicate with terminals located near the vehicle using P2P (Peer To Peer) technology. Terminals located near the vehicle include, for example, terminals worn by mobile objects that move at relatively low speeds, such as pedestrians and cyclists, terminals that are fixedly installed in places such as stores, or MTC (Machine Type Communication) terminals. In addition, the communication unit 22 can also perform V2X communication. V2X communication refers to communication between the vehicle and other entities, such as vehicle-to-vehicle communication with other vehicles, vehicle-to-infrastructure communication with roadside devices, etc., vehicle-to-home communication with a house, and vehicle-to-pedestrian communication with terminals carried by pedestrians, etc.

[0122] The communication unit 22 can, for example, receive programs from an external source (over the air) to update the software that controls the operation of the vehicle control system 11. The communication unit 22 can also receive map information, traffic information, information about the vehicle 10's surroundings, etc., from an external source. Furthermore, the communication unit 22 can transmit information about the vehicle 10 and information about the vehicle 10's surroundings to an external source. Information about the vehicle 10 that the communication unit 22 transmits to an external source includes, for example, data indicating the status of the vehicle 10 and recognition results from the recognition unit 73. Furthermore, the communication unit 22 can perform communications corresponding to vehicle emergency notification systems such as e-Call.

[0123] A brief overview of the communication capabilities of the communication unit 22 with the vehicle interior will be provided. The communication unit 22 can communicate with various devices in the vehicle, for example, using wireless communication. The communication unit 22 can communicate wirelessly with devices in the vehicle using communication methods that enable digital bidirectional communication at a predetermined or higher communication speed via wireless communication, such as Wi-Fi, Bluetooth, NFC, and WUSB (Wireless USB). Not limited to these, the communication unit 22 can also communicate with various devices in the vehicle using wired communication. For example, the communication unit 22 can communicate with various devices in the vehicle via wired communication through a cable connected to a connection terminal (not shown). The communication unit 22 can communicate with various devices in the vehicle using communication methods that enable digital bidirectional communication at a predetermined or higher communication speed via wired communication, such as USB (Universal Serial Bus), HDMI (High-Definition Multimedia Interface) (registered trademark), and MHL (Mobile High-definition Link).

[0124] Here, "devices inside the vehicle" refers to, for example, devices inside the vehicle that are not connected to the communication network 41. Examples of devices inside the vehicle include mobile devices and wearable devices carried by passengers such as the driver, and information devices that are brought into the vehicle and temporarily installed.

[0125] For example, the communication unit 22 receives electromagnetic waves transmitted by road traffic information communication systems (VICS (Vehicle Information and Communication System) (registered trademark)) such as radio beacons, optical beacons, and FM multiplex broadcasting.

[0126] The map information storage unit 23 stores either or both maps acquired from external sources and maps created by the vehicle 10. For example, the map information storage unit 23 stores three-dimensional high-precision maps, global maps with lower precision than high-precision maps but covering a wide area, and so on.

[0127] High-precision maps include, for example, dynamic maps, point cloud maps, and vector maps. A dynamic map is, for example, a map consisting of four layers: dynamic information, semi-dynamic information, semi-static information, and static information, and is provided to the vehicle 10 from an external server. A point cloud map is a map composed of point clouds (point cloud data). Here, a vector map refers to a map adapted for ADAS (Advanced Driver Assistance System) that maps traffic information such as the location of lanes and traffic lights to a point cloud map.

[0128] Point cloud maps and vector maps may be provided, for example, from an external server, or they may be created in the vehicle 10 as maps for matching with the local map described later, based on sensing results from radar 52, LiDAR 53, etc., and stored in the map information storage unit 23. In addition, if high-precision maps are provided from an external server, in order to reduce communication capacity, map data of, for example, several hundred square meters relating to the planned route that the vehicle 10 will travel will be acquired from the external server.

[0129] The location information acquisition unit 24 receives GNSS signals from GNSS satellites and acquires the location information of the vehicle 10. The received GNSS signals are supplied to the driving support / automatic driving control unit 29. The location information acquisition unit 24 is not limited to using GNSS signals; for example, it may acquire location information using beacons.

[0130] The external recognition sensor 25 is equipped with various sensors used to recognize the external conditions of the vehicle 10, and supplies sensor data from each sensor to various parts of the vehicle control system 11. The types and number of sensors equipped in the external recognition sensor 25 are arbitrary.

[0131] For example, the external recognition sensor 25 includes a camera 51, a radar 52, a LiDAR (Light Detection and Ranging, Laser Imaging Detection and Ranging) 53, and an ultrasonic sensor 54. However, the external recognition sensor 25 may also be configured to include one or more of the cameras 51, radar 52, LiDAR 53, and ultrasonic sensor 54. The number of cameras 51, radar 52, LiDAR 53, and ultrasonic sensor 54 is not particularly limited as long as it is a number that can be realistically installed in the vehicle 10. Furthermore, the types of sensors included in the external recognition sensor 25 are not limited to this example, and the external recognition sensor 25 may include other types of sensors. Examples of the sensing areas of each sensor included in the external recognition sensor 25 will be described later.

[0132] The shooting method of camera 51 is not particularly limited as long as it is a shooting method capable of distance measurement. For example, camera 51 can be a camera of various shooting methods such as a ToF (Time Of Flight) camera, a stereo camera, a monocular camera, or an infrared camera, as needed. In addition, camera 51 may simply be for acquiring images, regardless of distance measurement.

[0133] Furthermore, for example, the external recognition sensor 25 may include an environmental sensor for detecting the environment relative to the vehicle 10. The environmental sensor is a sensor for detecting the environment such as weather, climate, and brightness, and may include various sensors such as a raindrop sensor, fog sensor, sunshine sensor, snow sensor, and illuminance sensor.

[0134] Furthermore, for example, the external recognition sensor 25 includes a microphone used for detecting sounds around the vehicle 10 and the location of sound sources.

[0135] The in-vehicle sensor 26 is equipped with various sensors for detecting information inside the vehicle and supplies sensor data from each sensor to various parts of the vehicle control system 11. The types and number of sensors equipped with the in-vehicle sensor 26 are not particularly limited as long as the number can realistically be installed in the vehicle 10.

[0136] For example, the in-vehicle sensor 26 can be equipped with one or more sensors from among a camera, radar, seat sensor, steering wheel sensor, microphone, and biosensor. The camera equipped in the in-vehicle sensor 26 can be a camera of various imaging types capable of distance measurement, such as a ToF camera, stereo camera, monocular camera, or infrared camera. However, it is not limited to these, and the camera equipped in the in-vehicle sensor 26 may be one that simply acquires images, regardless of distance measurement. The biosensor equipped in the in-vehicle sensor 26 is installed, for example, on the seat or steering wheel, and detects various biometric information of the driver or other passengers.

[0137] The vehicle sensor 27 is equipped with various sensors for detecting the state of the vehicle 10 and supplies sensor data from each sensor to various parts of the vehicle control system 11. The types and number of sensors equipped with the vehicle sensor 27 are not particularly limited as long as the number can realistically be installed on the vehicle 10.

[0138] For example, the vehicle sensor 27 includes a speed sensor, an acceleration sensor, an angular velocity sensor (gyro sensor), and an inertial measurement unit (IMU) that integrates them. For example, the vehicle sensor 27 includes a steering angle sensor for detecting the steering angle of the steering wheel, a yaw rate sensor, an accelerator sensor for detecting the amount of operation of the accelerator pedal, and a brake sensor for detecting the amount of operation of the brake pedal. For example, the vehicle sensor 27 includes a rotation sensor for detecting the rotation speed of the engine or motor, an air pressure sensor for detecting the air pressure of the tires, a slip ratio sensor for detecting the slip ratio of the tires, and a wheel speed sensor for detecting the rotation speed of the wheels. For example, the vehicle sensor 27 includes a battery sensor for detecting the remaining charge and temperature of the battery, and an impact sensor for detecting external impacts.

[0139] The recording unit 28 includes at least one of a non-volatile storage medium and a volatile storage medium, and stores data and programs. The recording unit 28 can be used as, for example, an EEPROM (Electrically Erasable Programmable Read Only Memory) and a RAM (Random Access Memory), and the storage medium can be a magnetic storage device such as an HDD (Hard Disk Drive), a semiconductor storage device, an optical storage device, or a magneto-optical storage device. The recording unit 28 records various programs and data used by each part of the vehicle control system 11. For example, the recording unit 28 includes an EDR (Event Data Recorder) and a DSSAD (Data Storage System for Automated Driving), and records information about the vehicle 10 before and after an event such as an accident, as well as biometric information acquired by the in-vehicle sensors 26.

[0140] The driving assistance / automatic driving control unit 29 controls the driving assistance and automatic driving of the vehicle 10. For example, the driving assistance / automatic driving control unit 29 includes an analysis unit 61, an action planning unit 62, and an operation control unit 63.

[0141] The analysis unit 61 performs analysis processing on the vehicle 10 and its surroundings. The analysis unit 61 includes a self-position estimation unit 71, a sensor fusion unit 72, and a recognition unit 73.

[0142] The self-position estimation unit 71 estimates the vehicle 10's position based on sensor data from the external recognition sensor 25 and a high-precision map stored in the map information storage unit 23. For example, the self-position estimation unit 71 generates a local map based on the sensor data from the external recognition sensor 25 and estimates the vehicle 10's position by matching the local map with the high-precision map. The position of the vehicle 10 is based on, for example, the center of the rear wheel relative to the axle.

[0143] Local maps are, for example, three-dimensional high-precision maps created using technologies such as SLAM (Simultaneous Localization and Mapping), or occupancy grid maps. Three-dimensional high-precision maps are, for example, the point cloud maps mentioned above. Occupancy grid maps divide the three-dimensional or two-dimensional space around the vehicle 10 into grids of a predetermined size and show the occupancy status of objects on a grid-by-grid basis. The occupancy status of objects is indicated, for example, by the presence or absence of an object or the probability of its existence. Local maps are also used, for example, in the detection and recognition processing of the external conditions of the vehicle 10 by the recognition unit 73.

[0144] The self-position estimation unit 71 may estimate the vehicle 10's position based on the GNSS signal and sensor data from the vehicle sensor 27.

[0145] The sensor fusion unit 72 performs sensor fusion processing to obtain new information by combining multiple different types of sensor data (for example, image data supplied from the camera 51 and sensor data supplied from the radar 52). Methods for combining different types of sensor data include integration, fusion, and union.

[0146] The recognition unit 73 performs a detection process to detect the external conditions of the vehicle 10, and a recognition process to recognize the external conditions of the vehicle 10.

[0147] For example, the recognition unit 73 performs detection and recognition processing of the external conditions of the vehicle 10 based on information from the external recognition sensor 25, information from the self-position estimation unit 71, information from the sensor fusion unit 72, etc.

[0148] Specifically, for example, the recognition unit 73 performs detection and recognition processing of objects around the vehicle 10. Object detection processing includes, for example, detecting the presence, size, shape, position, and movement of objects. Object recognition processing includes, for example, recognizing attributes such as the type of object or identifying a specific object. However, detection processing and recognition processing are not necessarily clearly separated and may overlap.

[0149] For example, the recognition unit 73 detects objects around the vehicle 10 by performing clustering, which classifies the point cloud based on sensor data from LiDAR 53 or radar 52 into clusters of points. This allows the presence, size, shape, and position of objects around the vehicle 10 to be detected.

[0150] For example, the recognition unit 73 detects the movement of objects around the vehicle 10 by performing tracking that follows the movement of clusters of points classified by clustering. This allows the speed and direction of travel (movement vector) of objects around the vehicle 10 to be detected.

[0151] For example, the recognition unit 73 detects or recognizes vehicles, people, bicycles, obstacles, structures, roads, traffic lights, traffic signs, road markings, etc., from the image data supplied from the camera 51. It may also recognize the types of objects around the vehicle 10 by performing recognition processing such as semantic segmentation.

[0152] For example, the recognition unit 73 can perform recognition processing of traffic rules around the vehicle 10 based on the map stored in the map information storage unit 23, the self-position estimation result by the self-position estimation unit 71, and the recognition result of objects around the vehicle 10 by the recognition unit 73. Through this processing, the recognition unit 73 can recognize the location and status of traffic signals, the content of traffic signs and road markings, the content of traffic regulations, and the lanes that can be driven on.

[0153] For example, the recognition unit 73 can perform recognition processing of the environment surrounding the vehicle 10. The surrounding environment that the recognition unit 73 is intended to recognize may include weather, temperature, humidity, brightness, and road surface conditions.

[0154] The action planning unit 62 creates an action plan for the vehicle 10. For example, the action planning unit 62 creates an action plan by performing route planning and route following processes.

[0155] Global path planning is the process of planning a rough route from the start to the goal. This path planning also includes a process called local path planning, which involves generating a track that allows the vehicle 10 to move safely and smoothly in its vicinity, taking into account the motion characteristics of the vehicle 10 along the planned path. Path planning may be distinguished as long-term path planning, and starting point generation as short-term path planning or local path planning. Safety-prioritized paths represent a similar concept to starting point generation, short-term path planning, or local path planning.

[0156] Route following is the process of planning actions to safely and accurately travel the route planned by route planning within a planned time. The action planning unit 62 can, for example, calculate the target speed and target angular velocity of the vehicle 10 based on the results of this route following process.

[0157] The motion control unit 63 controls the operation of the vehicle 10 in order to realize the action plan created by the action planning unit 62.

[0158] For example, the motion control unit 63 controls the steering control unit 81, brake control unit 82, and drive control unit 83, which are included in the vehicle control unit 32 described later, to perform acceleration / deceleration control and direction control so that the vehicle 10 moves along the trajectory calculated by the trajectory plan. For example, the motion control unit 63 performs coordinated control for the purpose of realizing ADAS functions such as collision avoidance or impact mitigation, follow driving, vehicle speed maintenance, collision warning for the vehicle, and lane departure warning for the vehicle. For example, the motion control unit 63 performs coordinated control for the purpose of autonomous driving, such as driving autonomously without driver operation.

[0159] The DMS30 performs driver authentication and driver status recognition based on sensor data from the in-vehicle sensors 26 and input data input to the HMI31, which will be described later. In this case, the driver status to be recognized by the DMS30 is expected to include, for example, physical condition, level of alertness, level of concentration, level of fatigue, gaze direction, level of intoxication, driving operation, and posture.

[0160] Furthermore, the DMS30 may perform authentication processing for passengers other than the driver and recognition processing for the status of said passengers. Also, for example, the DMS30 may perform recognition processing of the conditions inside the vehicle based on sensor data from the in-vehicle sensor 26. Examples of conditions inside the vehicle to be recognized include temperature, humidity, brightness, and odor.

[0161] HMI31 handles the input of various data and instructions, and presents various data to the driver and other users.

[0162] A brief explanation of data input by HMI31 is provided. HMI31 is equipped with an input device for human data input. HMI31 generates input signals based on data and instructions input by the input device and supplies them to each part of the vehicle control system 11. HMI31 is equipped with operators such as a touch panel, buttons, switches, and levers as input devices. However, HMI31 may also be equipped with input devices that allow information to be input by methods other than manual operation, such as voice or gestures. Furthermore, HMI31 may use external connected devices such as a remote control device using infrared or radio waves, or a mobile device or wearable device that corresponds to the operation of the vehicle control system 11, as input devices.

[0163] A brief overview of data presentation by HMI31 is provided below. HMI31 generates visual, auditory, and tactile information for the occupant or outside the vehicle. HMI31 also performs output control, managing the output, output content, output timing, and output method of each of these generated pieces of information. As visual information, HMI31 generates and outputs information indicated by images and light, such as operation screens, vehicle status displays, warning displays, and monitor images showing the surroundings of vehicle 10. As auditory information, HMI31 generates and outputs information indicated by sound, such as voice guidance, warning sounds, and warning messages. Furthermore, as tactile information, HMI31 generates and outputs information that is applied to the occupant's sense of touch, such as force, vibration, and movement.

[0164] As output devices for visual information output by HMI31, for example, a display device that presents visual information by displaying images itself, or a projector device that presents visual information by projecting images, can be applied. In addition to display devices with ordinary displays, the display device may also be a device that displays visual information within the passenger's field of view, such as a head-up display, a transparent display, or a wearable device with AR (Augmented Reality) functionality. Furthermore, HMI31 can also use display devices such as navigation devices, instrument panels, CMS (Camera Monitoring System), electronic mirrors, and lamps installed in the vehicle 10 as output devices for visual information output.

[0165] For HMI31, output devices that output auditory information can include, for example, audio speakers, headphones, and earphones.

[0166] As an output device for HMI31 to output tactile information, for example, a haptic element using haptic technology can be applied. The haptic element is installed in parts of the vehicle 10 that are in contact with by the occupants, such as the steering wheel and the seat.

[0167] The vehicle control unit 32 controls various parts of the vehicle 10. The vehicle control unit 32 includes a steering control unit 81, a brake control unit 82, a drive control unit 83, a body system control unit 84, a light control unit 85, and a horn control unit 86.

[0168] The steering control unit 81 detects and controls the state of the steering system of the vehicle 10. The steering system includes, for example, a steering mechanism with a steering wheel, an electric power steering system, etc. The steering control unit 81 includes, for example, a control unit such as an ECU that controls the steering system, an actuator that drives the steering system, etc.

[0169] The brake control unit 82 detects and controls the state of the vehicle's brake system. The brake system includes, for example, a brake mechanism including a brake pedal, an ABS (Antilock Brake System), a regenerative braking mechanism, etc. The brake control unit 82 also includes, for example, a control unit such as an ECU that controls the brake system.

[0170] The drive control unit 83 detects and controls the state of the vehicle 10's drive system. The drive system includes, for example, an accelerator pedal, a drive force generating device for generating driving force such as an internal combustion engine or drive motor, and a drive force transmission mechanism for transmitting driving force to the wheels. The drive control unit 83 also includes, for example, a control unit such as an ECU that controls the drive system.

[0171] The body system control unit 84 detects and controls the state of the body system of the vehicle 10. The body system includes, for example, a keyless entry system, a smart key system, power window devices, power seats, an air conditioning system, airbags, seat belts, a shift lever, etc. The body system control unit 84 also includes, for example, a control unit such as an ECU that controls the body system.

[0172] The light control unit 85 detects and controls the status of various lights on the vehicle 10. Examples of lights to be controlled include headlights, taillights, fog lights, turn signals, brake lights, projection lights, and bumper displays. The light control unit 85 includes a control unit such as an ECU that controls the lights.

[0173] The horn control unit 86 detects and controls the status of the vehicle's car horn. The horn control unit 86 includes, for example, a control unit such as an ECU that controls the car horn.

[0174] Figure 22 shows an example of the sensing area of ​​the external recognition sensor 25 in Figure 21, including the camera 51, radar 52, LiDAR 53, and ultrasonic sensor 54. In Figure 22, the vehicle 10 is schematically shown as viewed from above, with the left end being the front end of the vehicle 10 and the right end being the rear end of the vehicle 10.

[0175] Sensing regions 101F and 101B show examples of sensing regions of the ultrasonic sensor 54. Sensing region 101F covers the area around the front end of the vehicle 10 by multiple ultrasonic sensors 54. Sensing region 101B covers the area around the rear end of the vehicle 10 by multiple ultrasonic sensors 54.

[0176] The sensing results in sensing area 101F and sensing area 101B are used, for example, for parking assistance of vehicle 10.

[0177] Sensing areas 102F to 102B show examples of sensing areas for short-range or medium-range radar 52. Sensing area 102F covers a position further in front of the vehicle 10 than sensing area 101F. Sensing area 102B covers a position further in rear of the vehicle 10 than sensing area 101B. Sensing area 102L covers the rear periphery of the left side of the vehicle 10. Sensing area 102R covers the rear periphery of the right side of the vehicle 10.

[0178] The sensing results in sensing region 102F are used, for example, to detect vehicles or pedestrians in front of vehicle 10. The sensing results in sensing region 102B are used, for example, to prevent collisions behind vehicle 10. The sensing results in sensing regions 102L and 102R are used, for example, to detect objects in blind spots to the sides of vehicle 10.

[0179] Sensing areas 103F to 103B show examples of sensing areas by camera 51. Sensing area 103F covers a position further in front of vehicle 10 than sensing area 102F. Sensing area 103B covers a position further in rear of vehicle 10 than sensing area 102B. Sensing area 103L covers the periphery of the left side of vehicle 10. Sensing area 103R covers the periphery of the right side of vehicle 10.

[0180] The sensing results in sensing region 103F can be used, for example, for recognition of traffic lights and traffic signs, lane departure prevention support systems, and automatic headlight control systems. The sensing results in sensing region 103B can be used, for example, for parking assistance and surround view systems. The sensing results in sensing regions 103L and 103R can be used, for example, for surround view systems.

[0181] Sensing area 104 shows an example of the sensing area of ​​LiDAR 53. Sensing area 104 covers a position further in front of the vehicle 10 than sensing area 103F. On the other hand, sensing area 104 has a narrower range in the left-right direction than sensing area 103F.

[0182] The sensing results in the sensing region 104 can be used, for example, to detect objects such as surrounding vehicles.

[0183] Sensing area 105 shows an example of the sensing area of ​​the long-range radar 52. Sensing area 105 covers a position further in front of the vehicle 10 than sensing area 104. On the other hand, sensing area 105 has a narrower range in the left-right direction than sensing area 104.

[0184] The sensing results in sensing area 105 are used, for example, for ACC (Adaptive Cruise Control), emergency braking, collision avoidance, etc.

[0185] Furthermore, the sensing areas of the camera 51, radar 52, LiDAR 53, and ultrasonic sensor 54 included in the external recognition sensor 25 may take various configurations other than those shown in Figure 2. Specifically, the ultrasonic sensor 54 may be configured to sense the sides of the vehicle 10, or the LiDAR 53 may be configured to sense the rear of the vehicle 10. Also, the installation positions of each sensor are not limited to the examples described above. In addition, there may be one or more sensors.

[0186] The correspondence between the vehicle 10 in Figure 21 and the real-time processing unit 100 in Figure 3 is as follows: The vehicle 10 and the vehicle control system 11 correspond to the real-time processing unit 100. The processor 21 corresponds to the control unit 120, the real-time processing unit 121, and the safety MCU 122. The communication unit 22 corresponds to the communication unit 160. The map information storage unit 23 corresponds to the storage unit 170. The GNSS receiver 24 corresponds to the first sensor 110 and the GPS 113. The external recognition sensor 25 corresponds to the first sensor 110 and the second sensor 130. The in-vehicle sensor 26 and the vehicle sensor 27 correspond to the interior sensor 140. The recording unit 28 corresponds to the storage unit 170. The driving assistance / autonomous driving control unit 29 corresponds to the control unit 120, the real-time processing unit 121, and the safety MCU 122. The analysis unit 61 corresponds to the control unit 120, the real-time processing unit 121, and the safety MCU 122. The self-position estimation unit 71 corresponds to the self-position estimation unit 1212 and the self-position correction unit 1213. The sensor fusion unit 72 corresponds to the control unit 120, the real-time processing unit 121, and the safety MCU 122. The recognition unit 73 corresponds to the safety MCU 122 and the object detection unit 1221. The action planning unit 62 corresponds to the safety mechanism 1222 and the path following unit 1211. The motion control unit 63 corresponds to the path following unit 1211, the safety mechanism 1222, and the accelerator / brake / steering unit 150. The DMS 30 corresponds to the memory unit 170. The HMI 31 corresponds to the control unit 120. The vehicle control unit 32 corresponds to the control unit 120.

[0187] (2-2. Others) Of the processes described in each of the above embodiments, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically by known methods. In addition, the processing procedures, specific names, and information including various data and parameters shown in the above documents and drawings can be changed at will unless otherwise specified. For example, the various information shown in each figure is not limited to the information shown.

[0188] Furthermore, the components of each illustrated device are functionally conceptual and do not necessarily need to be physically configured as shown. In other words, the specific forms of distribution and integration of each device are not limited to those shown, and all or part of them can be functionally or physically distributed and integrated in any unit according to various loads and usage conditions.

[0189] Furthermore, the embodiments and modifications described above can be combined as appropriate, provided that the processing content is not inconsistent. In addition, although an automobile was given as an example of a mobile body in the above embodiments, the information processing disclosed herein is applicable to mobile bodies other than automobiles. For example, the mobile body may be a small vehicle such as a motorcycle or a three-wheeled vehicle, a large vehicle such as a bus or a truck, or an autonomous mobile body such as a robot or a drone.

[0190] Furthermore, the effects described herein are merely illustrative and not limiting; other effects may also occur.

[0191] (3. Effects of the motion control system related to this disclosure) As described above, the mobile control system 1 according to this disclosure comprises a real-time processing unit 100 and a non-real-time processing unit (cloud server 200), and the real-time processing unit 100 and the non-real-time processing unit communicate with each other. The real-time processing unit 100 performs real-time processing with the constraint of guaranteeing response time, and the non-real-time processing unit performs non-real-time processing without the constraint of guaranteeing response time. The real-time processing unit 100 may be the mobile device itself or may be installed in the mobile device.

[0192] The non-real-time processing unit performs various processes using sensor information acquired from the real-time processing unit 100 through non-real-time processing. For example, the non-real-time processing unit generates route information for the real-time processing unit 100's travel path. The real-time processing unit 100 then performs travel control that follows the route information through real-time processing.

[0193] Thus, the mobility control system 1 according to this disclosure generates route information using a non-real-time processing unit and executes driving that follows the route information using a real-time processing unit 100. As a result, the mobility control system 1 can perform highly safe mobility control that combines high accuracy and responsiveness.

[0194] Furthermore, the real-time processing unit 100 detects objects on or near its travel path based on sensor information and performs emergency control, which includes either emergency braking control of the real-time processing unit 100 or an instruction to the cloud server 200 to replan the travel path (one or more of the above). The real-time processing unit 100 also performs emergency control if it cannot communicate with the cloud server 200 or if the cloud server 200 is not functioning properly. This further enhances safety in emergency situations.

[0195] Furthermore, the real-time processing unit 100 assigns a timestamp to the sensor information and transmits the timestamped sensor information to the cloud server 200. Based on the timestamped sensor information, the cloud server 200 recognizes the location of the real-time processing unit 100 and generates route information for the real-time processing unit 100. This enables accurate information processing without temporal errors by performing information processing at the time the timestamp was assigned.

[0196] Furthermore, the real-time processing unit 100 corrects the position estimation result based on the position of the real-time processing unit 100 recognized by the cloud server 200. In addition, the real-time processing unit 100 re-estimates the position of the real-time processing unit 100 at the current time using the corrected position estimation result. As a result, the real-time processing unit 100 can estimate its position more accurately and drive in accordance with the route information more accurately.

[0197] Furthermore, the movement control device according to this disclosure (real-time processing device 100 in the embodiment) includes a communication unit (communication unit 160 in the embodiment) that communicates with a first processing unit (non-real-time processing unit 220 or non-real-time processing unit 200A in the embodiment), and a second processing unit (real-time processing unit 121 in the embodiment) that controls the movement of the movement device. The communication unit receives path information from the first processing unit, which generates path information for controlling the movement path of the movement device using sensor information acquired from the second processing unit based on non-real-time processing that has no constraints on the response time required for processing. The second processing unit controls the movement of the movement device in accordance with the path information received by the communication unit from the first processing unit, based on real-time processing that has constraints on the response time required for processing. In this way, the movement control device can perform highly safe movement control that achieves both high accuracy and responsiveness by controlling movement based on path information generated with high accuracy based on non-real-time processing.

[0198] Furthermore, the information processing device according to this disclosure (cloud server 200 in the embodiment) includes a communication unit (communication unit 210 in the embodiment) that communicates with a second processing unit (real-time processing unit 121 in the embodiment), and a first processing unit (non-real-time processing unit 220 or non-real-time processing unit 200A in the embodiment) that generates route information for a mobile device. The first processing unit generates route information for controlling the mobile device's movement path using sensor information acquired from the second processing unit, based on non-real-time processing which has no constraints on response time for processing. The communication unit transmits the route information to the second processing unit. In this way, the information processing device can perform highly safe movement control that achieves both high accuracy and responsiveness by transmitting route information, which is generated with high accuracy based on non-real-time processing, to the first processing unit that actually controls the mobile device.

[0199] (4. Hardware Configuration) The information devices such as the real-time processing unit 100 according to each embodiment described above are realized by a computer 1000 having a configuration such as that shown in Figure 23. The real-time processing unit 100 according to the embodiment will be described below as an example. Figure 23 is a hardware configuration diagram showing an example of a computer 1000 that realizes the functions of the real-time processing unit 100. The computer 1000 has a CPU 1100, RAM 1200, ROM (Read Only Memory) 1300, HDD (Hard Disk Drive) 1400, communication interface 1500, and input / output interface 1600. The various parts of the computer 1000 are connected by a bus 1050.

[0200] The CPU 1100 operates based on programs stored in the ROM 1300 or HDD 1400, and controls various parts. For example, the CPU 1100 loads the programs stored in the ROM 1300 or HDD 1400 into the RAM 1200 and executes processing corresponding to various programs.

[0201] ROM1300 stores boot programs such as the BIOS (Basic Input Output System) executed by CPU1100 when computer 1000 starts up, as well as programs that depend on the computer 1000's hardware.

[0202] The HDD1400 is a computer-readable recording medium that non-temporarily records programs executed by the CPU1100 and data used by such programs. Specifically, the HDD1400 is a recording medium that records a movement control program according to this disclosure, which is an example of program data 1450.

[0203] The communication interface 1500 is an interface for the computer 1000 to connect to an external network 1550 (e.g., the Internet). For example, the CPU 1100 can receive data from other devices or transmit data it generates to other devices via the communication interface 1500.

[0204] The input / output interface 1600 is an interface for connecting the input / output device 1650 and the computer 1000. For example, the CPU 1100 receives data from input devices such as a keyboard or mouse via the input / output interface 1600. The CPU 1100 also transmits data to output devices such as a display, speaker, or printer via the input / output interface 1600. The input / output interface 1600 may also function as a media interface for reading programs recorded on a predetermined recording medium (media). Examples of media include optical recording media such as DVDs (Digital Versatile Discs) and PDs (Phase Change Rewritable Disks), magneto-optical recording media such as MOs (Magneto-Optical Disks), tape media, magnetic recording media, or semiconductor memory.

[0205] For example, when computer 1000 functions as a real-time processing unit 100 according to the embodiment, the CPU 1100 of computer 1000 realizes functions such as the control unit 130 by executing a movement control program loaded on RAM 1200. The HDD 1400 stores the movement control program according to this disclosure and data in the storage unit 170. The CPU 1100 reads and executes the program data 1450 from HDD 1400, but as another example, these programs may be obtained from other devices via an external network 1550.

[0206] Furthermore, this technology can also be configured as follows. (1) It comprises a first processing unit and a second processing unit that communicate with each other, The first processing unit is, Based on non-real-time processing that has no constraints on response time for processing, route information for controlling the movement path of the mobile device is generated using sensor information acquired from the second processing unit. The second processing unit is: Based on real-time processing with constraints on response time for processing, the movement of the mobile device is controlled along the path information generated by the first processing unit. A mobile control system. (2) The first processing unit is provided on a cloud server, The second processing unit is provided in the mobile device, The movement control system described in (1) above. (3) The second processing unit is: Based on the internal sensor information, which is sensor information related to the behavior of the mobile device among the aforementioned sensor information, the position of the mobile device is estimated, and based on the result of the position estimation and the path information, the mobile device is controlled to follow the movement path. The movement control system described in (1) or (2) above. (4) The second processing unit detects objects on or around the movement path based on the sensor information, and based on the behavior of the moving device and the distance from the moving device to the object, it performs one or more of the following: emergency braking control of the moving device or instructions to the first processing unit to replan the movement path. The movement control system described in (3) above. (5) The second processing unit adds a timestamp to the sensor information, The first processing unit performs position recognition of the moving device at the time of the timestamp based on the sensor information to which the timestamp has been assigned. The movement control system described in (1), (2), (3), or (4) above. (6) The second processing unit is: Based on the position of the moving device recognized by the first processing unit, the position estimation result at the time of the timestamp is corrected, and the position of the moving device at the current time is re-estimated using the corrected position estimation result. The movement control system described in (5) above. (7) The second processing unit is: Based on the re-estimated position of the moving device at the current time, the movement of the moving device is controlled along the path information. The movement control system described in (6) above. (8) The first processing unit is, Based on the aforementioned sensor information, object recognition is performed, including at least one of the following: recognition of the color of a traffic light, recognition of a road sign, recognition of a white line, recognition of an animal, and prediction of the behavior of an animal. A movement control system as described in any one of (1) to (7) above. (9) The first processing unit is, Based on the results of the position recognition and the results of the object recognition, the path information is generated. The movement control system described in (8) above. (10) The first processing unit is, While the mobile device is traveling along the aforementioned travel path, it generates a second travel path that connects to the aforementioned travel path. A movement control system as described in any one of (1) to (9) above. (11) The second processing unit is: If the generation of the second movement path by the first processing unit is delayed, the moving device is controlled to stop on or near the movement path. The movement control system described in (10) above. (12) The first processing unit and the second processing unit are provided in the mobile device, A movement control system according to any one of the above (1) or (3) to (11). (13) A movement control method performed by a first processing unit and a second processing unit that communicate with each other, The first processing unit generates path information for controlling the movement path of the mobile device using sensor information acquired from the second processing unit, based on non-real-time processing that has no constraints on the response time required for processing. The second processing unit controls the movement of the mobile device along the path information generated by the first processing unit, based on real-time processing with constraints on the response time required for processing. A method for controlling movement. (14) A communication unit that communicates with the first processing unit, It includes a second processing unit that controls the movement of the mobile device, The communication unit receives the route information from the first processing unit, which generates route information for controlling the movement path of a mobile device using sensor information acquired from the second processing unit based on non-real-time processing that has no constraints on the response time for processing. The second processing unit controls the movement of the mobile device in accordance with the route information received by the communication unit from the first processing unit, based on real-time processing with constraints on the response time required for processing. Mobile control device. (15) A communication unit that communicates with the second processing unit, It comprises a first processing unit that generates route information for a mobile device, The first processing unit generates path information for controlling the movement path of the mobile device using sensor information acquired from the second processing unit, based on non-real-time processing that has no constraints on the response time required for processing. The communication unit transmits the route information to the second processing unit. Information processing device. [Explanation of symbols]

[0207] 1. Mobility control system 100 Real-time Processing Units 110 First Sensor 120 Control Unit 130 Second Sensor 140 Internal Sensors 150 Accelerator / Brake / Steering 160 Communications Department 170 Storage section 200 cloud servers 210 Communications Department 220 Non-real-time processing unit 230 Storage section

Claims

1. It comprises a first processing unit and a second processing unit that communicate with each other, The first processing unit is, Based on non-real-time processing that has no constraints on response time for processing, route information for controlling the movement path of the mobile device is generated using sensor information acquired from the second processing unit. The second processing unit is, Based on real-time processing with constraints on the response time required for processing, the movement of the mobile device is controlled along the path information generated by the first processing unit. A movement control system, The second processing unit is, Based on the internal sensor information, which is sensor information related to the behavior of the mobile device among the aforementioned sensor information, the position of the mobile device is estimated, and based on the result of the position estimation and the path information, the mobile device is controlled to follow the movement path, and further, Based on the sensor information, objects are detected on or around the travel path, and based on the behavior of the mobile device and the distance from the mobile device to the object, the travel distance when emergency braking control is applied to the mobile device and the travel distance when emergency braking control is not applied to the mobile device are calculated. If the travel distance without emergency braking control on the moving device is shorter than the distance to the object, the first processing unit is instructed to replan the travel path without performing emergency braking control on the moving device. If the travel distance without emergency braking control to the moving device is longer than the distance to the object, and the travel distance with emergency braking control to the moving device is shorter than the distance to the object, the system instructs the first processing unit to replan the travel path while performing emergency braking control on the moving device. If the distance traveled when emergency braking control is applied to the moving device is longer than the distance to the object, the moving device is stopped by emergency braking control. A mobile control system.

2. The first processing unit is provided on the cloud server, The second processing unit is provided in the mobile device, The motion control system according to claim 1.

3. The second processing unit adds a timestamp to the sensor information, The first processing unit performs position recognition of the moving device at the time of the timestamp based on the sensor information to which the timestamp has been assigned. The motion control system according to claim 2.

4. The second processing unit is, Based on the position of the moving device recognized by the first processing unit, the position estimation result at the time of the timestamp is corrected, and the position of the moving device at the current time is re-estimated using the corrected position estimation result. The motion control system according to claim 3.

5. The second processing unit is, Based on the re-estimated position of the moving device at the current time, the movement of the moving device is controlled along the path information. The motion control system according to claim 4.

6. The first processing unit is, Based on the aforementioned sensor information, object recognition is performed, including at least one of the following: recognition of the color of a traffic light, recognition of a road sign, recognition of a white line, recognition of an animal, and prediction of the behavior of an animal. The motion control system according to claim 3.

7. The first processing unit is, Based on the results of the position recognition and the results of the object recognition, the path information is generated. The motion control system according to claim 6.

8. The first processing unit and the second processing unit are provided in the mobile device, The motion control system according to claim 1.

9. A movement control method performed by a first processing unit and a second processing unit that communicate with each other, The first processing unit generates path information for controlling the movement path of the mobile device using sensor information acquired from the second processing unit, based on non-real-time processing that has no constraints on the response time required for processing. The second processing unit controls the movement of the mobile device along the path information generated by the first processing unit, based on real-time processing with constraints on the response time required for processing. A method for controlling movement, The second processing unit, Based on the internal sensor information, which is sensor information related to the behavior of the mobile device among the aforementioned sensor information, the position of the mobile device is estimated, and based on the result of the position estimation and the path information, the mobile device is controlled to follow the movement path. Furthermore, the second processing unit, Based on the sensor information, objects are detected on or around the travel path, and based on the behavior of the mobile device and the distance from the mobile device to the object, the travel distance when emergency braking control is applied to the mobile device and the travel distance when emergency braking control is not applied to the mobile device are calculated. If the travel distance without emergency braking control on the moving device is shorter than the distance to the object, the first processing unit is instructed to replan the travel path without performing emergency braking control on the moving device. If the travel distance without emergency braking control to the moving device is longer than the distance to the object, and the travel distance with emergency braking control to the moving device is shorter than the distance to the object, the system instructs the first processing unit to replan the travel path while performing emergency braking control on the moving device. If the distance traveled when emergency braking control is applied to the moving device is longer than the distance to the object, the moving device is stopped by emergency braking control. A method for controlling movement.

10. A communication unit that communicates with the first processing unit, It includes a second processing unit that controls the movement of the mobile device, The aforementioned communications unit is The first processing unit receives the route information from which the second processing unit generates route information for controlling the movement path of the mobile device using sensor information acquired from the second processing unit based on non-real-time processing that has no constraints on the response time for processing, and The second processing unit is, Based on real-time processing with constraints on the response time required for processing, the communication unit controls the movement of the mobile device in accordance with the route information received from the first processing unit. A mobile control device, The second processing unit is, Based on the internal sensor information, which is sensor information related to the behavior of the mobile device among the aforementioned sensor information, the position of the mobile device is estimated, and based on the result of the position estimation and the path information, the mobile device is controlled to follow the movement path, and further, Based on the sensor information, objects are detected on or around the travel path, and based on the behavior of the mobile device and the distance from the mobile device to the object, the travel distance when emergency braking control is applied to the mobile device and the travel distance when emergency braking control is not applied to the mobile device are calculated. If the travel distance without emergency braking control on the moving device is shorter than the distance to the object, the first processing unit is instructed to replan the travel path without performing emergency braking control on the moving device. If the travel distance without emergency braking control to the moving device is longer than the distance to the object, and the travel distance with emergency braking control to the moving device is shorter than the distance to the object, the system instructs the first processing unit to replan the travel path while performing emergency braking control on the moving device. If the distance traveled when emergency braking control is applied to the moving device is longer than the distance to the object, the moving device is stopped by emergency braking control. Mobile control device.

Images