Mobile body control device, mobile body control program, mobile body control method, and mobile body control system

Abstract

To enable a mobile body to be controlled in accordance with the stress of an occupant.SOLUTION: This mobile body control device comprises: an action determination unit that generates a reference path on which the mobile body can move avoiding dangers on the basis of the surrounding situation of the mobile body detected by a mobile body sensor; a body condition estimation unit that estimates the body condition of an occupant of the mobile body on the basis of a measurement value of a vital sensor to generate a body condition signal; and a path control unit that generates control information used for controlling the mobile body on the basis of the reference path and the body condition signal.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to a mobile object control device, a mobile object control program, a mobile object control method, and a mobile object control system for a mobile object such as a vehicle. [Background technology]

[0002] Patent document 1 discloses a vehicle control device that can detect when a vehicle occupant is feeling stressed based on the occupant's biometric information, and apply speed control that is appropriate for that occupant based on a recorded profile. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2023-46098 Summary of the Invention [Problem to be solved by the invention]

[0004] However, controlling speed alone is expected to have little effect on reducing stress. In addition, speed control based on a pre-recorded profile may not be effective in reducing stress due to changes in the driver's physical condition at the time. Furthermore, no effect can be expected for passengers who have not been recorded.

[0005] In view of the above circumstances, an object of the present disclosure is to make it possible to control a moving body in accordance with the stress of an occupant. [Means for solving the problem]

[0006] A mobile object control device according to an embodiment of the present disclosure includes: a behavior determination unit that generates a reference route that is a route that the moving object can travel based on the surrounding conditions of the moving object detected by the moving object sensor; a physical condition estimation unit that estimates the physical condition of an occupant of the vehicle based on measurements from a vital sensor and generates a physical condition signal; a route control unit that generates control information used to control the moving object based on the reference route and the physical condition signal; It is equipped with:

[0007] According to this embodiment, the route control unit generates control information for defining the trajectory of the moving object that matches the physical condition signal at each time, using the reference route as a reference, based on the physical condition signal generated by the physical condition estimation unit from real-time measurements of the vital sensors. This enables the moving object to travel under optimal conditions for the physical condition and stress level of the occupant without significantly deviating from the reference route (while staying within a predetermined range).

[0008] The behavior determination unit may generate the reference path, which is a set of points along which the moving object can move at a predetermined interval with a predetermined resolution.

[0009] By generating a reference path at a predetermined resolution, the amount of calculation processing in the path determination unit at the subsequent stage can be reduced.

[0010] The behavior determination unit may generate the reference route using an inference system capable of generating a route that avoids danger based on surrounding conditions.

[0011] For example, a reference route can be generated using a neural network or CNN (convolutional network), which is an inference system that can generate a route that avoids danger based on the surrounding situation.

[0012] The route control unit a constraint condition generation unit that generates constraint conditions from the physical condition signal; a control information calculation unit that generates the control information by performing model predictive control using the reference path and the constraint conditions; and The constraint generating unit may further generate the constraint from the physical condition signal that changes in accordance with the generated control information.

[0013] According to this embodiment, the constraint generation unit generates constraints to be used in model predictive control based on a physical condition signal obtained by real-time measurement values ​​of a vital sensor. The control information calculation unit uses a reference path as a future reference and executes model predictive control in accordance with constraints that match the physical condition signal at each time, thereby generating control information for defining the trajectory of the moving object. This enables the moving object to travel under optimal conditions for the physical condition and stress level of the occupant without significantly deviating from the reference path (while remaining within a predetermined range). Furthermore, the constraints are generated based on the physical condition signal based on the measurement values ​​of the vital sensor, and a control signal is generated based on the constraints. Since the control signal is based on the constraints based on the physical condition signal, it has the effect of alleviating stress and awakening the occupant. Therefore, by controlling the moving object using a control signal that changes according to the physical condition, the measurement values ​​of the vital sensor and the physical condition signal tend to change in a positive direction (stress is alleviated, drowsiness is awakened). As the physical condition signal changes in a positive direction, the constraints and control signal generated by the constraint generation unit also change to match the physical condition signal. In this way, the control signal is fed back to the occupant's physical condition, and the physical condition is fed back to the control signal. By repeating this feedback loop, it is possible to achieve driving that always matches the occupant's physical condition. Furthermore, even if the occupant's physical condition changes in a negative direction, the negative physical condition is fed back and a control signal is generated that will restore the occupant's physical condition, thereby achieving driving that matches the occupant's physical condition.

[0014] The constraint condition generation unit a reference value generating unit that generates a reference value for each occupant who is a measurement target of the vital sensor, the reference value being based on a driving history (which may include surrounding conditions, weather, road conditions, etc.) that is accumulated for each individual and includes a history of changes in the measurement values ​​or physical condition signals of the vital sensor, A different constraint may be generated for each occupant based on the reference value for each occupant.

[0015] If individual vital indicators differ in what makes a person comfortable, some occupants may feel comfortable when the LF / HF is slightly high. In this case, applying control to equalize the LF / HF may result in an operation to lower the LF / HF, which may then increase depending on the occupant, resulting in so-called oscillation. Therefore, such oscillation can be prevented by generating an optimal reference value for each occupant whose vital sensor is being measured and generating different constraints for each occupant based on the reference value for each occupant. The reference value is generated based on the driving history (which may include surrounding conditions, weather, road conditions, etc.) accumulated for each individual (i.e., a history of changes in vital sensor measurements or physical condition signals associated with the driving route, driving environment, and constraints acquired during past driving), and the conditions may vary for each individual occupant. This allows the accumulated driving history to be fed forward to change the neural network gain or constraints for each individual, thereby changing the control signal for each individual.

[0016] The control information may be an automatic driving signal for driving the moving body.

[0017] The control information may be a driving assistance signal that is notified to a driver who drives the moving object.

[0018] a condition monitoring unit that monitors changes in the constraint conditions, predicts changes in the physical condition of the occupant from the time-dependent changes in the constraint conditions, and generates predicted information on changes in physical condition; Further comprising: The constraint generating unit may change the constraint based on the predicted information on changes in physical condition.

[0019] For example, if the constraint condition suddenly changes to one that requires a sudden decrease in rotational speed, the condition monitoring unit can infer that a change in the physical condition of the occupant has occurred and generate a feedback signal to create a constraint condition that reduces the speed as much as possible.

[0020] A mobile object control program according to an embodiment of the present disclosure includes: The computer of the mobile control device, a behavior determination unit that generates a reference route that is a route that the moving object can travel based on the surrounding conditions of the moving object detected by the moving object sensor; a physical condition estimation unit that estimates the physical condition of an occupant of the vehicle based on measurements from a vital sensor and generates a physical condition signal; a route control unit that generates control information used to control the moving object based on the reference route and the physical condition signal; Operate as.

[0021] A mobile object control method according to an embodiment of the present disclosure includes: The computer of the mobile object control device executes the mobile object control program, generating a reference route that is a route along which the mobile object can travel based on the surrounding conditions of the mobile object detected by the mobile object sensor; generating a physical condition signal by estimating a physical condition of an occupant of the vehicle based on measurements of the vital sensors; Control information used to control the moving object is generated based on the reference route and the physical condition signal.

[0022] A mobile object control system according to an embodiment of the present disclosure includes: A mobile sensor; A vital sensor, a behavior determination unit that generates a reference route that is a route along which the mobile object can travel, based on the surrounding conditions of the mobile object detected by the mobile object sensor; a physical condition estimation unit that estimates the physical condition of the occupant of the vehicle based on the measurement value of the vital sensor and generates a physical condition signal; a route control unit that generates control information used to control the moving object based on the reference route and the physical condition signal; a mobile object control device having It is equipped with: [Effects of the Invention]

[0023] According to the present disclosure, a moving body can be controlled in accordance with the stress of the occupant.

[0024] The effects described here are not necessarily limited to those described herein, and may be any of the effects described in this disclosure. [Brief explanation of the drawings]

[0025] [Figure 1] FIG. 1 is a block diagram illustrating a functional configuration of a mobile object control system according to an embodiment of the present disclosure. [Figure 2] 1 shows a front image generated by the object detection unit. [Figure 3] 10 shows an overhead image generated by a mapping unit. [Figure 4] 10 shows a reference route generated by a route determination unit. [Figure 5] 1 shows the relationship between the reference route and the trajectory of the moving object based on the control information generated by the route control unit. [Figure 6] A mobile model is shown. [Figure 7] The relationship between the trajectory of a moving object based on control information that varies depending on constraints and the reference path is shown. [Figure 8] The results of a simulation of the running behavior of a moving object based on speed constraints are shown below. [Figure 9] The results of a simulation of the running motion of a moving object based on the constraints on the rotation angle are shown below. [Figure 10] The results of a simulation of the running motion of a moving object based on the constraints on the rotation angle are shown below. [Figure 11] The concept of this embodiment will be described. DETAILED DESCRIPTION OF THE INVENTION

[0026] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings.

[0027] 1. Functional configuration of the mobile control system

[0028] FIG. 1 is a block diagram showing the functional configuration of a mobile object control system according to an embodiment of the present disclosure.

[0029] The mobile object control system 1 is mounted, for example, on a mobile object 2 in which a passenger rides. The mobile object 2 is typically a vehicle, including an autonomous vehicle and a non-autonomous vehicle. The mobile object control system 1 may also be mounted on a mobile object 2 other than a vehicle (for example, an air vehicle) in which a passenger rides. The mobile object control system 1 may also be mounted on a virtual mobile object 2. For example, the mobile object 2 may be a virtual mobile object that is virtually driven by a user (virtual passenger) wearing a head-mounted display. In this specification, the "passenger" refers to at least the driver and may also include a passenger.

[0030] The mobile object control system 1 includes a mobile object control device 10, a mobile object sensor 20, a vital sensor 30, and an output unit 40.

[0031] The mobile body sensor 20 detects the surrounding conditions of the mobile body 2 and movement information of the mobile body 2. For example, the mobile body sensor 20 includes a camera (a ToF ranging camera, a visible light image sensor, a stereo camera, an infrared camera, etc.), radar, LiDAR, an ultrasonic sensor, an environmental sensor (various sensors for detecting environmental information such as weather, climate, and brightness, such as a raindrop sensor, a fog sensor, a sunshine sensor, a snow sensor, and an illuminance sensor), a microphone, a speed sensor, an acceleration sensor, an angular velocity sensor (gyro sensor), an inertial measurement unit (IMU), a steering wheel steering angle sensor, a yaw rate sensor, an accelerator pedal sensor, a brake pedal sensor, an engine or motor rotation speed sensor, a tire pressure sensor, a tire slip ratio sensor, a wheel rotation speed sensor, etc.

[0032] The vital sensor 30 detects vital data such as the heartbeat and breathing of the occupant. The vital sensor 30 is, for example, a driver monitor radar unit (millimeter-wave radar) installed in the interior of the moving body 2, and detects the vital data in a non-contact manner. The millimeter-wave radar may be, for example, a 60 GHz band radar. The millimeter-wave radar may be any of SISO, SIMO, MISO, and MIMO. The vital sensor 30 may include an image sensor camera unit (visible light image sensor) and may measure blood flow information, facial expression, posture, etc. of the occupant's face. The vital sensor 30 may also include, for example, a smart watch or a wristband-type wearable device.

[0033] The mobile object control device 10 is a computer that executes a mobile object control program stored in memory. By executing the mobile object control program, the mobile object control device 10 operates as a behavior determination unit 100, a physical condition estimation unit 200, and a route control unit 300. The mobile object control device 10 may further include a condition monitoring unit 400. The route control unit 300 includes a constraint condition generation unit 310 and a control information calculation unit 320. The constraint condition generation unit 310 may include a reference value generation unit 311 and an additional table 312. The mobile object control device 10 generates control information used to control the mobile object control system 1 based on inputs from the mobile object sensors 20 and the vital sensor 30. The mobile object control device 10 may be installed in a mobile object control system 1 installed in a mobile object 2 (the mobile object control system 1 is a local system), or may be installed in a server device (not shown) and communicate with a driving control unit of the mobile object 2 (the mobile object control system 1 is a cloud system).

[0034] The output unit 40 outputs the control information generated by the mobile object control device 10. If the mobile object 2 is capable of automatic driving, the control information is an automatic driving signal for driving the mobile object 2. If the mobile object 2 is capable of manual driving, the control information is a driving assistance signal notified to the driver of the mobile object 2.

[0035] 2. Mobile control device

[0036] Next, the calculation process of the mobile object control device 10 will be described in more detail.

[0037] 2-1. Action Decision-Making Department

[0038] FIG. 2 shows a forward image generated by the object detection unit.

[0039] The object detection unit 110 of the behavior decision unit 100 performs object detection using detection information (e.g., camera images) from the mobile sensor 20, and detects an obstacle 113 (another mobile object, etc.) ahead of the mobile object 2. The object detection unit 110 can use object detection processing such as YOLO (You only look once). As shown in FIG. 2 , the object detection unit 110 generates a front image 111 by adding information such as coordinates, size (coordinates of a bounding box 112), reliability, class, etc. to the obstacle 113, such as a mobile object, ahead of the mobile object 2. The object detection unit 110 outputs the front image 111 to the mapping unit 120.

[0040] FIG. 3 shows an overhead image generated by the mapping unit.

[0041] The mapping unit 120 of the behavior determination unit 100 performs coordinate transformation on the forward image 111 based on the camera parameters. As a result, an overhead image (viewed from directly above) of an obstacle 113 included in the forward image 111 is generated, as shown in FIG. 3. The mapping unit 120 adds external information 50, such as map information, to the image including the obstacle 113 to generate an overhead image 121, and outputs the overhead image 121 to the path determination unit 130. The mapping unit 120 performs mapping on a mesh grid 122 of, for example, 100×100. The resolution of the mesh grid 122 is, for example, 1 m×1 m, i.e., 1 grid = 1 m×1 m. This reduces the amount of calculation processing in the subsequent path determination unit 130.

[0042] FIG. 4 shows a reference route generated by the route determination unit.

[0043] The path determination unit 130 of the behavior determination unit 100 generates a reference path 131 from the overhead image 121. The reference path 131 is a path that the moving object 2 can travel while avoiding danger (i.e., on a road lane, keeping a predetermined distance from an obstacle 113) based on the surrounding conditions of the moving object 2 (including information on approach from the front, rear, left, and right) detected by the mobile object sensor 20 (e.g., a camera). The reference path 131 is a set of points 132 that the moving object 2 can travel at intervals of 1 m x 1 m with a resolution of 1 m x 1 m (the reference path 131 does not include lines). The points 132 that make up the reference path 131 coincide with points 133 (intersections of coordinate axes) of the mesh grid 122. The path determination unit 130 outputs the reference path 131 to the path control unit 300.

[0044] Here, the route determination unit 130 may generate the reference route 131 using, for example, a neural network or a CNN (convolutional neural network), which is a hierarchical inference system capable of generating a route that avoids danger based on the surrounding conditions. The hierarchical inference system may have a hierarchical inference structure from an object detection unit to a route determination unit. In addition to the object detection unit and the route determination unit, an estimation unit may be included that tracks detected objects and estimates the time to collision (TTC). The surrounding conditions may be shared with other moving objects via the cloud. The neural network or CNN is generated by reinforcement learning. Reinforcement learning can be achieved by using a known reinforcement learning algorithm, such as PPO (proximal policy optimization). When performing reinforcement learning, a learning method can be adopted in which an overhead image 121 equivalent to the one generated by the mapping unit 120 is prepared, and based on this, the moving object 2 is assumed to be at the origin of the overhead image 121. A reward is given when the moving object moves forward, and the episode ends when the moving object comes into contact with an obstacle or the road.

[0045] 2-2.Health condition estimation section

[0046] The physical condition estimation unit 200 continues to acquire measurements from the vital sensor 30. The measurements from the vital sensor 30 include, for example, the heart rate and respiratory rate measured by radar, and dynamic recognition information measured by an image sensor (camera). The physical condition estimation unit 200 generates a physical condition signal indicating the stress level of the occupant. For example, the physical condition estimation unit 200 may perform frequency analysis of the heartbeat interval, calculate the ratio LF / HF of the low-frequency component LF and the high-frequency component HF, and generate LF / HF as the physical condition signal. The value of LF / HF increases as stress increases. The physical condition estimation unit 200 outputs the physical condition signal to the route control unit 300. In addition to LF / HF, other indices such as HR (heart rate) and the standard deviation SDNN of RRI may also be used.

[0047] 2-3. Route control unit

[0048] FIG. 5 shows the relationship between the reference route and the trajectory of the moving object based on the control information generated by the route control unit.

[0049] The path control unit 300 generates control information used to control the traveling of the moving object 2 based on the reference path 131 input from the behavior determination unit 100 and the physical condition signal input from the physical condition estimation unit 200. The control information is information for realizing a trajectory 140 of the moving object 2 by controlling the traveling of the moving object 2 along the reference path 131 within the range of the rotation angle ω and the speed v determined by the constraint conditions (described later) generated by the constraint condition generation unit 310. The control information is used for the moving object 2 to follow the reference path 131, and uses the speed v and the steering angle δ. With the two signals of the speed v and the steering angle δ, it is possible to obtain information about the actual acceleration and steering.

[0050] 2-3-1. Constraint condition generator

[0051] The constraint condition generation unit 310 of the route control unit 300 generates constraint conditions for the rotation angle ω and the speed v using the physical condition signal, and outputs the generated constraint conditions to the control information calculation unit 320. The constraint conditions are used in the model predictive control executed by the control information calculation unit 320.

[0052] As a constraint, LF / HF=1 (reference value of physical condition signal where LF=HF) is set as the center (transformed to 0), and ±1 can be used as the transformed f(LH / HF). When the rotation angle ω is constrained to ±ω_max, it is possible to use the proportionality constant 1 / k as follows. However, since there are individual differences in stress level, it is also possible to shift by a deviation amount α. As stress increases, LF / HF increases, so when LF / HF exceeds a predetermined value, it is possible to control the rotation angle ω so that it is inversely proportional to LF / HF, as shown in formula (3). The maximum value ω of the rotation angle ω max is calculated using the following formula (Equation 1).

[0053]

number

[0054] Similarly, the maximum value of the velocity v is v max is calculated using the following formula (Equation 2).

[0055]

number

[0056]

number

[0057] In this way, as a constraint, U t =(ω max ,v max Since there are individual differences among occupants, the rotation angle ω may be made proportional to LF / HF, or may be shifted by the amount of shift α as described above.

[0058] If there are differences in what vital indicators each individual feels comfortable, some occupants may feel comfortable when the LF / HF is slightly high. In this case, if control is applied to make the LF / HF equal, the LF / HF may be lowered, and further, the LF / HF may be raised to match the occupant, which may cause so-called oscillation.

[0059] Therefore, in order to prevent such oscillation, the constraint condition generating unit 310 may include a reference value generating unit 311. The reference value generating unit 311 generates an optimum reference value according to the individual occupant who is the measurement target of the vital sensor 30. The constraint condition generating unit 310 generates a different constraint condition for each occupant based on the reference value for each occupant generated by the reference value generating unit 311.

[0060] For example, the reference value generation unit 311 receives a facial image as additional information in addition to the physical condition signal. The reference value generation unit 311 performs personal authentication based on the facial image and selects a preset reference value appropriate for the individual. Alternatively, the reference value generation unit 311 receives a vital signal including the facial image and generates a reference value using a neural network trained to output an optimal reference value from the vital signal. The reference value generation unit 311 may perform personal authentication using the facial image and use a preset offset signal, or may use information other than stress, such as facial color, from the facial image. The reference value generation unit 311 may also acquire a CAN signal and generate a reference value appropriate for each individual occupant based on agility, such as operational response. The reference value generation unit 311 may also generate a reference value appropriate for each individual occupant using sensing information, such as bone structure and steering operation, from the video signal. The reference value generation unit 311 may also generate a reference value based on changes in device operation or posture due to fatigue, poor physical condition, etc.

[0061] The constraint condition generation unit 310 may further include an additional table 312. The additional table 312 contains conditions for modifying the reference value generated by the reference value generation unit 311. The conditions in the additional table 312 may be, for example, conditions for limiting the speed or rotation angle within a predetermined range to consider comfort, fuel economy, etc. The offset signal used by the reference value generation unit 311 may be recorded as a reference value in the additional table 312. The offset signal as a reference value is generated based on a driving history (which may include surrounding conditions, weather, road conditions, etc.) accumulated for each individual (i.e., a history of changes in measurements or physical condition signals of the vital sensor 30 associated with the driving route, driving environment, and constraint conditions acquired during past driving), and may be a condition that differs for each individual occupant. This makes it possible to feed forward the driving history accumulated for each individual to change the gain of the neural network or change the constraint conditions for each individual, thereby changing the control signal for each individual.

[0062] The constraint condition generating unit 310 converts the reference value generated by the reference value generating unit 311 into a constraint condition. When an offset of the deviation amount γ is required, it is calculated by the following equation (Equation 4).

[0063]

number

[0064] 2-3-2. Control information calculation section

[0065] The control information calculation unit 320 of the route control unit 300 generates control information by executing model predictive control using the constraint conditions generated by the constraint condition generation unit 310. Model predictive control is a control method that performs optimization while predicting future responses at each time (each sampling time). Model predictive control solves an optimization problem at each time and uniquely determines a control input.

[0066] When model predictive control is used to control the moving object 2, the standard for future response (reference path 131) is provided separately in a hierarchical prediction system. Therefore, it is possible to generate control information that determines the future trajectory by referring to the reference path 131. On the other hand, one of the major features of model predictive control is that the controller itself can perform control within the range of constraints. For example, it is possible to determine an upper limit for the traveling angle θ of the moving object 2 and control the vehicle while operating within that range. Therefore, it is possible to estimate the stress of the occupants using the output of the vital sensor 30, and change, for example, the magnitude of the traveling angle θ of the vehicle depending on the stress of the occupants, thereby performing adaptive driving control to prevent stress.

[0067] The constraint conditions generated by the constraint condition generation unit 310 are based on the physical condition signals input from the physical condition estimation unit 200. Therefore, the control information calculation unit 320 performs model predictive control using the constraint conditions based on the physical condition signals at each time, thereby generating optimal control information for the moving object 2 at each time according to the physical condition of the occupant, which may change over time.

[0068] Model predictive control can keep control signals within a predetermined range by using constraints. For example, model predictive control can keep the rotation angle ω and velocity v of the moving object 2, which are determined by the steering angle δ, within a predetermined range. If the rotation angle ω or steering angle δ of the moving object 2 is larger than necessary, it will cause discomfort to the occupants and increase the stress index. To prevent this, it is effective to keep the rotation angle ω and velocity v within a predetermined range by using constraints based on the physical condition signals of the occupants.

[0069] Some of the advantages of model predictive control are as follows: It can control while observing constraints such as physics, performance, and safety. It is easy to apply to complex objects such as multi-input / multi-output systems. It has a practical feedback control correction effect for optimal control. It is a general-purpose method and can be applied to a wide range of applications.

[0070] FIG. 6 shows a mobile model.

[0071] The control information calculation unit 320 receives the reference path 131 and the constraint conditions as input, calculates and outputs control information for the moving object 2 under the constraint conditions. To perform control, a moving object model must be specified. The moving object model includes a heading angle θ, a steering angle δ, a rotation angle ω, a speed v of the moving object 2, a position (x, y) of the moving object 2, a current position (xt, yt) of the moving object 2, a discrete time dt, a width L of the front and rear wheels, and an instantaneous center ICR. The moving object 2 travels from the current position (xt, yt) at a heading angle θ using the speed v and steering angle δ.

[0072]

number

[0073] In the above equation (Equation 5), the dot of θ represents a time change (differential), and the relationship with the steering angle δ is given by the following equation (Equation 6).

[0074]

number

[0075] If the final state of the moving object 2 is (x, y, θ), the state equation becomes a nonlinear state equation shown in the following equation (Equation 7).

[0076]

number

[0077] The travel angle θ and velocity v of the reference path 131 are given by the following equation (Equation 8).

[0078]

number

[0079] Then, the control amount is ω t ,u t and the state quantity x t ,y t ,θ t Using this, we get the formula (9). Here, u tis v t =v t ref +u t is a small manipulated variable defined as

[0080]

number

[0081] Rearranging equation (9) gives equation (10).

[0082]

number

[0083] The constraints are the rotation angle and velocity v, and u t Consider the upper limit of u t ,ω t If we limit the maximum and minimum of this, we get equation (Equation 11).

[0084]

number

[0085] The objective function is the sum of squares of the errors from the reference path 131 .

[0086]

number

[0087] Here, T is the number of time steps to be predicted, called the prediction horizon. The objective function, constraints, and state equations are a convex programming problem with quadratic inequality constraints, and solving this gives us u t ,ω t Calculate ω t Once calculated, the steering angle δ (steering amount) is calculated. t Once calculated, v t =v t ref +u tThe speed v is calculated from the above relational expression. By differentiating it, the acceleration (amount of acceleration) is calculated. This value (control information) can be used for autonomous driving, and can also be notified to the driver. Convex programming problems can be solved using, for example, CVXPY (a solution algorithm for convex programming using Python). Using CVXPY, the optimal solution (control signal) for the input can be obtained by writing the cost function and constraints in a Python program in a specified format.

[0088] FIG. 7 shows the relationship between the trajectory of a moving object based on control information that varies depending on constraints and the reference path.

[0089] The reference path 131 is generated by a hierarchical reasoning system, (x r t ,y r t ), (x r t+1 ,y r t+1 ) ···. From this coordinate information, the travel angle θ and speed v (Equation 8) of the reference path 131 are calculated. The control information calculation unit 320 of the path control unit 300 can generate a control signal using the constraint conditions generated by the constraint condition generation unit 310. For example, when the LF / HF ratio increases (stress increases), the constraint condition generation unit 310 reduces (stricts) the constraint condition for the travel angle θ. As a result, the control information calculation unit 320 generates a control signal for realizing the trajectory 141 with a smaller travel angle θ than the trajectory 142. When the stress of the occupant is high, the path control unit 300 generates control information for the moving object 2 to travel along the trajectory 141 (with small lateral acceleration) to relieve the stress of the occupant. Conversely, when the stress of the occupant decreases, the path control unit 300 generates control information for the moving object 2 to travel along the trajectory 142 (with large lateral acceleration) to wake the occupant up.

[0090] In this way, the constraint conditions are generated based on the physical condition signal based on the measurement values ​​of the vital sensor 30, and the control signal is generated based on the constraint conditions. Since the control signal is based on the constraint conditions based on the physical condition signal, it has the effect of alleviating stress and waking up the occupant. Therefore, by controlling the vehicle 2 using the control signal that changes according to the physical condition, the measurement values ​​of the vital sensor 30 and the physical condition signal tend to change in a positive direction (stress is alleviated, drowsiness is awakened). As the physical condition signal changes in a positive direction, the constraint conditions and the control signal generated by the constraint condition generation unit 310 also change to match the physical condition signal. In this way, the control signal is fed back to the physical condition of the occupant, and the physical condition is fed back to the control signal. By repeating this feedback loop, it is possible to constantly realize driving that matches the physical condition of the occupant. Furthermore, even if the occupant's physical condition changes in a negative direction, the negative physical condition is fed back, and a control signal that restores the physical condition is generated, thereby realizing driving that matches the occupant's physical condition.

[0091] The constraint condition generation unit 310 may function as a learning type constraint condition generation unit. For example, a mechanism can be constructed in which braking constraints based on model prediction and constraints based on physical condition signals interact with each other. For example, the constraint condition generation unit 310 optimizes the constraint conditions in response to the physical condition signal. The control information calculation unit 320 generates an optimized control signal based on the optimized constraint conditions. By controlling the moving object 2 using the optimized control signal, the physical condition of the occupant improves and the physical condition signal changes in a positive direction. In this way, it is possible to realize system integrated control brought about by the constraint conditions and the interrelationship between the constraint conditions and the constraint conditions. A learning function may be given to the route determination, and a constraint condition generation unit capable of learning may be realized by reinforcement learning. According to reinforcement learning, the input (state: Observation) is set to physical condition signals such as LF / HF, HR, SDNN, etc. The output (action) is set to ω max ,v maxThe constraints are as follows. The reward is "comfort" quantified from the variance of the physical condition signal. Based on these inputs, outputs, and rewards, a neural network or CNN can be trained using a known reinforcement learning algorithm such as PPO, and the coefficients of the neural network or CNN can be determined. When using a facial image or other input as input, it is also possible to use a network such as a transformer that is capable of multimodal input. The physical condition signal is input to the learning-type constraint condition generation unit, which outputs the constraint conditions.

[0092] 2-4. Condition monitoring section

[0093] The mobile body control device 10 may further include a condition monitoring unit 400. The condition monitoring unit 400 monitors changes in the constraint conditions generated by the constraint condition generation unit 310, and generates health condition change estimation information by inferring changes in the physical condition of the occupant from the time-dependent changes in the constraint conditions. The constraint condition generation unit 310 changes the constraint conditions based on the health condition change estimation information generated by the condition monitoring unit 400. This feeds forward the estimated changes in physical condition, changes the gain of the neural network for each individual, or changes the constraint conditions, thereby making it possible to change the control signal for each individual.

[0094] The temporal change in the constraint conditions generated by the constraint condition generation unit 310 serves as an indicator of changes in the physical condition of the occupant. Therefore, the condition monitoring unit 400 may monitor the temporal change in the physical condition signal related to the constraint condition generation by the constraint condition generation unit 310 and generate a feedback signal so as to generate constraint conditions according to the physical condition of the occupant. For example, suppose that the constraint conditions generated by the constraint condition generation unit 310 suddenly change to constraint conditions that cause a sudden decrease in rotation speed. In this case, the condition monitoring unit 400 may infer that a change in the physical condition of the occupant has occurred and generate a feedback signal to the constraint condition generation unit 310 to generate constraint conditions that cause the speed to be reduced as much as possible.

[0095] The feedback of changes in physical condition by the condition monitoring unit 400 may be reinforcement learning or imitation learning, which performs offline learning based on an expert trajectory dependent on the vital sensor 30. The condition monitoring unit 400 may generate a behavioral trajectory that avoids danger using a hierarchical AI (behavioral decision-making AI). The condition monitoring unit 400 may perform feedback that can generate a control signal for generating a comfortable trajectory based on the output of the vital sensor 30, using the current output of the behavior-decision-making AI. A comfortable route can be generated using the offline learning of the condition monitoring unit 400 and optimal learning by model predictive control of the control information calculation unit 320.

[0096] In this embodiment, a reference path is determined based on safety, and model predictive control is determined based on constraints on the vehicle behavior. Feedback is then provided based on biometric sensing results for these constraints. In addition to changing the constraints on vehicle control, constraints on the safety margin may also be changed. Changing the allowable constraints on deviations from the reference path indirectly changes the constraints on the safety margin. Danger may be detected and linked to physical condition. For example, if the driver is in poor physical condition, the risk threshold may change and the control may be modified. For example, when the driver is in poor physical condition, the safety margin may be increased by slowing down and driving as close to the reference path as possible. When the driver is in good physical condition, the safety margin may be increased by increasing the speed appropriately and adjusting the acceleration / deceleration and lateral G forces to a comfortable level. However, deviations from the reference path may become slightly larger and the driving speed may also increase, thereby decreasing the safety margin. In addition to the behavior of the vehicle being controlled, the control results and the state of control may also be provided to the occupants via audio output or GUI image display. Note that the above specific examples of the safety margin are merely examples.

[0097] 3. Working Example

[0098] FIG. 8 shows the results of simulating the running behavior of a moving object based on speed constraints.

[0099] (A) shows the control result when the constraint condition for speed v is limited to 1.5. (B) shows the control result when the constraint condition for speed v is limited to 1.2. The upper part shows the speed v. The lower part shows the reference path 131 and the control result. The moving object 2 is controlled at the speed v under the constraint condition.

[0100] FIG. 9 shows the results of simulating the running motion of a moving object based on the constraints on the rotation angle.

[0101] (A) shows the control results when the constraint on the rotation angle ω is limited to ±0.7 radians. (B) shows the control results when the constraint on the rotation angle ω is limited to ±0.15 radians. The top row shows the rotation angle ω. The bottom row shows the reference path 131 and the control results. The moving object 2 is controlled with the rotation angle ω under the constraint.

[0102] FIG. 10 shows the results of simulating the running motion of a moving object based on the constraints on the rotation angle.

[0103] An example is shown in which the constraint condition for the rotation angle ω is changed from ±0.2 radians to ±0.5 radians at the 100th step. In this way, it is possible to change the limit value even during operation. As described above, the path control unit 300 can control the rotation angle ω in real time in response to feedback of changes in the physical condition of the occupant estimated by the condition monitoring unit 400.

[0104] 4. Conclusion

[0105] FIG. 11 shows the concept of this embodiment.

[0106] According to this embodiment, the behavior decision unit 100 uses a hierarchical prediction system to generate a reference path 131 (a set of points 132) that serves as a reference for future responses used in model predictive control. The constraint condition generation unit 310 of the path control unit 300 generates constraint conditions (speed v, rotation angle ω) used in model predictive control based on a physical condition signal generated by the physical condition estimation unit 200 using real-time measurements from the vital sensor 30. The control information calculation unit 320 of the path control unit 300 uses the reference path 131 as a future reference, executes model predictive control in accordance with constraint conditions that match the physical condition signal at each time, and generates control information for defining the trajectory 141 or 142 of the moving object 2. This enables the moving object 2 to travel under conditions (speed v, rotation angle ω) that are optimal for the physical condition and stress level of the occupant without significantly deviating from the set of points 132 of the reference path 131 (while remaining within a predetermined range).

[0107] As described above, according to this embodiment, it is possible to generate control information for the moving body 2 in accordance with the stress of the occupant, using the lateral acceleration, i.e., the rotational speed in the direction of the rotation angle ω, in addition to the speed v as a constraint condition.

[0108] Furthermore, there is no need for a pre-recorded profile as in Patent Document 1. Therefore, it is possible to flexibly generate control information according to the way in which the occupant feels stress due to changes in their physical condition at that time.

[0109] Although the embodiments and modified examples of the present technology have been described above, the present technology is not limited to the above-described embodiments, and it goes without saying that various modifications can be made within the scope of the gist of the present technology. [Explanation of symbols]

[0110] 1. Mobile control system 10 Mobile control device 100 Action Decision-Making Department 110 Object detection unit 111 Front image 112 Bounding Box 113 Obstacles 120 Mapping Section 121 overhead images 122 Mesh Grid 130 Route determination unit 131 Reference Routes 2. Mobile 20 Mobile Sensor 200 Physical Condition Estimation Department 30 Vital Sensor 300 Route control unit 310 Constraint condition generator 311 Reference Value Generation Unit 312 Additional Tables 320 Control Information Calculation Unit 40 Output section 50 External Information 400 Condition Monitoring Unit

Claims

1. a behavior determination unit that generates a reference route that is a route that the moving object can travel based on the surrounding conditions of the moving object detected by the moving object sensor; a physical condition estimation unit that estimates the physical condition of an occupant of the vehicle based on measurements from a vital sensor and generates a physical condition signal; a route control unit that generates control information used to control the moving object based on the reference route and the physical condition signal; A mobile object control device comprising:

2. The mobile object control device according to claim 1, The action determination unit generates the reference path, which is a set of points along which the moving object can move at a predetermined interval with a predetermined resolution. Mobile control device.

3. The mobile object control device according to claim 1, The action decision unit generates the reference route using an inference system that can generate a route that avoids danger based on the surrounding situation. Mobile control device.

4. The mobile object control device according to claim 1, The route control unit a constraint condition generation unit that generates constraint conditions from the physical condition signal; a control information calculation unit that generates the control information by performing model predictive control using the reference path and the constraint conditions; and The constraint generating unit further generates the constraint from the physical condition signal that changes in accordance with the generated control information. Mobile control device.

5. The mobile object control device according to claim 4, The constraint condition generation unit a reference value generating unit that generates a reference value for each occupant who is a measurement target of the vital sensor, the reference value being based on a driving history that is accumulated for each individual and includes a history of changes in the measurement values ​​of the vital sensor or the physical condition signal; generating a different constraint condition for each occupant based on the reference value for each occupant; Mobile control device.

6. The mobile object control device according to claim 1, The control information is an automatic driving signal for driving the moving body. Mobile control device.

7. The mobile object control device according to claim 1, The control information is a driving assistance signal notified to a driver who drives the moving object. Mobile control device.

8. The mobile object control device according to claim 4, a condition monitoring unit that monitors changes in the constraint conditions, predicts changes in the physical condition of the occupant from the time-dependent changes in the constraint conditions, and generates predicted information on changes in physical condition; Further comprising: The constraint condition generating unit changes the constraint conditions based on the predicted information on changes in physical condition. Mobile control device.

9. The computer of the mobile control device, a behavior determination unit that generates a reference route that is a route that the moving object can travel based on the surrounding conditions of the moving object detected by the moving object sensor; a physical condition estimation unit that estimates the physical condition of an occupant of the vehicle based on measurements from a vital sensor and generates a physical condition signal; a route control unit that generates control information used to control the moving object based on the reference route and the physical condition signal; A mobile control program that operates as a

10. The computer of the mobile object control device executes the mobile object control program, generating a reference route that is a route along which the mobile object can travel based on the surrounding conditions of the mobile object detected by the mobile object sensor; generating a physical condition signal by estimating a physical condition of an occupant of the vehicle based on measurements of the vital sensors; generating control information to be used for controlling the moving object based on the reference route and the physical condition signal; A mobile object control method.

11. A mobile sensor; A vital sensor, a behavior determination unit that generates a reference route that is a route along which the mobile object can travel, based on the surrounding conditions of the mobile object detected by the mobile object sensor; a physical condition estimation unit that estimates the physical condition of the occupant of the vehicle based on the measurement value of the vital sensor and generates a physical condition signal; a route control unit that generates control information used to control the moving object based on the reference route and the physical condition signal; a mobile object control device having A mobile object control system comprising:

Images