control device

Abstract

To improve the accuracy in performing control of a moving body in conformity with the behavior of a user.SOLUTION: A control device relating to one aspect in the disclosure generates one or more paths by executing calculation of one or more trained control models of a plurality of trained control models; monitors the behavior of a user to be monitored; selects one path from the generated one or more paths, in accordance with a motored result of the behavior of the user to be monitored; and controls the movement of a moving body, in accordance with the selected path. Each trained control model acquires a capability of generating a path for controlling the movement of the moving body in connection with corresponding behavior of the user, by machine learning.SELECTED DRAWING: Figure 1

Description

Technical Field

[0004] , , ,

[0005] , , , ,

[0001] This disclosure relates to control technologies for autonomous vehicles.

Background Art

[0002] In Patent Document 1, a system for autonomous vehicle control configured to determine vehicle commands from routes, GPS data, and sensor data using a trained neural network has been proposed.

Prior Art Documents

Patent Documents

[0003] <​​​​​​​​​​​​​​​​​​​ [Effects of the Invention]

[0006] According to this disclosure, it is possible to improve the accuracy of controlling a mobile object in a manner that is appropriate to the user's behavior. [Brief explanation of the drawing]

[0007] [Figure 1] Figure 1 schematically illustrates an example of a scenario in which this disclosure applies. [Figure 2] Figure 2 schematically shows an example of the control model configuration of this disclosure. [Figure 3] Figure 3 schematically shows an example of the configuration of the second control model of this disclosure. [Figure 4] Figure 4 schematically shows an example of the hardware configuration of the control device of this disclosure. [Figure 5] Figure 5 shows an example of a processing procedure related to control by the control device of this disclosure. [Modes for carrying out the invention]

[0008] According to conventional methods such as those described in Patent Document 1, an autonomous driving system can be constructed by using a trained machine learning model. However, the inventors of this invention have found the following problems with conventional methods. For example, it is assumed that specific types of driving, such as lane changes and evacuating to the shoulder, will be performed in response to user behavior. User behavior may include, for example, using the turn signal or an abnormality in the driver. Collecting training data for each type of driving, including such user behavior, is difficult because the number of patterns is enormous. Furthermore, the frequency of occurrence can vary greatly. Under the following assumption, we consider a scenario where user behavior is taken as input, and a trained machine learning model is generated to produce paths that match each behavior. In this case, it may be difficult for the resulting trained model to accurately perform vehicle control that matches the user's behavior. For example, extremely infrequent user behaviors may not be adequately learned, which could lead to a decrease in the accuracy of vehicle control by the trained model. This problem can occur regardless of the type of vehicle. Furthermore, this problem is not limited to vehicle control. The same issue arises with other moving objects besides vehicles when it comes to controlling movement. Therefore, similar problems can occur when controlling any moving object other than a vehicle.

[0009] In contrast, the control device according to the first aspect of this disclosure comprises a storage unit for storing a plurality of trained control models and a control unit. The control unit is configured to perform calculations on one or more of the plurality of trained control models to generate one or more paths, monitor the behavior of a target user, select one path from the generated one or more paths according to the results of monitoring the behavior of the target user, and control the movement of a mobile object according to the selected path. Each trained control model has acquired the ability through machine learning to generate paths for controlling the movement of a mobile object related to the corresponding behavior of the user.

[0010] In the first aspect of this disclosure, each trained control model is prepared according to the movement pattern associated with the user's behavior. That is, since each trained control model does not handle movement patterns other than those of other patterns, the mixing of data from other movement patterns can be suppressed in the machine learning of each trained control model. Ideally, training data may be collected specifically for the corresponding movement pattern, and the trained model may be generated using the obtained training data. In the first aspect of this disclosure, trained control models are prepared according to the movement pattern, and the path generated by the trained control model corresponding to the behavior of the target user is used for controlling the moving object. This is expected to improve the accuracy of performing control of the moving object that is adapted to the user's behavior.

[0011] Furthermore, as another form of the control device according to the above embodiment, one aspect of this disclosure may be an information processing method that implements all or part of the above components, a program, or a machine-readable storage medium that stores such a program. Here, a machine-readable storage medium is a medium that stores information such as programs by electrical, magnetic, optical, mechanical, or chemical action.

[0012] [1. Application Examples] Figure 1 schematically shows an example of a scenario in which the present disclosure is applied. The control device 1 according to this embodiment is one or more computers configured to control the automatic movement of a target mobile object M. In this embodiment, the control device 1 is mounted on the mobile object M and holds a plurality of trained control models 30. Each trained control model 30 has acquired the ability, through machine learning, to generate paths for controlling the movement of the mobile object in relation to the corresponding user behavior.

[0013] The control device 1 generates one or more paths 50 by performing calculations on one or more trained control models 35 from among multiple trained control models 30. The control device 1 monitors the behavior 40 of the target user OU and selects one path 55 from the one or more paths 50 generated according to the result of monitoring the behavior 40 of the target user OU. Then, the control device 1 controls the movement of the mobile object M according to the selected path 55.

[0014] In this embodiment, a trained control model 30 is prepared according to the movement pattern related to the user's behavior, and a path 55 generated by a trained control model 35 corresponding to the behavior 40 of the target user OU is used to control the mobile body M. Each trained control model 30 can be generated specifically for the corresponding movement pattern. Therefore, according to this embodiment, it is possible to adapt to the user's behavior. This can be expected to improve the accuracy of controlling the moving object.

[0015] In addition, in this embodiment, the behavior 40 of the target user OU may not be used as an input to each trained control model 30. That is, the behavior 40 of the target user OU may not be used as an explanatory variable for generating the path of each trained control model 30. As a result, the structure of the control model 30 can be simplified, and accordingly, an improvement in the path generation accuracy by each trained control model 30 can be expected.

[0016] Also, a form in which the operation of the trained control model is executed after selecting the trained control model corresponding to the user's behavior is conceivable as another form. However, in this form, after selecting the trained control model corresponding to the user's behavior, the time until the user's behavior is reflected in the control of the moving body becomes longer by the amount of time for executing the arithmetic processing of the trained control model. In contrast, the control device 1 according to this embodiment may execute all the operations of the trained control models 35 that are likely to be used regardless of whether they are used, and select the path 55 to be used from the operation results (one or more paths 50) according to the behavior 40 of the target user OU. Thereby, the time until the user's behavior is reflected in the control of the moving body can be shortened.

[0017] (Moving body) If the moving body can be automatically moved by machine control, the type of the moving body M may be appropriately selected according to the embodiment. The moving body M may be, for example, a movable device such as a vehicle, an aircraft, a ship, a robot device, or the like. The aircraft may be at least one of an unmanned aircraft such as a drone and a manned aircraft. In one example, as shown in FIG. 1, the moving body M may be a vehicle. In this case, an improvement in the accuracy of performing an automatic driving adapted to the user's behavior can be expected. Note that the type of the vehicle (number of wheels, power source, size, etc.) may be arbitrarily selected. As a typical example, the moving body M may be an automobile having an automatic driving ability of level 2 or higher.

[0018] (Controlling the operation) In one example, controlling the operation of the target mobile body M may be done by directly controlling the target mobile body M. In another example, the mobile body M may be equipped with a dedicated control device, such as a controller. In this case, controlling the operation of the target mobile body M by the control device 1 may be done by indirectly controlling the target mobile body M by providing the dedicated control device with a derived result. The control device 1 may be deployed in any location. In one example, as shown in Figure 1, the control device 1 may be mounted on the mobile body M. In another example, the control device 1 may be located away from the mobile body M and remotely control the mobile body M. The control device 1 may be configured to switch between the automatic control mode and manual control mode of the mobile body M at any time by any operation. When the mobile body M is a vehicle, typically the target user OU may be the driver or a passenger other than the driver. The target user OU may be determined as appropriate depending on the mobile body M.

[0019] (One or more pre-trained control models) In this embodiment, it is not always necessary to perform calculations on all of the provided trained control models 30 to generate a path. The most likely trained control model 35 to be used may be selected from the provided trained control models 30 in any way. For example, one or more trained control models 35 to be used to generate a path may be selected depending on the travel scene, such as location, route, and type of road (for vehicles, e.g., highway, general road). As a specific example, let's assume that the moving object M is a vehicle, and the control device 1 has a first control model that generates a path for changing lanes to the left, a second control model that generates a path for changing lanes to the right, and a third control model that generates a path for maintaining lanes as trained control models 30. In this scenario, when traveling in the leftmost lane, the control device 1 generates a path using the second and third control models respectively. Since there is no room to select the path of the first control model, the generation of the path by the first control model may be omitted. On the other hand, in a scene where lane changes are possible in both the left and right directions, the control device 1 may generate a path using each of the first to third control models.

[0020] (Path) The path (the output of the control model) may not be particularly limited in its form as long as it can control the operation of the moving body M, and may be appropriately determined according to the embodiment. In one example, the path (50, 55) may be composed of one or more control commands. The control command may be configured to indicate the control amount of the moving body M. In another example, the path (50, 55) may be configured to indicate the future movement path of the moving body M and may be used to derive one or more control commands. In this case, one or more control commands may be determined from the path 55 in an arbitrary manner. The control models (30, 35) may be referred to as path planners.

[0021] Note that the control command relates to the operation of the moving body M. The configuration of the control command may be appropriately selected according to the embodiment. In one example, the control command may be composed of acceleration, deceleration, steering, or a combination thereof. Acceleration and deceleration may include gear changes. The control command may be configured to indicate, for example, the control amount (control instruction value, control output amount) of the moving body M such as the accelerator control amount, the brake control amount, and the steering wheel steering angle. Further, the control command may further include a command related to the operation of the moving body M. As an example, when the moving body M is a vehicle, the control command may include vehicle operations such as a wiper, hazard, horn, communication processing (for example, transmitting data to the center, making an emergency call, etc.).

[0022] (Trained control model) The control model 30 is composed of a machine learning model having one or more computational parameters that can be adjusted by machine learning. One or more computational parameters are used for the calculation of the target inference (in this case, path derivation). Machine learning is the process of adjusting (optimizing) the values ​​of the computational parameters using training data. The configuration and type of the machine learning model are not particularly limited and may be appropriately selected depending on the embodiment. The machine learning model may consist of, for example, a neural network, a support vector machine, a regression model, a decision tree model, etc. The machine learning method may be appropriately selected depending on the machine learning model adopted (for example, backpropagation). As an example, at least one of the multiple control models 30 may be composed of at least part of a neural network. The structure of the neural network may be appropriately determined depending on the embodiment and may be specified, for example, by the number of layers from the input layer to the output layer, the type of each layer, the number of nodes (neurons) included in each layer, the connection relationships between the nodes in each layer, etc. As an example, the neural network may have a recursive structure. Furthermore, the neural network may include any layers such as fully connected layers, convolutional layers, pooling layers, deconvolutional layers, unpooling layers, normalization layers, dropout layers, and LSTM (Long Short-Term Memory). The neural network may also have any mechanisms such as an attention mechanism. The neural network is a GNN. The control model may include any model such as a graph neural network, diffusion model, or generative model (e.g., Generative Adversarial Network, Transformer, etc.). When a neural network is used as the control model, the weights of the connections between each node in the control model and the thresholds of each node are examples of computational parameters. When a machine learning model is used, the control model may be structured as an end-to-end model.

[0023] Each control model 30 is configured to derive a path according to the environment of the mobile body M. The environment is the events observed in at least one of the mobile body M itself and its surroundings. In one example, at least a portion of the environment may be observed by one or more sensors S placed inside or outside the mobile body M. The type of sensor S is not particularly limited as long as it is capable of observing any environment in which the mobile body M moves, and may be appropriately selected depending on the embodiment. In one example, one or more The sensor S above may include a camera (image sensor), radar, LiDAR (Light Detection and Ranging), sonar (ultrasonic sensor), infrared sensor, GNSS (Global Navigation Satellite System) / GPS (Global Positioning Satellite) module, etc.

[0024] The input / output configuration of the control model 30 is not particularly limited and can be appropriately selected depending on the embodiment, as long as a path can be derived from the environment of the mobile body M. In one example, at least one of the multiple control models 30 may be configured to derive a path from observation data 120 from sensor S at one or more time points. In another example, at least one of the multiple control models 30 may be configured to derive a path from the recognition result of the surrounding environment. In this case, the control device 1 may further include an analysis model that infers the recognition result of the surrounding environment from the observation data 120 of sensor S. Alternatively, the trained control model 30 may include such analysis model. The analysis model may be configured arbitrarily. In one example, the analysis model may be configured by a machine learning model. In addition, other information may be optionally added to the input of at least one of the multiple control models 30. At least one of the multiple control models 30 may be configured to further accept input of arbitrary information such as set speed, speed limit, position, map information, navigation information, etc. In another example, the control model 30 may be configured to directly output a path. In another example, the control model 30 may be configured to output a path indirectly, and the path may be obtained by performing arbitrary information processing (interpretation processing) on ​​the output of the control model 30.

[0025] Typically, multiple trained control models 30 may be configured separately (independently). However, the configuration of multiple trained control models 30 is not limited to this example. In one example of this embodiment, two or more control models among the multiple trained control models 30 may be configured integrally. For example, one model may include the above-mentioned analysis model placed on the input side, and n output parts that each derive a path from the output of the analysis model (n is a natural number of 2 or more). This one model may be considered as n trained control models 30. That is, in this embodiment, even if the structure of the model is integral, if the model has n output parts configured to each output a path, then holding the model may be considered as holding n trained control models 30. The same treatment may also apply to conditional models configured to output according to given conditions. That is, multiple trained control models 30 may include conditional models configured to output a path according to the input conditions (class / category). In this case, by changing the conditions given to the conditional model and repeating the calculations of the conditional model, multiple paths corresponding to each given condition can be derived. Therefore, if one conditional model is configured to correspond to n conditions, holding this one conditional model can be considered equivalent to holding n trained control models 30.

[0026] (Monitoring of behavior) Monitoring behavior 40 may consist of monitoring the absence of a specific input and monitoring the presence of a specific input. Monitoring the presence of a specific input may include identifying the type (indicator) of the input.

[0027] Furthermore, the behavior 40 of the target user OU may be monitored in any way. For example, the control device 1 may monitor the behavior 40 of the target user OU via the presence or absence of input such as images, sounds, or operators. When monitoring the behavior 40 via images, the behavior 40 may include gestures, the state of the target user OU (for example, the target user OU is in an abnormal state), etc. Operators may include any kind of device related to the operation of the mobile body M. For example, if the mobile body M is a vehicle, operators may include, for example, controls provided on the vehicle such as turn signals, hazard lights, and horns.

[0028] In one example of this embodiment, a monitoring device MD may be used to monitor the behavior 40. The monitoring device MD may include, for example, a camera (image sensor), a microphone, an operator, a sensor for measuring the amount of operation of the operator, etc. The behavior 40 of the target user OU may be identified by any method from the observation data 125 obtained in real time by the monitoring device MD. The behavior 40 of the target user OU may be identified from the observation data 125 by known analysis methods such as image analysis, voice analysis, analysis of the operation content of the operator, etc.

[0029] (Correspondence between behavior and control models) The correspondence between the prepared trained control model 30 and the user's behavior may be determined as appropriate depending on the embodiment. That is, the type of user behavior to which the trained control model 30 is prepared may be determined as appropriate depending on the embodiment. Furthermore, the ability to generate paths of various movement patterns for the prepared control model 30 corresponding to the behavior may also be determined as appropriate depending on the embodiment.

[0030] Figure 2 schematically shows an example of a control model configuration according to this embodiment. In the example in Figure 2, the multiple trained control models 30 include a first trained control model 31 that has acquired the ability to generate paths in situations without user instructions. Accordingly, one or more trained control models 35 may include the first trained control model 31. Selecting a path 55 may include selecting a path generated by the first trained control model 31 when there are no instructions from the target user OU. This enables automatic control of the mobile object M in situations where there are no arbitrary instructions as part of the behavior 40 of the target user OU. In one example, the absence of instructions may be defined as the absence of specific instructions to the mobile object M via input such as images, voice, or operators.

[0031] Furthermore, in the example shown in Figure 2, the multiple trained control models 30 further include one or more second trained control models 32, each of which has acquired the ability to generate paths in situations where there are corresponding instructions from the user. Accordingly, one or more trained control models 35 may include one or more second trained control models 32. Selecting a path 55 may include selecting a path generated by one or more second trained control models 32 in response to instructions from the target user OU. This enables automatic control of the mobile body M in accordance with any instructions given as part of the behavior 40 of the target user OU. In one example, the presence of instructions may consist of specific instructions given to the mobile body M via input such as images, voice, or operators. The content of the instructions and the capabilities of the second trained control models 32 may be determined as appropriate depending on the embodiment.

[0032] Figure 3 schematically shows an example of the configuration of the second trained control model 32 according to this embodiment. As shown in Figure 3, the moving object M may be a vehicle, and one or more second trained control models 32 may include a trained lane change model 33 that has acquired the ability to generate paths for controlling the movement of the moving object M (vehicle) in a situation in which a lane change instruction is given. Accordingly, selecting a path 55 may include selecting a path generated by the trained lane change model 33 in response to a lane change instruction from the target user OU.

[0033] The method for instructing a lane change (behavior 40) may be determined as appropriate depending on the embodiment. In one example, the instruction for a lane change may be given by any input such as gesture (image) input, voice input, or operation on a specific control (e.g., right or left turn signal operation). The trained lane change model 33 may also be configured to generate a path for performing at least one of changing to the right lane and changing to the left lane. If both right and left lane changes are to be performed, separate lane change models may be provided for right and left lane changes, respectively. Alternatively, one lane change model capable of handling both left and right lane changes may be used. It may be intended.

[0034] The type of the second trained control model 32 is not limited to the example of the trained lane change model 33, and may be determined as appropriate depending on the embodiment. As another example, when the moving object M is a vehicle, the second trained control model 32 may include a lane keeping model, an emergency stop model, etc. The lane keeping model may be configured to generate a path for maintaining the lane in which the vehicle is traveling. The lane keeping model may be included in the first trained control model 31 or the trained lane change model 33. The emergency stop model may be configured to generate a path for moving the vehicle to the shoulder of the road in response to an abnormal user condition, such as the driver losing consciousness. Furthermore, the correspondence between user behavior and the control model 30 is not limited to the above example. In another example, at least one of the first trained control model 31 and the second trained control model 32 may be omitted.

[0035] (Selecting a path) In Scene SC1 of Figure 3, since there are other lanes on both sides of the lane the vehicle is traveling in, lane keeping and lane changes in both directions are possible. On the other hand, in Scene SC2, since the vehicle is traveling in the leftmost lane, lane keeping and lane changes to the right are possible, but lane changes to the left are not. Therefore, in Scene SC2, it is preferable to exclude the path of lane changes to the left. Such movements that cannot be safely performed depending on the movement scene of a moving object M are not limited to when the moving object M is a vehicle. Similarly, when the moving object M is something other than a vehicle, movements that cannot be safely performed may occur depending on the movement scene of the moving object M.

[0036] To address this, in one example, when the control device 1 selects one or more trained control models 35 from a plurality of trained control models 30 as described above, it may exclude trained control models 30 that generate unexecutable paths depending on the movement scene. In other words, the control device 1 may eliminate unexecutable paths by not selecting trained control models 30 that generate unexecutable paths as trained control models 35 depending on the movement scene.

[0037] In another example, in addition to or instead of the said method, the control device 1 may evaluate the safety level of each of the one or more generated paths 50. Selecting one path 55 may consist of selecting one path 55 from the one or more generated paths 50 after excluding paths that are evaluated as having a low safety level. Whether or not a path has a low safety level may be determined by any method. Typically, whether or not a path has a low safety level may be determined by comparing the calculated safety level with a threshold. The threshold may be determined as appropriate depending on the embodiment. This makes it possible to eliminate paths that cannot be executed.

[0038] The method for evaluating safety can be appropriately selected depending on the embodiment. In one example, during machine learning, the training data is assigned a true value for safety (i.e., a safety level teacher signal is provided), so that the trained control models (30, 35) can derive safety along with the path. In this case, evaluating safety is performed by executing calculations on the trained control model 35 to calculate the safety of the derived path 50. In another example, the safety of the generated path 50 may be evaluated based on simulation. For example, the safety of the generated path 50 may be evaluated by simulating movement along the generated path 50 in real time using the control device 1 or an external computer (e.g., a server device). Alternatively, for example, by collecting the results of the simulation, multiple datasets consisting of combinations of the generated path (training data) and the evaluated safety level (teacher signal / label) may be obtained. The training data may further include the environment of the moving object M or the results of the environment recognition when the corresponding path was generated. A trained machine learning model that has acquired the ability to calculate safety from the generated path may be generated by machine learning using the multiple acquired datasets. The control device 1 generates one or more You can calculate the safety score for one or more of the 50 paths by feeding each of them to this trained machine learning model.

[0039] In another example, the safety level may be evaluated depending on whether the input data given to the trained control model 35 during the generation of path 50 (e.g., observation data 120 from sensor S, environmental recognition results, etc.) falls within the scope of machine learning. For example, an autoencoder may be generated by machine learning, either together with or separately from the trained control models (30, 35). If the input data falls within the scope of machine learning, the reconstruction error of the autoencoder will be small; otherwise, the reconstruction error of the autoencoder will be large. In other words, the larger the reconstruction error, the lower the safety level can be evaluated. Therefore, the control device 1 may perform the calculation of the autoencoder together with the calculation of one or more trained control models 35, and calculate the reconstruction error by calculating the difference between the input data and the output data. The control device 1 may evaluate the safety level of the generated path 50 according to the calculated reconstruction error. Simply put, the reconstruction error may be used directly as the safety level.

[0040] (Choose one path) Selecting one path 55 from one or more paths 50 may include selecting one path from a plurality of generated paths, and using the generated path as is. That is, if the control device 1 according to this embodiment has a first mode in which it generates a plurality of paths and selects one path from the plurality of generated paths, it may also have a second mode in which it generates one path 50 from only one trained control model 35, at least temporarily, and uses the generated path 50 as is as path 55 for controlling the mobile body M. The second mode may be omitted.

[0041] [2 Example Configurations] Figure 4 schematically shows an example of the hardware configuration of the control device 1 according to this embodiment. The control device 1 according to this embodiment is a computer in which a control unit 11, a storage unit 12, an external interface 13, an input device 14, an output device 15, and a drive 16 are electrically connected.

[0042] The control unit 11 includes a CPU (Central Processing Unit) and RAM (Random Access Memory). The memory unit 12 includes a RAM (Read Only Memory), ROM (Read Only Memory), etc., and is configured to perform arbitrary information processing based on the program and various data. The control unit 11 (CPU) is an example of a processor resource. The memory unit 12 may be composed of any storage device such as a hard disk drive or a solid-state drive. The memory unit 12 (and RAM, ROM) is an example of a memory unit in this disclosure. The memory unit 12 (and RAM, ROM) is also an example of a memory resource. In this embodiment, the memory unit 12 stores various information such as a control program 81 and learning result data 300.

[0043] The control program 81 is a program that causes the control device 1 to execute information processing related to the control of the mobile body M (Figure 5, described later). The control program 81 includes a series of instructions for said information processing. The learning result data 300 is configured to show information about the trained control model 30. In one example, the learning result data 300 may be provided for each trained control model 30. In another example, one learning result data 300 may be configured to show two or more trained control models 30. Furthermore, if the trained control model 30 can be reproduced, the data structure of the learning result data 300 may be appropriately determined according to the embodiment. In one example, the learning result data 300 may include information showing the values ​​of the computational parameters of the control model 30 adjusted by machine learning. The learning result data 300 may further include information showing the configuration of the machine learning model (e.g., the structure of the neural network).

[0044] External interface 13 may include, for example, a USB (Universal Serial Bus) port, a dedicated port The interface may be a wireless communication port or the like, and is configured to connect to an external device by wire or wireless connection. In this embodiment, the control device 1 may be connected to the sensor S and the monitoring device MD via the external interface 13. The input device 14 is a device for inputting data such as a mouse, keyboard, or control. The output device 15 is a device for outputting data such as a display or speaker. The input device 14 and the output device 15 may be integrated together by, for example, a touch panel display.

[0045] The drive 16 is a device for reading various information, such as programs, stored in the storage medium 91. At least one of the control program 81 and the learning result data 300 may be stored in the storage medium 91 instead of or together with the storage unit 12. The storage medium 91 is configured to store various information (stored programs, etc.) by electrical, magnetic, optical, mechanical, or chemical means so that a machine such as a computer can read the information. The control device 1 may acquire at least one of the control program 81 and the learning result data 300 from the storage medium 91. The storage medium 91 may be a disk-type storage medium such as a CD or DVD, or a non-disk-type storage medium such as semiconductor memory (e.g., flash memory). The type of drive 16 may be appropriately selected according to the type of storage medium 91.

[0046] Regarding the specific hardware configuration of the control device 1, components can be omitted, replaced, and added as appropriate depending on the embodiment. For example, the control unit 11 may include multiple hardware processors. Hardware processors include microprocessors, FPGAs (field-programmable gate arrays), DSPs (digital signal processors), and ECUs. It consists of components such as an Electronic Control Unit (ECC) and a Graphics Processing Unit (GPU). At least one of the external interface 13, input device 14, output device 15, and drive 16 may be omitted. The control device 1 may be a computer designed specifically for the service provided, a general-purpose computer, a terminal device, etc. If the mobile body M is a vehicle, the control device 1 may be an in-vehicle device.

[0047] [3 Examples of operation] Figure 5 shows an example of a processing procedure for controlling the mobile body M by the control device 1 according to this embodiment. The control unit 11 of the control device 1 executes instructions included in the control program 81 using the CPU. As a result, the control device 1 operates as a computer capable of performing the following information processing. The following processing procedure is an example of a control method executed by the computer. However, the following processing procedure is merely an example, and each step may be modified as much as possible. Furthermore, steps in the following processing procedure can be omitted, replaced, and added as appropriate, depending on the embodiment.

[0048] In step S101, the control unit 11 acquires observation data 120 from the sensor S. The control unit 11 may acquire observation data 120 directly or indirectly from the sensor S.

[0049] In step S102, the control unit 11 generates one or more paths 50 by performing calculations on one or more trained control models 35 from among a plurality of trained control models 30. In one example, the control unit 11 may derive paths 50 by providing at least a portion of the acquired observation data 120 to the trained control model 35 and performing calculations on the trained control model 35. Any preprocessing may be applied to the observation data 120 before providing it to the trained control model 35. In another example, the control unit 11 may recognize the environment of the mobile object M from the observation data 120 in any way, provide the recognition result of the environment to the trained control model 35, and derive paths 50 by performing calculations on the trained control model 35.

[0050] In one example of this embodiment, one or more trained control models 35 respond to a specific finger input from the user. The system may include a first trained control model 31 that has acquired the ability to generate paths for controlling the movement of a mobile object M in situations where no instructions are given. The system may also include one or more second trained control models 32 that have acquired the ability to generate paths for controlling the movement of a mobile object M in situations where there are corresponding specific instructions from the user. If the mobile object M is a vehicle, the one or more second trained control models 32 may include a trained lane change model 33.

[0051] Before executing step S102, the control unit 11 may, at any time, select one or more likely trained control models 35 from the multiple trained control models 30 depending on the movement scene of the mobile body M. In step S102, the control unit 11 may use the selected trained control model 35 to derive path 50.

[0052] In step S103, the control unit 11 monitors the behavior 40 of the target user OU. The behavior 40 of the target user OU may be monitored by any method. For example, the control unit 11 may monitor the behavior 40 of the target user OU by obtaining observation data 125 directly or indirectly from the monitoring device MD. In this case, the control unit 11 may identify the behavior 40 of the target user OU from the observation data 125 by any method.

[0053] In step S104, the control unit 11 evaluates the safety level of each of the one or more generated paths 50. In one example of this embodiment, the control unit 11 may evaluate the safety level of the paths 50 by any of the methods described above.

[0054] In step S105, the control unit 11 selects one path 55 from one or more generated paths 50 in accordance with the result of monitoring the behavior 40 of the target user OU. In one example of this embodiment, selecting one path 55 may include selecting a path generated by the first trained control model 31 when there is no specific instruction from the target user OU. Even if there is no trained control model 35 corresponding to the behavior 40 of the target user OU, the control unit 11 may select a path generated by the first trained control model 31, similar to the case where there is no specific instruction. In another example of this embodiment, selecting one path 55 may include monitoring instructions (behavior 40) from the target user OU and selecting a path generated by one or more second trained control models 32 in accordance with the result of said monitoring. If the moving object M is a vehicle, selecting one path 55 may include selecting a path generated by the trained lane change model 33 in response to a lane change instruction from the target user OU.

[0055] Furthermore, in one example of this embodiment, between steps S104 and S105, the control unit 11 may exclude paths that are evaluated as having a low level of safety from among the one or more generated paths 50, according to the result of evaluating the level of safety. Accordingly, selecting one path 55 may be performed by excluding paths that are evaluated as having a low level of safety and then selecting one path 55 from the one or more generated paths 50. An evaluation of a low level of safety may mean that the level of safety does not meet the criteria. In one example, whether or not the level of safety is low may be determined by a threshold comparison. In this case, the criteria for the level of safety is defined by a threshold.

[0056] For example, if the safety assessment results exclude a path of the trained control model corresponding to the behavior 40 of the target user OU, the control unit 11 may provide feedback to the target user OU that the corresponding path has been excluded, for example, by voice, image, or any other method. Furthermore, if a path corresponding to the behavior 40 of the target user OU is excluded, the control unit 11 may select the path with the highest safety level. If a path of the second trained control model 32 corresponding to a specific instruction by the target user OU is excluded, the control unit 11 may select a path generated by the first trained control model 31. If, as a result of the safety assessment, all paths 50 are excluded... If this occurs, the control unit 11 may switch from automatic control mode to manual control mode and output a notification to any user (typically, the target user OU) prompting them to manually operate the mobile unit M.

[0057] In step S106, the control unit 11 controls the movement of the mobile object M according to the selected path 55. Once the control of the target mobile object M is complete, the control unit 11 terminates the processing procedure of the control device 1 in this example of operation.

[0058] The control unit 11 may repeatedly execute the series of information processing steps S101 to S106 at any time. For example, the control unit 11 may repeatedly execute the series of information processing steps S101 to S106 for a predetermined period of time (for example, while the power source of the mobile body M is started and the automatic control mode is selected). This allows the control device 1 to continuously perform automatic control of the mobile body M.

[0059] Note that the processing order of each step is not limited to the example in Figure 5 and may be changed as appropriate depending on the embodiment. For example, the processing in step S103 may be executed at any time before the processing in step S105. The processing in step S104 may be executed after the processing in step S102 and at any time before the processing in step S105. The processing in step S104 may be omitted.

[0060] [Features] In this embodiment, multiple trained control models 30 are prepared according to the respective movement patterns associated with the user's behavior. Then, through the processing of steps S102 to S106 (excluding step S104), the path 55 generated by the trained control model 35 corresponding to the behavior 40 of the target user OU is used to control the mobile body M. This is expected to improve the accuracy of performing control of the mobile body M that is adapted to the user's behavior.

[0061] [4. Variant] While embodiments of this disclosure have been described in detail above, the above description is merely illustrative in all respects of this disclosure. Needless to say, various improvements or modifications can be made without departing from the scope of this disclosure. The processes and means described in this disclosure can be freely combined and implemented as long as no technical inconsistencies arise. [Explanation of Symbols]

[0062] 1...Control device, 11...Control unit, 12...Storage unit, 30-35…Trained control model, 50-55…Path, M...Mobile entity, OU...Target user, S...Sensor, MD...Monitoring device

Claims

1. A memory unit for storing multiple trained control models, and Control unit, A control device comprising, The control unit, To generate one or more paths by performing all operations of the plurality of trained control models, regardless of whether each of the plurality of trained control models is used or not. Monitoring the behavior of target users, Based on the results of monitoring the behavior of the target user, select one path from the one or more paths that have been generated, and To control the movement of a moving object according to the selected path. It is configured to perform, and Each of the aforementioned trained control models has acquired the ability, through machine learning, to generate paths for controlling the movement of the mobile object in relation to the corresponding user behavior. Control device.

2. The plurality of trained control models include a first trained control model that has acquired the ability to generate paths in situations without user instructions, and one or more second trained control models that have acquired the ability to generate paths in situations with corresponding user instructions. The control unit is further configured to evaluate the safety level of each of the one or more paths generated. Selecting one of the aforementioned paths means In response to specific instructions from the target user, the paths of the second trained control model that are evaluated as having a low level of safety are excluded, and In response to excluding the paths of the second trained control model that are evaluated as having a low safety level, select the one path with the highest safety level from among the one or more paths generated by the first trained control model that are included in the one or more paths generated. including, The control device according to claim 1.

3. The one or more trained control models include one or more second trained control models, each having acquired the ability to generate paths in situations with corresponding instructions from the user, Selecting one of the paths includes selecting a path generated by one of the one or more second trained control models in response to instructions from the target user, The aforementioned moving object is a vehicle, The one or more second trained control models include a trained lane change model, Selecting one of the paths includes selecting a path generated by the trained lane change model in response to a lane change instruction from the target user. The control device according to claim 1.

4. The control unit is further configured to evaluate the safety level of each of the one or more paths generated. The selection of the aforementioned path is comprised of excluding the paths evaluated as having a low safety level, and then selecting one path from the one or more paths generated. If, by excluding the paths that are evaluated as having a low level of safety, all of the one or more paths that were generated are excluded, The control unit, Switching the control mode of the aforementioned mobile body from automatic control mode to manual control mode, To output a notification to the target user prompting them to manually operate the mobile device, It is configured to perform further actions. The control device according to claim 1.

Images