Control device, computer program product comprising a control program, and control method
By adopting the combination of the first and second control models in the autonomous driving system, the problem of the autonomous driving system derives control instructions in abnormal environments is solved, reducing the frequency of manual operation and improving the reliability of the system.
Patent Information
- Application Number
- CN202410706119.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2023-06-05
- Filing Date
- 2024-06-03
- Publication Date
- 2024-12-06
AI Technical Summary
The existing autonomous driving system cannot export control commands normally when the camera lens is attached to dirt or the surrounding environment is not obvious, resulting in frequent switching to manual driving, affecting the reliability of autonomous driving.
Using a combination of the first control model and the second control model, the second control model is different from the first control model through its type and input data. It can normally export control instructions when the first control model is derived abnormally, or use the user's Manual operation instructions for control.
The control frequency based on manual operation is reduced, and the reliability and stability of the autonomous driving system in abnormal scenarios is improved.
Smart Images

Figure CN119078857A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to a control device, a computer program product including a control program, and a control method. Background Art
[0002] Patent Document 1 proposes a system for autonomous vehicle control, which is configured to use a trained neural network to determine vehicle commands based on a route, GPS data, and sensor data.
[0003] Prior art literature
[0004] Patent Document 1: Japanese Patent Application No. 2019-533810 Summary of the invention
[0005] Technical problem to be solved by the invention
[0006] One of the objects of the present disclosure is to provide a control technology for reducing the frequency of control by manual operation.
[0007] Technical solutions to solve problems
[0008] The control device involved in the first technical solution of the present disclosure includes a storage unit and a control unit for storing a first control model and a second control model. The second control model is configured to be different from the first control model by at least one of its type and input data, so that the control instruction can be normally derived in at least a part of the scene where the derivation of the control instruction based on the first control model is abnormal. The control unit is configured to execute: using the first control model, trial derivation of the first control instruction; when the derivation of the first control instruction based on the first control model is normal, control the movement of the mobile body according to the derived first control instruction; and when the derivation of the first control instruction based on the first control model is abnormal, control the movement of the mobile body according to the second control instruction derived by the second control model or the third control instruction from the manual operation of the user. In addition, in one example, at least one of the first control model and the second control model can be composed of a trained machine learning model.
[0009] The second technical solution of the present disclosure involves a storage medium storing a control program. The control program is a program for causing a control device that maintains a first control model and a second control model to perform the following processing, the processing including: using the first control model, trial-exporting a first control instruction; when the export of the first control instruction based on the first control model is normal, controlling the movement of a mobile body according to the exported first control instruction; and when an abnormality occurs in the export of the first control instruction based on the first control model, controlling the movement of the mobile body according to a second control instruction exported by the second control model or a third control instruction from a manual operation of a user, the second control model being configured to be different from the first control model by at least one of its type and input data, and being able to export control instructions normally in at least a portion of the scene where an abnormality occurs in the export of control instructions based on the first control model.
[0010] The control method involved in the third technical scheme of the present disclosure is an information processing method in which a control device that maintains a first control model and a second control model performs the following processing, the processing including: using the first control model, trying to export a first control instruction; when the export of the first control instruction based on the first control model is normal, controlling the movement of a mobile body according to the exported first control instruction; and when an abnormality occurs in the export of the first control instruction based on the first control model, controlling the movement of the mobile body according to a second control instruction exported by the second control model or a third control instruction from a manual operation of a user, the second control model being configured to be different from the first control model by at least any one of its type and input data, and being able to export control instructions normally in at least a portion of the scenarios where an abnormality occurs in the export of control instructions based on the first control model.
[0011] Effects of the Invention
[0012] According to the present disclosure, it is expected that the frequency of control by manual operation will be reduced. BRIEF DESCRIPTION OF THE DRAWINGS
[0013] Figure 1 An example of a scenario in which the present disclosure is applied is shown in a schematic manner.
[0014] Figure 2A An example of the relationship between the first control model and the second control model according to the present embodiment is schematically shown.
[0015] Figure 2B An example of the relationship between the first control model and the second control model according to the present embodiment is schematically shown.
[0016] Figure 3 An example of the control process in this embodiment is schematically shown.
[0017] Figure 4A An example of a scenario in which an abnormality occurs in the derivation of a control command in the present embodiment is schematically shown.
[0018] Figure 4B An example of a scenario in which an abnormality occurs in the derivation of a control command in the present embodiment is schematically shown.
[0019] Figure 4C An example of a scenario in which an abnormality occurs in the derivation of a control command in the present embodiment is schematically shown.
[0020] Figure 5 An example of the hardware configuration of the control device according to the present embodiment is schematically shown.
[0021] Figure 6 An example of the software configuration of the control device according to the present embodiment is schematically shown.
[0022] Figure 7 This is a flowchart showing an example of a processing procedure related to control performed by the control device according to the present embodiment.
[0023] Figure 8 An example of a control process in another embodiment is shown schematically.
[0024] Fig. 9 This is a flowchart showing an example of a processing procedure related to control performed by a control device according to another embodiment.
[0025] Description of symbols
[0026] 1. Control device,
[0027] 11 control unit, 12 storage unit, 13 external interface,
[0028] 14 input device, 15 output device, 16 driver,
[0029] 81 control program, 91 storage medium,
[0030] 111 an acquisition unit, 112 a first derivation unit, 113 a second derivation unit,
[0031] 114 determination unit, 115 operation receiving unit, 116 action control unit,
[0032] 125 Observational data,
[0033] 30 1st control model, 300 1st model data,
[0034] 35 2nd control model, 350 2nd model data,
[0035] 40 Manual operation,
[0036] 50 1st control instruction, 55 2nd control instruction,
[0037] 60 3rd control instruction,
[0038] M moving body, S sensor DETAILED DESCRIPTION
[0039] Conventionally, rule-based autonomous driving systems are known. In addition, according to the method of Patent Document 1, an autonomous driving system can be constructed by using a trained machine learning model. However, the inventors of the present application have found that these existing methods have the following problems.
[0040] That is, no matter which method is used, the rule base model or the machine learning model, it is not necessarily always possible to derive control instructions normally. In one example, when the image obtained by the camera is used to derive the control instructions, the control instructions may not be derived normally because the camera lens is covered with dirt (such as rain, mud, etc.) and / or the surroundings are dark and the surrounding environment is not reflected in the image. In another example, when the position information is used to derive the control instructions, the position information may not be measured and the control instructions cannot be derived normally. In such a case, automatic driving cannot be performed, so the vehicle will be controlled by manual driving. In the case where the state in which the control instructions cannot be derived normally frequently occurs in the equipped control model, the frequency of control based on manual driving will increase.
[0041] In addition, this problem may occur regardless of the type of vehicle. In addition, the occurrence of such a problem is not limited to the scene of controlling the vehicle. The same is true for moving objects other than vehicles in terms of controlling the movement of movement. Therefore, even in the scene of controlling all moving objects other than vehicles (such as flying objects, ships, robot devices, etc.), the same problem may occur.
[0042] In this regard, the control device involved in the first technical solution of the present disclosure includes a storage unit for storing a first control model and a second control model, and a control unit. The second control model is configured to be different from the first control model by at least one of its type and input data, so that the control instruction can be normally derived in at least a part of the scene where the derivation of the control instruction based on the first control model is abnormal. The control unit is configured to execute: using the first control model, trial derivation of the first control instruction; when the derivation of the first control instruction based on the first control model is normal, control the movement of the mobile body according to the derived first control instruction; and when the derivation of the first control instruction based on the first control model is abnormal, control the movement of the mobile body according to the second control instruction derived by the second control model or the third control instruction from the manual operation of the user.
[0043] In the first technical solution of the present disclosure, a second control model is provided together with the first control model, and the second control model is different in at least one of the type of its model and the input accepted, so that the control instruction can be derived from a different point of view. By being configured to derive the control instruction from a different point of view, the range in which the control instruction can be normally derived by the second control model may be different from that of the first control model. That is, it can be expected that the second control model can normally derive the control instruction in at least a part of the scene where the control instruction cannot be normally derived by the first control model. Therefore, it is possible to perform automatic control based on the second control model in at least a part of the scene where the control instruction cannot be normally derived by the first control model. That is, it is possible to support at least a part of the range in which it is difficult to use the first control model by the second control model. Therefore, according to the first technical solution of the present disclosure, it is possible to seek to reduce the frequency of control based on manual operation.
[0044] As other forms of the control device involved in the above technical solution, one aspect of the present disclosure may be an information processing method that realizes all or part of the above components, or a program, or a storage medium that stores such a program and can be read by a computer or other device. Here, the storage medium that can be read by a device refers to a medium that stores information such as a program through electrical, magnetic, optical, mechanical or chemical effects.
[0045] For example, the storage medium involved in the second technical solution of the present disclosure stores a control program. The control program is a program for causing a control device that maintains a first control model and a second control model to perform the following processing, the processing including: using the first control model, trial-exporting a first control instruction; when the export of the first control instruction based on the first control model is normal, controlling the movement of a mobile body according to the exported first control instruction; and when an abnormality occurs in the export of the first control instruction based on the first control model, controlling the movement of the mobile body according to a second control instruction exported by the second control model or a third control instruction from a manual operation of a user, the second control model being configured to be different from the first control model by at least one of its type and input data, and being able to export control instructions normally in at least a portion of scenarios where an abnormality occurs in the export of control instructions based on the first control model.
[0046] In addition, the control method involved in the third technical scheme of the present disclosure is an information processing method in which a control device that maintains a first control model and a second control model performs the following processing, the processing including: using the first control model, trying to export a first control instruction; when the export of the first control instruction based on the first control model is normal, controlling the movement of the mobile body according to the exported first control instruction; and when an abnormality occurs in the export of the first control instruction based on the first control model, controlling the movement of the mobile body according to a second control instruction exported by the second control model or a third control instruction from a manual operation of a user, the second control model being configured to be different from the first control model in at least one of its type and input data, and being able to export control instructions normally in at least a portion of the scenarios where an abnormality occurs in the export of control instructions based on the first control model.
[0047] Hereinafter, an embodiment (hereinafter also referred to as "this embodiment") relating to one aspect of the present disclosure will be described based on the accompanying drawings. However, the present embodiment described below is merely an illustration of the present disclosure in all respects. Various improvements or modifications may be made without departing from the scope of the present disclosure. When implementing the present disclosure, a specific configuration corresponding to the embodiment may also be appropriately adopted. In addition, the data appearing in the present embodiment is described by natural language, but more specifically, it is specified by pseudo-language, commands, parameters, machine language, etc. that can be recognized by a computer.
[0048] [1 Application example]
[0049] Figure 1An example of a scenario to which the present disclosure is applied is schematically shown. The control device 1 according to the present embodiment is one or more computers configured to control the automatic movement of a moving object M as a target.
[0050] In the present embodiment, the control device 1 is mounted on the mobile body M and holds the first control model 30 and the second control model 35. The first control model 30 and the second control model 35 are respectively configured to be able to derive control instructions for automatically controlling the movement of the mobile body M. The second control model 35 is configured to be able to normally derive control instructions in at least a part of the scene where an abnormality occurs in the derivation of control instructions based on the first control model 30, because at least any one of the type of its model and the accepted input data is different from the first control model 30.
[0051] In the present embodiment, the control device 1 uses the first control model 30 to test the derivation of the first control instruction 50. In this test, when the first control model 30 is used under the appropriate conditions of the first control model 30, the first control instruction 50 can be normally derived. On the other hand, when the first control model 30 is not used under the appropriate conditions, for example, when an image cannot be obtained normally or position information cannot be obtained, the first control instruction 50 may not be normally derived (that is, an abnormality occurs in the derivation of the first control instruction 50).
[0052] When the derivation of the first control instruction 50 based on the first control model 30 is normal, the control device 1 controls the movement of the mobile body M according to the derived first control instruction 50. On the other hand, when an abnormality occurs in the derivation of the first control instruction 50 based on the first control model 30, the control device 1 controls the movement of the mobile body M according to the second control instruction 55 derived from the second control model 35 or the third control instruction 60 derived from the manual operation 40 of the user.
[0053] As described above, in the present embodiment, the second control model 35 capable of deriving control instructions from a different perspective is configured together with the first control model 30. Thus, it can be expected that the control instruction (second control instruction 55) can be normally derived by the second control model 35 in at least a part of the scene where the control instruction (first control instruction 50) cannot be normally derived by the first control model 30. Therefore, even if an abnormality occurs in the derivation of the first control instruction 50 based on the first control model 30, it can be expected to continue the automatic control based on the second control model 35. That is, at least a part of the range in which it is difficult to use the first control model 30 can be supported by the second control model 35. In the case where the automatic control can continue by the second control model 35, it is sufficient not to switch to the control based on the manual operation 40. Therefore, according to the present embodiment, it is possible to seek to reduce the frequency of control based on manual operation.
[0054] (Moving body)
[0055] As long as it can be automatically moved by mechanical control, the type of the mobile body M can be appropriately selected according to the embodiment. The mobile body M can be, for example, a vehicle, an aircraft, a ship, a robot device, or other movable device. The aircraft can be at least one of an unmanned aircraft such as a drone and a manned aircraft.
[0056] In an example, Figure 1 As shown, the mobile body M can be a vehicle. In this case, it is possible to seek to reduce the control frequency based on manual operation in the scenario of implementing automatic driving of the vehicle. In addition, when the mobile body M is a vehicle, the type of vehicle (number of wheels, power source, size, etc.) can be selected arbitrarily. The vehicle can be selected from, for example, a two-wheeled vehicle, a three-wheeled vehicle, a four-wheeled vehicle, etc. The power source of the vehicle can be selected from, for example, electricity, fuel, etc. In the case where the vehicle is a car, the size of the vehicle can be selected from large, medium, quasi-medium, ordinary, large special, small special, etc. In the case where the vehicle is a two-wheeled vehicle, the size of the vehicle can be selected from large, ordinary, etc. As a typical example, the mobile body M can be a car with an automatic driving capability of level 2 or above. In addition, the user can be any person participating in the mobile body M. In the case where the mobile body M is a vehicle, typically, the user can be a driver.
[0057] (Control)
[0058] In one example, the motion of the object's moving body M can be controlled by directly controlling the object's moving body M. That is, the control device 1 can be configured to directly control the object's moving body M. In another example, the moving body M can be provided with a dedicated control device such as a controller. In this case, the motion of the object's moving body M controlled by the control device can be controlled by providing the derived result to the dedicated control device, thereby indirectly controlling the object's moving body M.
[0059] In addition, the control device 1 can be installed in any place. Figure 1 As shown, the control device 1 can be mounted on the mobile body M. The mounting location can be determined arbitrarily, for example, the control device 1 can be installed inside or outside the mobile body M. In another example, the control device 1 can be configured away from the mobile body M and configured to control the mobile body M in a remote manner. The control device 1 can be connected to the mobile body M in a wired or wireless manner.
[0060] (Control Model)
[0061] The first control model 30 and the second control model 35 are respectively constructed to derive control instructions according to the environment of the mobile body M. The environment is an event observed by at least one of the mobile body M itself and the surroundings. In one example, at least a part of the environment can be observed by one or more sensors S arranged inside or outside the mobile body M. As long as the sensor S can observe any environment in which the mobile body M moves, its type is not particularly limited and can be appropriately selected according to the implementation method. In one example, the one or more sensors S may include a camera (image sensor), a radar, a LiDAR (Light Detection And Ranging)), a sonar (ultrasonic sensor), an infrared sensor, a GNSS (Global Navigation Satellite System) / GPS (Global Positioning Satellite) module, etc.
[0062] As long as the control instructions can be derived from the environment of the mobile body (mobile body M), the input and output methods of the control model (the first control model 30, the second control model 35) may not be particularly limited and may be appropriately selected according to the implementation method. In one example, at least one of the first control model 30 and the second control model 35 may be configured to derive the control instructions based on the observation data of the sensor at more than one time point. In another example, at least one of the first control model 30 and the second control model 35 may be configured to derive the control instructions based on the recognition result of the surrounding environment. The recognition result of the surrounding environment can be obtained by parsing the observation data of the sensor by any method (i.e., using the analytical model). The control device 1 may also have an analytical model that infers the recognition result of the surrounding environment based on the observation data of the sensor. In one example, the analytical model may be configured to obtain the recognition result of the surrounding environment from the observation data according to the rule base by performing analytical processing such as pattern matching and edge detection. In another example, the analytical model may be composed of a trained machine learning model. The analytical model can be regarded as a part of the control model, or it can be regarded as a component different from the control model. Other information may be added to the input of the control model as desired. For example, the control model may be configured to accept input of any information such as a set speed, a speed limit, a position, map information, and navigation information.
[0063] In addition, in one example, the control model (the first control model 30, the second control model 35) can be configured to directly output the control instruction. In another example, the control model can also be configured to indirectly output the control instruction, and the control instruction can be obtained by performing any information processing (such as judgment, selection and other interpretation processing) on the output of the control model. The control instruction can also be configured to directly represent the control amount (control instruction value, control output amount) of the mobile body M such as the accelerator control amount, the brake control amount, the steering wheel steering angle, etc. Or, the control instruction can be configured to indirectly represent the control amount of the mobile body M such as the path, the state after control, etc. In this case, the control amount of the mobile body M can be obtained from the control instruction by performing any information processing. In one example, when the mobile body M is a vehicle, the control amount of the vehicle can be obtained by applying the inference result obtained from the control model to the vehicle model. The vehicle model can have various parameters such as accelerator, brake, steering wheel steering, etc., and can be appropriately configured to derive the control amount from indirect information (path, state after control, etc.). The output form of each of the first control model 30 and the second control model 35 can be arbitrarily selected. In one example, one of the first control model 30 and the second control model 35 may be configured to directly output a control command, and the other may be configured to indirectly output a control command.
[0064] The configuration of each of the first control model 30 and the second control model 35 can be appropriately selected according to the implementation method. In one example, the control model (the first control model 30, the second control model 35) can be composed of a trained machine learning model (e.g., an end-to-end model), a rule base model, or a combination thereof.
[0065] The rule base model is configured to compare the input provided (e.g., observation data, recognition results of the surrounding environment, and other information representing the environment) with the rules, and derive control instructions based on the results of the comparison (according to the appropriate rules). The rules can be set manually or at least partially automatically.
[0066] The machine learning model is configured to have one or more operation parameters that can be adjusted by machine learning. One or more operation parameters are used for the operation of inference as the purpose (derivation of control instructions in the present disclosure). Machine learning refers to the use of learning data to adjust (optimize) the value of the operation parameter. The machine learning model can be composed of, for example, a neural network, a support vector machine, a regression model, a decision tree model, etc. The machine learning method can be appropriately selected according to the machine learning model adopted (such as the error back propagation method, etc.).
[0067] As an example, at least one of the first control model 30 and the second control model 35 can be composed of a neural network. The structure of the neural network can be appropriately determined according to the implementation method, for example, by the number of layers from the input layer to the output layer, the type of each layer, the number of nodes (neurons) contained in each layer, the connection relationship between the nodes of each layer, etc. In one example, the neural network can have a recursive structure. In addition, the neural network can include any layer such as a fully connected layer, a convolutional layer, a pooling layer, a deconvolutional layer, an unpooling layer, a normalization layer, a dropout layer, and an LSTM (Long short-term memory). The neural network can have any mechanism such as an attention mechanism. The neural network can include any model such as a GNN (Graph neural network), a diffusion model, a generative model (such as a Generative Adversarial Network, a Transformer, etc.). When a neural network is used in a control model, the weight of the connection between the nodes contained in the control model and the threshold of each node are examples of operation parameters. In addition, when a machine learning model is adopted, the control model can be constructed by an end-to-end model.
[0068] (Relationship between the first control model and the second control model)
[0069] In the present embodiment, the first control model 30 is mainly used in the control of the mobile body M, and the second control model 35 is used in an auxiliary manner in the control of the mobile body M as a substitute means when the output of the first control model 30 is poor (abnormal). That is, the second control model 35 is used to support the first control model 30. The first control model 30 and the second control model 35 are configured so that at least one of the model type and the accepted input data (provided data) is different from each other. Therefore, the second control model 35 is configured to be able to normally derive control instructions in at least a part of the scene where an abnormality occurs in the derivation of control instructions based on the first control model 30.
[0070] Different types of models refer to different computational steps of the inference processing. If the computational steps of the inference processing are different, the conditions for poor derivation of control instructions may be different. The different types of models can be achieved, for example, by using a machine learning model for the first control model 30, a rule base model for the second control model 35, or machine learning models with different structures for the first control model 30 and the second control model 35. In one example, a neural network with a recursive structure can be used for the first control model 30, and a neural network without a recursive structure can be used for the second control model 35.
[0071] The input data provided may be different, and the control model may be configured by at least one of the following: at least a part of the input data provided to the control model is different in type, and the input data is the same in type but the resolution (amount of information) is different. If at least a part of the input data is different in type, the conditions for obtaining the input data normally may be different, and therefore, the conditions for the poor derivation of the control command may also be different. The same is true in the case of different resolutions (generally speaking, low-resolution data is easier to obtain normally than high-resolution data). As an example, one of the first control model 30 and the second control model 35 may be configured to receive image data and other data (such as location information, map information, navigation information, etc.), while the other may be configured to receive only image data. As another example, one of the first control model 30 and the second control model 35 may be configured to receive high-resolution data (such as location information, image data, etc.), while the other may be configured to receive low-resolution data. In this case, the first control model 30 may be configured to receive high-resolution data input, and the second control model 35 may be configured to receive low-resolution data input. In the present embodiment, due to at least one of the difference in model type and the difference in input data, the first control model 30 and the second control model 35 may be configured so that at least a portion of the scenarios in which control commands can be normally derived may differ.
[0072] Figure 2A An example of the relationship between the first control model 30 and the second control model 35 according to the present embodiment is schematically shown. Figure 2AIn one example, the first control model 30 is composed of a trained machine learning model 30A, and the second control model 35 is composed of a rule base model 35A. In one example, the trained machine learning model 30A can have an end-to-end model structure. In addition, in one example, the rule base model 35A can be configured to identify the surrounding environment from at least a portion of the observation data obtained by the sensor, compare the recognition result of the surrounding environment with the rules, and derive control instructions based on the comparison result (according to the appropriate rules). In order to identify the surrounding environment, the rule base model 35A may include a parsing model.
[0073] In general, the rule base model derives control instructions according to the rules. Therefore, compared with the machine learning model, the process of deriving control instructions is clear and the action of the rule base model is easy to predict. Therefore, according to this example, in a scenario where the derivation of control instructions based on the first control model 30 fails, it is expected that the second control model 35 with high predictability of action will continue to perform stable automatic control of the moving body M. In addition, the inputs of the trained machine learning model 30A and the rule base model 35A can be appropriately selected according to the implementation method. The inputs of the trained machine learning model 30A and the rule base model 35A can be the same or different.
[0074] Figure 2B Another example of the relationship between the first control model 30 and the second control model 35 according to the present embodiment is schematically shown. Figure 2B In one example, the first control model 30 is composed of a trained machine learning model 30B. The second control model 35 is composed of a trained machine learning model 35B whose input data is at least partially different from that of the first control model 30 (trained machine learning model 30B).
[0075] In one example, the input data of one of the trained machine learning model 30B and the trained machine learning model 35B may be composed of only a portion of the input data of the other. Figure 2B As shown, the trained machine learning model 30B may be configured to accept input of multiple data (e.g., image data and other data), whereas the trained machine learning model 35B may only accept input of a portion of the data (e.g., only image data).
[0076] In another example, the input data of the trained machine learning model 30B and the trained machine learning model 35B may be configured to have a portion repeated and other portions different. As a specific example, the trained machine learning model 30B may be configured to accept inputs of the first data and the second data (e.g., image data and location information), whereas the trained machine learning model 35B may accept inputs of the first data and the third data (e.g., image data and LiDAR data). The first data to the third data may each be composed of more than one type of data.
[0077] In another example, the input data of the trained machine learning model 30B and the trained machine learning model 35B may be configured to be completely different. As a specific example, the trained machine learning model 30B may be configured to accept only the input of the first data (e.g., image data / position information), whereas the trained machine learning model 35B may be configured to accept only the input of the second data (e.g., position information / image data). The first data and the second data may each be composed of more than one type of data.
[0078] In the case of adopting a method in which the types of input data of the trained machine learning model (30B, 35B) are repeated, the resolution (amount of information) of each input data may also be different. As a specific example, one side of the trained machine learning model (30B, 35B) may be configured to accept the input of high-resolution data (such as image data, location information, etc.), while the other side may be configured to accept the input of low-resolution data. In one example, the trained machine learning model 30B may be configured to accept the input of high-resolution data, while the trained machine learning model 35B may be configured to accept the input of low-resolution data. High resolution and low resolution refer to the relationship between the data input to each other, and are formed by the amount of the same type of data input to one side being different from that of the other side. In the case of adopting a method in which different resolutions are adopted, the types of input data of the trained machine learning models (30B, 35B) may also be completely consistent.
[0079] By at least partially different input data between the first control model 30 (trained machine learning model 30B) and the second control model 35 (trained machine learning model 35B), the data (explanatory variables) used in the derivation (inference) of the control instructions are different. As a result, the conditions for obtaining input data that can be used for normal derivation of control instructions may be different. Therefore, the range of poor derivation of control instructions can be different between the first control model 30 and the second control model 35. Therefore, according to this example, in a scenario where at least a part of the derivation of control instructions based on the first control model 30 is poor, it is expected that automatic control can be continued through the second control model 35.
[0080] Furthermore, when this method is adopted, in one example, the second control model 35 (trained machine learning model 35B) can be configured to have a smaller computational load than the first control model 30 (trained machine learning model 30B) by at least one of a smaller amount of input data and a smaller number of computational parameters. Thus, it is possible to reduce the frequency of control based on manual operations while suppressing an increase in computational load.
[0081] In one example, the first control model 30 (trained machine learning model 30B) may be trained in a general manner, while the second control model 35 (trained machine learning model 35B) may be trained in a dedicated manner. Alternatively, the first control model 30 may also be trained in a dedicated manner, but the second control model 35 may be trained in a more dedicated manner than the first control model 30. Training in a dedicated manner may be configured, for example, by limiting the domain of training samples (using training samples from a local domain for machine learning) and training in a manner that makes it easier for the second control model 35 to perform better than the first control model 30 in a specific domain. As a specific example, when the moving body M is a vehicle, the domain of training samples may be limited by giving priority to training samples related to driving in a tunnel, giving priority to training samples related to driving in a specific area, and so on. For the domain of training samples used for machine learning, a domain in which a control model (trained machine learning model 30B) trained in a general manner is prone to poor derivation may be selected. As a result, when the first control model 30 (trained machine learning model 30B) encounters export failure in a specific domain, the probability of continuing automatic control through the second control model 35 (trained machine learning model 35B) can be increased.
[0082] (Control Instructions)
[0083] The control instructions (the first control instruction 50, the second control instruction 55, and the third control instruction 60) are related to the movement of the mobile body M. The composition of the control instructions can be appropriately selected according to the implementation method. In one example, the control instructions can be composed of acceleration, deceleration, steering, or a combination of these. Acceleration and deceleration may include shifting gears. In the case of including at least any one of acceleration, deceleration, and steering, the control instructions can be represented by a path. Accordingly, the control model (the first control model 30 and the second control model 35) can be represented as a path planner. In addition, the control instructions may also include instructions related to the operation of the mobile body M. As an example, when the mobile body M is a vehicle, the control instructions may include vehicle operations such as turn indicators, hazard indicators, horns, communication processing (such as sending data to a center, making an emergency call, etc.).
[0084] (2nd control model and manual operation allocation)
[0085] When an abnormality occurs in the derivation of the first control instruction 50 based on the first control model 30, the allocation of whether to execute the automatic control based on the second control model 35 or the control based on the manual operation 40 can be appropriately determined according to the implementation method. As a simple example, the movement of the mobile body M is controlled according to the second control instruction 55 or the third control instruction 60. The movement of the mobile body M can be controlled according to the derived second control instruction 55 when the derivation of the second control instruction 55 based on the second control model 35 is normal, and the movement of the mobile body M is controlled according to the third control instruction 60 from the manual operation 40 when an abnormality occurs in the derivation of the second control instruction 55 based on the second control model 35.
[0086] Figure 3 An example of a control process in the control device 1 according to the present embodiment is schematically shown. Figure 3 In one example, the control device 1 attempts to derive the first control instruction 50 based on the first control model 30. As a result of the trial derivation of the first control instruction 50, when the first control instruction 50 is normally derived, the control device 1 controls the movement of the mobile body M according to the derived first control instruction 50. On the other hand, when an abnormality occurs in the derivation of the first control instruction 50, the control device 1 attempts to derive the second control instruction 55 based on the second control model 35. The trial derivation of the second control instruction 55 based on the second control model 35 can be performed regardless of whether the derivation of the first control instruction 50 is successful or not.
[0087] As a result of the trial of deriving the second control command 55, when the second control command 55 is derived normally, the control device 1 controls the motion of the mobile body M according to the derived second control command 55. On the other hand, when an abnormality occurs in the derivation of the second control command 55, the control device 1 accepts the input of the manual operation 40, and controls the motion of the mobile body M according to the third control command 60 derived from the input manual operation 40. Thus, the control mode is switched in the order of the first control model 30, the second control model 35, and the manual operation 40, so that the possibility of continuing the automatic control by the second control model 35 can be ensured. As a result, the control frequency based on the manual operation can be reduced.
[0088] In addition, the acceptance of the manual operation 40 may be performed at any timing. The control device 1 may also accept the input of the manual operation 40 during the execution of the automatic control based on the first control model 30 or the second control model 35. The control device 1 may switch from the automatic control mode based on the first control model 30 or the second control model 35 to the control mode based on the manual operation 40 according to any operation of the user. When switching from the automatic control based on the first control model 30 or the second control model 35 to the manual control based on the manual operation 40, the control device 1 may discard (ignore) the control instruction derived from the control model, or overlap it with the derived control instruction, and control the movement of the mobile body M according to the third control instruction 60 derived from the manual operation 40. Discarding may be instant switching from automatic control to manual control, that is, instant switching from control based on the control instruction derived from the control model to control according to the manual operation 40 of the user. On the other hand, overlapping may be gradual switching from automatic control to manual control (user control).
[0089] (abnormal)
[0090] The abnormality in the derivation of the control instructions (first control instruction 50, second control instruction 55) based on the control models (first control model 30, second control model 35) may include all cases where normal control instructions cannot be obtained.
[0091] Figure 4A , Figure 4B as well as Figure 4C An example of a scenario in which an abnormality occurs in the derivation of a control command in the control device 1 according to the present embodiment is schematically shown.
[0092] exist Figure 4A In the example, an abnormality occurs in the derivation of the first control instruction 50 based on the first control model 30, including incomplete output of the first control instruction 50. Incomplete output may be at least one of failure to make a meaningful output capable of deriving the control instruction, outputting an error signal, and failure to make any output.
[0093] exist Figure 4B In the example, the derivation of the first control instruction 50 based on the first control model 30 is abnormal, including that the reliability of the output of the first control model 30 is less than a threshold. The threshold can be determined arbitrarily. The situation where the reliability is equal to the threshold can be classified as any of abnormal and normal situations.
[0094] Reliability can be obtained by any method. In one example, Figure 4BAs shown, the first control model 30 can be configured to output the reliability of the inference result together with the inference result by any method such as providing reliability as a teacher signal during machine learning training (i.e., giving reliability to training data).
[0095] In another example, the reliability can be inferred based on the output of the first control model 30. For example, when the first control model 30 includes an autoencoder, the control device 1 can calculate the reconstruction error of the autoencoder as the reliability. In this case, the larger the reconstruction error, the lower the reliability evaluated by the control device 1. In addition, for example, when the output of the first control model 30 includes the result of class recognition, the control device 1 can calculate the inter-class entropy as the reliability. In this case, the larger the entropy value, the lower the reliability evaluated by the control device 1. In another example, the control device 1 can calculate the reliability based on the first control instruction 50 derived through the first control model 30 through any analytical processing such as simulation.
[0096] exist Figure 4C In the example, an abnormality occurs in the derivation of the first control instruction 50 based on the first control model 30, including that the change per unit time of the output of the first control model 30 exceeds a threshold value. The threshold value can be determined arbitrarily. The situation where the change is equal to the threshold value can be classified as any of abnormal and normal situations. Basically, for stable automatic control, the change of the control instruction derived by the control model will not increase. Therefore, it is possible to determine whether the derivation of the control instruction is successful or not based on the change of the output of the control model.
[0097] In an example, Figure 4C As shown, the amount of change in the output can also be calculated based on the change in the derivation result of the first control command 50. Figure 4C An example of calculating the change in the speed of acceleration or deceleration as the change in output is shown. In another example, as a simple method, the change in output can be calculated based on the change in the output value of the first control model 30. In addition, the change in output can be calculated by any method. In one example, the change in output can be calculated by the time integral of the difference in a predetermined period (the sum of the differences between t-t+1).
[0098] In this embodiment, the abnormality in the derivation of the first control instruction 50 may include at least one of the above three situations. Thus, in at least one of the above three situations, it is possible to judge that the derivation of the control instruction based on the first control model 30 is poor, and according to the result of the judgment, the automatic control based on the first control model 30 is switched to the automatic control based on the second control model 35. In addition, the first control model 30 is replaced with the second control model 35, and the abnormality in the derivation of the second control instruction 55 may also be the same.
[0099] [2 Configuration example]
[0100] [Hardware configuration example]
[0101] Figure 5 An example of the hardware configuration of the control device 1 according to the present embodiment is schematically shown. The control device 1 according to the present embodiment is a computer in which a control unit 11, a storage unit 12, an external interface 13, an input device 14, an output device 15, and a driver 16 are electrically connected.
[0102] The control unit 11 includes a CPU (Central Processing Unit), a RAM (Random Access Memory), a ROM (Read Only Memory), etc. as a hardware processor, and is configured to execute arbitrary information processing based on programs and various data. The control unit 11 (CPU) is an example of a processor resource.
[0103] The storage unit 12 may be composed of, for example, a hard disk drive, a solid state drive, etc. The storage unit 12 (as well as RAM and ROM) is an example of a memory resource. In the present embodiment, the storage unit 12 stores various information such as the control program 81, the first model data 300, and the second model data 350.
[0104] The control program 81 is used to cause the control device 1 to execute information processing related to the control of the moving body M (described later). Figure 7 , Fig. 9) program. The control program 81 includes a series of commands for the information processing. The first model data 300 is configured to include information related to the first control model 30. The second model data 350 is configured to include information related to the second control model 35. In one example, when at least one of the first control model 30 and the second control model 35 is composed of a trained machine learning model, at least one of the first model data 300 and the second model data 350 may include information representing the value of the operation parameter adjusted by machine learning. Depending on the circumstances, at least one of the first model data 300 and the second model data 350 may also include information representing the structure of the machine learning model (such as the structure of a neural network, etc.). In another example, when at least one of the first control model 30 and the second control model 35 is composed of a rule base model, at least one of the first model data 300 and the second model data 350 may include information representing the rules in the rule base model.
[0105] The external interface 13 may be, for example, a USB (Universal Serial Bus) port, a dedicated port, a wireless communication port, etc., and is configured to be connected to an external device in a wired or wireless manner. In this embodiment, the control device 1 may be connected to the sensor S via the external interface 13. The control device 1 may also be connected to any configuration (e.g., an operating component) of the mobile body M via the external interface 13.
[0106] The input device 14 is a device for performing input, such as a mouse, a keyboard, or an operating element. The output device 15 is a device for performing output, such as a display, a speaker, or the like. The user can operate the control device 1 by using the input device 14 and the output device 15. The input device 14 and the output device 15 may be configured in an integrated manner, such as by a touch panel display, or the like.
[0107] The drive 16 is a device for reading various information such as programs stored in the storage medium 91. At least any one of the control program 81, the first model data 300, and the second model data 350 may be stored in the storage medium 91 instead of being stored in the storage unit 12, or may be stored in the storage unit 12 and the storage medium 91. The storage medium 91 is configured to store various information (stored programs, etc.) through electrical, magnetic, optical, mechanical or chemical effects so that a computer or other device can read the information. The control device 1 can obtain at least any one of the control program 81, the first model data 300, and the second model data 350 from the storage medium 91. In addition, the storage medium 91 may be a disk-type storage medium such as a CD or a DVD, or may be a storage medium other than a disk-type storage medium such as a semiconductor memory (e.g., a flash memory). The type of the drive 16 may be appropriately selected according to the type of the storage medium 91.
[0108] In addition, regarding the specific hardware structure of the control device 1, the components can be appropriately omitted, replaced, and added according to the implementation method. For example, the control unit 11 may also include multiple hardware processors. The hardware processor can be composed of a microprocessor, an FPGA (field-programmable gate array), a DSP (digital signal processor), an ECU (Electronic Control Unit), a GPU (Graphics Processing Unit), etc. At least any one of the external interface 13, the input device 14, the output device 15, and the driver 16 can also be omitted. In addition to being designed as a computer dedicated to the services provided, the control device 1 can be a general-purpose computer, a mobile phone including a smartphone, a tablet PC (Personal Computer), and other terminal devices. In the case where the mobile body M is a vehicle, the control device 2 can be a vehicle-mounted device.
[0109] [Software Configuration Example]
[0110] Figure 6An example of the software configuration of the control device 1 involved in the present embodiment is shown in a schematic manner. The control unit 11 of the control device 1 expands the control program 81 stored in the storage unit 12 in the RAM, and executes the commands included in the control program 81 through the CPU. Thus, the control device 1 operates as a computer, and the computer includes the acquisition unit 111, the first derivation unit 112, the second derivation unit 113, the determination unit 114, the operation acceptance unit 115, and the action control unit 116 as software modules. That is, in the present embodiment, each software module of the control device 1 is realized by the control unit 11 (CPU).
[0111] The acquisition unit 111 is configured to acquire the observation data 125 of the sensor S. The first derivation unit 112 includes the first control model 30 by holding the first model data 300. The first derivation unit 112 is configured to derive the first control instruction 50 using the first control model 30. In the present embodiment, the first derivation unit 112 may be configured to derive the first control instruction 50 from at least a portion of the acquired observation data 125 using the first control model 30. The second derivation unit 113 includes the second control model 35 by holding the second model data 350. The second derivation unit 113 is configured to derive the second control instruction 55 using the second control model 35. In the present embodiment, the second derivation unit 113 may be configured to derive the second control instruction 55 from at least a portion of the acquired observation data 125 using the second control model 35.
[0112] The determination unit 114 is configured to determine whether the derivation of the control instructions (first control instruction 50, second control instruction 55) based on the control model (first control model 30, second control model 35) is normal. The operation receiving unit 115 is configured to receive the input of the manual operation 40 of the user and obtain the third control instruction 60 derived from the input manual operation 40. The motion control unit 116 is configured to control the movement of the mobile body M according to the derived first control instruction 50 when the derivation of the first control instruction 50 based on the first control model 30 is normal. Furthermore, the motion control unit 116 is configured to control the movement of the mobile body M according to the second control instruction 55 derived by the second control model 35 or the third control instruction 60 derived from the manual operation 40 of the user when an abnormality occurs in the derivation of the first control instruction 50 based on the first control model 30.
[0113] In addition, in this embodiment, an example is described in which each software module of the control device 1 is implemented by a general-purpose CPU. However, part or all of the above software modules may also be implemented by one or more dedicated processors. The above modules may also be implemented as hardware modules. Regarding the software configuration of the control device 1, modules may be omitted, replaced, or added as appropriate.
[0114] [3 Action Examples]
[0115] Figure 7 An example of a processing step related to the control of the mobile body M by the control device 1 involved in this embodiment is shown. The following processing steps are an example of a control method executed by a computer. However, the following processing steps are only an example, and each step can be changed as long as it is feasible. In addition, regarding the following processing steps, steps can be appropriately omitted, replaced, and added according to the embodiment.
[0116] (Step S101)
[0117] In step S101, the control unit 11 operates as the acquisition unit 111 to acquire the observation data 125 of the sensor S. The acquisition path of the observation data 125 can be arbitrarily selected. In one example, the control device 1 can be directly connected to the sensor S, and the control unit 11 can directly acquire the observation data 125 from the sensor S. In another example, the control unit 11 can indirectly acquire the observation data 125 from the sensor S via another computer. After acquiring the observation data 125, the control unit 11 causes the process to proceed to the next step S102.
[0118] (Step S102)
[0119] In step S102, the control unit 11 operates as the first derivation unit 112, and attempts to derive the first control command 50 using the first control model 30. The process of deriving the first control command 50 may be appropriately determined according to the embodiment.
[0120] As an example, when the first control model 30 includes a trained machine learning model, the control unit 11 may input at least a portion of the acquired observation data 125 to the first control model 30, and perform arithmetic processing (e.g., forward propagation arithmetic processing) of the first control model 30. As a result of the arithmetic processing, the control unit 11 may obtain an output value corresponding to the inference result (direct or indirect derivation result of the control instruction) from the first control model 30. In this case, the control unit 11 may refer to the first model data 300 at any timing before processing step S102, and set the first control model 30 (trained machine learning model) to a state capable of performing arithmetic processing.
[0121] As another example, when the first control model 30 includes a rule base model, the control unit 11 may refer to the first model data 300 and derive the first control instruction 50 according to the rules specified by the rule base model based on the environment represented by at least a part of the observation data 125. When the rule base model includes an analytical model, the control unit 11 may use the analytical model to identify the surrounding environment of the mobile body M from at least a part of the observation data 125, and derive the first control instruction 50 from the identified result according to the rule.
[0122] After the first control command 50 is tried out, the control unit 11 advances the process to the next step S103 .
[0123] (Step S103)
[0124] In step S103, the control unit 11 acts as the determination unit 114 to determine whether the first control instruction 50 is normally derived through the first control model 30. In the present embodiment, the abnormality in the derivation of the first control instruction 50 may include at least one of the following: the output of the first control instruction 50 is incomplete, the reliability of the output of the first control model 30 is less than a threshold value, and the change per unit time of the output of the first control model 30 exceeds a threshold value. If it is determined that the first control instruction 50 is normally derived, the control unit 11 causes the process to proceed to step S104. On the other hand, if it is determined that the derivation of the first control instruction 50 is abnormal, the control unit 11 causes the process to proceed to step S105.
[0125] (Step S104)
[0126] In step S104, the control unit 11 operates as the motion control unit 116, and controls the motion of the moving body M according to the derived first control instruction 50. The control unit 11 can directly or indirectly control the motion of the target moving body M. When the control of the target moving body M is completed, the control unit 11 ends the processing steps of the control device 1 involved in this action example.
[0127] (Step S105)
[0128] In step S105, the control unit 11 acts as the second derivation unit 113, and uses the second control model 35 to try to derive the second control instruction 55. In the present embodiment, the second control model 35 is different from the first control model 30 by at least one of the type and the input data, and is configured to be able to normally derive the control instruction in at least a part of the scene where an abnormality occurs in the derivation of the control instruction based on the first control model 30. In one example, the first control model 30 may be composed of a trained machine learning model, and the second control model 35 may be composed of a rule base model. In another example, the first control model 30 may be composed of a trained machine learning model, and the second control model 35 may be composed of a trained machine learning model whose input data is at least partially different from that of the first control model 30.
[0129] The process of deriving the second control instruction 55 can be appropriately determined according to the embodiment, similarly to the first control instruction 50. As an example, when the second control model 35 includes a trained machine learning model, the control unit 11 can obtain the derivation result of the second control instruction 55 by inputting at least a part of the obtained observation data 125 into the second control model 35, executing the calculation process of the second control model 35. The control unit 11 can refer to the second model data 350 at any timing before the processing step S105, and set the second control model 35 to a state in which the calculation process can be performed. As another example, when the second control model 35 includes a rule base model, the control unit 11 can refer to the second model data 350, and derive the second control instruction 55 from the environment represented by at least a part of the observation data 125 according to the rules specified by the rule base model. When the rule base model includes an analytical model, the control unit 11 can use the analytical model to recognize the surrounding environment of the mobile body M from at least a part of the observation data 125, and derive the second control instruction 55 from the recognized result according to the rule.
[0130] After the second control instruction 55 is tried out, the control unit 11 proceeds to the next step S106. Figure 7 In the example, as a result of the determination of step S103, the process of step S105 is executed based on the abnormality that occurs in the derivation of the first control instruction 50. However, the processing timing of step S105 is not limited to such an example. The process of step S105 may be executed at any timing regardless of the determination result of step S103. As another example, the process of step S105 may be executed at least partially in parallel with the process of step S102.
[0131] (Step S106)
[0132] In step S106, the control unit 11 operates as the determination unit 114 to determine whether the second control instruction 55 is normally derived by the second control model 35. In the present embodiment, similarly to the first control instruction 50, the abnormality in the derivation of the second control instruction 55 may include at least one of the incomplete output of the second control instruction 55, the reliability of the output of the second control model 35 being less than a threshold value, and the amount of change per unit time of the output of the second control model 35 exceeding a threshold value. If it is determined that the second control instruction 55 is normally derived, the control unit 11 causes the process to proceed to step S107. On the other hand, if it is determined that the derivation of the second control instruction 55 is abnormal, the control unit 11 causes the process to proceed to step S108.
[0133] (Step S107)
[0134] In step S107, the control unit 11 operates as the motion control unit 116, and controls the motion of the moving body M according to the derived second control command 55. The control unit 11 can directly or indirectly control the motion of the target moving body M. When the control of the target moving body M is completed, the control unit 11 ends the processing steps of the control device 1 involved in this action example.
[0135] (Step S108)
[0136] In step S108, the control unit 11 operates as the operation accepting unit 115, accepts the input of the manual operation 40 by the user, and obtains the third control command 60 from the input manual operation 40. After accepting the input of the manual operation 40 and obtaining the third control command 60, the control unit 11 advances the processing to the next step S109.
[0137] In addition, Figure 7 In the example, for the results of the determinations of step S103 and step S106, the process of step S108 is executed based on the fact that both the derivation of the first control instruction 50 and the derivation of the second control instruction 55 are abnormal. However, the processing timing of step S108 is not limited to such an example. The process of step S108 can be executed at any timing regardless of the determination result of at least one of step S103 and step S106. In another example, the process of step S108 can be executed regardless of any of the processes of step S101 to step S107.
[0138] In addition, in the present embodiment, when the step S108 is reached, the action mode of the mobile body M is switched from automatic control based on the control model (the first control model 30, the second control model 35) to manual control. The control unit 11 may perform any transition processing (e.g., notification processing of switching to manual control) when switching to manual control. The execution timing of any transition processing may be appropriately determined according to the implementation method. Furthermore, in the case where the manual operation 40 cannot be obtained, for example, because the user is in a state where the operation is impossible, the control unit 11 may perform any emergency processing (e.g., parking the vehicle on the shoulder of the road, etc.). In this case, the control unit 11 may omit the processing of the next step S109.
[0139] (Step S109)
[0140] In step S109, the control unit 11 operates as the motion control unit 116, and controls the motion of the moving body M according to the accepted manual operation 40. That is, the control unit 11 controls the motion of the moving body M according to the obtained third control instruction 60. The control unit 11 can directly or indirectly control the motion of the target moving body M. When the control of the target moving body M is completed, the control unit 11 ends the processing steps of the control device 1 involved in this action example.
[0141] (other)
[0142] In addition, the control unit 11 may repeatedly perform a series of information processing from step S101 to step S109. The timing of the repetition may be appropriately determined according to the implementation. In one example, the control unit 11 may repeatedly perform a series of information processing from step S101 to step S109 during a predetermined period (e.g., from the start to the stop of the power source of the moving body M). Thus, the control device 1 can continuously perform the motion control of the moving body M.
[0143] [feature]
[0144] In the present embodiment, the second control model 35 capable of deriving control instructions from a different perspective is provided together with the first control model 30. Thus, in at least a portion of the scene where the control instructions cannot be derived normally using the first control model 30, it is expected that the control instructions can be derived normally using the second control model 35. In the process of step S102, even if an abnormality occurs in the derivation of the first control instruction 50 based on the first control model 30, it is expected that the automatic control can be continued by using the processes of step S105 and step S107 of the second control model 35. Thus, even in the case where it is difficult to continue the automatic control of the first control model 30, there may be a situation where the control based on the manual operation 40 may not be switched to step S108 and step S109. Therefore, according to the present embodiment, it is possible to seek to reduce the frequency of control based on manual operation.
[0145] [4 Modifications]
[0146] The embodiments of the present disclosure have been described in detail above, but the descriptions so far are merely illustrative of the present disclosure in all aspects. It goes without saying that various improvements or modifications can be made without departing from the scope of the present disclosure. For example, the following changes can be made. In addition, the same figure marks are used for the same components as the above-mentioned embodiments, and the descriptions of the same points as the above-mentioned embodiments are appropriately omitted. The following modifications can be appropriately combined.
[0147] <4.1>
[0148] In the above-mentioned embodiment, the allocation of which of the second control model 35 and the manual operation 40 to be executed is determined according to whether the derivation of the second control instruction 55 is successful or not. However, the allocation method may not be limited to such an example, and may be appropriately determined according to the embodiment. In another example, on the basis of or in place of the above-mentioned embodiment, the movement of the mobile body M is controlled according to the second control instruction 55 or the third control instruction 60 by determining which of the second control model 35 and the manual operation 40 to adopt according to the movement environment of the mobile body M, deriving the second control instruction 55 using the second control model 35 when it is determined to adopt the second control model 35 and controlling the movement of the mobile body M according to the derived second control instruction 55, and controlling the movement of the mobile body M according to the third control instruction 60 derived from the manual operation 40 when it is determined to adopt the manual operation 40. In this mode, the mobile body M may be a vehicle, and the movement environment of the mobile body M may include the type of road on which the vehicle travels.
[0149] Figure 8 An example of a control process in the control device 1 according to this modification is schematically shown. Figure 8In the example, the control device 1 attempts to derive the first control instruction 50 based on the first control model 30. As a result of the trial derivation of the first control instruction 50, when the first control instruction 50 is derived normally, the control device 1 controls the movement of the mobile body M according to the derived first control instruction 50. On the other hand, when an abnormality occurs in the derivation of the first control instruction 50, the control device 1 determines which of the second control model 35 and the manual operation 40 to adopt according to the movement environment of the mobile body M at the control time point. In one example, the determination unit 114 may be configured to determine which of the second control model 35 and the manual operation 40 to adopt.
[0150] When it is determined to adopt the second control model 35, the control device 1 derives the second control instruction 55 using the second control model 35. Then, the control device 1 controls the motion of the mobile body M according to the derived second control instruction 55. On the other hand, when it is determined to adopt the manual operation 40, the control device 1 receives the input of the manual operation 40 and obtains the third control instruction 60 derived from the input manual operation 40. Then, the control device 1 controls the motion of the mobile body M according to the obtained third control instruction 60.
[0151] Fig. 9 An example of a processing step related to the control of the mobile body M by the control device 1 involved in this modification is shown. The following processing steps are an example of a control method executed by a computer. However, the following processing steps are only an example, and each step can be changed as long as it is feasible. In addition, regarding the following processing steps, steps can be appropriately omitted, replaced, and added according to the implementation method.
[0152] for Fig. 9 The processing steps shown in the figure can be similar to the above-mentioned embodiment (except that the processing of step S200 is executed before the processing of step S105 after it is determined that an abnormality has occurred in the derivation of the first control instruction 50 by the processing of step S103. Figure 7 ) is the same.
[0153] In step S200, the control unit 11 acts as the determination unit 114 to determine which of the second control model 35 and the manual operation 40 to adopt based on the moving environment of the moving body M at the control time point. In one example, the control unit 11 may determine which of the second control model 35 and the manual operation 40 to adopt based on the environment represented by at least a portion of the acquired observation data 125.
[0154] In this modification, a range of environments that can be handled can be set for the second control model 35. The control unit 11 can determine whether to use the second control model 35 or the manual operation 40 according to whether the moving environment of the mobile body M is within the range. That is, the control unit 11 can determine to use the second control model 35 if the moving environment of the mobile body M is within the range in which the second control model 35 can be used, and determine to use the manual operation 40 otherwise.
[0155] In one example, when the moving body M is a vehicle, the second control model 35 can be configured to be able to perform automatic driving on a specific type of road. In this case, the control unit 11 can decide which of the second control model 35 and the manual operation 40 to adopt according to the type of road on which the vehicle is traveling. As a specific example, the second control model 35 can be configured to perform constant speed driving and inter-vehicle distance control (such as ACC: Adaptive cruise control, etc.) on a highway. In this case, the control unit 11 can decide to adopt the second control model 35 if the type of road on which the vehicle (moving body M) is traveling is a highway at the control time point, otherwise, decide to adopt the manual operation 40. In addition, the control device 1 can maintain a blacklist that specifies the range in which constant speed driving and inter-vehicle distance control are not performed on the highway. The blacklist can be appropriately generated (for example, it can be obtained by the user's setting). In this case, the control unit 11 can decide to adopt the second control model 35 if the road on which the vehicle (moving body M) is traveling is within the range of highways that are not in the blacklist at the control time point. On the other hand, the control unit 11 may determine to adopt the manual operation 40 if the road the vehicle is traveling on is within the range of expressways in the blacklist or the type of road the vehicle is traveling on is other than an expressway (eg, a general road).
[0156] In another example, the control unit 11 may determine whether to use the second control model 35 or the manual operation 40, depending on whether the moving environment of the mobile body M is within a range where data used by the second control model 35 can be obtained. As a specific example, when the second control model 35 uses map information for deriving the second control instruction 55, the control unit 11 may determine to use the second control model 35 if the current position of the vehicle (mobile body M) is within the corresponding range of the map information, and otherwise determine to use the manual operation 40. In addition, when the sensor S includes a GPS module and the second control model 35 uses the position information measured by the GPS module for deriving the second control instruction 55, the control unit 11 may determine to use the second control model 35 if the current position of the vehicle (mobile body M) is within the measurement interval of the GPS module, and otherwise determine to use the manual operation 40.
[0157] If the second control model 35 is determined to be adopted, the control unit 11 advances the processing to step S105. On the other hand, if the manual operation 40 is determined to be adopted, the control unit 11 advances the processing to step S108. Fig. 9 In the example, for the result of the determination of step S103, the process of step S200 is executed according to the abnormality that occurred in the derivation of the first control instruction 50. However, the processing timing of step S200 may not be limited to such an example. The process of step S200 may be executed at any timing regardless of the determination result of step S103. As another example, the process of step S200 may be executed at least partially in parallel with at least any one of the processes in steps S101 to S103. According to this variation, in a scenario where automatic control based on the second control model 35 cannot be reliably performed, the determination process of step S106 can be omitted and switched to control based on manual operation 40. In this way, the efficiency of the control process can be sought.
[0158] <4.2>
[0159] In the above embodiment, for the sake of convenience, the number of the first control model 30 and the number of the second control model 35 are each one. However, the number of the first control model 30 and the number of the second control model 35 is not limited to one, and can be determined arbitrarily. The number of at least one of the first control model 30 and the second control model 35 provided in the control device 1 can be two or more.
[0160] In one example, the control device 1 may maintain different first control models 30 and second control models 35 for each scenario. When the moving body M is a vehicle, the scenario may be, for example, lane change, lane keeping, emergency stop (EDSS: Emergency Driving Stop System), etc. In this case, the control device 1 may switch the first control model 30 and the second control model 35 to be used for each scenario.
[0161] In addition, the control device 1 may also maintain multiple second control models 35 relative to one first control model 30. That is, the number of second control models 35 used to support one first control model 30 may not be limited to one, and may be determined arbitrarily. Multiple second control models 35 are provided to support one first control model 30. In this variation, the control device 1 may derive the second control instruction 55 through one or more second control models 35 among the multiple second control models 35 when an abnormality occurs in the derivation of the first control instruction 50 based on the first control model 30. In addition, the multiple second control models 35 may also be referred to as the second control model, the third control model, ..., the Nth control model, etc. in sequence.
[0162] [5 Supplement]
[0163] The processes and means described in the present disclosure can be freely combined and implemented as long as no technical contradiction occurs.
[0164] Furthermore, a process described as being performed by one device may be shared and executed by a plurality of devices. Alternatively, a process described as being performed by different devices may be executed by one device. The hardware configuration used to implement each function in a computer system can be flexibly changed.
[0165] The present disclosure can also be implemented by providing a computer with a computer program having the functions described in the above-mentioned embodiments installed therein and one or more processors of the computer reading out the program and executing it. Such a computer program can be provided to the computer via a non-transient computer-readable storage medium that can be connected to the system bus of the computer, or can be provided to the computer via a network. Non-transient computer-readable storage media include, for example, any type of disks such as magnetic disks (floppy (registered trademark) disks, hard disk drives (HDDs), etc.), optical disks (CD-ROMs, DVD disks, Blu-ray disks, etc.), read-only memories (ROMs), random access memories (RAMs), EPROMs, EEPROMs, magnetic cards, flash memories, optical cards, semiconductor drives (solid-state drives, etc.), and any type of medium suitable for storing electronic commands.
Claims
1. A control device comprising a storage unit storing a first control model and a second control model and a control unit, The second control model is configured to be different from the first control model in at least one of its type and input data, and is capable of normally deriving control instructions in at least a portion of the scenarios in which an abnormality occurs in the derivation of control instructions based on the first control model. The control unit is configured to execute: Using the first control model, trial deriving a first control instruction; When the derivation of the first control instruction based on the first control model is normal, controlling the movement of the moving body according to the derived first control instruction; and When an abnormality occurs in the derivation of the first control command based on the first control model, the movement of the moving body is controlled according to the second control command derived from the second control model or the third control command from a manual operation of a user.
2. The control device according to claim 1, The first control model is composed of a trained machine learning model, The second control model is composed of a rule base model.
3. The control device according to claim 1, The first control model is composed of a trained machine learning model, The second control model is comprised of a trained machine learning model having input data at least partially different from that of the first control model.
4. The control device according to claim 1, Controlling the movement of the moving body according to the second control instruction or the third control instruction is performed by the following processing: include: determining which of the second control model and the manual operation to adopt according to the moving environment of the moving body; When it is determined that the second control model is to be adopted, the second control instruction is derived using the second control model, and the movement of the moving body is controlled according to the derived second control instruction; as well as When it is determined that the manual operation is to be adopted, the movement of the moving body is controlled according to a third control command derived from the manual operation.
5. The control device according to claim 4, The mobile body is a vehicle, The moving environment of the moving object includes the type of the road on which the vehicle travels.
6. The control device according to claim 1, Controlling the movement of the moving body according to the second control instruction or the third control instruction is performed by the following processing: include: When the derivation of the second control instruction based on the second control model is normal, controlling the movement of the moving body according to the derived second control instruction; and When an abnormality occurs in the derivation of the second control command based on the second control model, the movement of the moving body is controlled according to the third control command derived from the manual operation.
7. The control device according to claim 1, The abnormality in the derivation of the first control command includes incomplete output of the first control model.
8. The control device according to claim 1, The abnormality in derivation of the first control command includes that the reliability of the output of the first control model is less than a threshold value.
9. The control device according to claim 1, The abnormality in derivation of the first control command includes that a change per unit time of an output of the first control model exceeds a threshold value.
10. A computer program product, comprising a control program, wherein the control program is configured to cause a control device that maintains a first control model and a second control model to execute: Using the first control model, trial deriving a first control instruction; When the derivation of the first control instruction based on the first control model is normal, controlling the movement of the moving body according to the derived first control instruction; and When an abnormality occurs in the derivation of the first control instruction based on the first control model, the movement of the moving body is controlled according to the second control instruction derived from the second control model or the third control instruction from the manual operation of the user, The second control model is configured to be different from the first control model in at least one of its type and input data, and is capable of normally deriving control instructions in at least a portion of the scenarios where an abnormality occurs in the derivation of control instructions based on the first control model.
11. The computer program product according to claim 10, The first control model is composed of a trained machine learning model, The second control model is composed of a rule base model.
12. The computer program product according to claim 10, The first control model is composed of a trained machine learning model, The second control model is comprised of a trained machine learning model having input data at least partially different from that of the first control model.
13. The computer program product according to claim 10, Controlling the movement of the moving body according to the second control instruction or the third control instruction is performed by the following processing: include: determining which of the second control model and the manual operation to adopt according to the moving environment of the moving body; When it is determined that the second control model is to be adopted, the second control instruction is derived using the second control model, and the movement of the moving body is controlled according to the derived second control instruction; as well as When it is determined that the manual operation is to be adopted, the movement of the moving body is controlled according to a third control command derived from the manual operation.
14. The computer program product according to claim 13, The mobile body is a vehicle, The moving environment of the moving object includes the type of the road on which the vehicle travels.
15. The computer program product according to claim 10, Controlling the movement of the moving body according to the second control instruction or the third control instruction is performed by the following processing: include: When the derivation of the second control instruction based on the second control model is normal, controlling the movement of the moving body according to the derived second control instruction; and When an abnormality occurs in the derivation of the second control command based on the second control model, the movement of the moving body is controlled according to the third control command derived from the manual operation.
16. The computer program product according to claim 10, The abnormality in the derivation of the first control command includes incomplete output of the first control model.
17. The computer program product according to claim 10, The abnormality in derivation of the first control command includes that the reliability of the output of the first control model is less than a threshold value.
18. The computer program product according to claim 10, The abnormality in derivation of the first control command includes that a change per unit time of an output of the first control model exceeds a threshold value.
19. A control method, The control device that maintains the first control model and the second control model executes: Using the first control model, trial deriving a first control instruction; When the derivation of the first control instruction based on the first control model is normal, controlling the movement of the moving body according to the derived first control instruction; and When an abnormality occurs in the derivation of the first control instruction based on the first control model, the movement of the moving body is controlled according to the second control instruction derived from the second control model or the third control instruction from the manual operation of the user, The second control model is configured to be different from the first control model in at least one of its type and input data, and is capable of normally deriving control instructions in at least a portion of the scenarios where an abnormality occurs in the derivation of control instructions based on the first control model.
20. The control method according to claim 19, Controlling the movement of the moving body according to the second control instruction or the third control instruction is performed by the following processing: include: determining which of the second control model and the manual operation to adopt according to the moving environment of the moving body; When it is determined that the second control model is to be adopted, the second control instruction is derived using the second control model, and the movement of the moving body is controlled according to the derived second control instruction; as well as When it is determined that the manual operation is to be adopted, the movement of the moving body is controlled according to a third control command derived from the manual operation.
