Control device, control program and control method

Abstract

To provide a control technique for suppressing frequencies at which control is performed by manual operation.SOLUTION: A control device relating to one aspect in the disclosure tries to derive a first control command, using a first control model, and when the first control command was normally derived using the first control mode, controls a moving body, in accordance with the derived first control command. When an abnormality occurs in the derivation of the first control command using the first control model, the control device controls the moving body, in accordance with a second control command derived from a second control mode or a third control command derived from manual operation by a user. The second control model is configured to differ in at least either of type and input data from the first control model, so as to be able to normally derive the control command in at least some situation where an abnormality occurs in the derivation of the control command using the first control model.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to a control device, a control program, and a control method. [Background technology]

[0002] Patent Document 1 proposes a system for autonomous vehicle control configured to determine vehicle commands from a route, GPS data, and sensor data using a trained neural network. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Special Publication No. 2019-533810 Summary of the Invention [Problem to be solved by the invention]

[0004] One object of the present disclosure is to provide a control technique for reducing the frequency of occurrence of control by manual operation. [Means for solving the problem]

[0005] The control device according to the first aspect of the present disclosure includes a storage unit that stores a first control model and a second control model, and a control unit. The second control model is configured to be able to normally derive a control command in at least some scenes where an abnormality occurs in the derivation of a control command by the first control model due to at least one of a type and input data being different from that of the first control model. The control unit is configured to execute the following: attempting to derive a first control command using the first control model; controlling the movement of a moving object according to the derived first control command when the derivation of the first control command by the first control model is normal; and controlling the movement of the moving object according to a second control command derived by the second control model or a third control command derived from a manual operation by a user when an abnormality occurs in the derivation of the first control command by the first control model. In one example, at least one of the first control model and the second control model may be configured by a trained machine learning model.

[0006] A control program according to a second aspect of the present disclosure is a program for causing a control device holding a first control model and a second control model configured to be able to normally derive a control command in at least some situations in which an abnormality occurs in the derivation of a control command by the first control model due to at least one of a type and input data being different from the first control model, to attempt to derive a first control command using the first control model, and, if the derivation of the first control command by the first control model is normal, to control movement of a moving body in accordance with the derived first control command, and, if an abnormality occurs in the derivation of the first control command by the first control model, to control the movement of the moving body in accordance with the second control command derived by the second control model or a third control command derived from manual operation by a user.

[0007] A control method according to a third aspect of the present disclosure includes the steps of: a control device that holds a first control model and a second control model that is configured to be able to normally derive a control command in at least some situations in which an abnormality occurs in derivation of a control command by the first control model due to at least one of a type and input data being different from the first control model, attempting to derive a first control command using the first control model; controlling a movement of a moving object in accordance with the derived first control command when derivation of the first control command by the first control model is normal; and controlling the movement of the moving object in accordance with a second control command derived by the above or a third control command derived from a manual operation by a user. Effect of the Invention

[0008] According to the present disclosure, it is expected that the frequency of occurrence of control by manual operation will be reduced. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 illustrates an example of a situation to which the present disclosure is applied. [Figure 2A] FIG. 2A illustrates an example of a relationship between a first control model and a second control model according to the present embodiment. [Figure 2B] FIG. 2B illustrates an example of the relationship between the first control model and the second control model according to the present embodiment. [Diagram 3] FIG. 3 shows a schematic diagram of an example of a control process in this embodiment. [Figure 4A] FIG. 4A illustrates an example of a situation in which an abnormality occurs in the derivation of a control command in this embodiment. [Figure 4B] FIG. 4B illustrates an example of a situation in which an abnormality occurs in the derivation of a control command in this embodiment. [Figure 4C] FIG. 4C illustrates an example of a situation in which an abnormality occurs in the derivation of a control command in this embodiment. [Diagram 5] FIG. 5 illustrates an example of a hardware configuration of the control device according to the present embodiment. [Figure 6] FIG. 6 shows an example of a software configuration of the control device according to the present embodiment. [Figure 7] FIG. 7 is a flowchart showing an example of a processing procedure related to control by the control device according to the present embodiment. [Figure 8] FIG. 8 shows a schematic diagram of an example of a control process in another embodiment. [Figure 9] FIG. 9 is a flowchart illustrating an example of a process procedure regarding control by a control device according to another embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Conventionally, a rule-based automated driving system has been known. According to the method of Patent Document 1, etc., an automated driving system can be constructed by using a trained machine learning model. However, the present inventors have found that these conventional methods have the following problems.

[0011] That is, in both the rule-based model and the machine learning model, the control command may not always be derived normally. In one example, when an image obtained by a camera is used to derive a control command, the surrounding environment may not be captured in the image due to dirt (e.g., rain, mud, etc.) on the camera lens or darkness, and the control command may not be derived normally. In another example, when position information is used to derive a control command, the position information may not be measured and the control command may not be derived normally. In such a case, automatic driving cannot be performed, and the vehicle is controlled by manual driving. When a state in which a control command cannot be derived normally occurs frequently in the deployed control model, the frequency of control by manual driving increases.

[0012] This problem can occur regardless of the type of vehicle. Furthermore, such problems do not occur only when controlling a vehicle. The same applies to moving objects other than vehicles in terms of controlling the movement of the object. Therefore, the same problem can occur when controlling any moving object other than a vehicle (for example, an aircraft, a ship, a robot device, etc.).

[0013] In contrast, a control device according to a first aspect of the present disclosure includes a storage unit that stores a first control model and a second control model, and a control unit. The second control model is configured to be able to normally derive a control command in at least some situations where an abnormality occurs in the derivation of a control command by the first control model due to at least one of the type and input data being different from that of the first control model. The control unit is configured to execute the following: attempting to derive a first control command using the first control model; controlling the movement of the moving body according to the derived first control command when the derivation of the first control command by the first control model is normal; and controlling the movement of the moving body according to the second control command derived by the second control model or a third control command derived from a manual operation by a user when an abnormality occurs in the derivation of the first control command by the first control model.

[0014] In the first aspect of the present disclosure, a second control model capable of deriving a control command from a different perspective due to differences in at least one of the model type and the input received is deployed together with the first control model. Since the second control model is configured to derive a control command from a different perspective, the range in which the second control model can normally derive a control command may differ from that of the first control model. That is, the second control model can be expected to be able to normally derive a control command in at least a part of the scenes in which the first control model cannot normally derive a control command. Therefore, automatic control by the second control model can be executed in at least a part of the scenes in which the first control model cannot normally derive a control command. That is, the second control model can support at least a part of the range in which it is difficult to operate the first control model. Therefore, according to the first aspect of the present disclosure, it is possible to reduce the frequency of control by manual operation.

[0015] As another form of the control device according to the above aspect, one aspect of the present disclosure may be an information processing method or a program for implementing all or part of the above components, or may be a storage medium that stores such a program and is readable by a machine such as a computer. Here, the machine-readable storage medium is a medium that stores information such as a program by electrical, magnetic, optical, mechanical, or chemical action.

[0016] For example, a control program according to a second aspect of the present disclosure is a program for causing a control device holding a first control model and a second control model configured to be able to normally derive a control command in at least some situations in which an abnormality occurs in the derivation of a control command by the first control model due to at least one of a type and input data being different from the first control model, to attempt to derive a first control command using the first control model, and, if the derivation of the first control command by the first control model is normal, to control movement of a moving body in accordance with the derived first control command, and, if an abnormality occurs in the derivation of the first control command by the first control model, to control the movement of the moving body in accordance with the second control command derived by the second control model or a third control command derived from manual operation by a user.

[0017] Furthermore, a control method according to a third aspect of the present disclosure is an information processing method in which a control device holding a first control model and a second control model configured to be able to normally derive a control command in at least some situations in which an abnormality occurs in the derivation of a control command by the first control model due to at least one of a type and input data being different from the first control model attempts to derive a first control command using the first control model, controls movement of a moving body in accordance with the derived first control command if the derivation of the first control command by the first control model is normal, and controls the movement of the moving body in accordance with the second control command derived by the second control model or a third control command derived from manual operation by a user if an abnormality occurs in the derivation of the first control command by the first control model.

[0018] Hereinafter, an embodiment according to one aspect of the present disclosure (hereinafter also referred to as "the present embodiment") will be described with reference to the drawings. The following description will be given based on the above aspects. However, the present embodiment described below is merely an example of the present disclosure in all respects. Various improvements or modifications may be made without departing from the scope of the present disclosure. In implementing the present disclosure, a specific configuration according to the embodiment may be appropriately adopted. Note that, although the data appearing in the present embodiment is described in natural language, more specifically, it is specified in pseudo-language, commands, parameters, machine language, etc. that can be recognized by a computer.

[0019] [1 Application example] 1 is a schematic diagram showing an example of a situation in which the present disclosure is applied. The control device 1 according to the present embodiment is one or more computers configured to control the automatic movement of a target moving object M.

[0020] In this embodiment, the control device 1 is mounted on a moving body M and holds a first control model 30 and a second control model 35. The first control model 30 and the second control model 35 are each configured to be capable of deriving a control command in order to automatically control the movement of the moving body M. The second control model 35 is configured to be capable of normally deriving a control command in at least some situations where an abnormality occurs in the derivation of a control command by the first control model 30 due to at least one of the model type and the input data received being different from that of the first control model 30.

[0021] In this embodiment, the control device 1 uses the first control model 30 to attempt to derive the first control command 50. In this attempt, if the first control model 30 is operated under appropriate conditions for the first control model 30, the first control command 50 can be derived normally. In contrast, if the first control model 30 is not operated under appropriate conditions, for example, an image cannot be obtained normally, position information cannot be obtained, or the like, the first control command 50 may not be derived normally (i.e., an abnormality may occur in the derivation of the first control command 50).

[0022] When the derivation of the first control command 50 by the first control model 30 is normal, the control device 1 controls the movement of the moving object M in accordance with the derived first control command 50. On the other hand, when an abnormality occurs in the derivation of the first control command 50 by the first control model 30, the control device 1 controls the movement of the moving object M in accordance with the second control command 55 derived by the second control model 35 or the third control command 60 derived from the manual operation 40 by the user.

[0023] As described above, in this embodiment, the second control model 35 capable of deriving a control command from a different perspective is deployed together with the first control model 30. As a result, it can be expected that the second control model 35 can normally derive a control command (second control command 55) in at least a part of the scenes in which the first control model 30 cannot normally derive a control command (first control command 50). Therefore, even if an abnormality occurs in the derivation of the first control command 50 by the first control model 30, it is possible to expect continuation of automatic control by the second control model 35. That is, the second control model 35 can support at least a part of the range in which it is difficult to operate the first control model 30. When automatic control can be continued by the second control model 35, it is not necessary to switch to control by manual operation 40. Therefore, according to this embodiment, it is possible to reduce the frequency of control by manual operation.

[0024] (Mobile) As long as the moving object M can move automatically by mechanical control, the type of the moving object M may be appropriately selected according to the embodiment. The moving object M may be, for example, a mobile device such as a vehicle, an aircraft, a ship, or a robot device. The aircraft may be at least one of an unmanned aircraft such as a drone and a manned aircraft.

[0025] In one example, as shown in FIG. 1, the moving object M may be a vehicle. In this case, In a situation where automatic driving is performed, the frequency of control by manual operation can be reduced. In addition, when the moving body M is a vehicle, the type of the vehicle (number of wheels, power source, size, etc.) may be selected arbitrarily. The vehicle may be selected from, for example, a two-wheeled vehicle, a three-wheeled vehicle, a four-wheeled vehicle, etc. The power source of the vehicle may be selected from, for example, electricity, fuel, etc. In the case where the vehicle is an automobile, the size of the vehicle may be selected from large, medium, semi-medium, normal, large special, small special, etc. In the case where the vehicle is a two-wheeled vehicle, the size of the vehicle may be selected from large, normal, etc. As a typical example, the moving body M may be an automobile having an automatic driving capability of level 2 or higher. In addition, the user may be any person involved in the moving body M. In the case where the moving body M is a vehicle, the user may typically be a driver.

[0026] (Control) In one example, controlling the operation of the target moving body M may be configured by directly controlling the target moving body M. That is, the control device 1 may be configured to directly control the target moving body M. In another example, the moving body M may be equipped with a dedicated control device such as a controller. In this case, controlling the operation of the target moving body M by the control device 1 may be configured by indirectly controlling the target moving body M by providing the derivation result to the dedicated control device.

[0027] The control device 1 may be installed in any location. In one example, as shown in FIG. 1, the control device 1 may be mounted on a moving body M. The mounting location may be determined arbitrarily, and for example, the control device 1 may be attached inside or outside the moving body M. In another example, the control device 1 may be disposed away from the moving body M and configured to remotely control the moving body M. The control device 1 may be connected to the moving body M by wire or wirelessly.

[0028] (Control Model) The first control model 30 and the second control model 35 are each constructed to derive a control command according to the environment of the moving body M. The environment is an event observed by at least one of the moving body M itself and the surroundings. In one example, at least a part of the environment may be observed by one or more sensors S arranged inside or outside the moving body M. The type of the sensor S is not particularly limited as long as it can observe any environment in which the moving body M moves, and may be appropriately selected according to the embodiment. In one example, the one or more sensors S may include a camera (image sensor), a radar, a LiDAR (Light Detection And Ranging), a sonar (ultrasonic sensor), an infrared sensor, a GNSS (Global Navigation Satellite System) / GPS (Global Positioning Satellite) module, etc.

[0029] As long as a control command can be derived from the environment of the moving body (moving body M), the form of input and output of the control models (first control model 30, second control model 35) is not particularly limited and may be appropriately selected according to the embodiment. In one example, at least one of the first control model 30 and the second control model 35 may be configured to derive a control command from observation data of a sensor at one or more points in time. In another example, at least one of the first control model 30 and the second control model 35 may be configured to derive a control command from a recognition result of the surrounding environment. The recognition result of the surrounding environment may be obtained by analyzing the observation data of the sensor in an arbitrary manner (i.e., using an analytical model). The control device 1 may further include an analytical model that infers the recognition result of the surrounding environment from the observation data of the sensor. In one example, the analytical model may be configured to obtain the recognition result of the surrounding environment from the observation data on a rule base by performing an analytical process such as pattern matching or edge detection. In another example, the analytical model may be configured by a trained machine learning model. The analytical model may be considered as part of the control model or as a separate component from the control model. Other information may be optionally added to the input of the control model. The control model may be configured to further accept input of any information, such as, for example, a set speed, a speed limit, a location, map information, navigation information, etc. good.

[0030] In one example, the control model (first control model 30, second control model 35) may be configured to directly output a control command. In another example, the control model may be configured to indirectly output a control command, and the control command may be obtained by performing any information processing (e.g., interpretation processing such as judgment and selection) on the output of the control model. The control command may be configured to directly indicate the control amount (control instruction value, control output amount) of the moving body M, such as, for example, an accelerator control amount, a brake control amount, and a steering wheel steering angle. Alternatively, the control command may be configured to indirectly indicate the control amount of the moving body M, such as, for example, a path, a state after control, and the like. In this case, the control amount of the moving body M may be obtained from the control command by performing any information processing. In one example, when the moving body M is a vehicle, the control amount of the vehicle may be obtained by applying the inference result obtained from the control model to the vehicle model. The vehicle model may have various parameters such as an accelerator, a brake, and a steering wheel steering, and may be appropriately configured to derive the control amount from indirect information (path, state after control, and the like). The output format of each of the first control model 30 and the second control model 35 may be selected arbitrarily. In one example, one of the first control model 30 and the second control model 35 may be configured to directly output a control command, and the other may be configured to indirectly output a control command.

[0031] The configuration of each of the first control model 30 and the second control model 35 may be appropriately selected depending on the embodiment. In one example, the control models (the first control model 30 and the second control model 35) may be configured as a trained machine learning model (e.g., an end-to-end model), a rule-based model, or a combination thereof.

[0032] The rule-based model is configured to match a given input (e.g., information indicating the environment, such as observation data, a result of recognition of the surrounding environment, etc.) with rules and derive a control command depending on the result of the match (according to the matching rule). The rules may be set manually or at least partially automatically.

[0033] The machine learning model is configured to have one or more calculation parameters that can be adjusted by machine learning. The one or more calculation parameters are used for calculation of a target inference (in this disclosure, derivation of a control command). Machine learning is to adjust (optimize) the value of the calculation parameter using learning data. The machine learning model may be configured, for example, by a neural network, a support vector machine, a regression model, a decision tree model, etc. The machine learning method may be appropriately selected according to the machine learning model to be adopted (for example, backpropagation method, etc.).

[0034] As an example, at least one of the first control model 30 and the second control model 35 may be configured with a neural network. The structure of the neural network may be appropriately determined depending on the embodiment, and may be specified, for example, by the number of layers from the input layer to the output layer, the type of each layer, the number of nodes (neurons) included in each layer, the connection relationship between the nodes in each layer, etc. In one example, the neural network may have a recursive structure. In addition, the neural network may include any layer, such as a fully connected layer, a convolutional layer, a pooling layer, a deconvolutional layer, an unpooling layer, a normalization layer, a dropout layer, or a Long short-term memory (LSTM). The neural network may have any mechanism, such as an attention mechanism, etc. Neural networks are GNN (Graph neural network), diffusion model, Any model, generative model (e.g., Generative Adversarial Network, Transformer, etc.) Any model may be included. When a neural network is used for the control model, the weight of the connection between each node included in the control model and the threshold value of each node are examples of the calculation parameters. When a machine learning model is adopted, the control model may be configured in the structure of an end-to-end model.

[0035] (Relationship between the first control model and the second control model) In this embodiment, the first control model 30 is used mainly for controlling the moving object M, and the second control model 35 is used secondary for controlling the moving object M as an alternative means in the event of an output failure (abnormality) of the first control model 30. In other words, the second control model 35 is used to support the first control model 30. The first control model 30 and the second control model 35 are configured to differ from each other in at least one of the model type and the input data (given data) that they accept. As a result, the second control model 35 is configured to be able to normally derive a control command in at least some scenes where an abnormality occurs in the derivation of a control command by the first control model 30.

[0036] The different types of models mean that the calculation steps of the inference process are different. If the calculation steps of the inference process are different, the conditions under which a control command is derivation failure occurs may be different. The different types of models may be realized, for example, by adopting a machine learning model for the first control model 30 and a rule-based model for the second control model 35, or by adopting machine learning models with different structures for the first control model 30 and the second control model 35. In one example, a neural network having a recurrent structure may be adopted for the first control model 30, and a neural network without a recurrent structure may be adopted for the second control model 35.

[0037] The difference in input data given may be at least one of the following: the type of at least a part of the input data given to the control model is different, and the type of the input data is the same but the resolution (amount of information) is different. If the type of at least a part of the input data is different, the condition for obtaining the input data normally may be different, and therefore the condition for the derivation failure of the control command may be different. The same applies to the case where the resolution is different (in general, low-resolution data is more likely to be obtained normally than high-resolution data). As an example, one of the first control model 30 and the second control model 35 may be configured to receive image data and other data (e.g., position information, map information, navigation information, etc.), while the other may be configured to receive only image data. As another example, one of the first control model 30 and the second control model 35 may be configured to receive high-resolution data (e.g., position information, image data, etc.), while the other may be configured to receive low-resolution data. In this case, the first control model 30 may be configured to receive high-resolution data, and the second control model 35 may be configured to receive low-resolution data. In this embodiment, the first control model 30 and the second control model 35 may be configured such that at least some of the situations in which control commands can be successfully derived may differ due to at least one of the different types of models and different input data.

[0038] FIG. 2A is a schematic diagram showing an example of the relationship between the first control model 30 and the second control model 35 according to the present embodiment. In the example of FIG. 2A, the first control model 30 is configured by a trained machine learning model 30A, and the second control model 35 is configured by a rule-based model 35A. In one example, the trained machine learning model 30A may have an end-to-end model structure. In one example, the rule-based model 35A may be configured to recognize the surrounding environment from at least a part of the observation data obtained by the sensor, match the recognition result of the surrounding environment with a rule, and derive a control command according to the result of the match (according to the matching rule). To recognize the surrounding environment, the rule-based model 35A may include an analytical model.

[0039] Generally, a rule-based model derives control commands according to rules, so the process of deriving control commands is clearer than that of a machine learning model, and the operation of the rule-based model is easier to predict. Therefore, according to this example, in a situation where a problem occurs in the derivation of control commands by the first control model 30, it is expected that the second control model 35, which has high predictability of operation, will continue to perform stable automatic control of the moving object M. Note that the inputs of the trained machine learning model 30A and the rule-based model 35A may be appropriately selected depending on the embodiment. The trained machine learning model The inputs of the model 30A and the rule-based model 35A may be the same or may be different from each other.

[0040] 2B is a schematic diagram showing another example of the relationship between the first control model 30 and the second control model 35 according to the present embodiment. In the example shown in FIG. 2B, the first control model 30 is configured by a trained machine learning model 30B. The second control model 35 is configured by a trained machine learning model 35B whose input data is at least partially different from that of the first control model 30 (trained machine learning model 30B).

[0041] In one example, the input data of one of the trained machine learning models 30B and 35B may be composed of only a portion of the input data of the other. As a specific example, as shown in FIG. 2B, the trained machine learning model 30B may be configured to accept input of multiple types of data (e.g., image data and other data), whereas the trained machine learning model 35B may be configured to accept input of only a portion of the data (e.g., only image data).

[0042] In another example, the input data of the trained machine learning model 30B and the trained machine learning model 35B may be configured to overlap in some parts and differ in other parts. As a specific example, the trained machine learning model 30B is configured to receive input of first data and second data (e.g., image data and location information), whereas the trained machine learning model 35B is configured to receive input of first data and third data (e.g., image data and LiDAR data). Each of the first to third data may be configured to include one or more types of data.

[0043] In yet another example, the input data of the trained machine learning model 30B and the trained machine learning model 35B may be configured to be completely different. As a specific example, the trained machine learning model 30B may be configured to accept only the input of first data (e.g., image data / location information), whereas the trained machine learning model 35B may be configured to accept only the input of second data (e.g., location information / image data). Each of the first and second data may be composed of one or more types of data.

[0044] When adopting a form in which the types of input data of the trained machine learning models (30B, 35B) overlap, the resolution (amount of information) of each input data may be different. As a specific example, one of the trained machine learning models (30B, 35B) may be configured to accept input of high-resolution data (e.g., image data, location information, etc.), while the other may be configured to accept input of low-resolution data. In one example, the trained machine learning model 30B may be configured to accept input of high-resolution data, while the trained machine learning model 35B may be configured to accept input of low-resolution data. High resolution and low resolution refer to the relationship between data input to each other, and are configured by the amount of the same type of data input to one being different from that of the other. When adopting a form in which the resolutions are different, the types of input data of the trained machine learning models (30B, 35B) may be completely consistent.

[0045] Since the input data differs at least partially between the first control model 30 (trained machine learning model 30B) and the second control model 35 (trained machine learning model 35B), the data (explanatory variables) used to derive (infer) the control commands are different. This may result in different conditions for obtaining input data that can be used to successfully derive the control commands. Therefore, the range in which poor derivation of the control commands occurs can be made different between the first control model 30 and the second control model 35. Therefore, according to this example, in at least some situations in which poor derivation of the control commands occurs by the first control model 30, continuation of automatic control by the second control model 35 can be prevented. can be expected.

[0046] In addition, when this embodiment is adopted, in one example, the second control model 35 (trained machine learning model 35B) may be configured to have a smaller calculation load by at least one of a smaller amount of input data and a smaller number of calculation parameters compared to the first control model 30 (trained machine learning model 30B). This makes it possible to reduce the frequency of control by manual operation while suppressing an increase in the calculation load.

[0047] In addition, in one example, the first control model 30 (trained machine learning model 30B) may be trained generically, whereas the second control model 35 (trained machine learning model 35B) may be trained specifically. Alternatively, the first control model 30 may also be trained specifically, but the second control model 35 may be trained more specifically than the first control model 30. Dedicated training may be configured by training the second control model 35 to be more likely to exhibit performance in a specific domain compared to the first control model 30, for example, by limiting the domain of the training sample (using a training sample derived from a local domain for machine learning). As a specific example, when the moving body M is a vehicle, the domain of the training sample may be limited by preferentially training a training sample related to driving in a tunnel, preferentially training a training sample related to driving in a specific area, or the like. A domain in which a generically trained control model (trained machine learning model 30B) is likely to fall into derivation failure may be selected as the domain of the training sample used for machine learning. This increases the probability that automatic control can be continued by the second control model 35 (trained machine learning model 35B) when the first control model 30 (trained machine learning model 30B) experiences a derivation failure in a particular domain.

[0048] (Control command) The control commands (first control command 50, second control command 55, third control command 60) relate to the operation of the moving object M. The configuration of the control commands may be appropriately selected depending on the embodiment. In one example, the control commands may be configured by acceleration, deceleration, steering, or a combination of these. Acceleration and deceleration may include gear changes. When at least one of acceleration, deceleration, and steering is included, the control commands may be expressed by a path. Accordingly, the control models (first control model 30 and second control model 35) may be expressed as a path planner. The control commands may further include commands related to the operation of the moving object M. As an example, when the moving object M is a vehicle, the control commands may include vehicle operations such as turn signals, hazards, horn, and communication processing (e.g., transmitting data to a center, making an emergency call, etc.).

[0049] (Allocation of second control model and manual operation) When an abnormality occurs in the derivation of the first control command 50 by the first control model 30, whether to execute automatic control by the second control model 35 or control by the manual operation 40 may be determined appropriately depending on the embodiment. As a simple example, controlling the movement of the moving body M in accordance with the second control command 55 or the third control command 60 may be configured by controlling the movement of the moving body M in accordance with the derived second control command 55 when the derivation of the second control command 55 by the second control model 35 is normal, and controlling the movement of the moving body M in accordance with the third control command 60 derived from the manual operation 40 when an abnormality occurs in the derivation of the second control command 55 by the second control model 35.

[0050] FIG. 3 shows a schematic diagram of an example of a control process in the control device 1 according to this embodiment. In the example shown in FIG. 3, the control device 1 attempts to derive a first control command 50 using a first control model 30. If the first control command 50 is normally derived as a result of the attempt to derive the first control command 50, the control device 1 controls the operation of the moving object M in accordance with the derived first control command 50. On the other hand, if an abnormality occurs in the derivation of the first control command 50, the control device 1 attempts to derive a second control command 55 using a second control model 35. The derivation attempt may be performed regardless of whether the derivation of the first control command 50 is successful or not.

[0051] If the second control command 55 is normally derived as a result of the attempt to derive the second control command 55, the control device 1 controls the operation of the moving object M according to the derived second control command 55. On the other hand, if an abnormality occurs in the derivation of the second control command 55, the control device 1 accepts the input of the manual operation 40 and controls the operation of the moving object M according to the third control command 60 derived from the input manual operation 40. As a result, the control mode is switched in the order of the first control model 30, the second control model 35, and the manual operation 40, so that the possibility of continuing the execution of automatic control by the second control model 35 can be ensured. As a result, the frequency of control by manual operation can be reduced.

[0052] The manual operation 40 may be accepted at any timing. While executing the automatic control by the first control model 30 or the second control model 35, the control device 1 may accept the input of the manual operation 40. The control device 1 may switch from the automatic control mode by the first control model 30 or the second control model 35 to the control mode by the manual operation 40 in response to any operation by the user. When switching from the automatic control by the first control model 30 or the second control model 35 to the manual control by the manual operation 40, the control device 1 may control the operation of the moving body M according to the third control command 60 derived from the manual operation 40 by discarding (ignoring) the control command derived by the control model or overlapping the derived control command. The discarding may be an immediate switch from the automatic control to the manual control, that is, an immediate switch from the control by the control command derived by the control model to the control according to the manual operation 40 of the user. On the other hand, the overlapping may be a gradual switch from the automatic control to the manual control (user control).

[0053] (Abnormal) An abnormality occurring in the derivation of the control commands (first control command 50, second control command 55) by the control models (first control model 30, second control model 35) may include any case in which a normal control command cannot be obtained.

[0054] 4A, 4B, and 4C each show an example of a situation in which an abnormality occurs in the derivation of a control command in the control device 1 according to this embodiment.

[0055] 4A, the occurrence of an abnormality in the derivation of the first control command 50 by the first control model 30 includes the occurrence of an output failure of the first control command 50. The output failure may be at least any one of a failure to provide a meaningful output that enables a control command to be derived, an output of an error signal, and no output at all.

[0056] In the example of Fig. 4B, the occurrence of an abnormality in the derivation of the first control command 50 by the first control model 30 includes the reliability of the output of the first control model 30 being less than a threshold. The threshold may be determined arbitrarily. A case in which the reliability is equal to the threshold may be classified as either an abnormal case or a normal case.

[0057] The confidence level may be acquired by any method. In one example, as shown in FIG. 4B, the first control model 30 may be configured to be capable of outputting the confidence level of the inference result together with the inference result by any method, such as by providing the confidence level as a teacher signal during machine learning training (i.e., the confidence level is assigned to the training data).

[0058] In another example, the reliability may be calculated from the output of the first control model 30. For example, when the first control model 30 includes an autoencoder, the control device 1 may calculate the reconstruction error of the autoencoder as the reliability. In this case, the control device 1 may calculate the reliability as the reconstruction error becomes larger. The reliability may be evaluated as low. Also, for example, when the output of the first control model 30 includes the result of class classification, the control device 1 may calculate the inter-class entropy as the reliability. In this case, the control device 1 may evaluate that the reliability is lower as the entropy value is larger. In yet another example, the control device 1 may calculate the reliability from the first control command 50 derived by the first control model 30 in any analysis process such as a simulation.

[0059] In the example of FIG. 4C , an abnormality in the derivation of the first control command 50 by the first control model 30 includes the change amount per unit time of the output of the first control model 30 exceeding a threshold value. The threshold value may be determined arbitrarily. A case in which the change amount is equal to the threshold value may be classified as either an abnormal case or a normal case. Basically, for stable automatic control, the change amount of the control command derived by the control model does not become large. Therefore, it is possible to determine whether the derivation of the control command is successful or not according to the change amount of the output of the control model.

[0060] In one example, as shown in FIG. 4C, the change in output may be calculated from the fluctuation of the derived result of the first control command 50. FIG. 4C shows an example in which the change in acceleration or deceleration speed is calculated as the change in output. In another example, as a simple method, the change in output may be calculated from the fluctuation of the output value of the first control model 30. Also, the change in output may be calculated by any method. In one example, the change in output may be calculated by the time integral of the difference over a predetermined period (the sum of the differences between t-t+1).

[0061] In this embodiment, the occurrence of an abnormality in the derivation of the first control command 50 may include at least any of the above three cases. As a result, in at least any of the above three cases, it is possible to determine a failure in the derivation of a control command by the first control model 30, and switch from automatic control by the first control model 30 to automatic control by the second control model 35 according to the result of the determination. Note that the same applies to the occurrence of an abnormality in the derivation of the second control command 55 when the first control model 30 is replaced with the second control model 35.

[0062] [2 Configuration example] [Hardware configuration example] 5 is a schematic diagram showing an example of a hardware configuration of the control device 1 according to the present embodiment. The control device 1 according to the present embodiment is a computer to which a control unit 11, a storage unit 12, an external interface 13, an input device 14, an output device 15, and a drive 16 are electrically connected.

[0063] The control unit 11 includes a hardware processor, such as a CPU (Central Processing Unit), It includes a RAM (Random Access Memory), a ROM (Read Only Memory), etc., and is configured to execute any information processing based on programs and various data. The control unit 11 (CPU) is an example of a processor resource.

[0064] The storage unit 12 may be configured, for example, by a hard disk drive, a solid state drive, or the like. The storage unit 12 (and RAM, ROM) is an example of a memory resource. In this embodiment, the storage unit 12 stores various information such as a control program 81, first model data 300, second model data 350, and the like.

[0065] The control program 81 is a program for causing the control device 1 to execute information processing (FIGS. 7 and 9 described below) related to the control of the moving body M. The control program 81 includes a series of instructions for the information processing. The first model data 300 is configured to include information related to the first control model 30. The second model data 350 is configured to include information related to the second control model 35. In one example, when at least one of the first control model 30 and the second control model 35 is configured by a trained machine learning model, at least one of the first model data 300 and the second model data 350 indicates values ​​of calculation parameters adjusted by machine learning. In some cases, at least one of the first model data 300 and the second model data 350 may further include information indicating a configuration of a machine learning model (e.g., a neural network structure, etc.). In another example, when at least one of the first control model 30 and the second control model 35 is configured by a rule-based model, at least one of the first model data 300 and the second model data 350 may include information indicating a rule in the rule-based model.

[0066] The external interface 13 may be, for example, a Universal Serial Bus (USB) port, a dedicated port, a wireless communication port, or the like, and is configured to connect to an external device in a wired or wireless manner. In this embodiment, the control device 1 may be connected to a sensor S via the external interface 13. The control device 1 may be connected to any configuration of a moving object M (for example, an operating tool) via the external interface 13.

[0067] The input device 14 is a device for performing input such as a mouse, a keyboard, an operator, etc. The output device 15 is a device for performing output such as a display, a speaker, etc. A user can operate the control device 1 by using the input device 14 and the output device 15. The input device 14 and the output device 15 may be integrally configured using, for example, a touch panel display, etc.

[0068] The drive 16 is a device for reading various information such as a program stored in a storage medium 91. At least one of the control program 81, the first model data 300, and the second model data 350 may be stored in the storage medium 91 instead of the storage unit 12 or together with the storage unit 12. The storage medium 91 is configured to accumulate various information (stored programs, etc.) by electrical, magnetic, optical, mechanical, or chemical action so that a machine such as a computer can read the information. The control device 1 may acquire at least one of the control program 81, the first model data 300, and the second model data 350 from the storage medium 91. The storage medium 91 may be a disk-type storage medium such as a CD or a DVD, or may be a storage medium other than a disk-type storage medium such as a semiconductor memory (for example, a flash memory). The type of the drive 16 may be appropriately selected according to the type of the storage medium 91.

[0069] In addition, regarding the specific hardware configuration of the control device 1, components can be omitted, replaced, or added as appropriate depending on the embodiment. For example, the control unit 11 may include multiple hardware processors. The hardware processors may be a microprocessor, a field-programmable gate array (FPGA), a digital signal processor (DSP), an ECU, (Electronic Control Unit), GPU (Graphics Processing Unit), etc. At least one of the external interface 13, the input device 14, the output device 15, and the drive 16 may be omitted. The control device 1 may be a computer designed specifically for the service to be provided, a general-purpose computer, a mobile phone including a smartphone, a tablet PC (Personal Computer), or other terminal device. When the moving object M is a vehicle, The control device 2 may be an in-vehicle device.

[0070] [Software configuration example] 6 shows an example of a software configuration of the control device 1 according to this embodiment. The control unit 11 of the control device 1 expands the control program 81 stored in the storage unit 12 into the RAM, and executes instructions included in the control program 81 by the CPU. As a result, the control device 1 operates as a computer including an acquisition unit 111, a first derivation unit 112, a second derivation unit 113, a determination unit 114, an operation acceptance unit 115, and an operation control unit 116 as software modules. That is, in this embodiment, each software module of the control device 1 is realized by the control unit 11 (CPU).

[0071] The acquisition unit 111 is configured to acquire the observation data 125 of the sensor S. The first derivation unit 112 includes a first control model 30 by holding the first model data 300. The first derivation unit 112 is configured to derive a first control command 50 by using the first control model 30. In the present embodiment, the first derivation unit 112 may be configured to derive the first control command 50 from at least a part of the acquired observation data 125 by using the first control model 30. The second derivation unit 113 includes a second control model 35 by holding the second model data 350. The second derivation unit 113 is configured to derive a second control command 55 by using the second control model 35. In the present embodiment, the second derivation unit 113 may be configured to derive the second control command 55 from at least a part of the acquired observation data 125 by using the second control model 35.

[0072] The determination unit 114 is configured to determine whether the derivation of the control commands (first control command 50, second control command 55) by the control models (first control model 30, second control model 35) is normal. The operation reception unit 115 is configured to receive an input of a manual operation 40 by a user, and to acquire a third control command 60 derived from the input manual operation 40. The operation control unit 116 is configured to control the movement of the moving object M according to the derived first control command 50 when the derivation of the first control command 50 by the first control model 30 is normal. Furthermore, the operation control unit 116 is configured to control the movement of the moving object M according to the second control command 55 derived by the second control model 35 or the third control command 60 derived from the manual operation 40 by the user when an abnormality occurs in the derivation of the first control command 50 by the first control model 30.

[0073] In this embodiment, an example is described in which each software module of the control device 1 is realized by a general-purpose CPU. However, some or all of the above software modules may be realized by one or more dedicated processors. Each of the above modules may be realized as a hardware module. With regard to the software configuration of the control device 1, modules may be omitted, replaced, or added as appropriate.

[0074] [3 Example of operation] FIG. 7 shows an example of a processing procedure for controlling a moving body M by the control device 1 according to this embodiment. The following processing procedure is an example of a control method executed by a computer. However, the following processing procedure is merely an example, and each step may be modified as much as possible. Furthermore, steps in the following processing procedure may be omitted, replaced, or added as appropriate depending on the embodiment.

[0075] (Step S101) In step S101, the control unit 11 operates as the acquisition unit 111 and acquires the observation data 125 of the sensor S. The acquisition path of the observation data 125 may be selected arbitrarily. In one example, the control device 1 may be directly connected to the sensor S, and the control unit 11 may acquire the observation data 125 directly from the sensor S. In another example, the control unit 11 may acquire the observation data 125 indirectly from the sensor S via another computer. Upon acquiring the observation data 125, the control unit 11 proceeds to the next step S102.

[0076] (Step S102) In step S102, the control unit 11 operates as the first derivation unit 112, and attempts to derive the first control command 50 using the first control model 30. The process of deriving the first control command 50 may be appropriately determined depending on the embodiment.

[0077] As an example, when the first control model 30 includes a trained machine learning model, the control unit 11 At least a part of the acquired observation data 125 may be input to the first control model 30, and calculation processing (for example, forward propagation calculation processing) of the first control model 30 may be executed. As a result of this calculation processing, the control unit 11 may acquire an output value corresponding to an inference result (a direct or indirect derivation result of a control command) from the first control model 30. In this case, the control unit 11 may refer to the first model data 300 at any timing before processing step S102, and set up the first control model 30 (trained machine learning model) in a state in which calculation processing can be executed.

[0078] As another example, when the first control model 30 includes a rule-based model, the control unit 11 may refer to the first model data 300 and derive the first control command 50 from the environment indicated by at least a portion of the observation data 125 in accordance with rules defined by the rule-based model. When the rule-based model includes an analytical model, the control unit 11 may use the analytical model to recognize the surrounding environment of the moving object M from at least a portion of the observation data 125, and derive the first control command 50 from the recognition result in accordance with the rules.

[0079] After attempting to derive the first control command 50, the control unit 11 advances the process to the next step S103.

[0080] (Step S103) In step S103, the control unit 11 operates as a judgment unit 114 and judges whether or not the first control command 50 has been normally derived by the first control model 30. In this embodiment, the occurrence of an abnormality in the derivation of the first control command 50 may include at least one of the following: an output failure of the first control command 50 occurs, the reliability of the output of the first control model 30 is less than a threshold, and the change amount per unit time of the output of the first control model 30 exceeds a threshold. If it is determined that the first control command 50 has been normally derived, the control unit 11 advances the process to step S104. On the other hand, if it is determined that an abnormality has occurred in the derivation of the first control command 50, the control unit 11 advances the process to step S105.

[0081] (Step S104) In step S104, the control unit 11 operates as the operation control unit 116 and controls the operation of the moving object M in accordance with the derived first control command 50. The control unit 11 may directly or indirectly control the operation of the target moving object M. When the control of the target moving object M is completed, the control unit 11 ends the processing procedure of the control device 1 according to this operation example.

[0082] (Step S105) In step S105, the control unit 11 operates as the second derivation unit 113 and attempts to derive the second control command 55 using the second control model 35. In this embodiment, the second control model 35 is configured to be able to normally derive a control command in at least some scenes where an abnormality occurs in the derivation of a control command by the first control model 30 due to at least one of the type and input data being different from that of the first control model 30. In one example, the first control model 30 may be configured by a trained machine learning model, whereas the second control model 35 may be configured by a rule-based model. In another example, the first control model 30 may be configured by a trained machine learning model, and the second control model 35 may be configured by a trained machine learning model whose input data is at least partially different from that of the first control model 30.

[0083] The process of deriving the second control command 55 may be appropriately determined depending on the embodiment, similar to the first control command 50. As an example, when the second control model 35 includes a trained machine learning model, the control unit 11 may input at least a part of the acquired observation data 125 to the second control model 35 and execute arithmetic processing of the second control model 35 to acquire the derivation result of the second control command 55. The control unit 11 may acquire the derivation result of the second control command 55 at any timing before processing step S105. Then, the control unit 11 may set up the second control model 35 in a state in which calculation processing can be executed, by referring to the second model data 350. As another example, when the second control model 35 includes a rule-based model, the control unit 11 may refer to the second model data 350 and derive the second control command 55 from the environment indicated by at least a part of the observation data 125 in accordance with the rules defined by the rule-based model. When the rule-based model includes an analytical model, the control unit 11 may use the analytical model to recognize the surrounding environment of the moving object M from at least a part of the observation data 125, and derive the second control command 55 from the recognition result in accordance with the rules.

[0084] When the derivation of the second control command 55 is attempted, the control unit 11 advances the process to the next step S106. In the example of FIG. 7, the process of step S105 is executed in response to the determination result of step S103 that an abnormality has occurred in the derivation of the first control command 50. However, the processing timing of step S105 is not limited to this example. The processing of step S105 may be executed at any timing regardless of the determination result of step S103. As another example, the processing of step S105 may be executed at least partially in parallel with the processing of step S102.

[0085] (Step S106) In step S106, the control unit 11 operates as the determination unit 114 and determines whether or not the second control command 55 has been normally derived by the second control model 35. In this embodiment, similar to the first control command 50, the occurrence of an abnormality in the derivation of the second control command 55 may include at least one of the following: an output failure of the second control command 55 occurs, the reliability of the output of the second control model 35 is less than a threshold, and the change amount per unit time of the output of the second control model 35 exceeds a threshold. If it is determined that the second control command 55 has been normally derived, the control unit 11 advances the process to step S107. On the other hand, if it is determined that an abnormality has occurred in the derivation of the second control command 55, the control unit 11 advances the process to step S108.

[0086] (Step S107) In step S107, the control unit 11 operates as the operation control unit 116 and controls the operation of the moving object M in accordance with the derived second control command 55. The control unit 11 may directly or indirectly control the operation of the target moving object M. When the control of the target moving object M is completed, the control unit 11 ends the processing procedure of the control device 1 according to this operation example.

[0087] (Step S108) In step S108, the control unit 11 operates as the operation reception unit 115, receives an input of the manual operation 40 by the user, and acquires a third control command 60 derived from the input manual operation 40. Upon receiving the input of the manual operation 40 and acquiring the third control command 60, the control unit 11 advances the process to the next step S109.

[0088] In the example of Fig. 7, the process of step S108 is executed in response to the determination results of steps S103 and S106 that an abnormality has occurred in both the derivation of the first control command 50 and the derivation of the second control command 55. However, the timing of the process of step S108 is not limited to this example. The process of step S108 may be executed at any timing regardless of the determination results of at least one of steps S103 and S106. In another example, the process of step S108 may be executed regardless of any of the processes of steps S101 to S107.

[0089] In this embodiment, when step S108 is reached, the operation mode of the moving object M is switched from automatic control by the control models (first control model 30, second control model 35) to manual control. When switching to manual control, the control unit 11 may execute any transition process (for example, a notification process to switch to manual control). Timing of Execution of Any Transition Process may be appropriately determined depending on the embodiment. Furthermore, for example, when the manual operation 40 cannot be obtained because the user is in a state where he / she cannot operate, the control unit 11 may execute any emergency process (for example, parking the vehicle on the roadside, etc.). In this case, the control unit 11 may omit the process of the next step S109.

[0090] (Step S109) In step S109, the control unit 11 operates as the operation control unit 116 and controls the operation of the moving object M in accordance with the accepted manual operation 40. That is, the control unit 11 controls the operation of the moving object M in accordance with the acquired third control command 60. The control unit 11 may directly or indirectly control the operation of the target moving object M. When the control of the target moving object M is completed, the control unit 11 ends the processing procedure of the control device 1 according to this operation example.

[0091] (others) The control unit 11 may repeatedly execute a series of information processing from step S101 to step S109. The timing of the repetition may be appropriately determined depending on the embodiment. In one example, the control unit 11 may repeatedly execute a series of information processing from step S101 to step S109 for a predetermined period (for example, from when the power source of the moving body M is started to when it is stopped). In this way, the control device 1 may continuously perform the operation control of the moving body M.

[0092] [Features] In this embodiment, the second control model 35 capable of deriving a control command from a different perspective is deployed together with the first control model 30. As a result, it is possible to expect the second control model 35 to derive a control command normally in at least some of the scenes in which the first control model 30 cannot derive a control command normally. Even if an abnormality occurs in the derivation of the first control command 50 by the first control model 30 in the process of step S102, it is possible to expect the continuation of automatic control by the processes of steps S105 and S107 using the second control model 35. As a result, even if it is difficult to continue the automatic control of the first control model 30, there may be cases where it is not necessary to switch to control by manual operation 40 in steps S108 and S109. Therefore, according to this embodiment, it is possible to reduce the frequency of control by manual operation.

[0093] [4 Variations] Although the embodiment of the present disclosure has been described in detail above, the above description is merely an example of the present disclosure in every respect. It goes without saying that various improvements or modifications can be made without departing from the scope of the present disclosure. For example, the following modifications are possible. In the following, the same reference numerals are used for the same components as in the above embodiment, and the description of the same points as in the above embodiment is omitted as appropriate. The following modifications can be combined as appropriate.

[0094] <4.1> In the above embodiment, the allocation of which of the second control model 35 and the manual operation 40 to execute is determined depending on whether the derivation of the second control command 55 is successful. However, the allocation method is not limited to such an example, and may be appropriately determined depending on the embodiment. In another example, together with or instead of the above embodiment, controlling the movement of the moving body M in accordance with the second control command 55 or the third control command 60 may be configured by determining which of the second control model 35 and the manual operation 40 to adopt in accordance with the environment of the movement of the moving body M, deriving the second control command 55 using the second control model 35 when it is decided to adopt the second control model 35, and controlling the movement of the moving body M in accordance with the derived second control command 55, and controlling the movement of the moving body M in accordance with the third control command 60 derived from the manual operation 40 when it is decided to adopt the manual operation 40. In this embodiment, the moving body M may be a vehicle, and the environment of the movement of the moving body M may include the type of road on which the vehicle travels.

[0095] FIG. 8 is a schematic diagram showing an example of a control process in the control device 1 according to this modified example. In the example of FIG. 8, the control device 1 attempts to derive the first control command 50 by the first control model 30. If the first control command 50 is normally derived as a result of the attempt to derive the first control command 50, the control device 1 controls the operation of the moving object M according to the derived first control command 50. On the other hand, if an abnormality occurs in the derivation of the first control command 50, the control device 1 determines whether to adopt the second control model 35 or the manual operation 40 depending on the environment of the movement of the moving object M at the time of control. In one example, the determination unit 114 may be configured to determine whether to adopt the second control model 35 or the manual operation 40.

[0096] When it is decided to adopt the second control model 35, the control device 1 derives a second control command 55 using the second control model 35. Then, the control device 1 controls the operation of the moving object M in accordance with the derived second control command 55. On the other hand, when it is decided to adopt the manual operation 40, the control device 1 accepts an input of the manual operation 40 and acquires a third control command 60 derived from the input manual operation 40. Then, the control device 1 controls the operation of the moving object M in accordance with the acquired third control command 60.

[0097] FIG. 9 shows an example of a processing procedure for controlling a moving body M by a control device 1 according to this modified example. The following processing procedure is an example of a control method executed by a computer. However, the following processing procedure is merely an example, and each step may be modified as much as possible. Furthermore, steps in the following processing procedure may be omitted, replaced, or added as appropriate depending on the embodiment.

[0098] The processing procedure shown in FIG. 9 may be similar to that of the above embodiment (FIG. 7), except that after it is determined in the processing of step S103 that an abnormality has occurred in the derivation of the first control command 50, the processing of step S200 is executed before the processing of step S105.

[0099] In step S200, the control unit 11 operates as the determination unit 114 and determines whether to adopt the second control model 35 or the manual operation 40 depending on the environment of the movement of the moving object M at the time of control. In one example, the control unit 11 may determine whether to adopt the second control model 35 or the manual operation 40 depending on the environment indicated by at least a portion of the acquired observation data 125.

[0100] In this modification, a range of a compatible environment may be set for the second control model 35. The control unit 11 may determine whether to adopt the second control model 35 or the manual operation 40 depending on whether the environment in which the moving object M moves is within that range. That is, the control unit 11 may determine to adopt the second control model 35 if the environment in which the moving object M moves is within a usable range of the second control model 35, and may determine to adopt the manual operation 40 if not.

[0101] In one example, when the moving object M is a vehicle, the second control model 35 may be configured to be capable of performing automatic driving on a specific type of road. In this case, the control unit 11 may determine whether to adopt the second control model 35 or the manual operation 40 depending on the type of road on which the vehicle is traveling. As a specific example, the second control model 35 may be configured to perform constant speed driving and vehicle distance control (e.g., ACC: Adaptive cruise control, etc.) on a highway. In this case, the control unit 11 may determine to adopt the second control model 35 if the type of road on which the vehicle (moving object M) is traveling is a highway at the time of control, and may determine to adopt the manual operation 40 if not. In addition, the control device 1 may hold a blacklist that specifies a range in which constant speed driving and vehicle distance control are not performed on a highway. The blacklist may be generated as appropriate (for example, may be obtained by a user setting). In this case, the control unit 11 may determine whether the road on which the vehicle (moving object M) is traveling is included in the blacklist at the time of control, and may determine whether to adopt the manual operation 40 if not. If the road on which the vehicle is traveling is within a range of a blacklisted expressway, or if the type of road on which the vehicle is traveling is other than an expressway (e.g., an ordinary road), the control unit 11 may determine to employ the manual operation 40.

[0102] In another example, the control unit 11 may determine whether to adopt the second control model 35 or the manual operation 40 depending on whether the environment in which the moving object M moves is within a range in which data used by the second control model 35 can be acquired. As a specific example, when the second control model 35 uses map information to derive the second control command 55, the control unit 11 may determine to adopt the second control model 35 if the current position of the vehicle (moving object M) is within the corresponding range of the map information, and may otherwise determine to adopt the manual operation 40. In addition, when the sensor S includes a GPS module and the second control model 35 uses position information measured by the GPS module to derive the second control command 55, the control unit 11 may determine to adopt the second control model 35 if the current position of the vehicle (moving object M) is within the measurement section of the GPS module, and may otherwise determine to adopt the manual operation 40.

[0103] When it is determined that the second control model 35 is adopted, the control unit 11 advances the process to step S105. On the other hand, when it is determined that the manual operation 40 is adopted, the control unit 11 advances the process to step S108. In the example of FIG. 9, the process of step S200 is executed in response to the occurrence of an abnormality in the derivation of the first control command 50 as a result of the determination of step S103. However, the processing timing of step S200 is not limited to such an example. The process of step S200 may be executed at any timing regardless of the determination result of step S103. As another example, the process of step S200 may be executed at least partially in parallel with at least any of the processes of steps S101 to S103. According to this modified example, in a situation where automatic control by the second control model 35 is reliably impossible, the determination process of step S106 can be omitted and the control can be switched to the manual operation 40. This can improve the efficiency of the control process.

[0104] <4.2> In the above embodiment, for convenience of explanation, the number of each of the first control model 30 and the second control model 35 is one. However, the number of the first control model 30 and the second control model 35 is not limited to one and may be determined arbitrarily. The number of at least one of the first control model 30 and the second control model 35 arranged in the control device 1 may be two or more.

[0105] In one example, the control device 1 may hold a first control model 30 and a second control model 35 that are different for each scene. When the moving object M is a vehicle, the scene may be, for example, a lane change, lane keeping, an emergency stop (EDSS: Emergency Driving Stop System), etc. In this case, the control device The control device 1 may switch between the first control model 30 and the second control model 35 to be used for each scene.

[0106] Furthermore, the control device 1 may hold a plurality of second control models 35 for one first control model 30. That is, the number of second control models 35 used in support of one first control model 30 may not be limited to one, and may be determined arbitrarily. A plurality of second control models 35 may be deployed as support for one first control model 30. In this modification, when an abnormality occurs in the derivation of the first control command 50 by the first control model 30, the control device 1 may derive the second control command 55 by one or more second control models 35 among the plurality of second control models 35. Note that the plurality of second control models 35 may be referred to in order as the second control model, the third control model, ..., the Nth control model, etc.

[0107] [5 Supplementary Note] The processes and means described in the present disclosure can be freely combined and implemented as long as no technical contradiction occurs.

[0108] Furthermore, a process described as being performed by one device may be shared and executed by multiple devices. Or, a process described as being performed by different devices may be executed by one device. In a computer system, the hardware configuration for implementing each function can be flexibly changed.

[0109] The present disclosure can also be realized by supplying a computer program implementing the functions described in the above embodiments to a computer, and having one or more processors of the computer read and execute the program. Such a computer program may be provided to the computer by a non-transitory computer-readable storage medium connectable to the system bus of the computer, or may be provided to the computer via a network. Non-transitory computer-readable storage media include, for example, any type of disk, such as a magnetic disk (e.g., a floppy disk, a hard disk drive (HDD), etc.), an optical disk (e.g., a CD-ROM, a DVD disk, a Blu-ray disk, etc.), a read-only memory (ROM), a random access memory (RAM), an EPROM, an EEPROM, a magnetic card, a flash memory, an optical card, a semiconductor drive (e.g., a solid-state drive), and any type of medium suitable for storing electronic instructions. [Explanation of symbols]

[0110] 1...Control device, 11: control unit, 12: storage unit, 13: external interface, 14...input device, 15...output device, 16...drive, 81...control program, 91...storage medium, 111: Acquisition unit; 112: First derivation unit; 113: Second derivation unit; 114: Determination unit, 115: Operation reception unit, 116: Operation control unit, 125…observation data, 30...first control model, 300...first model data, 35...second control model, 350...second model data, 40...Manual operation, 50: first control command, 55: second control command, 60…Third control command, M: moving object, S: sensor

Claims

1. a storage unit that stores the first control model and the second control model; and Control unit, A control device comprising: the second control model is configured to be able to normally derive a control command in at least a part of situations in which an abnormality occurs in the derivation of a control command by the first control model due to at least one of a type and input data being different from that of the first control model, The control unit is Attempting to derive a first control command using the first control model; When the derivation of the first control command by the first control model is normal, controlling a movement of a moving object in accordance with the derived first control command; when an abnormality occurs in the derivation of the first control command by the first control model, controlling the movement of the moving object in accordance with a second control command derived by the second control model or a third control command derived from a manual operation by a user; configured to execute Control device.

2. The first control model is configured by a trained machine learning model; The second control model is configured by a rule-based model. The control device according to claim 1 .

3. The first control model is configured by a trained machine learning model; The second control model is configured by a trained machine learning model whose input data is at least partially different from that of the first control model. The control device according to claim 1 .

4. Controlling the movement of the moving object in accordance with the second control command or the third control command includes: determining whether to adopt the second control model or the manual operation depending on an environment in which the moving object moves; When it is determined to adopt the second control model, deriving the second control command using the second control model, and controlling the movement of the moving object in accordance with the derived second control command; and When it is determined that the manual operation is to be adopted, controlling the movement of the moving object in accordance with a third control command derived from the manual operation; Consists of: The control device according to claim 1 .

5. the moving object is a vehicle, The environment of the movement of the moving body includes the type of road on which the vehicle travels. The control device according to claim 4.

6. Controlling the movement of the moving object in accordance with the second control command or the third control command includes: When the derivation of the second control command by the second control model is normal, controlling the movement of the moving object in accordance with the derived second control command; and when an abnormality occurs in the derivation of the second control command by the second control model, controlling the movement of the moving object in accordance with the third control command derived from the manual operation; Consists of: The control device according to claim 1 .

7. The occurrence of an abnormality in the derivation of the first control command includes an occurrence of an output failure of the first control model. The control device according to claim 1 .

8. The occurrence of an abnormality in the derivation of the first control command includes a reliability of an output of the first control model being less than a threshold value. The control device according to claim 1 .

9. The occurrence of an abnormality in the derivation of the first control command includes a change amount per unit time of the output of the first control model exceeding a threshold value. The control device according to claim 1 .

10. A control device that holds a first control model and a second control model that is configured to be able to normally derive a control command in at least a part of situations in which an abnormality occurs in deriving a control command by the first control model due to at least one of a type and input data being different from that of the first control model, Attempting to derive a first control command using the first control model; When the derivation of the first control command by the first control model is normal, controlling a movement of a moving object in accordance with the derived first control command; when an abnormality occurs in the derivation of the first control command by the first control model, controlling the movement of the moving object in accordance with a second control command derived by the second control model or a third control command derived from a manual operation by a user; In order to execute Control program.

11. The first control model is configured by a trained machine learning model; The second control model is configured by a rule-based model. The control program according to claim 10.

12. The first control model is configured by a trained machine learning model; The second control model is configured by a trained machine learning model whose input data is at least partially different from that of the first control model. The control program according to claim 10.

13. Controlling the movement of the moving object in accordance with the second control command or the third control command includes: determining whether to adopt the second control model or the manual operation depending on an environment in which the moving object moves; When it is determined to adopt the second control model, deriving the second control command using the second control model, and controlling the movement of the moving object in accordance with the derived second control command; and When it is determined that the manual operation is to be adopted, controlling the movement of the moving object in accordance with a third control command derived from the manual operation; Consists of: The control program according to claim 10.

14. the moving object is a vehicle, The environment of the movement of the moving body includes the type of road on which the vehicle travels. The control program according to claim 13.

15. Controlling the movement of the moving object in accordance with the second control command or the third control command includes: When the derivation of the second control command by the second control model is normal, controlling the movement of the moving object in accordance with the derived second control command; and when an abnormality occurs in the derivation of the second control command by the second control model, controlling the movement of the moving object in accordance with the third control command derived from the manual operation; Consists of: The control program according to claim 10.

16. The occurrence of an abnormality in the derivation of the first control command includes an occurrence of an output failure of the first control model. The control program according to claim 10.

17. The occurrence of an abnormality in the derivation of the first control command includes a reliability of an output of the first control model being less than a threshold value. The control program according to claim 10.

18. The occurrence of an abnormality in the derivation of the first control command includes a change amount per unit time of the output of the first control model exceeding a threshold value. The control program according to claim 10.

19. A control device that holds a first control model and a second control model that is configured to be able to normally derive a control command in at least a part of situations in which an abnormality occurs in the derivation of a control command by the first control model due to at least one of a type and input data being different from that of the first control model, Attempting to derive a first control command using the first control model; When the derivation of the first control command by the first control model is normal, controlling a movement of a moving object in accordance with the derived first control command; when an abnormality occurs in the derivation of the first control command by the first control model, controlling the movement of the moving object in accordance with a second control command derived by the second control model or a third control command derived from a manual operation by a user; Execute Control methods.

20. Controlling the movement of the moving object in accordance with the second control command or the third control command includes: determining whether to adopt the second control model or the manual operation depending on an environment in which the moving object moves; When it is determined to adopt the second control model, deriving the second control command using the second control model, and controlling the movement of the moving object in accordance with the derived second control command; and When it is determined that the manual operation is to be adopted, controlling the movement of the moving object in accordance with a third control command derived from the manual operation; Consists of:

20. The control method of claim 19.

Images