AUTONOMOUS MOBILE DEVICE, SERVER DEVICE, ANOMALYSIS DETECTION METHOD, AND PROGRAM

Abstract

To detect an abnormal state that requires countermeasures in an autonomous mobile device under a determination condition desired by an operator.SOLUTION: An autonomous mobile device 1 includes a sensor group 1a, a state estimation section 1b, a first abnormality determination section 1c, a second abnormality determination section 1d, an output section 1e, and a reference setting section 1f. The state estimation section 1b estimates, based on the detection result, which of a plurality of predetermined state labels the current state corresponds to, and the first abnormality determination section 1c determines whether or not there is an abnormality based on the estimation result. Based on results estimated multiple times at different times, the second abnormality determination section 1d determines which of a plurality of countermeasure labels or a normal label corresponds. The plurality of countermeasure labels are labels predetermined for each countermeasure to indicate a history abnormal state that requires countermeasures, and the normal label is a label that does not require countermeasures. The output section 1e outputs determination results of the first abnormality determination section 1c and the second abnormality determination section 1d. The reference setting section 1f sets the determination reference of the second abnormality determination section 1d.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to an autonomous mobile device, a server device, a learning device, an anomaly detection method, and a program. [Background technology]

[0002] In recent years, autonomously moving devices such as drones and self-driving cars have been actively researched and are being used in some areas. In addition, autonomous moving devices are expected to become more widespread in the future due to the decline in the working population, cost reduction, and business efficiency. For example, autonomous moving devices are expected to be used in situations closely related to people's lives as tasks such as transporting and delivering luggage are replaced by autonomous moving devices from people and cars driven by people. In that case, autonomous moving devices will be operated in people's living spaces, so they will be required to operate safely as well as reliably and stably.

[0003] Patent Document 1 describes an emergency response instruction device for a drone that includes a receiving means, a determining means, a forming means, and a transmitting means. The receiving means receives a status notification signal, which is a signal from a drone that is an unmanned aerial vehicle and includes status information indicating the status of the drone. When the determining means receives the status notification signal through the receiving means, the determining means determines whether the status of the drone is an emergency based on the status information included in the status notification signal. When the determining means determines that the status of the drone is an emergency, the forming means forms response instruction information for the drone. The transmitting means transmits the response instruction information formed by the forming means.

[0004] Regarding abnormality detection, Patent Document 2 describes a plant operation support device that includes an abnormal state database, an operation manual database, a plant monitoring unit, an abnormal state discrimination unit, a countermeasure search unit, and a notification unit. The abnormal state database records the range of parameters when the plant body is in an abnormal state, and the operation manual database records the operation procedure for eliminating the abnormal state. The plant monitoring unit collects parameters from the plant body. The abnormal state discrimination unit refers to the abnormal state database and determines whether the plant body is in an abnormal state based on the collected parameters. The countermeasure search unit extracts the operation procedure from the operation manual database when the plant body is in an abnormal state. The notification unit notifies a terminal device of the operation procedure. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2018-165115 A [Patent Document 2] JP 2003-005828 A Summary of the Invention [Problem to be solved by the invention]

[0006] The present inventors have considered safe and stable operation of an autonomous mobile device. In order to operate an autonomous mobile device safely, that is, to make it difficult for a collision to occur in the first place even if an abnormality occurs in the autonomous mobile device or the surrounding environment, the following safety measures can be considered: That is, as safety measures, possible measures include operating the autonomous mobile devices with a sufficiently large distance between each other and between the autonomous mobile device and surrounding structures or people, operating at a sufficiently slow speed, and operating only in areas where a crash or collision will not cause any problems.

[0007] However, assuming that autonomous mobile devices are to be fully used for purposes such as home delivery, the autonomous mobile devices need to be able to operate in large numbers, at high density, and at high speed in urban areas where people are present, but the above safety measures do not meet these requirements. In other words, autonomous mobile devices cannot be operated in large numbers, at high density, and at high speed by ensuring safety through operation in limited areas, operation with a sufficiently wide gap between devices, or operation at low speed. Therefore, a mechanism is required that allows operation in large numbers, at high density, and at high speed, and that also provides safety measures in the event of an emergency.

[0008] On the other hand, some autonomous mobile devices have a mechanism to stop the device for safety reasons with a very simple judgment. For example, there is a mechanism to monitor the operating status of the autonomous mobile device's operating parts and stop the device if a malfunction is suspected, or a mechanism to monitor the thresholds of sensor values ​​such as attitude sensors and acceleration sensors and stop the device if a certain sensor exceeds a certain threshold.

[0009] However, because autonomous mobile devices are used in a variety of situations, immediately stopping movement due to overdetection or detection of a malfunction or abnormal value may be preventative and excessive, failing to meet operational requirements, or may not necessarily be safe.

[0010] For example, when an autonomous vehicle, which is a type of autonomous mobile device, is traveling in the passing lane on a highway, stopping the vehicle on the spot even if an abnormality is suspected is dangerous as it could lead to a collision with the vehicle behind. Therefore, unless there is an emergency that outweighs the risk, the vehicle should continue to drive to a safe place where it is safe to stop.

[0011] As another example, drones, which are a type of autonomous mobile device, operated in urban areas for transportation purposes, etc., are at risk of contact with people and vehicles on the ground, power lines, important facilities, etc., so even if an abnormality is suspected, it is not desirable to immediately land the drone on the spot. In other words, unless there is an emergency that outweighs the risk, the drone should be operated to a safe place to land. As mentioned above, it is expected that high-density, high-speed operation will be required in the future, so there is a risk that a sudden landing operation will cause a collision with other drones in the vicinity, just like the example of an autonomous vehicle on a highway.

[0012] In addition, when the operational requirements are to transport important goods for transportation purposes, etc., even in an emergency, it may not be desirable to stop or land the aircraft in a location where recovery would be difficult or impossible, such as deep in the mountains, in remote areas, on the sea, on a lake, or on a river.

[0013] In other words, excessive emergency stops of an autonomous mobile device due to suspected abnormalities are inconvenient from an operational standpoint, and emergency stops may not be the best way to deal with certain operational requirements. In such cases, it is desirable for judgment criteria and countermeasures to be defined by the operator, rather than the developer, according to operational requirements.

[0014] Another possible method is to have humans remotely operate the autonomous mobile devices only in emergencies, but this method requires a large number of personnel in response to the large number of autonomous mobile devices in operation, which can lead to the following problems: That is, problems arise in terms of cost and manpower, and in the event of a multiple accident, there is a limit to the number of autonomous mobile devices that can respond.

[0015] We have explained the issues from the perspective of safety, but problems may also arise from the perspective of stable operation if extreme measures such as emergency stop or emergency landing are implemented to halt operations due to simple criteria such as suspected abnormalities, the presence or absence of a single failure in a redundant part, or thresholds in sensor information. This problem frequently interferes with the operation of the autonomous mobile device, such as transportation and movement, which are the original purposes of the autonomous mobile device, and frequently makes it impossible to meet operational requirements, such as delivering transported goods by a set time.

[0016] For this reason, it is desirable for autonomous mobile devices to have a means of determining the degree and type of abnormality, and, in the case of minor abnormalities, to allow operators, rather than developers, to flexibly and easily set the conditions for determining abnormalities and methods of dealing with them in accordance with operational requirements.

[0017] In contrast, the technology described in Patent Document 1 can take measures after detecting an abnormality as a response to an abnormality in a drone, but cannot improve the accuracy of abnormality determination before the abnormality occurs, avoid emergency landings as much as possible, and increase the availability of the drone. Also, the technology described in Patent Document 2 can determine an abnormal state, search for the cause of the abnormality, and extract measures to address the cause of the abnormality, but the extracted measures are based on data in each database, making it difficult for the operator to change them for each plant, and lacking versatility.

[0018] An object of the present disclosure is to provide an autonomous mobile device, a server device, a learning device, an anomaly detection method, a program, etc. that are capable of detecting abnormal conditions that require countermeasures based on judgment conditions desired by the operator of the autonomous mobile device. [Means for solving the problem]

[0019] The autonomous mobile device according to a first aspect of the present disclosure includes a sensor group for detecting a current state of the autonomous mobile device, a state estimation unit for estimating which of a plurality of predetermined state labels the current state of the autonomous mobile device corresponds to based on the detection result by the sensor group, a first abnormality determination unit for determining whether the current state is an abnormal state that requires a measure to change the state based on the estimation result by the state estimation unit, a second abnormality determination unit for determining which of a plurality of countermeasure labels or a normal label the current state corresponds to based on the results estimated multiple times by the state estimation unit, an output unit for outputting the determination results by the first abnormality determination unit and the determination results by the second abnormality determination unit, and a criterion setting unit for setting a judgment criterion for the second abnormality determination unit. The multiple countermeasure labels are labels that are predetermined for each of the countermeasures to indicate that the state of the autonomous mobile device indicated by the results estimated multiple times is a historical abnormal state that requires a measure to change the state, and the normal label is a label that does not require a measure.

[0020] A server device according to a second aspect of the present disclosure includes a receiving unit that receives, from an autonomous mobile device, a detection result obtained by a sensor group that detects a current state of the autonomous mobile device, a state estimation unit that estimates which of a plurality of predetermined state labels the current state of the autonomous mobile device corresponds to based on the detection result by the sensor group, a first abnormality determination unit that determines whether the current state is an abnormal state that requires a measure to change the state based on the estimation result by the state estimation unit, a second abnormality determination unit that determines which of a plurality of countermeasure labels or a normal label the current state corresponds to based on the results estimated multiple times by the state estimation unit, an output unit that outputs the determination results by the first abnormality determination unit and the determination results by the second abnormality determination unit, and a criterion setting unit that sets a judgment criterion for the second abnormality determination unit. The multiple countermeasure labels are labels that are predetermined for each countermeasure to indicate that the state of the autonomous mobile device indicated by the results estimated multiple times is a historical abnormal state that requires a countermeasure to change the state, and the normal label is a label that does not require a countermeasure.

[0021] A learning device according to a third aspect of the present disclosure includes an acquisition unit that acquires detection results acquired by a group of sensors that detect a current state of an autonomous mobile device, and a control unit, wherein the control unit inputs the detection results acquired by the acquisition unit multiple times and outputs one of multiple countermeasure labels and a normal label, generating a trained model for countermeasure label output. The multiple countermeasure labels are labels that are predetermined for each countermeasure to indicate that the state of the autonomous mobile device indicated by the results estimated multiple times is a historical abnormal state that requires a countermeasure to change the state, and the normal label is a label that does not require a countermeasure.

[0022] The anomaly detection method according to a fourth aspect of the present disclosure includes an acquisition step of acquiring a result of detection of a current state of an autonomous mobile device by a group of sensors in the autonomous mobile device, a state estimation step of estimating which of a plurality of predetermined state labels the current state of the autonomous mobile device corresponds to based on the detection result by the group of sensors, a first anomaly determination step of judging whether the current state is an abnormal state that requires a measure to change the state based on the estimation result in the state estimation step, a second anomaly determination step of judging which of a plurality of countermeasure labels or a normal label the current state corresponds to based on the result estimated multiple times at different times in the state estimation step, an output step of outputting the judgment result in the first anomaly determination step and the judgment result in the second anomaly determination step, and a criterion setting step of setting a judgment criterion for the second anomaly determination step. The multiple countermeasure labels are labels that are predetermined for each of the countermeasures to indicate that the state of the autonomous mobile device indicated by the results estimated multiple times is a historical abnormal state that requires a countermeasure to change the state, and the normal label is a label that does not require a countermeasure.

[0023] A program according to a fifth aspect of the present disclosure causes a computer provided in an autonomous mobile device or connected to the autonomous mobile device to execute the following steps: acquiring a result of detecting a current state of the autonomous mobile device by a group of sensors in the autonomous mobile device; a state estimation step of estimating which of a plurality of predetermined state labels the current state of the autonomous mobile device corresponds to based on the detection result by the group of sensors; a first abnormality determination step of determining whether the current state is an abnormal state that requires a measure to change the state based on the estimation result in the state estimation step; a second abnormality determination step of determining which of a plurality of countermeasure labels or a normal label the current state corresponds to based on the results estimated multiple times at different times in the state estimation step; an output step of outputting the determination results in the first abnormality determination step and the determination results in the second abnormality determination step; and a criterion setting step of setting a judgment criterion for the second abnormality determination step. The multiple countermeasure labels are labels that are predetermined for each countermeasure to indicate that the state of the autonomous mobile device indicated by the results estimated multiple times is a historical abnormal state that requires a countermeasure to change the state, and the normal label is a label that does not require a countermeasure. Effect of the Invention

[0024] The present disclosure makes it possible to provide an autonomous mobile device, a server device, a learning device, an anomaly detection method, a program, etc. that are capable of detecting abnormal conditions that require countermeasures based on judgment conditions desired by the operator of the autonomous mobile device. [Brief description of the drawings]

[0025] [Figure 1] 1 is a block diagram showing an example of the configuration of an autonomous moving device according to a first embodiment. [Diagram 2] FIG. 2 is a block diagram showing an example of the configuration of a learning device that generates a trained model used in the autonomous mobile device according to the first embodiment. [Diagram 3] FIG. 11 is a block diagram showing an example of the configuration of an autonomous moving device according to a second embodiment. [Figure 4]4 is a block diagram showing an example of the configuration of a state estimation unit in the autonomous mobile device of FIG. 3. [Diagram 5] FIG. 11 is a diagram illustrating an example of learning data used when generating a state model. [Figure 6] FIG. 13 is a diagram illustrating an example of state classification. [Figure 7] FIG. 13 is a conceptual diagram for explaining an example of a procedure for generating a state model 122 when a correct label is unknown. [Figure 8] FIG. 13 is a conceptual diagram for explaining an example of a procedure for generating a state model 122 when a correct label is known. [Figure 9] FIG. 11 is a schematic diagram for explaining an example of state estimation. [Figure 10] FIG. 11 is a diagram illustrating an example of data. [Figure 11] 4 is a block diagram showing an example of the configuration of an abnormal state determination unit in the autonomous mobile device of FIG. 3. [Figure 12] FIG. 11 is a diagram illustrating an example of a status abnormality rule. [Figure 13] FIG. 11 is a diagram illustrating an example of an abnormality determination. [Figure 14] 4 is a block diagram showing an example of the configuration of a state history abnormality determination unit in the autonomous mobile device of FIG. 3. [Figure 15] FIG. 13 is a diagram illustrating an example of a state history anomaly rule. [Figure 16] 4 is a block diagram showing an example of the configuration of a test execution unit in the autonomous mobile device of FIG. 3. [Figure 17] FIG. 13 is a diagram illustrating an example of a test rule. [Figure 18] 4 is a block diagram showing an example of the configuration of a countermeasure decision unit in the autonomous mobile device of FIG. 3. [Figure 19] FIG. 13 is a diagram illustrating an example of a countermeasure rule. [Figure 20] 4 is a flow diagram for explaining an example of the operation of the autonomous mobile device of FIG. 3. [Figure 21] 5 is a flow diagram for explaining an example of the operation of the state estimation unit 12 in FIG. 4. [Figure 22] 12 is a flow chart for explaining an example of the operation of the abnormal state determination unit in FIG. 11. [Figure 23] 15 is a flow chart for explaining an example of the operation of the state history abnormality determination unit in FIG. 14. [Figure 24] 17 is a flow chart for explaining an example of the operation of the test execution unit in FIG. 16. [Diagram 25] 4 is a flow diagram for explaining an example of the operation of a countermeasure decision unit in the autonomous mobile device of FIG. 3. [Figure 26] FIG. 11 is a block diagram showing an example of the configuration of an autonomous mobile system according to a third embodiment. [Figure 27] FIG. 11 is a block diagram showing another configuration example of the autonomous mobile system according to the third embodiment. [Figure 28] FIG. 2 illustrates an example of a hardware configuration of the apparatus. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0026] Hereinafter, an embodiment will be described with reference to the drawings. In the embodiments, the same or equivalent elements may be denoted by the same reference numerals, and duplicated descriptions will be omitted as appropriate.

[0027] <Embodiment 1> An autonomous mobile device (autonomous mobile equipment) according to the first embodiment will be described with reference to Fig. 1. Fig. 1 is a block diagram showing an example of the configuration of the autonomous mobile device according to the first embodiment.

[0028] 1, the autonomous mobile device 1 according to this embodiment includes a plurality of sensors (sensor group) 1a, a state estimation unit 1b, a first abnormality determination unit 1c, a second abnormality determination unit 1d, an output unit 1e, and a reference setting unit 1f. The autonomous mobile device 1 may also include a communication unit (not shown) that communicates with an external device. This communication unit may be at least one of a wireless communication unit and a wired communication unit depending on the type of the autonomous mobile device 1, the timing of data communication with the outside, and the like.

[0029] The autonomous mobile device 1 is a device that is capable of moving autonomously, and can detect whether or not there is an abnormal state while it is moving (operating) based on the detection results (sensor data) output from the sensor group 1a.

[0030] The autonomous mobile device 1 according to this embodiment can be applied to any mobile device that requires monitoring (detection) of whether the autonomous mobile device 1 is in an abnormal state while moving, and can be suitably used for managing such devices. The autonomous mobile device 1 has information processing capabilities, controls actuators and the like, moves autonomously, and can be any device that is responsible for transportation or movement.

[0031] Examples of the autonomous mobile device 1 include an autonomous automobile, an autonomous train, an autonomous ship, an autonomous flying aircraft, a drone, an AGV (Automated Guided Vehicle) used in a factory or a warehouse, a robot with a running function or a leg-based moving function, a robot electric wheelchair, a motorcycle, and the like. Here, the drone is not limited to an air vehicle such as an unmanned aerial vehicle flying in the sky, but may be operated in any environment such as on the ground, in the sky, on the water, underwater, or in space. In addition, the robot may be moved by any method such as running or walking. Furthermore, examples of the autonomous mobile device 1 include construction vehicles such as forklifts, construction machines, and heavy machinery, vehicles used in logistics such as railroad vehicles, taxis, and trucks, police vehicles, and fire engines. Here, the railroad vehicles may be moved by any method such as light rail, iron wheel type, new transportation system, monorail, and magnetic levitation type. The autonomous mobile device 1 also includes devices with mode switching that sometimes operate autonomously and sometimes operate according to instructions or operations from a human, and devices that move part of their autonomous operation in cooperation with human operations or instructions. In other words, the autonomous mobile device 1 only needs to be capable of moving autonomously, and can also have the function of moving according to control from an external controller or according to the operation of the passenger.

[0032] The sensor group 1a detects the current state of the autonomous mobile device 1. Here, the current state can refer to the sensor value of the sensor included in the sensor group 1a, which is the result of measuring the detection target at the installation location, or the state indicated by the sensor value, and mainly includes the moving state (operating state). The moving state can include the tilt and position of the autonomous mobile device 1. The installation location and the detection target generally differ depending on the type and shape of the autonomous mobile device 1. The sensor group 1a can be composed of multiple sensors necessary for controlling the autonomous mobile device 1, and can include not only a sensor group that collects the state of the autonomous mobile device 1 itself, but also a sensor group that collects the state of the surroundings (surrounding environment). The detection interval in the sensor group 1a does not matter, but it is desirable for the detection to be short for quick detection. The detection intervals of the sensors belonging to the sensor group 1a are preferably the same, but may be different, and detection data with a slight difference in detection time in each sensor can be treated as the same time.

[0033] Some or all of the units 1b to 1f can be configured as a control unit (not shown) that controls the entire autonomous mobile device 1. This control unit can be realized, for example, by a CPU (Central Processing Unit), a working memory, and a non-volatile storage device that stores a program. This program can be a program that executes processing for realizing the functions of some or all of the units 1b to 1f. Furthermore, this control unit can also be realized, for example, by an integrated circuit.

[0034] The state estimation unit 1b estimates which of a plurality of predetermined state labels the current state of the autonomous mobile device 1 corresponds to, based on the detection results of the sensor group 1a. The state estimation by the state estimation unit 1b makes it possible to convert a combination of information from a plurality of sensors (e.g., each numerical value, ON / OFF, mode, etc.) into a state label that represents the state of the autonomous mobile device 1, which is easy for humans to understand. This makes it simple and easy to define a method of dealing with the state of the autonomous mobile device 1. Furthermore, this allows the user of the autonomous mobile device 1 to flexibly define emergency operations at the operation stage, rather than the developer incorporating them in advance as control rules, mechanisms, circuits, and software at the design and development stage of the autonomous mobile device 1. Here, the definition of a method of dealing with an emergency refers to the definition of a countermeasure, which will be described later, and the definition of a criterion for determining when the countermeasure is implemented.

[0035] The first abnormality judgment unit 1c judges whether or not the current state is an abnormal state (state requiring countermeasures) that requires countermeasures to change the state, based on the estimation result by the state estimation unit 1b (i.e., the state label estimated by the state estimation unit 1b).

[0036] The second abnormality determination unit 1d determines whether the state corresponds to one of a plurality of countermeasure labels or a normal label based on the results (a plurality of times or a plurality of types of state labels) estimated a plurality of times at different times by the state estimation unit 1b. For this determination, the estimation results by the state estimation unit 1b may be stored as history.

[0037] The multiple countermeasure labels are labels that are predefined for each countermeasure to indicate that the state of the autonomous mobile device 1 indicated by the results of multiple estimations is a historical abnormal state (state requiring countermeasure) that requires countermeasures to change the state. That is, a countermeasure label is set for each countermeasure to be executed according to the results of multiple estimations, or in other words, a countermeasure is set for each countermeasure label indicated by the judgment result. The normal label is a label that does not require countermeasures, and if it is judged to correspond to the normal label, it means that no countermeasures are required.

[0038] The output unit 1e outputs the judgment result of the first abnormality judgment unit 1c and the judgment result of the second abnormality judgment unit 1d. The output destination of these may include, for example, a movement control unit (drive control unit) (not shown) that controls the movement of the autonomous mobile device 1, thereby making it possible to control the movement according to the judgment results.

[0039] The standard setting unit 1f sets a judgment standard for the second abnormality judgment unit 1d. The standard setting unit 1f can be an operation unit (operation receiving unit) provided on the main body of the autonomous mobile device 1, a receiving unit that receives a setting signal so that the standard can be set by communication from an external device, or both.

[0040] The second abnormality determination unit 1d can also perform the determination using a first trained model that inputs the results of estimation performed multiple times at different times by the state estimation unit 1b, inputs the estimation result by the state estimation unit 1b, and outputs one of multiple countermeasure labels and a normal label. This trained model and the algorithms in the trained model described later are not important as long as the determination can be performed. Of course, the second abnormality determination unit 1d can also perform the determination using a predetermined definition formula without using a trained model, and this point is the same for the first abnormality determination unit 1c and the state estimation unit 1b.

[0041] In addition, it is preferable that the first abnormality judgment unit 1c also judges whether the current state corresponds to one of a plurality of countermeasure labels predefined for each type of countermeasure to change the current state or a normal label that does not require countermeasures. The countermeasure labels and normal labels here can be defined in the same way as those in the second abnormality judgment unit 1d, but the difference is that the judgment is based on a single state detection result, so the judgment criteria are basically different. In other words, the label group of countermeasure labels and normal labels in the first abnormality judgment unit 1c can have labels in common with the label group of countermeasure labels and normal labels in the second abnormality judgment unit 1d, but at least the judgment criteria are different.

[0042] The first abnormality judgment unit 1c can also perform judgment using a second trained model that inputs the estimation result from the state estimation unit 1b and outputs one of a plurality of countermeasure labels and a normal label.

[0043] The state estimation unit 1b can also perform estimation using a third trained model that inputs the detection results from the sensor group 1a and outputs one of a plurality of state labels. In this way, state estimation by the state estimation unit 1b using machine learning can convert a combination of information from a plurality of sensors (such as numerical values, ON / OFF, and mode) into a state label that represents the state of the autonomous mobile device that is easy for humans to understand. By using machine learning, it is possible to estimate the state label more accurately.

[0044] Each trained model that can be used for determination and estimation will be described later with reference to FIG. 2. Each trained model can be constructed as one trained model. Each trained model can be stored inside the autonomous mobile device 1, such as by storing it inside the control unit, but can also be stored in an external server device. When each trained model is stored in the server device, the autonomous mobile device 1 can transmit the detection result of the sensor group 1a and obtain the result of determination or estimation as a response.

[0045] As described above, by using state labels, the user (operator) of the autonomous mobile device 1 can flexibly define emergency actions (actions when measures need to be taken) at the operational stage, and this is also true when a trained model is not used. Furthermore, by using state labels that represent the states of the autonomous mobile device that are easy for humans to understand, the user can define more complicated emergency actions for the autonomous mobile device 1. For example, the following becomes possible.

[0046] That is, it is possible to provide a means for humans to easily define a countermeasure for a state label that represents a state of the autonomous mobile device 1 that is easy for humans to understand, and a means for changing the operation of the autonomous mobile device 1 based on the definition. In addition, as for this countermeasure, since the state of the autonomous mobile device 1 can be expressed by a state label, it becomes possible for the criterion setting unit 1f to define a more complex and abnormal way of using the autonomous mobile device and a countermeasure for it that take into account the passage of time, as in the judgment by the second abnormality judgment unit 1d. In addition, the criterion setting unit 1f is configured to be able to set the judgment criterion of the second abnormality judgment unit 1d, but it is preferable to configure it so that it is also able to set the judgment criterion of the first abnormality judgment unit 1c. In this case, if the judgment result is output as a label by the first abnormality judgment unit 1c, the definition can be easily changed.

[0047] These mechanisms not only enable the autonomous mobile device 1 to automatically take appropriate countermeasures when it falls into a specific state, such as when an abnormality occurs, but also allow the operator to easily change the countermeasures that had to be pre-installed at the time of design and development of the autonomous mobile device later. These changes can be made by the operator depending on, for example, the value and characteristics of the cargo at the time of operation, the contract conditions for operation, etc.

[0048] As described above, the state of the autonomous mobile device 1 is expressed in a form that is easy for people to understand through state estimation by the state estimation unit 1b, so that the method of dealing with the state can be easily defined, that is, various rules can be described concisely. Here, it is expressed that the rules can be described concisely in that they can be described in highly readable combination labels, rather than being defined for a large number of sensor values, status values, and other values, combinations and ranges.

[0049] For example, by describing the rules concisely, the behavior of the autonomous mobile device in the event of an abnormality can be varied according to the transported goods and the surrounding environment, the price of the vehicle and cargo, the purpose of operation, etc. Furthermore, since the state of the autonomous mobile device 1 is expressed in a format that is easy for people to understand, it is easy to define countermeasures for that state and define how to judge them (for example, the definition of the test and the definition of the judgment that will be explained in the second embodiment).

[0050] Furthermore, by being able to concisely describe rules, it becomes possible for the user to concisely describe anomaly determination rules as a definition file, rather than implementing them as control laws in the control unit (which requires specialized knowledge in the design and development of the autonomous mobile device 1). Furthermore, by being able to concisely describe rules, it becomes possible to concisely define not only simple anomaly determination rules, but also more complicated conditions that take history into consideration.

[0051] For example, as an example of a judgment rule based on the number of occurrences, it can be defined so that if state A occurs a certain number of times within a certain time period, it is judged to be an anomaly. Also, as an example of a rule for combinations that occur in no particular order, it can be defined so that if states A, B, and C occur within a certain time period, it is judged to be an anomaly. Also, as an example of a rule for combinations where the order is important, it can be defined so that if state A, state B, and state C occur within a certain time period, it is judged to be an anomaly. As shown in these examples, complex conditions can be easily defined. Of course, the combinations of states are not limited to A, B, and C, and combinations of two or more states can be easily defined.

[0052] Furthermore, by allowing the operator to easily set rules in this manner, there is no need to excessively falsely detect abnormalities and take immediate emergency action, thereby improving the operational availability of the autonomous mobile device 1.

[0053] As described above, in the autonomous mobile device 1 according to this embodiment, the operator, not the developer, can define the classification of anomalies and the corresponding actions in a more detailed and flexible manner and dynamically in accordance with business requirements. In other words, according to this embodiment, it becomes possible to detect an abnormal state that requires measures using a judgment condition desired by the operator of the autonomous mobile device 1. This makes the autonomous mobile device 1 more available than autonomous mobile devices that can only handle routine responses such as moving to a specified location or making an emergency stop (landing) when an abnormality occurs.

[0054] Next, a trained model used in the autonomous mobile device 1 will be described with reference to Fig. 2. Fig. 2 is a block diagram showing an example of the configuration of a learning device that generates a trained model used in the autonomous mobile device 1.

[0055] 2, the learning device 2 can include an acquisition unit 2a and a control unit 2b. Here, the control unit 2b can include a storage unit 2c. The learning device 2 can also be provided inside the autonomous mobile device 1.

[0056] The acquisition unit 2a acquires detection results acquired by a group of sensors that detect the current state of the autonomous mobile device 1. When the learning device 2 is outside the autonomous mobile device 1, the acquisition unit 2a can be a reading unit that reads in the detection results via a portable recording medium, or a receiving unit that receives the detection results via wired or wireless communication. When the learning device 2 is provided inside the autonomous mobile device 1, the acquisition unit 2a may acquire the detection results from the group of sensors 1a via internal wiring.

[0057] The control unit 2b receives the detection results acquired by the acquisition unit 2a multiple times, and generates a trained model for outputting countermeasure labels, which outputs one of multiple countermeasure labels and a normal label. Note that the multiple countermeasure labels and the normal label are as described above.

[0058] The trained model 2e for outputting countermeasure labels can be a trained model constructed by inputting multiple detection results into the untrained model 2d stored in the memory unit 2c and performing machine learning, and this can also be stored in the memory unit 2c.

[0059] The control unit 2b can also be configured to generate a trained model for outputting a state label, which inputs the detection result acquired by the acquisition unit 2a and outputs one of a plurality of state labels predetermined to indicate the current state of the autonomous mobile device 1. Similarly, the trained model for outputting a state label can also be a trained model constructed by inputting the detection result to an untrained model 2d stored in the storage unit 2c and performing machine learning, and this can also be stored in the storage unit 2c.

[0060] The control unit 2b can be realized by, for example, a CPU, a working memory, and a non-volatile storage device that stores a program, similar to the control unit that controls the entire autonomous mobile device 1. This program can be a trained model generation program.

[0061] In this way, the learning device 2 of this embodiment can generate a learned model that can be used in an autonomous mobile device that can detect abnormal conditions that require countermeasures using judgment conditions desired by the operator of the autonomous mobile device.

[0062] <Embodiment 2> The second embodiment will be described with reference to Figures 3 to 25, focusing on the differences from the first embodiment, but the various examples described in the first embodiment can also be applied. The following description will be given with an example in mind in which a drone is used as the autonomous mobile device according to this embodiment, but the present invention will be essentially the same even if the drone is replaced with another type of autonomous mobile device.

[0063] Fig. 3 is a block diagram showing an example of a configuration of an autonomous mobile device according to the second embodiment. As shown in Fig. 3, the autonomous mobile device 10 according to this embodiment can include a sensor group 11, a state estimation unit 12, a state abnormality determination unit 13, a state history abnormality determination unit 14, a test implementation unit 15, a countermeasure decision unit 16, a movement control unit 17, and a standard setting unit 18. As the autonomous mobile device 10, various examples exemplified as the autonomous mobile device 1 according to the first embodiment can be applied. Generally, an autonomous mobile device is provided with a sensor group 11, a movement control unit 17, and a simple abnormality detection means and only performs autonomous operation, but the autonomous mobile device 10 according to this embodiment further includes each unit 12 to 16, 18.

[0064] The sensor group 11 corresponds to the sensor group 1a, the state estimation unit 12 is an example of the state estimation unit 1b, the state abnormality determination unit 13 is an example of the first abnormality determination unit 1c, the state history abnormality determination unit 14 and the test implementation unit 15 are an example of the second abnormality determination unit 1d, and the countermeasure decision unit 16 is an example of the output unit 1e. The standard setting unit 18 is an example of the standard setting unit 1f. Note that only one of the state history abnormality determination unit 14 and the test implementation unit 15 may be installed.

[0065] Each component shown in FIG. 3 may be a program on a computer, may be composed of an electric / electronic circuit, or may be a combination of a program and an electric / electronic circuit. Information can be exchanged between components connected by lines in FIG. 3. If the components are programs, information between the components may be exchanged via a shared memory or storage, or if the units are separate for implementation, information may be exchanged via communication. In addition, all or a part of the components other than the sensor group 11 in FIG. 3 may be mounted on an external device connected via communication to the autonomous mobile device 10 and processed, and this example will be described later as embodiment 3. For example, a server device (which may be a cloud server device) such as a management device connected via communication to the autonomous mobile device 10 may execute the same process instead of the autonomous mobile device 10. Details of each component will be described below.

[0066] (Sensor group 11) First, the sensor group 11 is a group of sensors mounted on the autonomous mobile device 10. Not limited to the autonomous mobile device 10, the sensor group 11 can be a plurality of various sensors that are generally mounted on an autonomous mobile device to collect information on its own state and the state of the surroundings required for its control.

[0067] An example of the sensor group 11 will be described using a drone, which is a type of autonomous mobile device, as an example. The sensor group 11 can include a speed sensor that observes the speed of the aircraft, an acceleration sensor that observes the acceleration of the aircraft, a gyro sensor that observes the roll, pitch, and yaw of the aircraft, and a rotation sensor that observes the number of rotations of each rotor used for flight. The sensor group 11 can also include a pressure sensor and an altitude sensor that observe the altitude of the aircraft, a geomagnetic sensor that observes the direction from the aircraft, and a temperature sensor that detects heat generation in the circuit and rotor. Of course, the sensor group 11 does not need to include all of these examples, and can include only a part of them, and can also include other types of sensors without being limited to these examples.

[0068] For example, another sensor included in the sensor group 11 may be a sensor that acquires position information in a global navigation satellite system (GNSS). The GNSS is a general term for satellite positioning systems such as a global positioning system (GPS), a global navigation satellite system (GLONASS), Galileo, and a quasi-zenith satellite system. Here, the quasi-zenith satellite system is a quasi-zenith satellite system (QZSS). With this sensor, the autonomous mobile device 10 can receive a GNSS signal, measure its own position in space, and obtain position information. The sensor included in the sensor group 11 may be an optical camera or a stereo camera, and the position and attitude of the autonomous mobile device may be obtained by processing the image acquired by the camera. The sensor included in the sensor group 11 may be an acoustic wave sensor, an ultrasonic sensor, a 2D-LiDAR, a 3D-LiDAR, or the like. In this case, the autonomous mobile device 10 can process waveform data and point cloud data obtained from a sensing device using sound waves or lasers to obtain its own position, attitude, surrounding conditions, etc., and the information obtained in this way can also be called sensor information. Therefore, such sensors that require information processing can also be included in the sensor group 11.

[0069] In addition, the operating mode of the autonomous mobile device 10 (mode On / Off, current mode, etc.) and the status of other mounted devices (e.g., the rotor, etc.) and log data of a control device used for autonomous control can also be considered sensor information, so devices that provide such information can also be included in the sensor group 11. Here, examples of the rotor status include valid / invalid, normal / abnormal, rotation speed, output, temperature, etc.

[0070] In the case of a general autonomous mobile device rather than this embodiment, the sensing results collected by the sensor group 11 are input to a mobile control unit 17, which will be described next, and simple autonomous control is performed, but the autonomous mobile device 10 of this embodiment is equipped with each unit 12 to 16, 18.

[0071] (Movement control unit 17) The movement control unit 17 is a control unit that controls the movement of the autonomous mobile device 10, and can also be called a drive control unit. The movement control unit 17 is a part that corresponds to the brain that controls the actuators such as rotors mounted on the autonomous mobile device 10 based on the sensing information provided by the sensor group 11, and performs autonomous control. Since the autonomous mobile device 10 performs autonomous control so as to move along the route, coordinates, attitude, and speed provided at the start of operation, even if the position is slightly shifted due to control errors or disturbances such as wind, the actuators can be controlled based on the sensing information to maintain the original position, attitude, and speed by itself. This is a function that is generally provided in autonomous mobile devices. In addition, in general autonomous mobile devices, for simple abnormalities that are easy to detect, such as the stop of some rotors, loss of communication with the ground, and large deviation from the route, a minimum response function may be built in, such as moving to predetermined coordinates, lowering the altitude and making an emergency landing on the spot. However, the autonomous mobile device 10 according to this embodiment enables control as desired by the operator even in situations where this response function is excessively executed.

[0072] (State Estimation Unit 12) The state estimation unit 12 will be described with reference to Fig. 4 to Fig. 10. Fig. 4 is a block diagram showing an example of the configuration of the state estimation unit 12 in the autonomous mobile device 10 of Fig. 3.

[0073] The state estimation unit 12 is a functional unit that converts a combination of various sensor information sent from the sensor group 11 into a name expressing a state that is easy for people to understand and its accuracy. As in the first embodiment, the name expressing this state is called a state label.

[0074] 4, the state estimation unit 12 can be composed of an estimation unit 121 and a state model 122. The state model 122 can be constructed as shown in the following example. The creation procedure of the state model 122 differs depending on whether the correct label is unknown or known.

[0075] An example of learning data and states used in generating the state model 122 will be described with reference to Fig. 5 and Fig. 6. Fig. 5 is a diagram showing an example of learning data used in generating the state model 122, and Fig. 6 is a diagram showing an example of classification of states.

[0076] Drone sensor data includes acceleration, gyro, magnetism, etc. For convenience of explanation, the raw data acquired from the drone is called raw sensor data. It is desired that the state model 122 is one that can classify the state well based on the raw sensor data. This part of "wanting to classify the state well" is called the task for explanation. In order to improve the accuracy of the task, the analyst can process the sensor data 40 in addition to using the sensor data 40 as it is, as shown in FIG. 5. For convenience of explanation, this is called processed sensor data 41. The raw sensor data 40 and the processed sensor data 41 can each be called a feature amount (feature vector). The state model 122 is constructed as a model for classifying the state. For this reason, it is necessary to know in which state the sensor data acquired in advance is data, and this is called a correct answer label. Data that combines the correct answer label and the feature amount is called training data 42.

[0077] In Fig. 6, an example is given in which two feature quantities are used to classify each state into a normal state, a state where the vehicle is not moving forward, and a state where the propeller is stopped. The feature vectors indicated by circles in Fig. 6 indicate the correct labels for each state. Note that, for the sake of simplicity, two feature quantities are given here for explanation, but the raw sensor data and processed sensor data for each output value of each sensor can also be used as feature quantities.

[0078] An example of generating the state model 122 will be described in more detail with reference to Fig. 7 and Fig. 8. Fig. 7 is a conceptual diagram for explaining an example of a procedure for generating the state model 122 when the correct label is unknown, and Fig. 8 is a conceptual diagram for explaining an example of a procedure for generating the state model 122 when the correct label is known.

[0079] When the correct label is known, the correct label is given as an answer and the model is trained. This is called supervised learning, and a model predicting a state is generated by combining a teacher (correct label) and a feature quantity and performing machine learning. The learning algorithm (learning device) can be determined arbitrarily by the analyst. The learning result is output as a state model 122. If the learning device or parameters are changed and trained, the same number of state models 122 can be generated. The analyst can select and use the best model from the multiple state models generated. Alternatively, the state estimation unit 12 can perform state estimation by taking a majority vote among the multiple state models 122 generated. When generating the state model 122, as described above, the method of selecting the algorithm or its parameters, the method of selecting from the multiple state models, and the like are not important, and the same applies to the case of training other models described below.

[0080] When the correct labels are unknown, they must first be created. Clustering is one approach to this. Clustering is the process of grouping data that is close to each other into a single cluster. A correct label is then attached to indicate the state of the data in each cluster created in this way. This correct label can be attached by an expert, or it can be attached using a small amount of data with correct labels collected in advance. The data with the correct labels created in this way becomes raw sensor data, and training data is created based on this data. The procedure from this point on is the same as when the correct labels are known.

[0081] The estimation unit 121 will be described with reference to Fig. 9 and Fig. 10. Fig. 9 is a schematic diagram for explaining an example of state estimation, and Fig. 10 is a diagram showing an example of data. The estimation unit 121 can be constructed as follows.

[0082] For convenience of explanation, the sensor data of the drone currently flying is called current data. In the same manner as when the raw sensor data was processed to create the state model 122, the current data is also processed (converted into feature quantities) and input. As shown in FIG. 9, the estimation unit 121 receives the feature quantities of the current data and the state model 122 as inputs and obtains an estimation result of the state as an output. A trained model for estimation can be used for this estimation, and various inference algorithms (classification algorithms) and parameters can be used as the method of this model. Also, as described above, the state model 122 can be created by changing and learning the learning device and its parameters, and the number of state models 122 is the same as the number of state models 122. The estimation unit 121 can obtain the result of state estimation by adopting the best model out of the multiple state models 122 as the state model 122 to be input to the estimation model. Alternatively, the estimation unit 121 can obtain the result of state estimation by taking a majority vote based on the result of estimation using the estimation model.

[0083] In the estimation unit 121, since the state is represented as multiple values ​​rather than binary, the class into which the current data is classified is output as a probability. For example, as shown in Fig. 10, when there are seven samples of current data, the estimation unit 121 outputs the probability that each sample is a normal class (0), a probability that each sample is a no-movement class (1), and a probability that each sample is a propeller-stopped class (2). The class (state) into which each sample is classified is determined by using the class with the largest probability value.

[0084] (Status abnormality determination unit 13) The abnormal status determination unit 13 will be described with reference to Fig. 11 to Fig. 13. Fig. 11 is a block diagram showing an example of the configuration of the abnormal status determination unit 13.

[0085] The abnormal state determination unit 13 is a functional unit that detects the necessity of countermeasures or testing when it can be determined immediately by looking at only one state label issued by the state estimation unit 12, and requests countermeasures or testing to the countermeasure decision unit 16 and the test implementation unit 15. In this embodiment, a case where it is determined that a countermeasure or testing is necessary is called an abnormality, which means a state in which control different from normal autonomous driving operation is required, and this is the same in the first embodiment.

[0086] As shown in Fig. 11, the abnormal status determination unit 13 can be composed of a determination unit 131 and an abnormal status rule 132. The abnormal status rule 132 can be determined, for example, as shown in Fig. 12. Fig. 12 is a diagram showing an example of the abnormal status rule 132.

[0087] The abnormal state rule 132 is a rule for determining that each state (each state label) in the state estimation unit 12 is abnormal. This rule can be determined by a person and can be input and registered in advance before the autonomous mobile device 10 starts to operate. The abnormal state rule 132 defines conditions for determining that a state label notified from the state estimation unit 12 is abnormal (a state that requires control different from normal autonomous operation), and an abnormality label that indicates the type of abnormality when the condition is met. The abnormality label can also include a label (called a warning label) that causes the test execution unit 15, which will be described later, to perform a test.

[0088] In the example of Figure 12, for the no forward movement state label, "speeding faster than the speed limit" is used as an example of a condition rule, and "headwind abnormality" is used as an example of an abnormality label that is output as a result. Also, for the propeller stopped state label, "propeller motor not working" is used as an example of a condition rule, and "propeller failure abnormality" is used as an example of an abnormality label that is output as a result.

[0089] In the example of FIG. 12, an example is given in which data is described in a JSON (JavaScript Object Notation) format, but as long as necessary items are present and their relationships can be described, any data format is acceptable. For example, CSV (Comma Separated Value) format, XML (Extensible Markup Language), or a format in which data is stored in a table format in an RDB (Relational Database) may be used. The storage location may also be in a working memory or in a storage section of a database. Anomaly labels can be defined as names for informing the countermeasure decision unit 16 of the type of anomaly, and can also be defined to request the test implementation unit 15 to perform processing.

[0090] The judgment unit 131 judges whether or not there is an abnormality according to the rules defined in the abnormal status rules 132, and issues an instruction to the test execution unit 15 or issues an instruction to the countermeasure decision unit 16. If none of the abnormal status rules apply, nothing is done.

[0091] The abnormality determination in the determination unit 131 will be described with reference to Fig. 13. Fig. 13 is a diagram showing an example of the abnormality determination. In the example of Fig. 13, the case of a headwind and the case of a propeller failure are given.

[0092] The determination unit 131 determines that there is a headwind warning when, for example, the vehicle is moving forward, there is swaying, and the speed is high. For example, it determines that the vehicle is moving forward when the forward propulsive force applied is 40% or more of the maximum value, and determines that there is swaying when the left-right or up-down swaying is 60% or more of a threshold value. It determines that the speed is also high when the speed is greater than a threshold value. When these conditions overlap, the determination unit 131 determines that there is a headwind warning, and outputs a warning label indicating this. The determination unit 131 then notifies the test execution unit 15, which will be described later, of this warning label, and when the test execution unit 15 stops the forward propulsive force and performs a test, and as a result, it determines that there is swaying when the swaying reaches 100%, and in that case outputs an abnormality label indicating an abnormality in the headwind.

[0093] The determination unit 131 determines that there is a propeller failure warning when, for example, the vehicle is moving forward, there is shaking, or there is rotation. For example, it determines that the vehicle is moving forward when the forward propulsive force is 40% or more of the maximum value, determines that there is shaking when the left-right or up-down shaking is 30% or more of a threshold value, and determines that there is rotation when any of the roll, pitch, and yaw is 30% or more of a threshold value. When these conditions overlap, the determination unit 131 determines that there is a propeller failure warning and outputs a warning label indicating this. The determination unit 131 then notifies the test execution unit 15, which will be described later, of this warning label, and when the test execution unit 15 references the history and the result exceeds a threshold value, it outputs an abnormality label indicating a propeller failure, assuming that the propeller has deteriorated. Examples of the history referenced here include the number of warnings in the last few seconds, abnormal rotation in the last few seconds, and past collision history.

[0094] (Status history abnormality determination unit 14) The status abnormality determination unit 13 will be described with reference to Fig. 14 to Fig. 17. Fig. 14 is a block diagram showing an example of the configuration of the status history abnormality determination unit 14, and Fig. 15 is a diagram showing an example of a status history abnormality rule.

[0095] The state history abnormality determination unit 14 executes a determination as to whether or not the state is in an abnormal state by referring to the state history, which is the history (history information indicating the background and history) of the state label estimated by the state estimation unit 12. The state history abnormality determination unit 14 can also be called a countermeasure state history abnormality determination unit or a history abnormality determination unit. The state history referred to here desirably includes state history of other autonomous mobile devices of the same type.

[0096] Specifically, the state history anomaly judgment unit 14 is a functional unit that looks at multiple state labels issued by the state estimation unit 12 and, when it determines that countermeasures or testing are necessary, detects this and requests countermeasures decision unit 16 and test implementation unit 15 to take countermeasures or tests.

[0097] The cases where multiple state labels are examined and detection is performed and it is determined that countermeasures or testing are necessary (although this is referred to as an abnormality here, this means that the state requires control different from normal autonomous operation) can refer to the following cases. For example, there are cases where the detection condition is the order in which state A occurs and then state B occurs within a certain period of time, and cases where the detection condition is a combination of state A and state B occurring in any order within a certain period of time. There are also cases where the detection condition is the number of times or frequency that state A occurs a certain number of times within a certain period of time, and cases where these three types of cases are combined. Note that the explanation has been given using two states, state A and state B, but there is no intention to limit the relationship to two states, and more than two states are acceptable.

[0098] As shown in Fig. 14, the state history anomaly determination unit 14 can be composed of a determination unit 141, a state history 142, and a state history anomaly rule 143. The state history 142 receives the state label issued by the state estimation unit 12 via the determination unit 141, and holds and stores it together with a timestamp. The state history anomaly rule 143 can be determined, for example, as illustrated in Fig. 15. Fig. 15 is a diagram showing an example of the state history anomaly rule 143. Note that, in the example of Fig. 15, the data format does not matter as in the example of Fig. 12, and the same applies to the examples of Figs. 17 and 19 described below.

[0099] The state history abnormality rule 143 is a rule for determining an abnormality from the state history for each state (each state label) in the state estimation unit 12. This rule can be determined by a person and can be input and registered in advance before the autonomous mobile device 10 starts operation. The state history abnormality rule 143 defines the conditions under which a plurality of state labels notified a plurality of times from the state estimation unit 12 should be determined to be abnormal (a state that requires control different from normal autonomous operation). The state history abnormality rule 143 further defines an abnormality label that indicates the type of abnormality when the condition is met. This abnormality label can also include a label (called a warning label or a test label) that causes the test implementation unit 15, which will be described later, to perform a test.

[0100] In the example of FIG. 15, for the state label of not moving forward, an example of a condition rule is adopted that it occurs "10 times or more" or that it occurs "5 times or more in 10 seconds", and in either case, an example of an abnormal label output as a result is adopted that "headwind abnormality". For the state label of not moving forward, if the condition other than those conditions is met and it cannot be determined, an example of an abnormal label is adopted that a "stop test" is requested to the test execution unit 15. Also, for the propeller stop state label, an example of a condition rule is adopted that "two propellers are stopped" or that it occurs "2 times or more in 10 seconds", and in either case, an example of an abnormal label output as a result is adopted that a "propeller failure abnormality" is requested to the test execution unit 15. For the propeller stop state label, if the condition other than those conditions is met and it cannot be determined, an example of an abnormal label is adopted that a "rotation test" is requested to the test execution unit 15. Also, for the state label of being swept to the right, an example of a condition rule is adopted that "was swept to the left last time", and an example of an abnormal label output as a result is adopted that a "propeller failure abnormality (wandering abnormality)" is used.

[0101] In this way, the condition for determining whether countermeasures or testing are necessary by looking at multiple status labels is defined in the status history anomaly rule 143. The output anomaly label can be defined as a name for informing the countermeasure decision unit 16 of the type of anomaly, and can also be defined to request processing from the test execution unit 15 (send a test instruction).

[0102] The judgment unit 141 judges whether or not the state history anomaly rule 143 is satisfied, and if it is satisfied, that is, if the above condition (detection condition) is met, it outputs an anomaly label. That is, the judgment unit 141 judges whether or not there is an anomaly according to the rules defined in the state history anomaly rules 143, and issues an instruction to the test execution unit 15 or issues an instruction to the countermeasure decision unit 16. If none of the state history anomaly rules 143 is satisfied, nothing is done.

[0103] (Testing Division 15) The test execution unit 15 will be described with reference to Fig. 16 and Fig. 17. Fig. 16 is a block diagram showing an example of the configuration of the test execution unit 15, and Fig. 17 is a diagram showing an example of test rules.

[0104] The test execution unit 15 is a functional unit that receives a test request and performs a test operation and a test to determine whether or not there is an abnormality when the abnormal status determination unit 13 or the abnormal status history determination unit 14 determines that it is necessary to perform a test operation and then determine whether or not there is an abnormality. Therefore, the output destination of the determination result of the abnormal status determination unit 13 includes the second abnormality determination unit 1d (here, the test execution unit 15).

[0105] When the determination result by the abnormal state determination unit 13 indicates an abnormal state, the test execution unit 15 executes a test to change and operate the control of the autonomous mobile device 10 on a trial basis. The test execution unit 15 executes a determination as to whether or not there is a historical abnormal state based on the determination result by the abnormal state determination unit 13 before the test and the determination result by the abnormal state determination unit 13 after the test.

[0106] Alternatively, when the judgment result by the abnormal state judgment unit 13 indicates an abnormal state or when the judgment result by the abnormal state judgment unit 14 indicates an abnormal history state, the test execution unit 15 executes a test to change and operate the control of the autonomous mobile device 10 on a trial basis. In this case, the test execution unit 15 executes a final judgment as to whether or not it is an abnormal history state, based on the judgment results by the abnormal state judgment unit 13 or the abnormal state history judgment unit 14 before and after the test.

[0107] In this case, it is preferable to use a label. That is, it is preferable for the abnormal status determination unit 13 to determine whether the current status corresponds to a plurality of countermeasure labels predetermined for each type of countermeasure for changing the current status or a normal label that does not require any countermeasure. These plurality of countermeasure labels include one or more types of test labels indicating that a test needs to be performed. When the abnormal status determination unit 13 indicates a result that corresponds to one or more test labels, the test execution unit 15 executes a test corresponding to the test label indicated by the determination result.

[0108] 16, the test execution unit 15 can be composed of an execution unit 151, test rules (test execution abnormality rules) 152, test history 153, and result determination unit 154. The execution unit 151 accepts a test request (or a label corresponding thereto) from the status abnormality determination unit 13 and the status history abnormality determination unit 14, and makes a test operation request to the mobile control unit 17 based on the test rules 152. The execution unit 151 also registers a record of the test operation request in the test history 153 together with the request. The test history 153 holds a history (history of test labels) of the test operation request made by the execution unit 151 to the mobile control unit 17 based on the test rules 152, with a timestamp.

[0109] The test rules 152 are rules for determining an abnormality based on a test performed for each state (for each state label) of the state estimation unit 12 or for each warning label in the state history abnormality determination unit 14. This rule can be determined manually and can be input and registered in advance before the autonomous mobile device 10 is operated. The test rules 152 define test labels, which are the names of tests to be performed, for multiple state labels notified from the state estimation unit 12 and warning labels notified from the state history abnormality determination unit 14. Furthermore, the test rules 152 define, for each test label, a test operation, an expected operation for the test operation, and, when the operation is detected, whether it is normal or abnormal, and, if it is abnormal, the abnormality label. The abnormality label here is a label that indicates the type of abnormality if it corresponds to an abnormality.

[0110] In the example of Figure 17, for the test label showing test number 1 (headwind), an example of a condition rule is "If stopped, it will be swept backwards," and an example of an abnormal label output as a result of this condition being satisfied is "headwind abnormality." Also, for the test label showing test number 2 (propeller failure), an example of a condition rule is "If rotated, it will not rotate," and an example of an abnormal label output as a result of this condition being satisfied is "propeller failure abnormality."

[0111] The result determination unit 154 determines the state resulting from the test operation performed in response to the test operation request, and outputs the determination result to the countermeasure decision unit 16, while also registering the determination result in the test history 153. Specifically, the result determination unit 154 reads out the test rules 152 and compares the labels using the state label received from the state estimation unit 12, the warning label received from the state history anomaly determination unit 14, and the test history in the test history 153 as keys. The result determination unit 154 checks the test result in this way, determines whether it is abnormal or normal, and if it determines it to be abnormal, it makes a countermeasure request to the countermeasure decision unit 16 and transmits an abnormal label. If it determines it to be normal, it does nothing.

[0112] In this embodiment, as explained in embodiment 1, the rules can be written succinctly, so that countermeasures for a certain state and their judgment conditions can be defined succinctly, and these countermeasures can include carrying out tests. By carrying out tests, it is possible to isolate abnormalities and their causes, and thus it is possible to distinguish between true abnormalities that require the operation to be stopped and states that are similar to abnormalities but in which operation can be continued, thereby reducing overdetection of abnormalities.

[0113] In this manner, in this embodiment, a human can easily define a test operation method and a method for determining the operation test result for a state label that represents a state of the autonomous mobile device 10 that is easy for a human to understand, and the operation of the autonomous mobile device 10 can be changed based on the operation test result. This allows a more precise determination to be made based on the results of the test as to whether an abnormality should be determined and a countermeasure should be taken, or whether nothing needs to be done. In other words, according to this embodiment, excessive preventive measures can be reduced, contributing to the stability of operation. In addition, by taking the test into consideration, even in a situation where an abnormality is excessively falsely detected and an immediate emergency action is taken, this is no longer necessary, and the availability of the operation of the autonomous mobile device 10 can be improved.

[0114] (Countermeasures Decision-Making Division 16) The countermeasure decision unit 16 will be described with reference to Fig. 18 and Fig. 19. Fig. 18 is a block diagram showing an example of the configuration of the countermeasure decision unit 16, and Fig. 19 is a diagram showing an example of countermeasure rules.

[0115] The countermeasure decision unit 16 receives an abnormal label as an abnormality notification from the status abnormality determination unit 13, the status history abnormality determination unit 14, and the test implementation unit 15, determines a countermeasure by checking the received abnormal label against the countermeasure rule, and issues a countermeasure request to the movement control unit 17. As shown in Fig. 18, the countermeasure decision unit 16 can be composed of a decision unit 161 and countermeasure rules 162 which are the above-mentioned countermeasure rules.

[0116] The countermeasure rule 162 is a rule that specifies which countermeasure should be taken for each abnormality (each abnormal label). This rule can be determined by a person, and can be input and registered in advance before the autonomous mobile device 10 starts operating.

[0117] In the example of FIG. 19, for the headwind anomaly label, an example of a condition rule is "not reported to air traffic control in the past," and the result of this condition being met is "report that the wind is strong to air traffic control." For the headwind anomaly label, an example of a condition rule is "receive a route change order from air traffic control," and the result of this condition being met is "fly in a place where the wind is not blowing." For the headwind anomaly label, if the condition is not met and a judgment cannot be made, "land." For the propeller failure anomaly label, an example of a condition rule is "is about to crash," and the result of this condition being met is "emergency landing." For the propeller failure anomaly label, an example of a condition rule is "can fly for a little while," and the result of this condition being met is "land at a nearby landing point." For the propeller failure anomaly label, an example of a condition rule is "can fly for a little while," and the result of this condition being met is "land at a nearby landing point." For the propeller failure anomaly label, if the condition is not met and a judgment cannot be made, "open the parachute."

[0118] The decision unit 161 refers to the countermeasure rule 162 to decide a countermeasure for the received abnormal label, and instructs the movement control unit 17. The countermeasure decision unit 16 is an example of the output unit 1e, and as in this example, can include the movement control unit 17 as its output destination. In particular, the decision unit 161 of the countermeasure decision unit 16 can be configured to output a control command corresponding to the necessary countermeasure to the movement control unit 17 as a determination result. Of course, the movement control unit 17 may interpret the determination result from the decision unit 161 and execute the corresponding control.

[0119] (Standard setting section 18) The criterion setting unit 18 sets the judgment criteria for the state history abnormality judgment unit 14 and the test execution unit 15. The criterion setting unit 18 can be an operation unit (operation receiving unit) provided on the main body of the autonomous mobile device 1, a receiving unit that receives a setting signal so that the criterion can be set by communication from an external device, or both. This allows the operator to set the above-mentioned state history abnormality rules 143 and test rules 152, and the output of processing results in accordance with them.

[0120] The autonomous mobile device 10 may also include a setting unit that sets the judgment criteria of the abnormal status judgment unit 13, and here, an example is given in which the setting unit is provided in the criteria setting unit 18. This allows the operator to set the above-mentioned abnormal status rules 132 and the output of the processing results in accordance with them. The autonomous mobile device 10 preferably also includes a countermeasure setting unit that sets a control command in accordance with the necessary countermeasure output by the countermeasure decision unit 16. This countermeasure setting unit may be provided as a part of the criteria setting unit 18. This allows the operator to set the above-mentioned countermeasure rules 162 and the output (command output) of the processing results in accordance with them.

[0121] Moreover, the state model 122 of the state estimation unit 12, the state abnormality rule 132 of the state abnormality determination unit 13, the state history abnormality rule 143 of the state history abnormality determination unit 14, the test rule 152 of the test implementation unit 15, and the countermeasure rule 162 of the countermeasure decision unit 16 can be written as follows: That is, a setting file or the like can be created in advance and written in advance from the standard setting unit 18 as rules for each unit of the autonomous mobile device 10, which makes it even easier to update the settings.

[0122] (Operation Example of Autonomous Mobile Device 10) An example of the internal operation of the autonomous mobile device 10 will be described with reference to Fig. 20 to Fig. 25. Fig. 20 is a flow diagram for explaining an example of the operation of the autonomous mobile device 10, and the entire sequence is expressed here in the notation of UML (Unified Modeling Language) 2.0. Fig. 20 shows the flow of a series of operations that are repeatedly performed each time sensor information (detection results from the sensor group 11) occurs during operation of the autonomous mobile device 10, or each time the sensor information is notified from the sensor group 11 to other functional units at regular intervals, etc.

[0123] In the autonomous mobile device 10, every time the sensor group 11 notifies the sensor information (step S11), the state estimation unit 12 performs state estimation, and the result is notified to one or more of the state abnormality determination unit 13, the state history abnormality determination unit 14, and the test execution unit 15 (step S12). Then, any one of the state abnormality determination unit 13, the state history abnormality determination unit 14, and the test execution unit 15 that has received the notification can execute an abnormality determination.

[0124] If the autonomous mobile device 10 determines that an abnormality has occurred as a result of the determination, the autonomous mobile device 10 transmits an abnormality notification to the countermeasure decision unit 16 (steps S13, S16, S18). The countermeasure decision unit 16 receives the abnormality notification, decides a countermeasure, and causes the autonomous mobile device 10 to perform the countermeasure operation via the mobile control unit 17 (step S14), thereby performing a countermeasure operation against the abnormal operation.

[0125] Furthermore, if it is determined that there is an abnormality (warning) that requires a test as a countermeasure, the status abnormality determination unit 13 transmits a test request to the test execution unit 15 (step S15). If it is determined that there is an abnormality (warning) that requires a test as a countermeasure, the status history abnormality determination unit 14 transmits a test request to the test execution unit 15 (step S110). The test execution unit 15 receives the test request and transmits a test operation request to the movement control unit 17 (step S113), and then the abnormality determination is performed.

[0126] Next, an example of the internal operation of the state estimation unit 12 will be described with reference to Fig. 21. Fig. 21 is a flow diagram for explaining an example of the operation of the state estimation unit 12 in Fig. 4. Prior to this series of processes, a state model that has been learned with each state as a correct label has already been stored in the state model.

[0127] The estimation unit 121 receives a sensor information notification (step S11) from the sensor group 11 (step S21), estimates a state based on the received sensor information notification against the state model 122, and obtains state label information (step S22). When the estimation unit 121 receives a state notification (step S12), it transmits the state label information to the state abnormality determination unit 13, the state history abnormality determination unit 14, and the test implementation unit 15 (step S23), and ends the process.

[0128] An example of the internal operation of the abnormal condition determination unit 13 will be described with reference to Fig. 22. Fig. 22 is a flow chart for explaining an example of the operation of the abnormal condition determination unit 13 in Fig. 11 .

[0129] The judgment unit 131 receives a state notification (step S12) from the state estimation unit 12 and receives state label information (step S31). The judgment unit 131 checks the state label information against the state abnormality rule 132, and if there is a match, obtains an abnormal label or a test label (step S32). Here, if there is no match, nothing is obtained.

[0130] Next, the judgment unit 131 judges whether a label has been obtained (step S33), and if NO, ends the process, and if YES, judges the type of the obtained label (step S34). In step S34, it is judged whether the label is an abnormal label or a test label. If an abnormal label has been obtained, the judgment unit 131 notifies the countermeasure decision unit 16 of the abnormal label in an abnormality notification (step S13) (step S35) and ends the process. If a test label has been obtained, the judgment unit 131 notifies the test implementation unit 15 of the test label in a test request (step S15) (step S36) and ends the process.

[0131] An example of the internal operation of the state history abnormality determination unit 14 will be described with reference to Fig. 23. Fig. 23 is a flow chart for explaining an example of the operation of the state history abnormality determination unit 14 in Fig. 14 .

[0132] The determination unit 141 receives a sensor information notification (step S11) from the sensor group 11 (step S41), and adds the received sensor information notification to the state history 142 (step S42). Next, the determination unit 141 reads out the history of the sensor information notification from the state history 142, and obtains an abnormal label or a test label if there is a match in accordance with the rule read out from the state history anomaly rule 143 (step S43). Here, if there is no match, nothing is obtained.

[0133] Next, the judgment unit 141 judges whether a label has been obtained (step S44), and if NO, ends the process, and if YES, judges the type of the obtained label (step S45). In step S45, it is judged whether the label is an abnormal label or a test label. If an abnormal label has been obtained, the judgment unit 141 notifies the countermeasure decision unit 16 of the abnormal label in the abnormality notification (step S13) (step S46) and ends the process. The judgment unit 141 notifies the test implementation unit 15 of the test label in a test request (step S15) (step S47) and ends the process.

[0134] An example of the internal operation of the test execution unit 15 will be described with reference to Fig. 24. Fig. 24 is a flow diagram for explaining an example of the operation of the test execution unit 15 of Fig. 16.

[0135] The implementation unit 151 receives a test request from the state abnormality determination unit 13 (step S15) or a test request from the state history abnormality determination unit 14 (step S110) (step S51). Next, the implementation unit 151 refers to the test rules 152 and reads out a test rule corresponding to the test request from the test rules 152 (step S52). The implementation unit 151 requests the mobility control unit 17 to perform a test operation of the autonomous mobile device 10 according to the read test rule by a test operation request (step S113), and registers the test operation in the test history 153 (step S53).

[0136] Next, the result determination unit 154 receives a state notification (step S12) from the state estimation unit 12 (step S54). In step S54, the operation of the autonomous mobile device 10 in response to the test request is received. Next, the result determination unit 154 compares the received state notification (step S12) with the pair of the test history 153 and the test rule 152 (step S55) and determines whether the test result is in accordance with the test rule (normal) or not (step S56). If the result is YES in step S56, the result determination unit 154 ends the process. If the result is NO in step S56, that is, if the test result is different from the test rule (abnormal), the result determination unit 154 notifies the countermeasure decision unit 16 of an abnormality label in the abnormality notification (step S16) (step S57) and ends the process.

[0137] An example of the internal operation of the countermeasure decision unit 16 will be described with reference to Fig. 25. Fig. 25 is a flow diagram for explaining an example of the operation of the countermeasure decision unit 16.

[0138] The decision unit 161 receives any one of the abnormality notification from the test execution unit 15 (step S16), the abnormality notification from the state abnormality determination unit 13 (step S13), and the abnormality notification from the state history abnormality determination unit 14 (step S18) (step S61). Next, the decision unit 161 determines a countermeasure based on the received abnormality notification (steps S16, S13, S18) against the countermeasure rule 162 (step S62). Then, the decision unit 161 transmits a countermeasure request (steps S14, S17) to the movement control unit 17, and the movement control unit 17 issues a control instruction for a countermeasure operation to the autonomous mobile device 10 (step S63), and the process ends.

[0139] As described above, according to this embodiment, similar to the first embodiment, it is possible to detect an abnormal state that requires countermeasures under a judgment condition desired by the operator of the autonomous mobile device 10, and also to obtain the effects of the components individually provided. For example, by providing at least one of the state history abnormality judgment unit 14 and the test implementation unit 15, it is possible to improve the accuracy of detection of an abnormal state and increase the availability of the autonomous mobile device 10.

[0140] In this embodiment, it is possible to maintain a certain level of accuracy in detecting abnormal states even when, for example, the autonomous mobile device 10 is configured without any of the test implementation unit 15, the state abnormality determination unit 13, and the state history abnormality determination unit 14. Even in this case, the advantage that the operator can easily define abnormality determination criteria and countermeasures by the criteria setting unit 18 is maintained.

[0141] Also, any one or more of the status abnormality rule 132, the status history abnormality rule 143, and the countermeasure rule 162 may be replaced with a status abnormality model, a status history abnormality model, and a countermeasure rule model, respectively. Any of the models can be generated, for example, by using the same method as the generation method of the status model 122. In this case, since the operator needs to make the model learn instead of creating each definition file, it is preferable to make at least one rule easily settable from the standard setting unit 18.

[0142] <Embodiment 3> The third embodiment will be described with reference to Fig. 26 and Fig. 27, focusing on the differences from the second embodiment. The third embodiment can appropriately use various examples described in the first and second embodiments. Fig. 26 is a block diagram showing an example of a configuration of an autonomous mobile system according to the third embodiment, and Fig. 27 is a block diagram showing another example of a configuration of an autonomous mobile system according to the third embodiment.

[0143] The autonomous mobile system according to this embodiment shown in Fig. 26 includes an autonomous mobile device 10, and a server device 50 connected thereto for managing the operation of the autonomous mobile device 10. The server device 50 can have a management unit (operation management unit) 51 for managing the operation of one or more autonomous mobile devices 10. The management method used by the operation management unit 51 does not matter as long as it can manage the operation of the autonomous mobile device 10. The operation management unit 51 manages the operation of multiple autonomous mobile devices 10, for example, in the same way as air traffic control in an aircraft.

[0144] The server device 50 can also include a standard setting unit 18. This allows the state model 122 of the state estimation unit 12, the state abnormality rule 132 of the state abnormality determination unit 13, the state history abnormality rule 143 of the state history abnormality determination unit 14, the test rule 152 of the test implementation unit 15, and the countermeasure rule 162 of the countermeasure decision unit 16 to be registered from the server device 50. These can also be rewritten from the server device 50. In this way, the standard setting unit 18 can be provided in the server device 50, and the server device 50 can be configured to add, delete, or edit definitions from, for example, a user-oriented setting screen. In this case, the standard setting unit 18 on the autonomous mobile device 10 side can also be omitted.

[0145] The operation management unit 51 can be configured as a control unit (not shown) that controls the entire server device 50 and a communication unit (not shown) that communicates with the autonomous mobile device 10. This control unit can be realized, for example, by a CPU, a working memory, and a non-volatile storage device that stores a program. This program can be a operation management program that executes processing for realizing the functions of the operation management unit 51 in cooperation with the communication unit. In addition, this control unit can also be realized, for example, by an integrated circuit.

[0146] The autonomous mobile system according to this embodiment shown in Fig. 27 includes an autonomous mobile device 10a and a server device 50a connected thereto for managing the operation of the autonomous mobile device 10a. The server device 50a can have an operation management unit 51 for managing the operation of one or more autonomous mobile devices 10a. The method of management by the operation management unit 51 does not matter as long as it can manage the operation of the autonomous mobile device 10a.

[0147] The server device 50a also includes a state estimation unit 12, a state abnormality determination unit 13, a state history abnormality determination unit 14, a test implementation unit 15, and a countermeasure decision unit 16, and the autonomous mobile device 10a has these functions removed from the autonomous mobile device 10. That is, the autonomous mobile device 10a includes a sensor group 11 and a mobile control unit 17, and executes processes such as transmitting information from the sensor group 11 and receiving command signals to the mobile control unit 17 while communicating with the server device 50a through a communication unit (not shown). Therefore, the server device 50a does not include a sensor group, but includes a receiving unit (communication unit) that receives from the autonomous mobile device 10a the detection results acquired by the sensor group 11 that detects the current state of the autonomous mobile device 10a. The server device 50a can also transmit command signals to the mobile control unit 17 through this communication unit.

[0148] In addition, each unit such as the operation management unit 51, the state estimation unit 12, etc. can be configured as a control unit (not shown) that controls the entire server device 50a and a communication unit (not shown) that communicates with the autonomous mobile device 10a, similar to the case described for the server device 50.

[0149] In addition, the server device 50a has been exemplified as having the state estimation unit 12, the state abnormality determination unit 13, the state history abnormality determination unit 14, the test implementation unit 15, and the countermeasure decision unit 16, but it is also possible for the server device 50a to have only one or more of them, with the remaining components being provided on the autonomous mobile device 10a side. Also, the autonomous mobile system can be constructed as a system not including the operation management unit 51 in the configuration example of FIG. 26 or FIG. 27.

[0150] <Other embodiments> [a] In each embodiment, the functions of an autonomous mobile device, a learning device, a server device, etc. have been described, but each device is not limited to the configuration examples shown, and it is sufficient if each device can realize these functions.

[0151] [b] Each device according to each embodiment may have the following hardware configuration. Fig. 28 is a diagram showing an example of the hardware configuration of the device. The same applies to the other embodiment [a] described above.

[0152] The device 100 shown in FIG. 28 may have a processor 101, a memory 102, and an interface 103. The processor 101 may be, for example, a microprocessor, a microprocessor unit (MPU), or a CPU. The processor 101 may include a plurality of processors. The memory 102 is, for example, configured by a combination of a volatile memory and a non-volatile memory. The functions of each device described in each embodiment are realized by the processor 101 reading and executing a program stored in the memory 102. At this time, input and output of information can be performed via an interface 103 such as a communication interface that communicates with other internal parts or other external devices. For example, when the device 100 is an autonomous mobile device, the interface 103 may include at least an interface with each sensor.

[0153] In the above examples, the program includes instructions (or software code) that, when loaded into a computer, cause the computer to perform one or more functions described in the embodiments. The program may be stored in a non-transitory computer-readable medium or a tangible storage medium. By way of example and not limitation, the computer-readable medium or tangible storage medium includes random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drive (SSD) or other memory technology, CD-ROM, digital versatile disc (DVD), Blu-ray® disk or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage device. The program may be transmitted on a transitory computer-readable medium or communication medium. By way of example and not limitation, the transitory computer-readable medium or communication medium includes electrical, optical, acoustic, or other forms of propagated signals.

[0154] [c] Further, as the procedures of the processing in the autonomous mobile device or the server device are exemplified in each of the above-mentioned embodiments, the present disclosure also includes a form as an anomaly detection method in the autonomous mobile device or the server device. This anomaly detection method can include the following acquisition step, state estimation step, first anomaly determination step, second anomaly determination step, output step, and reference setting step. The acquisition step acquires a result of the current state of the autonomous mobile device detected by a group of sensors in the autonomous mobile device. The state estimation step estimates which of a plurality of predetermined state labels the current state of the autonomous mobile device corresponds to based on the detection result by the group of sensors. The first anomaly determination step determines whether or not the current state is an abnormal state that requires measures to change the state based on the estimation result in the state estimation step. The second anomaly determination step determines whether or not the state indicated by the results of the multiple estimations at different times in the state estimation step is a historical abnormal state that requires measures to change the state. The output step outputs the determination result in the first anomaly determination step and the determination result in the second anomaly determination step. The above-mentioned criterion setting step sets a criterion for the second abnormality determination step. Note that other examples are as described in the above-mentioned embodiments. In addition, the present disclosure may include a safe automatic control method for an autonomous mobile device, including such an abnormality detection method.

[0155] The program for the autonomous mobile device can be said to be a program for causing a computer (control computer) provided in the autonomous mobile device to execute the above-mentioned acquisition step, state estimation step, first abnormality determination step, second abnormality determination step, output step, and reference setting step.The program for the server device can be said to be a program for causing a computer connected to the autonomous mobile device to execute the above-mentioned acquisition step, state estimation step, first abnormality determination step, second abnormality determination step, output step, and reference setting step.

[0156] As the processing procedure in the learning device has been described in each of the above-mentioned embodiments, the present disclosure also includes a form as a learning method in a learning device. This learning method includes the following acquisition step and generation step. This acquisition step acquires detection results acquired by a group of sensors that detect the current state of the autonomous mobile device. This generation step inputs the detection results acquired in the acquisition step multiple times, and generates a trained model for outputting countermeasure labels, which outputs one of multiple countermeasure labels and a normal label. Note that other examples are as described in each of the above-mentioned embodiments. In addition, it can be said that the program related to the learning device is a program for causing a computer provided in the autonomous mobile device or connected to the autonomous mobile device to execute the above acquisition step and generation step.

[0157] The present disclosure is not limited to the above-described embodiment, and may be modified as appropriate without departing from the spirit and scope of the present disclosure. In addition, the present disclosure may be implemented by appropriately combining the respective embodiments.

[0158] A part or all of the above-described embodiments can be described as, but is not limited to, the following supplementary notes. (Appendix 1) A group of sensors that detects a current state of the autonomous mobile device; a state estimation unit that estimates which of a plurality of predetermined state labels a current state of the autonomous mobile device corresponds to based on a detection result by the group of sensors; a first abnormality determination unit that determines whether or not a current state is an abnormal state that requires a measure to change the current state, based on a result of estimation by the state estimation unit; A second anomaly determination unit that determines whether the state estimation unit corresponds to a plurality of countermeasure labels or a normal label based on the results of estimation performed a plurality of times at different times by the state estimation unit; an output unit that outputs a determination result of the first abnormality determination unit and a determination result of the second abnormality determination unit; a criterion setting unit that sets a judgment criterion for the second abnormality judgment unit; Equipped with the plurality of countermeasure labels are labels that are predetermined for each of the countermeasures to indicate that the state of the autonomous mobile device indicated by the results of the plurality of estimations is a historical abnormal state that requires a countermeasure to change the state, and the normal label is a label that does not require a countermeasure. Autonomous mobility device. (Appendix 2) The second anomaly determination unit inputs a result estimated multiple times at different times by the state estimation unit, inputs an estimation result by the state estimation unit, and performs determination using a first trained model that outputs one of the multiple countermeasure labels and the normal label. 2. An autonomous mobile device as described in claim 1. (Appendix 3) The second abnormality determination unit has a history abnormality determination unit that executes a determination as to whether or not the state is in the history abnormal state by referring to a state history that is a history of the state label estimated by the state estimation unit. 3. The autonomous mobile device according to claim 1 or 2. (Appendix 4) The state history also includes a history of state labels for other autonomous mobile devices of the same type as the autonomous mobile device. 4. The autonomous mobile device of claim 3. (Appendix 5) The output destination of the output unit regarding the determination result of the first abnormality determination unit includes the second abnormality determination unit, The second abnormality judgment unit has a test execution unit that, when the judgment result of the first abnormality judgment unit indicates the abnormal state or the judgment result of the history abnormality judgment unit indicates the history abnormal state, executes a test to change and operate the control of the autonomous mobile device on a trial basis, and executes a final judgment of whether or not the history abnormality state exists based on the judgment result of the first abnormality judgment unit or the history abnormality judgment unit before the test and the judgment result of the first abnormality judgment unit or the history abnormality judgment unit after the test. 5. The autonomous mobile device according to claim 3 or 4. (Appendix 6) The output destination of the output unit regarding the determination result of the first abnormality determination unit includes the second abnormality determination unit, The second abnormality determination unit has a test execution unit that, when the determination result by the first abnormality determination unit indicates an abnormal state, executes a test to change and operate the control of the autonomous mobile device on a trial basis, and executes a determination as to whether or not the historical abnormality state exists based on the determination result by the first abnormality determination unit before the test and the determination result by the first abnormality determination unit after the test. 5. The autonomous mobile device according to claim 1. (Appendix 7) the first abnormality determination unit determines whether the current state corresponds to a plurality of countermeasure labels predetermined for each type of countermeasure for changing the current state or a normal label indicating that no countermeasure is required; The plurality of countermeasure labels include one or more types of test labels indicating that the test needs to be performed, The test implementation unit, when the first abnormality determination unit indicates a result that corresponds to the one or more test labels, implements a test corresponding to the test label indicated by the determination result. 7. The autonomous mobile device according to claim 5 or 6. (Appendix 8) the first abnormality determination unit determines whether the current state corresponds to a plurality of countermeasure labels predetermined for each type of countermeasure for changing the current state or a normal label for which no countermeasure is required; 8. The autonomous mobile device according to claim 1, (Appendix 9) The first anomaly determination unit performs a determination using a second trained model that receives an estimation result from the state estimation unit and outputs one of the plurality of countermeasure labels and the normal label. 9. The autonomous mobile device of claim 8. (Appendix 10) A setting unit that sets a judgment criterion for the first abnormality judgment unit, 10. The autonomous mobile device according to any one of claims 1 to 9. (Appendix 11) A movement control unit that controls movement of the autonomous moving device, The output unit outputs a control command corresponding to a necessary measure to the movement control unit as a determination result. 11. The autonomous mobile device according to any one of claims 1 to 10. (Appendix 12) A countermeasure setting unit that sets a control command corresponding to the necessary countermeasure output by the output unit, 12. The autonomous mobile device of claim 11. (Appendix 13) The state estimation unit performs estimation using a third trained model that inputs the detection results by the sensor group and outputs one of the multiple state labels. 13. The autonomous mobile device according to any one of claims 1 to 12. (Appendix 14) a receiving unit that receives, from the autonomous mobile device, a detection result acquired by a group of sensors that detects a current state of the autonomous mobile device; a state estimation unit that estimates which of a plurality of predetermined state labels a current state of the autonomous mobile device corresponds to based on a detection result by the group of sensors; a first abnormality determination unit that determines whether or not a current state is an abnormal state that requires a measure to change the current state, based on a result of estimation by the state estimation unit; A second anomaly determination unit that determines whether the state estimation unit corresponds to a plurality of countermeasure labels or a normal label based on the results of estimation performed a plurality of times at different times by the state estimation unit; an output unit that outputs a determination result of the first abnormality determination unit and a determination result of the second abnormality determination unit; a criterion setting unit that sets a judgment criterion for the second abnormality judgment unit; Equipped with the plurality of countermeasure labels are labels that are predetermined for each of the countermeasures to indicate that the state of the autonomous mobile device indicated by the results of the plurality of estimations is a historical abnormal state that requires a countermeasure to change the state, and the normal label is a label that does not require a countermeasure. Server device. (Appendix 15) The autonomous mobile device further includes a management unit that manages the operation of the autonomous mobile device. 15. The server device according to claim 14. (Appendix 16) 16. An autonomous mobile system comprising: the server device according to claim 14 or 15; and an autonomous mobile device equipped with the group of sensors. (Appendix 17) 14. An autonomous mobile system comprising: an autonomous mobile device according to any one of claims 1 to 13; and a server device connected to the autonomous mobile device and managing operation of the autonomous mobile device. (Appendix 18) an acquisition unit that acquires detection results acquired by a group of sensors that detect a current state of the autonomous mobile device; A control unit; Equipped with The control unit generates a trained model for outputting countermeasure labels, the trained model inputting a plurality of detection results acquired by the acquisition unit and outputting one of a plurality of countermeasure labels and a normal label; the plurality of countermeasure labels are labels that are predetermined for each of the countermeasures to indicate that the state of the autonomous mobile device indicated by the results of the plurality of estimations is a historical abnormal state that requires a countermeasure to change the state, and the normal label is a label that does not require a countermeasure. Learning device. (Appendix 19) The control unit receives the detection result acquired by the acquisition unit, and outputs one of a plurality of predetermined state labels to indicate a current state of the autonomous mobile device, thereby generating a trained model for outputting a state label. 19. The learning device of claim 18. (Appendix 20) an acquisition step of acquiring a result of detection of a current state of the autonomous mobile device by a group of sensors in the autonomous mobile device; a state estimation step of estimating which of a plurality of predetermined state labels a current state of the autonomous mobile device corresponds to based on a detection result by the group of sensors; a first abnormality determination step of determining whether or not a current state is an abnormal state that requires a measure to change the current state based on a result of the estimation in the state estimation step; a second anomaly determination step of determining which of a plurality of countermeasure labels or a normal label the state corresponds to based on the results of estimation performed a plurality of times at different times in the state estimation step; an output step of outputting a determination result in the first abnormality determination step and a determination result in the second abnormality determination step; a criterion setting step of setting a criterion for the second abnormality determination step; Equipped with the plurality of countermeasure labels are labels that are predetermined for each of the countermeasures to indicate that the state of the autonomous mobile device indicated by the results of the plurality of estimations is a historical abnormal state that requires a countermeasure to change the state, and the normal label is a label that does not require a countermeasure. Anomaly detection methods. (Appendix 21) an acquisition step of acquiring detection results acquired by a group of sensors that detect a current state of the autonomous mobile device; A generation step of generating a trained model for outputting a countermeasure label, the trained model being configured to input a plurality of detection results acquired in the acquisition step and output one of a plurality of countermeasure labels and a normal label; Equipped with the plurality of countermeasure labels are labels that are predetermined for each of the countermeasures to indicate that the state of the autonomous mobile device indicated by the results of the plurality of estimations is a historical abnormal state that requires a countermeasure to change the state, and the normal label is a label that does not require a countermeasure. How to learn. (Appendix 22) A computer provided in the autonomous mobile device or connected to the autonomous mobile device, an acquisition step of acquiring a current state of the autonomous mobile device detected by a group of sensors in the autonomous mobile device; a state estimation step of estimating which of a plurality of predetermined state labels a current state of the autonomous mobile device corresponds to based on a detection result by the group of sensors; a first abnormality determination step of determining whether or not a current state is an abnormal state that requires a measure to change the current state based on a result of the estimation in the state estimation step; a second anomaly determination step of determining which of a plurality of countermeasure labels or a normal label the state corresponds to based on the results of estimation performed a plurality of times at different times in the state estimation step; an output step of outputting a determination result in the first abnormality determination step and a determination result in the second abnormality determination step; a criterion setting step of setting a criterion for the second abnormality determination step; Run the command, the plurality of countermeasure labels are labels that are predetermined for each of the countermeasures to indicate that the state of the autonomous mobile device indicated by the results of the plurality of estimations is a historical abnormal state that requires a countermeasure to change the state, and the normal label is a label that does not require a countermeasure. program. (Appendix 23) A computer provided in the autonomous mobile device or connected to the autonomous mobile device, an acquisition step of acquiring detection results acquired by a group of sensors that detect a current state of the autonomous mobile device; A generation step of generating a trained model for outputting a countermeasure label, the trained model being configured to input a plurality of detection results acquired in the acquisition step and output one of a plurality of countermeasure labels and a normal label; Run the command, the plurality of countermeasure labels are labels that are predetermined for each of the countermeasures to indicate that the state of the autonomous mobile device indicated by the results of the plurality of estimations is a historical abnormal state that requires a countermeasure to change the state, and the normal label is a label that does not require a countermeasure. program. [Explanation of symbols]

[0159] 1, 10, 10a Autonomous mobile device 1a, 11 Sensor group 1b, 12 State Estimation Unit 1c 1st abnormality determination section 1d 2nd abnormality determination section 1e Output section 1f, 18 Standard Setting Section 2 Learning device 2a Control section 2b Control section 2c storage section 2d untrained model 2e Pre-trained model 13. Status Abnormality Judgment Unit 14. Status history abnormality judgment section 15 Testing Department 16 Countermeasures Decision-Making Department 17 Movement control section 40 Raw Sensor Data 41 Processed sensor data 42 Training Data 50, 50a Server device 51 Operation Management Department 100 devices 101 Processor 102 Memory 103 Interface

Claims

1. A group of sensors that detects a current state of the autonomous mobile device; a state estimation unit that estimates which of a plurality of predetermined state labels a current state of the autonomous mobile device corresponds to based on a detection result by the group of sensors; a first abnormality determination unit that determines whether or not a current state is an abnormal state that requires a measure to change the current state, based on a result of estimation by the state estimation unit; a second abnormality determination unit that determines whether the state estimation unit corresponds to a plurality of countermeasure labels or a normal label based on the results of estimation performed a plurality of times at different times by the state estimation unit; an output unit that outputs a determination result of the first abnormality determination unit; a criterion setting unit that sets a judgment criterion for the second abnormality judgment unit; Equipped with the plurality of countermeasure labels are labels that are predetermined for each countermeasure to indicate that the state of the autonomous mobile device indicated by the results of the plurality of estimations is a historical abnormal state that requires a countermeasure to change the state, and the normal label is a label that does not require a countermeasure; the second abnormality determination unit has a history abnormality determination unit that executes a determination as to whether or not the state is in the history abnormal state by referring to a state history that is a history of state labels estimated by the state estimation unit, The state history also includes a history of state labels for other autonomous mobile devices of the same type as the autonomous mobile device. Autonomous mobility device.

2. An output destination of the output unit regarding the determination result of the first abnormality determination unit includes the second abnormality determination unit, The second abnormality determination unit has a test implementation unit that, when the determination result of the first abnormality determination unit indicates the abnormal state or the determination result of the history abnormality determination unit indicates the history abnormal state, executes a test to change and operate the control of the autonomous mobile device on a trial basis, and executes a final determination of whether or not the history abnormality state exists based on the determination result of the first abnormality determination unit or the history abnormality determination unit before the test and the determination result of the first abnormality determination unit or the history abnormality determination unit after the test. The autonomous mobile device according to claim 1 .

3. the first abnormality determination unit determines whether the current state corresponds to a plurality of countermeasure labels predetermined for each type of countermeasure for changing the current state or a normal label indicating that no countermeasure is required; The plurality of countermeasure labels include one or more types of test labels indicating that the test needs to be performed, the test execution unit, when the first abnormality determination unit indicates a result that corresponds to the one or more test labels, performs a test corresponding to the test label indicated by the determination result. The autonomous mobile device according to claim 2 .

4. A movement control unit that controls movement of the autonomous moving device, The output unit outputs a control command corresponding to a necessary measure to the movement control unit as a determination result. The autonomous mobile device according to any one of claims 1 to 3.

5. a receiving unit that receives, from the autonomous mobile device, a detection result acquired by a group of sensors that detects a current state of the autonomous mobile device; a state estimation unit that estimates which of a plurality of predetermined state labels a current state of the autonomous mobile device corresponds to based on a detection result by the group of sensors; a first abnormality determination unit that determines whether or not a current state is an abnormal state that requires a measure to change the current state, based on a result of estimation by the state estimation unit; a second abnormality determination unit that determines whether the state estimation unit corresponds to a plurality of countermeasure labels or a normal label based on the results of estimation performed a plurality of times at different times by the state estimation unit; an output unit that outputs a determination result of the first abnormality determination unit and a determination result of the second abnormality determination unit; a criterion setting unit that sets a judgment criterion for the second abnormality judgment unit; Equipped with the plurality of countermeasure labels are labels that are predetermined for each countermeasure to indicate that the state of the autonomous mobile device indicated by the results of the plurality of estimations is a historical abnormal state that requires a countermeasure to change the state, and the normal label is a label that does not require a countermeasure; the second abnormality determination unit has a history abnormality determination unit that executes a determination as to whether or not the state is in the history abnormal state by referring to a state history that is a history of state labels estimated by the state estimation unit, The state history also includes a history of state labels for other autonomous mobile devices of the same type as the autonomous mobile device. Server device.

6. an acquisition step of acquiring a result of detection of a current state of the autonomous mobile device by a group of sensors in the autonomous mobile device; a state estimation step of estimating which of a plurality of predetermined state labels a current state of the autonomous mobile device corresponds to based on a detection result by the group of sensors; a first abnormality determination step of determining whether or not a current state is an abnormal state that requires a measure to change the current state, based on a result of the estimation in the state estimation step; a second abnormality determination step of determining which of a plurality of countermeasure labels or a normal label the state corresponds to based on the results of estimation performed a plurality of times at different times in the state estimation step; an output step of outputting a determination result in the first abnormality determination step and a determination result in the second abnormality determination step; a criterion setting step of setting a judgment criterion for the second abnormality judgment step; Equipped with the plurality of countermeasure labels are labels that are predetermined for each countermeasure to indicate that the state of the autonomous mobile device indicated by the results of the plurality of estimations is a historical abnormal state that requires a countermeasure to change the state, and the normal label is a label that does not require a countermeasure; The second abnormality determination step includes a history abnormality determination step of determining whether or not the state is the history abnormal state by referring to a state history that is a history of the state label estimated in the state estimation step, The state history also includes a history of state labels for other autonomous mobile devices of the same type as the autonomous mobile device. Anomaly detection methods.

7. A computer provided in the autonomous mobile device or connected to the autonomous mobile device, an acquisition step of acquiring a current state of the autonomous mobile device detected by a group of sensors in the autonomous mobile device; a state estimation step of estimating which of a plurality of predetermined state labels a current state of the autonomous mobile device corresponds to based on a detection result by the group of sensors; a first abnormality determination step of determining whether or not a current state is an abnormal state that requires a measure to change the current state, based on a result of the estimation in the state estimation step; a second abnormality determination step of determining which of a plurality of countermeasure labels or a normal label the state corresponds to based on the results of estimation performed a plurality of times at different times in the state estimation step; an output step of outputting a determination result in the first abnormality determination step and a determination result in the second abnormality determination step; a criterion setting step of setting a judgment criterion for the second abnormality judgment step; Run the command, the plurality of countermeasure labels are labels that are predetermined for each countermeasure to indicate that the state of the autonomous mobile device indicated by the results of the plurality of estimations is a historical abnormal state that requires a countermeasure to change the state, and the normal label is a label that does not require a countermeasure; the second abnormality determination step includes a history abnormality determination step of determining whether or not the state is the history abnormal state by referring to a state history that is a history of the state label estimated in the state estimation step, The state history also includes a history of state labels for other autonomous mobile devices of the same type as the autonomous mobile device. program.

Images