Perception of an environment external to an autonomous system

Abstract

Disclosed is a computer-implemented method for an autonomous system, the autonomous system comprising a set of sensors and a set of actuators, the computer- implemented method comprising: receiving a first sensor input from the set of sensors, extracting from the first sensor input signal a first signal and a second signal, predicting a first prediction based on the first signal, wherein the first prediction is for predicting a state of the autonomous system after processing delay time, predicting a second prediction based on the second signal, wherein the second prediction is for predicting the external environment after the processing delay, combining the first prediction and the second prediction to obtain a combined prediction, and comparing the combined prediction to a second sensor input, which is sensor input received after the processing delay.

Description

[0001] PERCEPTION OF AN ENVIRONMENT EXTERNAL TO AN AUTONOMOUS SYSTEM

[0002] FIELD

[0003] The present disclosure relates to autonomous systems and perception of an environment external to an autonomous system.

[0004] BACKGROUND

[0005] Automation of various systems has led to systems being able to perform at least some functions without manual intervention. Such automation brings many benefits but must also be designed such that automated performance of functions avoids causing harm to the device or system itself, to the external environment, nor to the people around the device or system performing the functions in an automated manner.

[0006] BRIEF DESCRIPTION

[0007] The scope of protection sought for various embodiments is set out by the independent claims. Dependent claims define further embodiments included in the scope of protection. The embodiments and features, if any, described in this specification that do not fall under the scope of the independent claims are to be interpreted as examples useful for understanding various embodiments of the disclosure.

[0008] According to a first aspect there is provided a computer-implemented method for an autonomous system, the autonomous system comprising a set of sensors and a set of actuators, the set of actuators being for causing a change of state of the autonomous system, the computer-implemented method comprising: receiving a first sensor input from the set of sensors, extracting from the first sensor input signal a first signal, that is a signal caused by the change of state of the autonomous system, and a second signal, that is a signal caused by an external environment, predicting, by a first prediction unit, a first prediction based on the first signal, wherein the first prediction is for predicting a state of the autonomous system after processing delay time, predicting, by a second prediction unit, a second prediction based on the second signal, wherein the second prediction is for predicting the external environment after the processing delay time, combining the first prediction and the second prediction to obtain a combined prediction, comparing the combined prediction to a second sensor input, which is sensor input received after the processing delay time, and providing the result of the comparison for the first prediction unit and the second prediction unit. It is to be noted that the result of the comparison may additionally be provided to a logical unit, such as a signal splitter, that is configured to extract the first and the second signals from the sensor input. Additionally, or alternatively, the result of the comparison may further be provided to a logical unit, such as an action decision making unit, that is configured to determine functionality of the autonomous system.

[0009] In some exemplary embodiments according to the first aspect, a change in the environment external to the autonomous system is reflected in signals caused by the external environment, and captured by the set of sensors, as a part of a sensor input, and the change in the external environment may be perceived by the autonomous system. This may be achieved as over time, as the procedure according to the first aspect is repeated, a change in the environment is reflected in the sensor inputs received by the autonomous system using the set of sensors and thus the autonomous system is capable of perceiving a change in the external environment.

[0010] According to a second aspect, there is provided an autonomous system comprising a set of sensors and a set of actuators, the set of actuators being for causing a change of state of the autonomous system, the autonomous system comprising means for performing the following: receiving a first sensor input from the set of sensors, extracting from the first sensor input signal a first signal, that is a signal caused by the change of state of the autonomous system, and a second signal, that is a signal caused by an external environment, predicting, by a first prediction unit, a first prediction based on the first signal, wherein the first prediction is for predicting a state of the autonomous system after processing delay time, predicting, by a second prediction unit, a second prediction based on the second signal, wherein the second prediction is for predicting the external environment after the processing delay time, combining the first prediction and the second prediction to obtain a combined prediction, comparing the combined prediction to a second sensor input, which is sensor input received after the processing delay time, and providing the result of the comparison for the first prediction unit and the second prediction unit. It is to be noted that the result of the comparison may additionally be provided to a logical unit, such as a signal splitter, that is configured to extract the firstand the second signals from the sensor input. Additionally, or alternatively, the result of the comparison may further be provided to a logical unit, such as an action decision making unit, that is configured to determine functionality of the autonomous system. According to a third aspect, there is provided a computing system comprising at least one computing device that comprises means for performing the following: receiving a first sensor input from the set of sensors, extracting from the first sensor input signal a first signal, that is a signal caused by the change of state of the autonomous system, and a second signal, that is a signal caused by an external environment, predicting, by a first prediction unit, a first prediction based on the first signal, wherein the first prediction is for predicting a state of the autonomous system after processing delay time, predicting, by a second prediction unit, a second prediction based on the second signal, wherein the second prediction is for predicting the external environment after the processing delay time, combining the first prediction and the second prediction to obtain a combined prediction, comparing the combined prediction to a second sensor input, which is sensor input received after the processing delay time, and providing the result of the comparison for the first prediction unit and the second prediction unit. It is to be noted that the result of the comparison may additionally be provided to a logical unit, such as a signal splitter, that is configured to extract the first and the second signals from the sensor input. Additionally, or alternatively, the result of the comparison may further be provided to a logical unit, such as an action decision making unit, that is configured to determine functionality of the autonomous system.

[0011] In some examples according to the second, or third, aspect, the means comprises at least one processor; and at least one memory including computer program code which, when executed by the at least one processor, causes the performance of the computing device.

[0012] According to a fourth aspect there is provided computer program product comprising instructions, which, when executed by a computing device, cause the computing device to perform a computer-implemented method comprising at least the following: receiving a first sensor input from the set of sensors, extracting from the first sensor input signal a first signal, that is a signal caused by the change of state of the autonomous system, and a second signal, that is a signal caused by an external environment, predicting, by a first prediction unit, a first prediction based on the first signal, wherein the first prediction is for predicting a state of the autonomous system after processing delay time, predicting, by a second prediction unit, a second prediction based on the second signal, wherein the second prediction is for predicting the external environment after the processing delay time, combining the first prediction and the second prediction to obtain a combined prediction, comparing the combined prediction to a second sensor input, which is sensor input received after the processing delay time, and providing the result of the comparison for the first prediction unit and the second prediction unit. It is to be noted that the result of the comparison may additionally be provided to a logical unit, such as a signal splitter, that is configured to extract the first and the second signals from the sensor input. Additionally, or alternatively, the result of the comparison may further be provided to a logical unit, such as an action decision making unit, that is configured to determine functionality of the autonomous system.

[0013] According to a fifth aspect there is provided a non-volatile computer- readable medium comprising program instructions stored thereon which, when executed on a computing device, cause the computing device to perform a computer-implemented method comprising at least the following: receiving a first sensor input from the set of sensors, extracting from the first sensor input signal a first signal, that is a signal caused by the change of state of the autonomous system, and a second signal, that is a signal caused by an external environment, predicting, by a first prediction unit, a first prediction based on the first signal, wherein the first prediction is for predicting a state of the autonomous system after processing delay time, predicting, by a second prediction unit, a second prediction based on the second signal, wherein the second prediction is for predicting the external environment after the processing delay time, combining the first prediction and the second prediction to obtain a combined prediction, comparing the combined prediction to a second sensor input, which is sensor input received after the processing delay time, and providing the result of the comparison for the first prediction unit and the second prediction unit. It is to be noted that the result of the comparison may additionally be provided to a logical unit, such as a signal splitter, that is configured to extract the firstand the second signals from the sensor input. Additionally, or alternatively, the result of the comparison may further be provided to a logical unit, such as an action decision making unit, that is configured to determine functionality of the autonomous system.

[0014] LIST OF DRAWINGS

[0015] The present disclosure discusses below some exemplary embodiments in greater detail with reference to the enclosed drawings, in which:

[0016] Fig. 1 illustrates examples of environments in which there may not be preconfigured reaction available for an automated system. Fig. 2A illustrates a linear model of perception.

[0017] Fig. 2B and Fig. 2C illustrate models of perception with feedback.

[0018] Fig. 3 illustrates an exemplary embodiment of functionality of a processing unit comprised in an autonomous system.

[0019] Fig. 4 illustrates an exemplary embodiment of an autonomous system.

[0020] Fig. 5 illustrates an exemplary embodiment of a processing unit comprised in an autonomous system.

[0021] Fig. 6 illustrates an exemplary embodiment of a device.

[0022] DETAILED DESCRIPTION

[0023] Reducing need for manual work has been, and continuous to be, an area of great focus in development of various systems. For example, in an industrial environment there may be an aim to have systems, such as robots, performing functions without manual intervention. As another example, cars are being developed such that they may perform more and more functions without the driver. As a further example, household chores may be automated thus reducing need for manual labour in a household. There may be various benefits that can be achieved when manual work is not required, for example increased reliability, safety and / or predictability.

[0024] In some examples, a system may be considered as an automated system, when there are certain, one or more, pre-determined functionalities that the system is capable of performing automatically without a user having to intervene. This may be achieved by configuring the automated system to perform a specific functionality at a specific situation. Thus, the automated system may be programmed to perform in a pre-determined manner in a certain situation. It is to be noted that a system, that may be an automated system, may be understood as a device or as a group of devices that are interconnected to perform functionalities together. The automated system thus has clear instructions regarding what to do next and thus its decisions regarding which functionality to perform in which situation are pre-determined.

[0025] In case it is pre-determined which functionality the automated system is to perform in each situation, the automated system may be trained to recognize such situations. For example, the automated system may perceive its external environment based on input it receives from sensors that are comprised in the system and / or are connected to the system. The input may thus be understood as sensor input. The sensor input may then be processed by the automated system and the automated system may determine if the sensor input is to be interpreted as corresponding to a known object and / or situation, which may be understood as a label. In other words, if the sensor input corresponds to a situation to which the system has been pre-configured to react to in a certain manner. If not, the system may not react to the sensor input as the system has no pre-configuration to do so.

[0026] The automated system may be well suited in many situations, for example, if the environment is such that it is predictable and thus preconfiguration, which requires manual work and is required to reacting to the environment, can be considered as feasible. Yet, this may not be always the case. Fig. 1 illustrates examples of environments in which there may not be preconfigured reaction available for an automated system. Scenario 100 illustrates an automated system, which is a bus 105. The bus 105 in this example embodiment is configured to drive automatically and may thus be considered as an automated vehicle. The bus is driving on a road that has trees around the road, but not many buildings. In this example scenario, there is a barrier 104 on the road because of an accident 102, which requires the road around the area of the accident 102 to be kept clear. Thus, the barrier 104 is temporarily on the road and is intended to be moved away from the road as soon as the accident 102 has been cleared. Therefore, if the bus has been pre-configured with images, and optionally also other information regarding the road, there has not been knowledge regarding the barrier 104 at the time of the pre-configuration and thus the bus 105 may not have any pre-configuration according to which to function. This may cause the bus 105 for example not to slow down and stop, but instead to drive through the barrier.

[0027] Scenario 120 of the Fig. 1 illustrates an example of a rural environment. In this scenario 120, there is an automated system 125, which is a tractor, that is operating in an automated manner on the field. Information regarding the field has been used to pre-configure the automated system 125 such that it is capable of performing functionalities on the field without a driver. Thus, the automated system 125 has been pre-configured such that upon sensing a certain situation, such as sensing and recognizing a certain object, that recognition triggers the automated system to automatically perform a pre-configured functionality. Yet, there may be an unexpected wild animal 122, such as a moose, on the field when the automated system 125 is performing the functionality. The wild animal 122 may be a rare one and not expected to be seen on the field. Therefore, the automated system 125 may not be pre-configured to react to the wild animal 122 even if sensor input received by the automated system 125 could sense the wild animal 122. This may lead to a situation in which the automated system 125 does not react to the wild animal 122 at all, which then may lead to unsafe consequences.

[0028] Scenario 140 illustrates an environment with buildings, for example, a suburb, in which there is an automated system 145, which is a car capable of driving without a driver. The map as well as images of the area may have been utilized to pre-configure the automated system 145 to be capable of driving without a driver in that environment. Thus, if a crocodile 142 suddenly appeared on the streets of that environment, the automated system 145 would not necessarily react to it as the pre-configuration did not consider a crocodile 142 appearing on the streets. This may however be an unwanted situation causing unsafe consequences to the automated system 145 and to the crocodile 142.

[0029] Scenario 160 illustrates an environment that is an industrial environment in which there are automated systems 165, 170 and 175. The industrial environment may be considered as a predictable environment and therefore, when the automated systems 165, 170 and 175 have been preconfigured to operate in an automated manner within the industrial environment, a camel 162 wondering around the industrial environment may not have been taken into account. Thus, the automated systems 165, 170 and 175 may not have a pre-configuration that would guide them to react to the camel 162. Yet, the camel 162 may cause serious damage to the automated systems 165, 170 and 175, not to mention the injuries caused to the camel 162 itself unless a collision between the camel 162 and the automated systems 165, 170 and 175 is avoided.

[0030] The scenarios discussed above describe situations in which an automated system fails to detect and / or react to something it should react to. As the consequences may be severe, such situations may be addressed by increasing sensitivity of sensors of the automated system as well as adjusting threshold for determining that pattern recognition has recognized an object for example. This way it may be enabled that the automated system errs on the side of recognizing and / or reacting to rather than not recognizing and / or reacting as that may be considered to be a safer alternative causing less severe situations than erring on the side of not recognizing.

[0031] For example, a car may be configured to receive, from a set of sensors, sensor input based on which it may perform recognition of an object and / or situation, and then based on the recognition automatically perform one or more functionalities such as braking and / or providing a warning to the driver. For example, a car may be configured to cause braking in case it detects, based on the input from the sensors, that there is something on a road. To prevent an accident, the car may be configured to perform an emergency braking without any input from the driver. In other words, the car may then brake without the driver having any involvement in the braking. Yet, in case the car is driving when it is snowing, the snowflakes may easily be interpreted by the sensitive sensors and sensitive pattern recognition as an object causing the car to brake unnecessarily, and unexpectedly. This may cause the driver not to be able to steer the car as expected, and the risk of rear collision may be increased as well.

[0032] As another example, the car may be configured to sense how close the car is to a wall or another object such as a pillar when the driver is parking. To warn the driver, the car may cause an indication, which may comprise audio and / or visual output to be rendered to the driver. The indication may become more notable as the car approaches the object. For example, the car may be parking next to a wall and the sensors help the driver to perceive how close to the wall the car is. However, if there are for example a lot of fallen leaves on the ground, next to the wall, the sensors detect that there is an object. As the leaves are closer to the car than the wall, the car may provide strong warnings even though there is still sufficiently room between the car and the wall. This may cause inconvenient user experience in case there are loud warning sounds even though there is no need for those while the driver tries to focus on parking the car, and the leaves are not an object that should block the parking, unlike the wall. In some examples, the car may even prevent driving any closer as it is not capable of recognizing a difference between leaves and the wall.

[0033] Thus, it may be understood that the automated functionality of a system is based on binary recognition, so that either the system recognizes and reacts to something or does not, and then causes corresponding functionality to be performed. Yet, it would be beneficial if the system was also caused to detect that there is a situation in the environment external to the system that is new to it and which it does not fully recognize at once, and thus the functionality of the system would be adjusted accordingly. It may also be desirable that the recognition is improved over time such that the system may learn about the new situation. In other words, it would be beneficial if the system could autonomously perceive the environment even if the environment is not previously known. Further, it may be desirable that the system is able to react even though the situation may not be completely recognized yet. It is to be noted that the scenarios described above may be applicable to various types of automated systems, such as to nautical vessels and airplanes as well.

[0034] There may be various other scenarios as well in which an automated system has not been pre-configured to react to an occurrence within the environment in which the automated system operates, or it is configured to be too sensitive to react. For example, there may be a cleaning robot operating in a household and there may be an occurrence in the household to which the cleaning robot has not been configured to react to. To address this issue, more information can be gathered regarding unexpected occurrences in different environments. Based on those, the occurrences may be labelled, in other words, recognizing those occurrences based on sensor input may be pre-configured to the automated systems. This approach requires a person to have a knowledge of a possible occurrence beforehand, which may be challenging in a real-life environment. The particular understanding of perception that underlies this approach can be generalized to follow the Shannon Weaver model that is illustrated in Fig. 2A.

[0035] Fig. 2A describes a linear model of perception in which there is first an information source 210. The information source 210 may be understood as an entity that provides a message, which may also be understood as a sequence of individual messages, that are to be transmitted. The message is then provided to the transmitter 215, which may be understood as an entity that converts the message into suitable signals that are then transmitted by the transmitter 215 to a channel 220. The channel 220 is a communication channel in which the transmitted signals can proceed. The channel 220 may be for example air. As the transmitted signals proceed in the channel 220 there maybe noise from a noise source 225, and the noise causes distortion to the transmitted signals. The amount of distortion caused to the signals in the channel 220 may be dependent on the conditions of the channel 220. The signals, and the noise causing the distortion to the signals in the channel 220, are then received by a receiver 230. The receiver 230 may be considered as an entity that is configured to convert the signals received back to the message. The receiver 230 may also be configured to perform error correction to mitigate the effects of the noise, as the noise in the channel distorts the signals and thus may cause errors in the conversion of the signals back to the original message unless the effects are mitigated. Once the message is decoded by the receiver 230, the message may be provided to the destination 240, which is the destination for which the message was intended.

[0036] As illustrated with the example model of Fig. 2A, it is known beforehand what is the message that the receiver should receive, so when correcting errors caused by the noise added in the channel, the decoded message can be compared to the original message to know if the decoding successfully corrected the errors caused by the noise added in the channel.

[0037] In a real-life situation however, the automated system is to perceive an environment external to it, and the external environment does not intentionally send messages to the automated system. Thus, in case the automated system is preconfigured to recognize certain sensor input as a pre-determined object, for example as a cat, as a car, as a building, as a human etc, the automated system may perform processing of the sensor input such that it may be understood to correspond to decoding a recognizable message. The recognizable message of course is then something that has been taught to the automated system before hand while configuring the automated system. Thus, the automated system is able to decode sensor input to a meaningful perception when the sensor input corresponds to pre-configured objects. The pre-configuration may be understood as training and the pre-configured objects may be understood as training data, which may also be referred to as labelled training data or labelled input. Thus, the automated system may be capable of recognizing from the sensor input data such objects that correspond to training data that has been used to train the automated system. It is to be noted that a situation may be understood as a combination of a plurality of objects that are recognized based on a pre-configuration and the training data.

[0038] Yet, in case an automated system changes its own state, for example by rendering audio, moving, and / or turning on or off lights, then the automated system can perceive the effects of those changes, as the sensors of the automated system can detect the changes in the state of the automated system. Therefore, the automated system may detect, using its sensors, its own change of state, at least partly. As the automated system has the information (actuator signals] that led to its change of state, then as the change of state is detected, the change of state may be considered as a message that was sent by the automated system to itself, and as such, it is also known, what the transmitted message was.

[0039] Yet, if the sensor input data represents a relevant perception that has not been present in the training data, the automated system may fail to perceive a relevant element in the external environment, as described above. As the external environment may be constantly changing and there may be occurrences of surprising events, the task of training all possible scenarios is never really finished and may therefore not be obtainable at all. As discussed above, the automated system is able to react to the environment if it has been pre-configured to do so. On the other hand, in some exemplary embodiments, a system may be able to perform, additionally or alternatively, at least some autonomous perception regarding the environment that may not be completely known to the system, and to determine a functionality with respect to the environment although while the perception of the environment may remain at least partly unclear to the autonomous system. Unclear may be understood as the perception being at least partly uncertain in such a manner that the autonomous system is not capable of determining what exactly the perception corresponds to. It is to be noted that there is always uncertainty involved in the perception of the environment the autonomous system obtains using its sensors, as the system may utilize modelling of the environment and as a generic rule, models tend to be wrong to varying degrees as the perfect modelling of a constantly evolving environment is hardly possible. The level of uncertainty regarding the perception of the environment may however vary, and even though there would be a rather high uncertainty, the autonomous system may still be able to determine on some level what is taking place in its environment and determine its functionality based on that. For example, the autonomous system may determine based on its perception of the external environment that a car moves at a certain speed. If the real speed of the car differs only slightly compared to the determination of the autonomous system, it may be determined that the uncertainty is low. It is to be noted that the determination may be a prediction determined by the autonomous system. On the other hand, in case the autonomous system has failed to determine an object that crosses a street or that lights of its environment suddenly go out, then it may be considered that the uncertainty is high.

[0040] In other words, the autonomous system that may detect, based on sensor input received, that there is something in its surrounding environment, although the autonomous system may not be able to determine exactly what it is. Yet, the autonomous system may still be able to determine a functionality to be performed by the autonomous system even though the uncertainty is high enough to prevent the autonomous system completely determining what is taking place in the environment. Additionally, the autonomous system may be able to over time perceive the uncertainty in the environment and learn more regarding the uncertain aspects. Thus, the autonomous system is a system comprising one or more devices that are configured to perform functionalities together, and the system is capable of performing perception of the environment based at least partly on input received from sensor(s), and to determine an action to be performed in response to the perception, while the autonomous system determines some level of uncertainty with respect to the perception. The level of uncertainty may be understood in different manners. In the context of this document though, the level of uncertainty of a perception determined by an autonomous system may be understood as a measure of a gap between the autonomous system determining a prediction regarding the environment and then determining how accurate that prediction was based on sensor input received from the environment.

[0041] For example, the autonomous system may perceive an object it does not recognize, but it is able to recognize that there is an object. Over time then, the autonomous system may learn how the object behaves for example, in terms of changing its shape, illuminating light, moving, etc. Further, the autonomous system may have received a goal, which may also be understood as a task that the autonomous system is to perform. Then, in a real-life environment, which may be continuously changing, the autonomous system is capable of perceiving the environment using its sensors and processing capabilities regarding the sensor input, and determining functionalities, based on the perception that cause the autonomous system to progress towards the goal, also in case there is high uncertainty involved in the perception of the external environment, such as there being objects that are notyet known to the autonomous system. The combination of functionalities that lead to accomplishing the goal of the autonomous system may be understood as a course of action. When the autonomous system operates in the autonomous manner, it may be understood as operating without human intervention or control and being capable of operating outside the parameters of the pre-configuration that has been made using training data.

[0042] To enable the autonomous performance of functionalities, the autonomous system is to be capable of sensing the environment, which is external to the autonomous system using a set of sensors. A set of sensors may be understood to comprise one or more sensors. The autonomous system is also to keep track of its current state and location. The autonomous system is also to perceive and understand disparate data sources, and to determine one or more functions it is to perform next and to make a plan with respect to achieving its aim. It would also be beneficial for the autonomous system to be able to learn from previously determined, and performed functionalities, and how successful those were.

[0043] For an autonomous system to be able to perceive the environment and to determine functions to be performed with respect both to the environment external to the autonomous system and to the goal, the linear model may be developed into a circular one with a feedback loop that allows to determine how well the autonomous system identifies the effects that its own actions have in the sensor input. It is to be noted that the sensor input may be understood as input from the sensors. It is also to be noted in the context of this document that a feedback loop may be understood to indicate that the procedure is to be performed at least once, and therefore may be repeated multiple consecutive times, such that continuous feedback can be achieved. With this feedback, the autonomous system may be enabled to distinguish between input originating from the external environment itself, and input caused by movement that is own movement of the autonomous system, as well as probe the external environment to form a perception of the external environment, in other words, to form a model of the external environment. This is necessary, because when sensor input is received by a system, that may be automated or autonomous system, the sensor input is affected by movement of the system itself. This may be understood as a relativity problem. The system may be considered to be in a different inertial system as the external environment and thus it is desirable to separate the effects of the movement of the system itself captured as part of the sensor input received and a part of the sensor input that originates from the external environment as such by maintaining a constant relationship between actuators, that impact the movement of the system, and sensors that provide the sensor input to the system. In other words, as the sensors of the system are in a different inertial system than the external environment, the movement of the system itself causes apparent movement perceived by the sensors and thus provided as sensor input to the system. Additionally, or alternatively, the autonomous system may change its state in other manners than by moving as well and thus the sensor input may also capture other changes of state than movement, for example, light, heat, and / or sound originating from the autonomous system itself may be captured by the set of sensors. Yet, in the sensor input this perceived movement is combined with the perception of the external environment and processing of the sensor input signals received is to be performed for the perceived own movement to be extracted from the perceptions caused by the external environment. This extraction is required to build an objective model of the external environment that can be relied on to make predictions, and thereby form a correct perception of the external environment.

[0044] In Fig. 2B, the general model for perception introduced in Fig. 2A, which is based on the classic Shannon Weaver model, is developed further such that there is a feedback loop that enables autonomous perception of the environment for an autonomous system. In this example embodiment, that illustrates an example model for autonomous perception, there is a computer processing unit [CPU] 250. The CPU 250 maybe understood as a processing unit that comprises, at least partly, the capability of the autonomous system to process information. Processing information may comprise computations performed by one or more computing devices in accordance with computer program instructions that may be part of a computer program stored in one or more memory devices. It is to be noted that units discussed in the context of this document may be understood as logical units the implementation of which may differ. Signals received by the sensors may be understood to be comprised in the information that is processed by the CPU 250. Thus, the CPU 250 comprises at least one computing device capable of performing information processing. The CPU 250, which may be comprised in an autonomous system, can thus determine a functionality to be performed, in other words, it causes one or more actuators comprised in the autonomous system to perform the functionality. Thus, for each actuator required in the functionality the CPU 250 provides an actuator signal 252. For the sake of simplifying explaining, in the context of this example embodiment only one actuator is discussed, but it is to be noted that the same approach is applicable also when there are multiple actuators.

[0045] The actuator signal 252 thus causes an actuator 255 to perform functionality that, either by itself or as a combination of other actuators performing their respective functionalities as well, cause the autonomous system to perform functionality, which causes a change of state of the autonomous system. The change of state of the autonomous system may be understood to comprise a physical change, such as, but not limited to, movement, emitting different amount of light than previously and / or causing a different level of sound to be output than previously. Thus, it may be considered that due to the functionality of the autonomous system caused by the actuator 255 the autonomous system provides a signal that goes through the channel 260. The signal may be understood as own signal 257 because the change of state of the autonomous system causes sensors 270 of the autonomous system to sense the change of state of the autonomous system.

[0046] The environment 265 which is external to the autonomous system is perceived by the sensors 270 as if it was noise of the channel 260. Thus, the sensors 270 sense a combined signal 267 that comprises the own signal 257 as well as external signal 262, which is signal for perceiving the external environment 265, which is received as if it was the noise. The sensors 270 then provide sensor input signal 275 to a perceptual learning controller (PLC) 280, which may be a unit separate from the CPU 250, as illustrated in this example embodiment, but may alternatively be comprised in the CPU 250.

[0047] The PLC 280, which may be understood as a logical unit, may then be configured to separate, in other words split or extract, the sensor input signal 275 into an extracted own signal 257b and an extracted external signal 262b and then provide to the CPU 250, as feedback, the split signals, that is, the extracted own signal 257b and the extracted external signal 262b. Additionally, the CPU 250 may provide to the PLC 280 the actuator signal 252. As the PLC 280 splits the sensor input signal 275, the PLC 280 may use the actuator signal as a reference that can be used as a basis for extracting the own signal 257b, and to determine correlation function with respect to the actuator signal and the own signal 257. This allows the PLC 280 to learn from possible errors. On the other hand, as the CPU receives as feedback the extracted own signal 257b and the extracted external signal 262b, the CPU 250 may determine a model of the external environment 265 as the CPU 250 is capable of determining the relationship between the actuators causing the own signals and external signals the sensor input in a continuous manner.

[0048] Fig. 2C illustrates an example model of autonomous perception. This example is a variation of the example discussed in the context of Fig. 2B. In this example, the perception model is like in the example of Fig. 2B except that the PLC 280 is comprised in the CPU 250. In this example model of autonomous perception, there is a feedback loop like in the example of Fig. 2B. Additionally, in this example, there is an action-decision making (ADM) unit 290, which may be understood as a logical unit. The ADM 290 is configured to determine the actuator signal 252 to the actuator 255 based on the input it receives from the PLC 280. The inputs are the external signal 262b and the own signal 257b. The ADM 290 then provides the actuator signal 252 to the PLC 280 as a key information to compute the extraction function for splitting the sensory input 275 into the extracted external signal 262b and the extracted own signal 257b. In this example, the ADM 290 thus determines the actuator signal based on the model it builds of the external environment based on the input.

[0049] It is to be noted that the extraction of the external signal and the own signal from each other requires processing, which in turn requires time. Also, the perception of the external environment is based, at least partly, on a model of the external environment built by the autonomous system, and the autonomous system also predicts how the external environment will develop. Based on the predictions, the CPU 250 of the autonomous system may then predict future sensor input as well. The prediction also requires time. Thus, a processing delay may be understood as a time that it takes for sensor input to be received by the set of sensors and then to be processed by the CPU 250. Therefore, the autonomous system has a perception of the external world that has a lag with respect to the real world, in other words, to the external environment. While this requires the ADM to determine actuator signals that are based on predicting how the real-world is after the delay, the feedback mechanism also allows to check how well the predictions are made. In other words, corrective measures may be determined in case there is error, that is determined as significant enough, as the error may be used as a basis to improve the predictions. Thus, the perception of the external environment determined by the CPU 250 may be validated using feedback that indicates if predicted input has been correct or not. This allows verification of perception made based on sensor input without external validation such as pre-configuration made using training data. Instead, the autonomous system is capable of detecting errors and then using the detected error as a feedback, make corrections. In other words, the processing delay allows predictions made based on perception of the external environment to be verified using real-world data. Thus, the verification indicates if the perception and predictions made based on it, are correct as predicting correctly ahead of time for the duration of the processing delay requires the model of the external environment to correspond to a correct perception of the external environment. It is to be noted that the process of extracting the signals, making predictions and comparing the predictions to real-time sensor input to validate the predictions is a procedure that may be repeated multiple times thus achieving continuous verification to the perceptions of the external environment as well as input based on which the perception may be modified if needed.

[0050] Fig. 3 illustrates an exemplary embodiment of functionality of a PLC, such as the PLC 280 discussed above. The units discussed in this, and other exemplary embodiments of this document, may be understood as logical units the implementation of which may vary, and the implementation may use software and / or hardware for performing the functionality of the logical unit. It is to be noted that two or more logical units may use the same hardware resources. Hardware resources may be comprised in different types of devices, such as, but not limited to, a computing system, an entity for edge computing, an entity used in cloud computing, a device comprised in an autonomous system as such, etc. It is also to be noted that even if some logical units are described as separate logical units, there may be implementations in which at least some of those logical units are part of the same, one logical unit, and vice versa. The PLC is comprised in, or connected to, the CPU of an autonomous system that is capable of forming a perception of an environment that is external to the autonomous system, and which may thus be understood as an external environment. Forming the perception can be done in an autonomous manner meaning that the autonomous system is capable of perceiving the environment even if some, or all aspects, regarding the external environment are not pre-configured to the autonomous system using training data.

[0051] The autonomous system comprises a set of sensors that comprises one or more sensors configured to observe the environment external to the autonomous system. The sensors, in this and other exemplary embodiments discussed in this document, may be for example physical sensors such as position sensors, pressure sensors, temperature sensors, force sensors, vibration sensors, proximity sensors, light sensors, touch sensors, infrared sensors, inertial, etc. It is to be noted that one autonomous system may comprise a plurality of different types of sensors. The autonomous system may also comprise a set of actuators that comprises one or more actuators. The actuators, in this and other exemplary embodiments discussed in this document, may be physical actuators such as hydraulic actuators, pneumatic actuators, electric actuators, inline actuators, parallel actuators, mechanical actuators, thermal actuators, magnetic actuators, etc. It is to be noted that the autonomous system may comprise a plurality of different types of actuators. As mentioned above, the CPU provides to the actuators actuator signals that cause the actuators to perform their respective actions that, by themselves or together cause such functionality of the autonomous system that causes the change of state of the autonomous system.

[0052] As also mentioned above, the sensor input provided by the set of sensors comprises also the sensor input caused by the own movement of the autonomous system, in other words, the sensor input comprises a signal that is caused by the actions of the autonomous system itself. Additionally, the sensor input comprises an input caused by the external environment as such, which is input that is independent of the actions of the autonomous system. Thus, the sensor input 257 received from the set of sensors may be considered as raw input data (RI), which comprises the combination of own signal (OS) and external signal (ES) captured by the set of sensors. This may be represented using the equation Rl = OS + ES. Thus, data streams received as an output from the set of sensors, which in other words are captured by the set of sensors, flow as a combined raw input data to the CPU of the autonomous system. Within the CPU, there is a PLC, which may be configured to receive the raw input data. It is to be noted that the data may be in the form of signals received and thus the raw input data may also be understood as raw input signal that is the sensor input signal comprising the combined signal.

[0053] In this exemplary embodiment, the PLC comprises a signal split unit 300, which is configured to extract the OS and the ES from the RI 257. The signal split unit 300 then provides the extracted ES 262b to an input prediction unit 315, which is configured to predict the ES ahead in time T, which corresponds to the processing delay time T caused by processing the raw input data. In parallel. The signal split unit 300 may provide the extracted OS 257b to a movement prediction unit 310, that is configured to predict the OS in further, after time T has passed. Thus, in this exemplary embodiment, the change in the state of the autonomous system comprises movement, but it is to be noted that it could also comprise other changes in the state of the autonomous system. The prediction 317 provided by the input prediction unit 315 and the prediction 312 provided by the movement prediction unit 310 are then provided as input to a combined prediction unit 320, which is configured to predict the raw input signal in future after the time T, which corresponds to the delay caused by processing of the raw input. Processing the raw input may be understood to comprise splitting the signals OS and ES from the combined signal and then using those signals for performing predictions and refining and correcting the model of the external environment.

[0054] The combined prediction unit 320 then combines the individual predictions and provides them to a reality model fitness checker unit 330, which is configured to compare the combined prediction to the actually received raw input after the delay T. The difference between the raw input and the combined prediction, in other words the results of the comparison, provides a delta value E, which may be considered to be an error of the prediction. Optionally, there may be a threshold value for the E in terms of its significance such that once the E is greater than the threshold value, it is considered as significant enough to be taken into account. In other words, in case the E is less than the threshold value, it may be considered as matching the raw input. The E may then be provided to an error classification unit 340, which is configured to classify the error E into different components and subsequently provide the classifications of the error E as feedback 345 to the signal split unit 300, to the input prediction unit 315, and to the movement prediction unit 310. The error classification unit 340 may identify a source for the error by for example determining a correlation of the error E with each of the predicted signals, and then feeding back the relevant portion to the respective prediction units. Additionally, in case both predicted signals are correlated with the error, then the classification unit 340 may perform additional correlations to distinguish between the combination of separate prediction errors of each predicted signal and an error in the signal split unit 300. This way a feedback loop may be achieved, and the error may be utilized by the prediction units to learn and improve the predictions. This allows there to be verification even if there is no validation through a preconfiguration made using training data. Thus, as the feedback is available, and as the procedure may be repeated multiple times, there may be continuous learning with the constant measurement of the difference between what is predicted and what is sensed by the set of sensors. It is to be noted that the classifications of the error E may also be provided to other units, for example, to units outside the PLC (like the ADM], which are configured to make decisions and cause functionalities of the actuators, thereby allowing them to take into account the level of uncertainty for its decision making functionality. It is to be noted that error classification may also be referred to as error labelling.

[0055] A benefit associated with the splitting of the signal is that it allows forming an autonomous perception of the external environment, which may be considered as a pre-requisite for achieving a truly autonomous system. As the prediction is then compared to the actual received input, the system may be considered to have a built-in error detection with respect to perceiving the environment. As the built-in error detection then allows performing error correction as well, the autonomous system may be considered as a stable system in which an error in predicting is not amplified, but instead corrected, and may be corrected autonomously.

[0056] Fig. 4 illustrates an exemplary embodiment of an autonomous system. The autonomous system may be for example a vehicle capable of moving in various environments. In this exemplary embodiment, the autonomous system comprises a CPU 410, which may be a CPU such as those described above. The autonomous system also comprises actuators 420 and 425, which are configured to cause change of state, which in this exemplary embodiment is movement, of the autonomous system in accordance with signals provided by the CPU to the actuators 420 and 425. The movement of the autonomous system caused by at least one of the actuators causes own signals 422 and 427. The autonomous system further comprises sensors 430, 431, 432, 433, 434 and 435, which are configured to sense sensor input 440, 441, 442, 443, 444, and 445.

[0057] In this exemplary embodiment, the autonomous system is capable of driving in different environments. As one example, the autonomous system may drive in a suburb 400. While in the suburb, there may be a helicopter 405 landing on the road and the autonomous system is to perceive that and to adjust its driving accordingly. Thus, the autonomous system is expected to perceive that there is an object, which is the helicopter 405, and that it is approaching the road the autonomous system is driving. Thus, the autonomous system is to avoid colliding with the helicopter 405, but also, it is not to avoid the collision in a manner that would cause any damage to the houses in the suburb 400. The autonomous system is expected to perform this even if it has not been pre-configured to recognize the helicopter 405.

[0058] As another example, the autonomous system may be driving along a road 404, when there is a drone 406 flying next to the road 404. The autonomous system is now expected to perceive the drone 406 and as the drone 406 is flying next to the road and not moving its trajectory such that it would be coming across the road 404 on such an altitude that it could collide into the autonomous system, the autonomous system is to continue driving normally along the road 404. The drone 406 is an object that is new to the autonomous system, in other words, the autonomous system has not been pre-configured to recognize the drone 406.

[0059] As yet another example, the autonomous system may be driving in a rural area 408 such that there is a sheep herd next to a road the autonomous system is driving along. In this example, the autonomous system is to perceive the herd and to be ready to react and brake in case the herd begins to move towards the road.

[0060] For the autonomous system to be able to function as expected in the examples described above, although the autonomous system may not have been trained how to react in those example situations, the autonomous system is to be able to autonomously perceive the environment, and also to change its course of action due to an object if needed even if the object is not fully recognized by the autonomous system. Thus, the autonomous system is to be able to predict the development of the external environment and to continuously observe the environment and based on the observations, correct and improve the predictions and to learn autonomously without external guidance.

[0061] Fig. 5 illustrates an exemplary embodiment of a CPU 500, that may be understood as a logical unit that may be comprised in an autonomous system such as those discussed in the exemplary embodiments above. In this exemplary embodiment the CPU 500 comprises two different units, which may be understood as logical units, the PLC 510 and a purpose fulfilment checker (PFC) unit 550. It is to be noted that the logical unit discussed herein may also be referred to using some other name. The PLC 510 is configured to receive raw input 505, which may also be understood as sensor input received from a set of sensors, to a sensorimotor correlator unit 512 comprised in the PLC 510. The sensorimotor correlator 512 is configured to perform correlation analysis such that it is capable of determining which part of the raw input 505 signal belongs to the effects of the own signal, in other words, to the effect caused by the own signal [OS] of the autonomous system, which in this exemplary embodiment is caused by movement of the autonomous system and the movement is caused by the set of actuators of the autonomous system. Once the own signal is identified, then as a consequence also the part of the raw input 505 that corresponds to the external signal (ES) may be determined. The movement of the autonomous system is caused by the set of actuators, and the actuators are controlled by providing to them actuator signals. The actuator signals that are for producing movement of the autonomous system may be understood as raw output.

[0062] A level of correlation between the external signal and a change of state of the autonomous system, such as the movement caused by the set of actuators, may vary. The variation may be due to the autonomous system learning more about the environment. For example, at beginning of the autonomous system learning more about the environment, there may be little, if any correlation as the system has little knowledge of the environment to which action decisions could be based on. Then as the autonomous system learns more about the external environment, some of the actuator signals may be correlated on purpose with the external environment. For example, the autonomous system may determine that it should bypass an object and thus the actuator signals have a correlation to the object of the external environment.

[0063] The correlation may be determined based on the following relations. As discussed previously, the raw input (RI) can be described as Rl-OS+ES. On the other hand, the relative movement of the autonomous system is correlated with the raw output, which leads to the following postulation: OS=Q(RO]. By combining these two equations, the result is that RI=fl(RO)+ES. Thus, the sensorimotor correlator 512 is configured to determine the fl, which is a correlation function. The sensorimotor correlator 512 may determine the function fl in any suitable manner. For example, the sensorimotor correlator 512 may perform multidimensional correlation analysis in which different types of signals with varying delays between them are considered. Examples of signal types are visual and auditive types, but there may be also other types of signals. Additionally, or alternatively, machine learning may be utilized in determining the function fl. For example, a neural network may be utilized to develop and update a model for correlation so that the function fl may then be determined using the model for correlation. In case machine learning model is used to determine the function fl, then the machine learning model may be trained using a training dataset that comprises known functionalities of the autonomous system and the commands provided to actuators of the autonomous system to achieve such functionalities. Thus, the dataset used for training the machine learning model may be selfgenerated by the system. It is also to be noted that in case machine learning is utilized for determining the function fl, the machine learning model may be any suitable machine learning model and it may be trained in any suitable manner.

[0064] As mentioned above, the correlation between the external signal and a change of state of the autonomous system, which then causes the own signal, may vary. Thus, the function fl may converge to a stable state as it learns about the external environment, and the autonomous system may then use the function fl to produce actuator signals that are correlated with the environmental signals. For example, the stable state may be required for the signal splitter 520 to be able to identify subtle changes that may be originating from the actuator signal determined and provided by the ADM 560.

[0065] Once the function fl is determined, it is provided as the output 515 to a signal splitter unit 520. The signal splitter may then use the function fl to extract from the raw input 505 the ES and the OS. As the autonomous system may be understood to transmit signals that it then receives as part of the sensor input, it may be understood that the autonomous system is configured to build, and over time also fine-tune, a virtual channel through which it transmits signals to itself. As these signals are received as part of the raw input 505, it may be determined that everything else originates from the external environment. Thus, the function fl, which may be built on finetuning of the channel from the autonomous system to itself, allows to understand the part of the raw input that originates from the autonomous system itself, and the part that originates from the external environment. The signal splitter unit 520 then provides the ES 524 to an external signal predictor unit 528, which is configured to perform computations that result in a prediction of the ES as a projection forward in time after a delay time T. The delay time T is the time that it takes to perform processing of the raw input 505 within the CPU 500. The signal splitter 520 then also provides the OS 522 to an own movement predictor unit 526, which is configured to perform computations that result in a prediction of the OS as a projection in time forward after the processing delay time T. The external predictor unit 528 provides prediction of the ES 529 and the own movement predictor unit 526 provides prediction OS 527, which are then combined into a combined prediction 532 that is then provided to a model fitness checker unit 535. The model fitness checker unit 535 also receives, as an input, the raw input 505, and the model fitness checker is configured to perform computations for determining a difference between the raw input 505 and the combined prediction 532. The difference may be understood as an error E. The determined error E 537 may then be provided to an error classifier unit 540, which is configured to perform one or more classifications for the error to determine and indicate a source, or sources, of the error. For example, the error classifier unit 540 may be configured to classify the error based on its likely source. Thus, the error classifier unit 540 may be configured to compute correlation of the error with each predicted signals to indicate a portion of the error that originates from the predictor unit of the predicted signal. Also, in case there is an inverse correlation between the errors due to the predictors, the portion of the error that originates from a misclassification in the signal splitter may be determined to have originated from the sensorimotor correlator unit 512 and its miscomputation of the correlation function £1 515. Once the classifications have been performed, the classified error 545 may be provided as an output to at least some of such units that are involved in decision making with respect to forming a model of the external environment and / or determining functions of the automated system that are to be performed. The determination regarding to which units the classified error 545 is to be provided as an input may be based on for example the source of the error. Thus, the classified error E may be provided as feedback to the own movement predictor unit 526, external predictor unit 528, the sensorimotor correlator 512, and / or the signal splitter 520. In this exemplary embodiment, the CPU 500 also comprises a purpose fulfilment checker [PFC] 550. It is to be noted that in some other exemplary embodiments, the PFC may be a unit that is connected to the CPU, while not necessarily comprised in the CPU. The autonomous system is associated with a purpose, which defines for what the autonomous system is intended. For example, if the autonomous system is a vehicle, it is intended for autonomous driving, if the autonomous system is a robot for household chores, then its purpose is to autonomously perform household chores, if the autonomous system is intended for industrial usage, then its purpose is autonomous performance of industrial functions, and so on. The goal of the autonomous system may thus be coherent with the purpose of the autonomous system. The autonomous system may also be associated with sub-goals, which are functionalities, that may be part of a course action of the autonomous system, or they may be independent course of action, that are coherent with the purpose of the autonomous system. The sub-goals may be prioritized with respect to the goal of the autonomous system at a given time. For example, if the autonomous system is a vehicle, the goal may be to drive to a certain destination. The sub-goals may then be for example one or more of the following: stay on the road, avoid collisions, follow the speed-limits, and / or do not run out of fuel or electricity. The prioritization may be determined in any suitable manner. The PFC 550 comprises an intentionality directives unit 590, which may be a logical unit. The intentionality directives unit 590 is for providing prioritization of the sub-goals for the autonomous system at a given time. Thus, a course of action at the given time may be evaluated, in other words rated, with respect to the priorities of the sub-goals having the highest importance at the given moment. The sub-goals may be pre-configured, or they may be learnt by the autonomous system, or a combination of both.

[0066] Thus, the PFC 550 may be understood as a unit, that may be a logical unit, for planning in different levels that projects probable scenarios and shortterm predictions, which may also be referred to as forecasts, evaluating various different courses of action of the autonomous system with respect to the external environment and the goal of the autonomous system, selecting and fine-tuning one course of action. The course of action may be with respect to achieving the goal or a part of the goal. The different levels may be understood for example as having different time spans. The probable scenarios and short-term predictions discussed in this exemplary embodiment may be determined in any suitable manner. For example, probabilistic modelling and / or Monte Carlo simulations may be used. The PFC 550 also comprises, in this exemplary embodiment, an actiondecision maker (ADM) unit 560, which may be a logical unit. The ADM 560 is for performing computations that result in one or more output signals that are then provided to the actuators of the autonomous system. The one or more output signals are signals that cause the actuators to perform a functionality such that the autonomous system is caused to perform its determined next functionality. The ADM 560 may determine the next functionality based on a prospective short-term prediction of the effects of its own movement and a prediction of the external environment. The prospective short-term prediction may be obtained from a prospective short-term prediction unit 570, which may be understood as a logical unit comprised in the ADM 560. The prospective short-term prediction unit 570 may receive as an input for determining the prediction of the own movement 527, as well as the classified error 545.

[0067] Additionally, the ADM 560 comprises a unit for environmental predictions 575, which may be a logical unit. The unit for environmental predictions determines how the external environment will develop in the same time horizon as the prospective short-term prediction. Both predictions 570 and 575 are used by the ADM 560 to determine, by performing suitable computations, output signals required for the actuators of the autonomous system to accomplish the course of action determined for that given time. Thus, the ADM unit 560, provides, as output, actuator output 565, which may be understood as one or more signals causing the actuators to perform the functionality of the autonomous system. Thus, causing for example the own movement of the autonomous system. The actuator output signals 565 may therefore be understood as raw output 565. The ADM 560 may therefore be considered to determine functionality of the autonomous system based on, at least partly, the prediction of the own movement, prediction of the external environment and the error.

[0068] As the ADM 560 receives, as an input, the classified error 545, the ADM 560 may determine if the error suggests that the course of action should be changed. For example, if the autonomous system is a vehicle, the error may be determined to suggest that the there is a risk the autonomous system will run over people. If such determination is made by the ADM 560, then the ADM may determine to provide such actuator output 565 that causes the current course of action to be interrupted and immediate reaction to ensure integrity of the autonomous system and avoidance of transgressions. Interrupting the current course of action and determining a new course of action may be understood as reconfiguring the course of action of the autonomous system to cause the autonomous system to perform the goal.

[0069] The prospective short-term prediction unit 570 and the environmental prediction unit 575 provide predictions of both, the system’s own movement and the external environment such that the ADM 560 may determine the immediate next functionality to be performed by the autonomous system and then provide the actuator signals causing the raw output 565 accordingly. It is to be noted that the raw output 565 is also provided to the sensorimotor correlators so that the correlation function fl 515 can be determined.

[0070] The PFC 550 may then also comprise an Alternative Action Evaluation unit 580 which may be a logical unit configured to build a model regarding the autonomous system itself and its physical capabilities. As the PFC 550 may receive as an input the function fl 515, which indicates the relationship between the raw output and the own signal comprised in the raw input, the alternative action evaluation unit 580 is capable of building a model regarding physical capabilities of the autonomous system itself. Thus, the Alternative Action Evaluation unit 580 provides evaluation regarding which alternative courses of action may be considered and one course of action may then be selected from the alternative courses of action. The output of unit 580 is fed to the Prospective short term prediction unit 570.

[0071] The PFC 550 then also comprises a unit for environment forecasting 585, which may be a logical unit. The unit for environment forecasting 585 may be configured to build a model of the external environment. As such, it may provide long term forecasting of the external environment, with different probable scenarios, which are ranked according to their respective probability. The ranking may be determined based on past experience for example. It is to be noted that when selecting the immediate next functionality to be performed by the autonomous system, as well as the course of action, the intentionality directive unit 590 provides the goals as well as the sub-goals and their prioritization based on which the selection may be done. The model of the external environment may be developed based on the extracted external signals as well as based on the model of the autonomous system. The output of unit 585 is fed to the Environmental prediction unit 575.

[0072] As the PFC 550 comprises the ADM 560, the Alternative Action Evaluation unit 580 and the Environment Forecasting unit 585, the autonomous system may be considered to comprise capability of performing different levels of predictions, as the time span of those predictions is different. The predictions determined by these units may be taken into account, as illustrated by the dashed arrow 577, by the unit 526 for own movement prediction and by the unit 528 for prediction of the external environment, which also provide predictions at different level than the predictions from the ADM 560. In this example embodiment, the predictions are taken into account, but are not necessarily input as such to the units 526 and 528. For example, the autonomous system may illuminate a light and thus, the Environment Forecasting unit 585 may provide a prediction regarding a pattern of a highly probable scenario that the autonomous system follows with respect to illuminating the light. The ADM 560 then on the other hand may provide a prediction regarding when the light is turned off next time and thus the illuminating is paused. Based on such predictions the own movement predictor unit 526 for example may determine a prediction with respect to receiving sensor input detecting the illuminated light. This may be beneficial for example in terms of the autonomous system being able to cope with an error detected by the PLC 510. The predictions provided by the ADM 560 may for example be used as a basis for determining if the error is to be disregarded or compensated for in some manner.

[0073] It is to be noted though that the model of the external environment may, in some exemplary embodiments, comprise a plurality of different models with respect to the external environment, and one model may model part of the external environment, not necessarily the whole external environment. As the model of the autonomous system indicates what its physical capabilities are, then the information can consequently also be used to identify what corresponds to the external environment thereby building a model of the external environment. It is to be noted that these both models are based on the separation of the own movement and external signals. It is also to be noted that as the model fitness checker unit 535 is capable of detecting errors in predictions, those errors may also be considered as indications that there may be defect(s) in the model of the external environment and / or the autonomous system itself. Thus, depending on the origin of an error, in case the error is indicative of a defect in a model, the model may be updated. Updating the model may comprise for example updating one or more parameters of the model. Thus, the models may be finetuned over time as the process of splitting the signals, making predictions and then comparing those results to the raw input to identify and classify errors is repeated multiple times.

[0074] In this exemplary embodiment, the PLC 510 is connected to the set of sensors of the autonomous system and thus receives the raw inputthat originates from the sensors. It is to be noted that although the raw output may comprise the actuators causing movement, additionally, or alternatively, there may be actuators that cause for example sound and / or lights to be output by the autonomous system. If the set of sensors of the autonomous system then comprises sensors for sound and / or light, the raw input thus comprises also the light and / or sound signals that originate from the actuators of the autonomous system. The processing of the raw input may then however be performed in a manner corresponding to what is discussed above. Thus, the processing of raw input signals is not limited to certain types of signals but may be applicable to various types of signals.

[0075] As described above, the autonomous system is capable of verifying its perception without external validation. Additionally, the autonomous system is capable of determining the next functionality based on the predictions and the goal and its related sub-goals. Thus, when looking at Fig. 4 and scenario 400 for example, the autonomous system is capable of detecting the sound of the helicopter 405 and optionally may also detect the helicopter 405 based on its shape that is moving. The feedback loop of the autonomous system allows the autonomous system to perceive over time how the movement of the helicopter 405 develops and also if the helicopter is approaching the autonomous system as it continues to proceed in the environment. If the autonomous system detects that the helicopter 405 is about to land in the direction to which the autonomous system is proceeding, the autonomous system determines the immediate functionality based on the goal, but also based on the sub-goals. For example, the goal may be to proceed to a certain destination and sub-goals may be not to collide to anything and not to drive on a lot on which there are houses. Thus, even if the autonomous system does not recognize helicopter 405, and thus there is no external validation that it is a helicopter 405 either, the autonomous system may be able to detect the sound of the helicopter 405 and whether the sound is getting louder or not, and it may then also sense that there is an object landing. As the autonomous system is capable of correcting its predictions, in case it did not predict the helicopter 405 to land, the autonomous system then is also capable of determining that due to the landing, there is a risk of collision unless the autonomous system slows down and / or finds a detour. Thus, the autonomous system may then determine the course of action accordingly. So even if the autonomous system would not have been trained to recognize a helicopter, the autonomous system is still capable of perceiving that in the external environment there is an object that is moving such that a collision is possible unless the course of action is changed and the autonomous system may then determine, with respect to the goal and sub-goals, what the next course of action could be.

[0076] If the scenario 404 of the Fig. 4 is looked at next, then the autonomous system may proceed towards a fork in the road. The autonomous system knows that it is about to stay on the left lane and thus veer towards the left. The autonomous system may then sense sound and optionally also movement coming from the right as that is where the drone 406 is flying. The autonomous system may perceive, using the feedback loop and error correction, that the drone stays on the right, not coming on or above the road, even though its altitude changes. As that is the case, the autonomous system may determine that it is safe to continue driving on the left lane normally as the drone is not about to disturb the course of action selected. Thus, the autonomous system does not need to recognize the drone 406 as such to be able to react accordingly to it.

[0077] If looking at the scenario 408 of the Fig. 4, the herd can be observed and through the feedback loop, the autonomous system learns that there may be sudden and unpredictable movements by some of the sheep in the herd. Thus, the autonomous system may determine to slow down and drive slowly enough to allow there to be enough reaction time in case a sheep suddenly jumps onto the road for example. Thus, the autonomous system does not have to recognize the sheep as such in order to be able to react to it accordingly.

[0078] When an autonomous system is capable of extracting the external signal from its own signal, and as there is a feedback loop allowing to detect, and correct errors in perception of the environment, the autonomous system is capable of building continuously a model reflecting its own physical capabilities as well as the external environment. This allows the autonomous system to autonomously perceive the environment without external validation. Also, as the signals are continuously extracted, using a signal splitter, and verification that the perception is correct is also continuously performed, the autonomous system is capable of avoiding a situation in which it would not be aware of changing conditions in the external environment and thus would not necessarily perceive them and / or know how to react to those.

[0079] It is to be noted that an autonomous system that is capable of perceiving the environment as described above, may also be trained for example to detect what is considered as normal in an external environment. Thus, using the approach described above to achieve the autonomous system can be combined with training, using labelled training data, the autonomous system to recognize some objects in the external environment. This may allow to build knowledge regarding what there can be in the external environment faster. Additionally, or alternatively, the autonomous system may be configured to learn from perceptions it has made already. Thus, as an object is perceived, its physical properties, as well as optionally sounds emitted by the object, and movement of the object may be stored and used as reference for perceiving similar objects in the external environments later on. Thus, when the autonomous system has achieved a goal, the perceptions made during the course of action required to achieve that goal may be utilized, by the autonomous system, later on thereby making the autonomous system capable of self-learning.

[0080] In general, the exemplary embodiments discussed above may have various benefits. For example, as the feedback loop allows constant verification of models regarding for example external environment of an autonomous system, the verification of models and predictions can be performed with respect to the real- world and therefore the models may be dynamically fine-tuned, and errors can be corrected in real-time, or at least substantially in real-time. This allows learning to be used instead of, or in addition to, pre-configured and pre-learned models regarding the external environment. As another benefit there may be that functionality of the autonomous system may be determined based on perceiving the environment and how it develops even if the situation is not pre-configured to the autonomous system.

[0081] The exemplary embodiments discussed above may be implemented using any suitable implementation. For example, a computing system may be configured to cause an autonomous system to perform functionality such as described above in the exemplary embodiments. A computing system may be understood as one or more computer devices that are configured to function in an interconnected manner. The computing system may be a system on a chip [SoC], or part of it may be on a SoC and part of it may be performed using cloud computing and / or edge computing. For example, in a nautical vessel the implementation may be comprised in a SoC, while a vehicle such as car may be connected to edge computing that implements at least part of the functionalities that are in accordance with the exemplary embodiments discussed above. Yet, as mentioned, the implementation may be in any suitable format and with any suitable devices forming the autonomous system.

[0082] Fig. 6 illustrates an exemplary embodiment of a device 600 that may be or may be comprised in a computing device or in a computing system comprising a plurality of computing devices. The computing device or a computing system are capable of performing computations such as those discussed above, and as such, this exemplary embodiment is compatible with the previous exemplary embodiments. The device 600 may thus be comprised in an autonomous system. In this exemplary embodiment, there is at least one processor 640, at least one memory 630, at least one connectivity unit 610 and at least one unit for receiving input and providing output 620. It is to be noted that the units described here are logical units and thus the actual implementation may vary. The units may also be connected to each other.

[0083] The at least one processor 640 may also be referred to as core, a central processing unit, microprocessor or graphical processing unit (GPU). A processor may be understood as an integrated circuit for performing calculations according to instructions provided using computer code. The at least one memory 630 may comprise volatile and / or non-volatile memory. Thus, the at least one memory 630 may be understood to be one block of memory or a combination of different blocks of memory. The memory may be for storing different types of data. The at least one memory 630 stores also computer program instructions, for example in the form of an application and / or an operating system. The at least one memory 630 provides computer program instructions to the at least one processor 640 for executing and the at least one processor 640 may then be configured to store data into the at least one memory 630. Some examples of memory are random access memories (RAMs), such as static RAM (SRAM) and dynamic RAM (DRAM), readonly memory (ROM), flash memories, optical discs, and magnetic computer storage devices, such as hard disk drives. The input and output unit 620 may allow user input, such as pressing a button, touch input and / or voice input, to be received by the device 600 and output such as audio, haptic or visual output to be provided to a user. The connectivity unit 610 allows connection to be formed between the device 600 and another device. The connectivity unit may allow wireless and / or wired connections to be formed between the device 600 and other devices. Examples of connection types that may be supported by the connectivity unit 610 are cellular communication -based connections, local area networks, Bluetoothconnections, Wi-Fi connections, etc.

[0084] The present disclosure has been described above with reference to the exemplary embodiments. However, a person skilled in the art will understand there may be embodiments that vary from the example embodiments discussed above within the scope of the claims. Thus, skilled person will understand that the exemplary embodiments described above may, but are not required to, be combined with other exemplary embodiments in various manners.

Claims

CLAIMS1. A computer-implemented method for an autonomous system, the autonomous system comprising a set of sensors and a set of actuators, the set of actuators being for causing a change of state of the autonomous system, the computer-implemented method comprising: receiving a first sensor input from the set of sensors, wherein the first sensor input comprises a first signal, that is a signal caused by the change of state of the autonomous system, combined with a second signal, that is a signal caused by an external environment, and wherein the second signal represents a perception caused by the external environment; extracting from the first sensor input signal the first signal, and the second signal, wherein the extracting is performed based on actuator signals provided to the set of actuators and which cause the change of state of the autonomous system; predicting, by a first prediction unit, a first prediction based on the first signal, wherein the first prediction is for predicting a state of the autonomous system after processing delay time; predicting, by a second prediction unit, a second prediction based on the second signal, wherein the second prediction is for predicting the external environment after the processing delay time; combining the first prediction and the second prediction to obtain a combined prediction; comparing the combined prediction to a second sensor input, which is sensor input received after the processing delay time, and wherein the second sensor input comprises a third signal, that is caused by the state of the autonomous system after processing delay time, and a fourth signal, that is caused by the external environment after the processing delay time; and providing the result of the comparison for the first prediction unit and the second prediction unit for using the result of the comparison as continuous feedback for learning and improving predictions of the first and the second prediction unit.

2. A computer-implemented method according to claim 1, wherein the result of the comparison is determined to be an error if it is greater than a threshold value.

3. A computer-implemented method according to claim 2, wherein the error is classified based on determining if the error is caused by the first or the second prediction.

4. A computer-implemented method according to any previous claim, wherein the automation system further comprises means for determining functionality of the autonomous system based on the first prediction and the second prediction.

5. A computer-implemented method according to claim 4, wherein the functionality of the autonomous system is further determined based on a course of action, wherein the course of action comprises functionalities that cause the autonomous system to perform a goal.

6. A computer-implemented method according to claim 5, wherein the functionality is further determined based on one or more sub-goals, and their respective priority.

7. A computer-implemented method according to claim 6, wherein the autonomous system further comprises means for reconfiguring the course of action based on the error.

8. A computer-implemented method according to claim 7, wherein reconfiguring the course of action comprises re-prioritizing the sub-goals.

9. A computer-implemented method according to any previous claim, wherein the autonomous system further comprises means for determining a correlation function indicating correlation between the first signal comprised in the first sensor input and one or more actuator signals that cause the set of actuators to cause the change of state of the autonomous system, and wherein the correlation function is used as a basis for extracting the first signal and the second signal from the first sensor input.

10. A computer-implemented method according to any previous claim, wherein the method further comprises determining functionality for the autonomous system based, at least partly, on first prediction, the second predictionand the result of the comparison.

11. A computer-implemented method according to any previous claim, wherein the method further comprises determining a model of physical capabilities of the autonomous system based on the first extracted signal, and / or determining the model of the external environment based on the second extracted signal.

12. A computer-implemented method according to any previous claim, wherein the change of state of the autonomous system comprises a physical change of the state of the autonomous system.

13. A computer program product comprising computer instructions stored in a memory which, when executed by at least one processor, cause a computing apparatus to perform a computer-implemented method according to any of claims 1 to 12.

14. A computing system comprising at least one computing device that comprises means for performing a computer-implemented method according to any of claims 1 to 12.

15. An autonomous system comprising a set of sensors and a set of actuators, the set of actuators being for causing a change of state of the autonomous system, the autonomous system comprising means for performing the computer- implemented unit according to any of claims 1 to 12.

Images