INTELLIGENT REMOTE SENSING SYSTEM FOR AUTONOMOUS ROBOT NAVIGATION IN DYNAMIC ENVIRONMENTS

Abstract

Herein disclosed are advanced intelligent remote sensing systems and methods for robots to navigate autonomously through dynamic / unstructured environments with only a reconnaissance sensor as a perception sensor and an inertial measurement unit as an orientation sensor. The proposed invention combines machine learning methods and computational navigation calculations in robot automation, to perform perception sensing, processing, and actuate a model robot predictive controller to enable autonomous robot navigation, while preserving user privacy and can be widely adopted by mobile autonomous robots.

Description

Description INTELLIGENT REMOTE SENSING SYSTEM FOR AUTONOMOUS ROBOT NAVIGATION IN DYNAMIC ENVIRONMENT Field of Invention Engineering The present invention relates generally to a system and method of an advanced intelligent remote sensing system for a mobile robot to autonomously navigate through a dynamic / unstructured environment using only a reconnaissance sensor as a perception sensor and an inertial measurement unit as an orientation sensor. The proposed system and method have the advantage of protecting the identity of surrounding humans because it optimizes the use of minimum sensors and can be widely adopted by any mobile robot. Background of the Invention The use of autonomous mobile robots has become increasingly advanced in recent years with the advancement of robot sensor capabilities. Currently, robot sensor capabilities are becoming richer and richer with more sensors embedded in robots to adapt to dynamic or unstructured environments. There are minimum requirements and / or capabilities of autonomous robots from both software and hardware perspectives. From a software perspective, minimum requirements should include the ability to handle computational complexity, such as how many operations the sensor processing algorithm requires and the ability to improve the reliability of interpretation; such as software interpretation problems; the difficulty of interpreting sensor readings as x-rays; and the difficulty of recognizing sensor errors, etc.Meanwhile, from a hardware perspective, at least, autonomous robots must have a field of view (FOV) and range, power consumption, reliability and hardware size, which must match the robot's payload and power capabilities. However, the challenges surrounding the usability of robot sensors are diverse. Autonomous mobile robots have been rapidly used for object tracking, guiding, and are widely used in understanding the surrounding / environment, recognizing objects and interacting with humans. Recently when robotics technology becomes more advanced, most robots use various sensors to perform sensing and locomotion. The most common practice of autonomous robots in performing “object detection” & “object tracking / following” is by using Camera & LiDAR sensors, as a minimum requirement. In some cases, there are other additional sensors used, such as Ultrasonic, Time of Flight (ToF) Camera, Radar, etc. With the emergence of autonomous mobile robots controlled through and / or by “Smart Devices”, there are growing concerns around the privacy of personal data collected by “Smart Devices”. When robots are used in public areas, there is a possibility that personal data can be exploited without the user’s consent. Conversely, when robots are used in private areas (such as homes, hospital rooms, etc.), people may feel uncomfortable with the presence of robots with cameras. Although robot technology is advancing towards personal and / or public assistant systems in the future, it is important to understand the privacy aspects surrounding their users. The proposed invention provides a system and method for performing sensor perception to generate real-time mapping, localization, and predictive control of a robot to enable tracking of moving objects, including maintaining distance to selected target objects in a dynamic environment. This includes maintaining user privacy aspects, when used in public or private areas. The technology disclosed in the invention can be applied to any type of mobile robot. There have been several patents and public papers exploring various machine learning technologies that complement computational calculations in robotic automation, such as real-time scene understanding by reconstructing maps, recognizing objects, and so on. However, no inventions have discussed methods on how to leverage all of these topics into an intelligent system that automatically performs perceptual sensing and drives predictive control of robots to enable autonomous robot navigation. The first prior art is Patent KR101726696B1 proposes a device, using a camera, to detect, recognize human identity when recognizing an environment, and follow moving objects in the mapped environment. While our proposed invention uses a machine learning (RNN) method to detect and recognize humans through the utilization of low-cost perception sensors and measurement sensors. The Second Prior Invention is US Patent 8830091B2 which proposes an apparatus and method for a mobile robot to perform object tracking, following, and guiding. This invention discloses a method and apparatus that uses visual sensors and computational sensors to process Simultaneous Localization and Mapping (SLAM), which will enable the operation of a mobile robot. In contrast, our invention does not use visual sensors to process SLAM, as our invention uses low-cost perception sensors and measurement sensors to create a dynamic map of the environment. There are also some studies related to techniques on how to calculate the momentum of moving objects, such as Kalman Filter and others. The Kalman Filter technique was recently disclosed in a publication by PR Gunjal, BR Gunjal, HA Shinde, SM Vanam and SS Aher, “Moving Object Tracking using Kalman Filter”. The International Conference on Advances in Communication and Computing Technology (ICACCT), Sangamner, 2018, discussed a method for detecting and tracking moving objects using the Kalman filter, which provides an estimate of the state of a process. Most of the above references only focus on providing some parts of our invention. Whereas, our invention proposes an intelligent robot navigation system and method that will combine machine learning methods and computational navigation to automatically provide autonomous navigation recommendations to a mobile robot, with only perception sensors and orientation sensors. Summary of Invention The present invention proposes a novel scheme of an advanced intelligent remote sensing system for a mobile robot to navigate autonomously through a dynamic / unstructured environment with a reconnaissance sensor as the perception and an inertial measurement unit as the orientation sensor. The present invention proposes a novel method of enabling an autonomous robot with minimum sensors, 1 perception / reconnaissance sensor & 1 orientation sensor / IMU, to perform perceptual sensing and drive a predictive controller of the model robot by providing the autonomous robot and its target object estimation from past, present, and future states.The present invention will provide real-time mapping and localization using the Simultaneous Localization and Mapping (SLAM) method and calculate a series of observed measurements over time to predict the movement trajectory of a targeted object while generating a trajectory path and maintaining position control of a mobile robot with respect to the selected target object. In this system, we detect all moving objects in an environment through Recurrent Neural Network (RNN). The RNN model will learn the movement pattern of a particular object from a set of data from a rangefinder with object position labels. The RNN model estimates the location of objects around the robot. With the position information, the distance measured between the robot and the selected target object, the system estimates the object’s trajectory to handle challenges such as the target object being out of range of the autonomous robot’s sensors. The object’s trajectory is derived from the position and velocity vectors, which are calculated by methods such as the Kalman Filter using historical data of the target’s movement and velocity. The system continuously processes and learns from the information fed into it to provide the autonomous robot with navigation to maintain the robot’s position with the selected target object. This invention also protects user privacy, including user identity because the system will not use visual sensors to identify, recognize, and understand all objects in an environment. INVENTION FEATURES This invention has two main features, which are better than conventional schemes, namely: First, the present invention will combine machine learning methods and computational model navigation by understanding and processing all information captured from perception sensors and orientation sensors, by: - apply reconnaissance sensors as perception sensors and inertial measurement units as orientation sensors; - utilizes the Simultaneous Localization and Mapping (SLAM) method to generate real-time mapping and localization of object coordinates such as distance and degree in the environment based on inputs generated by perception sensors and orientation sensors; - apply Recurrent Neural Network (RNN) to perform object detection in the environment by analyzing object movement patterns; - generate trajectory estimates by utilizing predictors using historical data of target movement and speed. Second, the present invention will provide recommendations for robots with autonomous navigation models to automatically navigate robots in dynamic and / or unstructured environments, by: - generate and continuously update target object trajectory predictions by calculating a series of observed measurements over time; - maintain control of the robot's position with the targeted object through various action recommendations to perform specific autonomous tasks such as tracking, following, and / or guiding the targeted object. Short Description of Image In order to understand the present invention and to see how it may be implemented in practice, several embodiments will be described with reference to the accompanying drawings, and wherein: Figure 1 is an overview of the invention of Intelligent Remote Sensing System for Autonomous Robot Navigation in Dynamic Environment using Simultaneous Localization and Mapping (SLAM), Moving Object Detection (MOD), Moving Object Prediction (MOP), and Moving Object Keeping methods according to the present invention. Figure 2 is a flow diagram of the overall system. Figure 3 is an example scenario of using an autonomous robot for elderly care. Figure 4 is a sample visualization of the robot's perception and possible moving objects. Figure 5 is an example scenario of using autonomous robots in a retail store. Figure 6 is an example scenario of using an autonomous robot as a hotel receptionist. Figure 7.a. is an example of a scenario of autonomous robot usage in a building. Figure 7.b. is an example scenario of using an autonomous robot that rescues a victim and finds a way out. Figure 8 is an example of a case scenario where an autonomous robot finds a moving object and triggers an alarm / event. Figure 9 is an overall process diagram with illustrations. Figure 10 is a Simultaneous Localization and Mapping Diagram. Figure 11 is an illustration of how the system scans the environment and builds an initial map. Figure 12 is an illustration of how the system sets the robot's initial position. Figure 13 is an illustration of how the system scans the environment and updates the map. Figure 14 is an illustration of how the system updates the current position of the robot. Figure 15 is a Multiple Object Tracking (MOT) Diagram. Figure 16 is the input of the Moving Object Detection (MOD) Diagram. Figure 17 is a Detailed Diagram of Moving Object Detection (MOD). Figure 18 is an illustration of Object A being blocked by Object C. Figure 19 is an illustration of the object's movement trajectory. Figure 20 is a sample blind spot area. Figure 21 is a diagram of keeping moving objects. Figure 22 is a Process Diagram for Keeping Moving Objects. Figure 23.a. is an illustration of how the system defines the initial moving object properties. Figure 23.b. is an illustration of how the system updates the properties of a moving object. Figure 23.c. is an illustration of how the system adjusts the robot's next position to avoid missing the target object with the motion equation. Figure 24 is an illustration of how the robot moves when the target object is blocked by another object while moving. Figure 25 is an illustration of how the robot moves when the target object has an unknown position point. Full Description of the Invention The preferred embodiments and their advantages are best understood by reference to FIGS. 1 through 25. It is to be understood, therefore, that the embodiments of the invention described herein are merely illustrations of the application of the principles of the present invention. Reference herein to the details of the illustrated embodiments is not intended to limit the scope of the claims, which themselves enumerate features considered essential to the present invention. Referring now to Figure 1, an overview of the Intelligent Remote Sensing System for Autonomous Robot Navigation in Dynamic Environments, hereinafter referred to as the Intelligent Robot Navigation System, according to the present invention is shown. As described in Figure 1, the present invention provides an intelligent robot navigation system that can autonomously navigate a robot through a dynamic environment using a minimum number of sensors. Generally, in the development of robots to obtain precise and accurate judgments and decisions, many sensors are installed on the robot body. However, there are several factors such as privacy, security, and cost that need to be considered; therefore in the present invention we only use surveillance sensors as perception sensors and inertial measurement unit sensors as orientation sensors. As described in Figure 1, there are three stages of the process in this invention; understanding, processing, and driving all available sensors and all data captured by the autonomous robot intelligent system. In the first stage, the Intelligent Robot Navigation System will perform sensing to obtain real-world conditions using reconnaissance sensors as perception sensors and inertial measurement units (IMUs) as orientation sensors to control the position and movement of the robot. In this part, all sensor data will go through an information extraction process to be fed to the next stage.In the second stage is the Intelligent Robot Navigation Recommendation System consisting of three main modules, Simultaneous Localization and Mapping Module, Multiple Object Tracking Module, and Moving Object Keeping Module, where the main focus of this module is to interpret real-world conditions by reconstructing maps without prior knowledge of real-time locations, including identifying objects, determining the trajectory of moving target objects, and finally providing recommendations for the robot navigation model. In this last stage, the robot will execute the results of the recommendation system, by performing certain tasks assigned to the robot, such as tracking, following / guiding target objects, and so on. Referring now to Figure 2, it depicts the overall system flow diagram of the intelligent robot navigation system. In the initial stage the user selects the robot’s main task from the available user interface (following, guiding, etc.). When the robot is started and ready, it will collect data from the surveillance sensors and the inertial measurement unit sensors. The robot will use both inputs to build a map and localize the robot’s position using SLAM as well as detect moving objects and track their position and momentum using MOT as explained earlier. Then, the robot will be given a target moving object to be maintained by the user. When the target moving object is out of reach, the robot will move to a certain position to get a better perception of the target moving object. If the task is completed the robot will stop maintaining the target object, otherwise the robot will continue to perform the maintenance procedure and repeat from the beginning. Referring now to Figures 3 to 8, various user scenarios of how the Intelligent Robot Navigation System is used. The present invention will enable autonomous robots to be used in various situations in human daily life, not only limiting the use of robots in factories to perform pre-programmed repetitive tasks. In factories, robots are designed and developed to perform specific repetitive tasks. Robots in a factory do not have the ability to perform an action with the situation that occurs in the environment. For example, in a car manufacturing plant, robots are used to weld several components to the chassis of the car. The robot will weld the components repeatedly, because that is its main task. When something happens, say an accident, the robot cannot react to stop the process, unless the operator turns off the robot. When a robot needs to interact with humans; it needs perception capability to understand the dynamically changing environment. Then intelligence needs to be added to make the robot perform actions, which can be a step for the robot, trigger a warning, or provide guidance or assistance. The present invention does not limit the actions of the robot, but the present invention allows autonomous robots to be able to have environmental perception, track or detect moving objects, and then perform some actions. Referring now to Figure 3, it depicts a sample scenario of autonomous robots used for elderly care at home. The need for robots at home may increase, due to changes in human lifestyles. The phenomenon of having small family members is increasing, and in some cases, new couples are having fewer children. Three-generation households were common in the past. Family members had more interactions by providing assistance to each other or sharing some tasks. For example, a grandmother / grandfather takes care of his / her grandchildren or grandchildren take care of their grandparents, by reminding the elderly of their medications, guiding them when they move, or simply chatting with them. But nowadays, when the number of family members is decreasing, the presence of autonomous robots that have the ability to take care of, take care of the elderly, act as butlers, or simply accompany family members will be very important. As illustrated in Figure 3, the main task of autonomous robots is to take care of the elderly, who are left alone at home because other family members may have to be out of the house or because they have their own errands. Both parents need to go to work, children are needed to go to school, and the elderly have to stay at home alone. Autonomous robots are expected to understand the layout of the house, understand and be able to follow moving objects, and autonomous robots can also act as walking aids to help the elderly with walking difficulties. Referring now to Figure 4, it visualizes the perception of an autonomous robot and the possible position of a moving object. When the robot is started or turned on, it will start using the surveillance sensor to measure the distance between its position and all objects surrounding the robot. The robot will then try to move around the room to have a better perception of the layout of the house using the SLAM method. At the same time, the robot will detect when there is a moving object or not. The method used to detect moving objects is to use surveillance data, converted into one-dimensional data, then analyzed using RNN to determine the possibility that the scanned object is a moving object. The result of the RNN process is an object with a probability value that the object is moving.The map visualization and possible objects are shown on the mobile device as shown in Figure 4. The robot can then be tasked to keep a distance from one of the objects. The robot will follow the object, which in this case is an elderly person, and ensure that the elderly person is safe at home. The robot can also be programmed to provide guidance, for cases where the robot needs to care for an elderly person with physical disabilities. The robot can maintain a certain distance, ensure that the robot can be used as a walking aid, and provide guidance when the elderly person needs to move between rooms in the house. Referring now to Figure 5 and Figure 6, illustrates the scenario of autonomous robots used in retail stores as salespersons or as receptionists in hotels or public buildings. When entering a public building, especially for the first visit to the building, people will ask the reception or information staff when they are in a hurry to find a certain room in the building. Most buildings provide a directory with a map of the building. But for most people reading this guide, understanding the map, and remembering the way to reach the destination is not easy. People feel more comfortable and easy by asking the information staff, and then the staff can easily provide guidance to where they are going. For some situations, it is even more convenient for people, if the reception staff can provide guidance assistance until they can reach the room.This kind of service is part of the much needed hospitality services for hotels, hospitals or retail stores. It generates additional costs to train staff, and manage them. Some public services, such as hotels, bus stations, airports, even need to have 24-hour reception or information staff. In this illustration, the hospitality industry can be easily replaced by autonomous robots. With this invention, autonomous robots can be trained and programmed to perform specific tasks. As a use case scenario, the robot can be used in a building to scan the entire room and layout. Using surveillance sensors, the robot can build a map of the building. Once the map is developed, the robot can be trained to know the name of each room in the building. Interaction, either by voice or touch mechanism, can be used to make the robot understand its task. The user can say the name of the room, and the robot can translate the voice into text, and find the matching room name in the building. Alternatively, the robot can be equipped with a display (such as an interactive directory), and the user can touch the room / destination he / she is looking for. The robot can then start providing guidance to the user to the desired room / place.The robot will start detecting moving objects and keep its distance from the moving objects. Referring now to Figure 7a & b, it depicts how autonomous robots are deployed in disaster areas. Every day disasters occur around the world. Disasters can occur due to natural, technological, or social hazards. Some examples of common disasters that occur around us are fire, flood, and earthquake. To prevent greater casualties, immediate action is needed. For example, when a fire breaks out in a building / house, a fire brigade team is usually deployed to find victims trapped inside the building. More lives are expected to be saved when the team tries to search every room of the building. In this situation, the rescue team is actually in a dangerous position. Many of them can get injured, or even lose their lives during the process. The building conditions can be toxic, flammable, have thick smoke, and may have materials that collapse at unexpected times.There are also rare disasters that occur, but result in high hazard environments. For example, when the earthquake occurred in Fukushima and the tsunami hit the reactor. The radioactive radiation emitted from the reactor made it impossible for humans to enter the building to analyze the victims, and make repairs to the nuclear power plant, or to search for victims and evacuate them from the nuclear power plant. These situations, where human lives can be at risk, can be prevented by using robots. Robots with special specifications can be made, for example, heat-resistant robots, robots that can handle the radiation emitted from nuclear reactors, etc. The purpose of the robot is only to scan all the rooms in the building, detect possible victims, and provide instructions for evacuation from the building. In this illustration, it explains the performance of autonomous robots when used in disaster areas. First, the robot will start the surveillance sensor, and start mapping the building. After a part of the building map is generated, it will move around the known area, until the unknown is to complete the map. During the movement of the robot, the same method of detecting an object is to move the object being calculated using the RNN method. Surveillance sensors have advantages over other methods of detecting objects, because the surveillance results are not disturbed by smoke. If the robot uses a camera to detect an object, the robot will have difficulty detecting anything because of the dense smoke. The robot's perception results can be transmitted wirelessly and displayed on the screen of the mobile device. When the robot detects a moving object, depending on the MOT probability result, the robot can approach the target, or continue to search the entire room in the building.In case the robot detects a high probability of a moving object, and the robot approaches the object, a button or voice can be used to confirm that the victim needs help as shown in Figure 7a. The robot then tries to find a route to the building entrance, based on the route / path that has been memorized since the robot entered the building. During the movement, the robot will keep a distance from the moving object, to ensure that the victim can follow the robot as shown in Figure 7b. Referring now to Figure 8, it illustrates how an autonomous robot is used as a surveillance tool and triggers an alarm or event when it finds a moving object. Many housing complexes in urban areas are left empty during working hours, as all residents are doing activities outside the home. Both parents are now working and have to spend more time in the office, rather than at home. Children also go to school in the morning, and return home in the afternoon or evening. Their homes will be left unattended in the afternoon. Conversely, there are many buildings that are abandoned at night. Offices, warehouses, schools, and many other public buildings use security guards to secure the building. The guards need to patrol on a schedule, moving from one room to another, checking to make sure no one is entering the building without permission. This repetitive task needs to be done every day, repeating every few hours at night.Meanwhile, for humans to work every day at night continuously will have an impact on their health problems. Humans naturally rest at night, spending six to eight hours to sleep. That is why for security officers, they usually have two shifts to work, during the day and at night. Their scheduled work is also rotated regularly, to prevent them from getting health problems from working all night every day. This task is suitable to be replaced by surveillance robots. Robots do not need to sleep at night, and they will not get tired even when they do repetitive tasks. Using robots also reduces safety risks when they find suspicious objects / people. Robots can be made more resistant to dangerous weapons, such as knives, explosives, or tear gas. People can be injured by these dangerous weapons, but not by robots.Robots can also be equipped with alarm or trigger systems, which automatically send data to the server for further analysis and recording. If the robot is unable to prevent an accident, the alarm and log data have been sent to the server. This data may be useful for further investigation. As shown in Figure 8, in this case an autonomous robot is deployed in a building or house. The robot will start using surveillance sensors to build a map and start learning the results of the sensor data, to identify objects in the room. The 360-degree data from the surveillance will be processed by the robot to identify objects. The robot will move from one room to another, to create a complete map of the building. Once the map is completely created, it will be used as a reference for objects in the building. The next task for the robot is to patrol the building. As the robot moves, the surveillance will collect data simultaneously. The resulting data will be compared again to previous data (as a reference), using RNN to determine if there is a moving object. The result of the RNN is the probability that the object is a moving object. A certain threshold can be set to determine that the robot needs to keep a distance from the object.When the robot finds a moving object, additional actions can be taken by the robot, such as sending an alarm event to the server or taking a photo of the suspected object. The moving object is then set as a target, and the robot must maintain a distance from the target. When the target object moves, the robot may remain in its position if the robot and the target are still within a certain distance. The robot is allowed to follow the target object, if the object moves away. The robot can predict the path and trajectory of the moving object, using the surveillance data collected from the sensors on the robot. As disclosed in Figure 9, shows the overall process flow diagram and its use in the system. The present invention is an overall system that combines machine learning methods and computing modules including Simultaneous Localization and Mapping (SLAM), Multiple Object Tracking (MOT), consisting of Moving Object Detection (MOD) and Moving Object Prediction (MOP) modules, and Moving Object Guarding methods to generate trajectory recommendations for autonomous robots to guard selected target objects. As disclosed in Figure 9, the system will generate a map using the SLAM module at position n=0, where no moving objects are detected by the system. At position n=1, using the Moving Object Detection module, it is depicted that there are possible movements of four objects, where all four objects have moved, compared to their positions at time n=1.Therefore, at the time n = 2, the system will recognize that there are three moving objects, namely objects A, B, C, with a high probability based on their movements using the MOD module and object A is selected by the user as the target object to be guarded. The system will generate a trajectory prediction for each object until position n = 7. In the MOP module as shown in Figure 9, it can be seen that at the time n = 7, object A will pass through the outer boundary and its position will be blocked by objects B and C at the time n = 4. By knowing the conditions as above, the robot will generate a map trajectory for itself in maintaining its distance from the target object A. The map trajectory generated by the robot can be seen in the Moving Object Guard module section as shown in Figure 9, where R0, R1, up to R7. After generating the map trajectory, the robot will move and follow the map trajectory.This process will continue to be repeated to maintain the distance between the robot and the target object A. As shown in Figure 10, shows the Simultaneous Localization and Mapping (SLAM) method used in the present invention. Simultaneous Localization and Mapping (SLAM) is the concept of constructing a map of an unknown environment by an autonomous robot while at the same time navigating the environment using the map. In SLAM both the robot’s trajectory and the map are estimated on-line without the need for prior knowledge of the location. The SLAM module will enable the autonomous robot to perceive its environment through input data from sensors. The proposed invention uses two sensors. Namely a rangefinder sensor as the perception sensor and an inertial measurement unit (IMU) as the orientation sensor. Perception can be done using only rangefinder sensors such as LIDAR, sonar etc. A rangefinder is a surveying instrument for quickly determining the distance, bearing and height of distant objects.Using distance data for multiple positions can provide the robot with information about its environment and use an Inertial Measurement Unit (IMU) for orientation. An IMU is an electronic device commonly used in robots to measure the angular velocity and orientation of the robot's internal body, using a combination of accelerometers, gyroscopes, and sometimes magnetometers. This unit can provide the robot with information about its position in its environment. Referring now to Figures 11 through 14, the process sequence of the disclosed SLAM method used in the present invention. In the first step, the autonomous robot will scan the environment using the surveillance sensors and construct a map. As disclosed in Figure 11, as a first step of the method, the present invention will obtain input from the surveillance sensors, and then construct a map based on that input. The surveillance sensors obtain ranges of nearby objects around the robot, so the right figure shows what the environment looks like from the robot's perspective. We can see that the robot cannot detect object E because it is blocked by object D. Referring now to Figure 12, reveals the second step of the present invention on how the autonomous robot sets its initial position (x0, y0). This process is actually being carried out simultaneously with the first step, as shown in Figure 11, but for ease of understanding we separate them. In this step, the robot initializes its position (defined as (x0, y0)). Referring now to Figure 13, discloses the third step of the present invention on how an autonomous robot re-scans the environment as it moves, and updates the map when it finds new information. For example, the robot moves from (x0, y0) to another point. Then the robot's perception will change as follows. After moving to the new point, the robot gets a new perspective of the objects. It gets the right side of objects B and D so that it gets the full shape of objects B and D. It can also now detect object E. This new perspective is then updated to the map stored in the robot. Referring now to Figure 14, the final step of this method, shows an illustration of how an autonomous robot will update its current position using the IMU. Similar to the second step, as shown in Figure 12, this step is actually performed simultaneously with the third step, as shown in Figure 13. In this step, the robot gets its current position using the IMU (say (x1, y1)). The robot lets the IMU know how long it has been moving and what direction it is moving. The third and fourth steps are repeated until the robot completes its main task. We can also see that the more the robot moves, the more perspectives it gets and that will result in a better map. Referring now to Figure 15, a Multiple Object Tracking (MOT) module, an essential component of the present invention, is disclosed, which comprises 2 main components: a Moving Object Detection Component, hereinafter referred to as MOD, and a Moving Object Prediction Component, hereinafter referred to as MOP. The Multiple Object Tracking, hereinafter referred to as MOT, takes distance data from the surveillance sensors to calculate the object position and the object momentum. The MOT uses the MOD module which uses a Recurrent Neural Network (RNN) to determine the object position. The object position is used by the MOP to determine the object momentum (speed and direction). Referring now to Figure 16, revealing the MOD diagram, the method of how the invention detects moving objects is shown. The invention uses a Recurrent Neural Network (RNN) to determine the probability of an object by analyzing its movement pattern. A series of object movement data is captured by a surveillance sensor. A series of such outputs, containing the location information of the object, are fed into the RNN network. The RNN network will determine the probability of an object at a certain location by analyzing their movement patterns. Each object (e.g., human, car, and animal) has a different movement pattern and the RNN neural network learns to recognize objects from their movement patterns. Referring now to Figure 16, it illustrates how in a circular space there are 3 objects, namely object A, object B, and object C. By using a distance sensor, the autonomous robot scouts the entire surrounding area producing distance data from 0 ° to 360 °. Each distance data obtained is normalized to the reference point, as shown in Figure 16, the robot is right at the reference point and then the information is fed into the RNN network. Changes in data distance are caused by the movement of moving objects such as human movement that has a certain movement pattern. Humans will not move with a random pattern, but their movements have a certain pattern, such as the pattern of human movement when running or walking. The RNN network will learn or find this movement pattern by performing a series of analyses of changes in data distance.The limitation of the proximity sensor that only captures distance information in one dimension is a challenge to distinguish and follow the movement of an object in the surrounding area. Problems such as two or more objects diverging from each other can be solved by calculating their historical movement data. The RNN network has a state vector that records all relevant historical information obtained from previous inputs. This state vector will help to follow the target object when an intersection occurs. Referring now to Figure 17, it illustrates how the present invention uses RNN to detect moving objects. As an illustration, an autonomous robot is used in a circular room with three objects (object A, object B, and object C). At tn, no changes are detected by the surveillance sensors, therefore the probabilities of objects A, B, and C are zero (0). At tn+±, the RNN will detect changes from the surveillance sensors for objects A and C. Since the RNN has learned the pattern of changes (motion patterns), the probabilities of objects A and C will approach 1. On the other hand, the probability for object B is zero since no changes are detected in object B. To train the RNN network to recognize objects, first, a series of outputs from the surveillance sensors are fed into the network. Each of these sets of sensor outputs is labeled with the position of the object.By forward propagating the output of the surveillance sensors and back propagating error corrections of the corresponding labels, the network will eventually be able to locate the object. Referring now to Figures 18 and 19, various illustrations of Moving Object Prediction are shown. The problem of tracking objects using a surveillance sensor is when an object is blocked by another object, as shown in Figure 18, where object “A is positioned behind object “C”. The surveillance sensor is unable to detect it; therefore we lose track of object A. To solve the problem, we need to predict and obtain the next possible vector position of each object in the blocked area or we can name it as moving momentum of the object. By knowing its momentum, we will know the tendency of direction and velocity of an object. In this invention, we generate object trajectories derived from the results of the prediction method. In this invention we use the Kalman filter method, one of the well-known methods for predicting and estimating object movement. The predictor uses each object's position and velocity from the previous method as input vectors for the predictor and gives you the next estimated position and velocity vector. When estimating the next vector, the predictor also updates its prediction matrix to get a better prediction. To generate the trajectory, we simulate the results of the prediction matrix repeatedly, depending on the length of the trajectory we need. The process will continue to repeat until the system stops. Referring now to Figure 19, it illustrates the trajectory of the object movement. In this illustration, X is the state matrix and in our case, X is the position and velocity of each moving object. The predictor will compute the historical data of the object's position and velocity, and generate the trajectory iteratively. The predictor simulates the results based on the prediction matrix. All steps are iterative and will be computed over time. Referring now to Figure 20, it illustrates how the Keep Moving Objects module is used to identify possible blind spot areas in an environment. In some cases, it may be difficult to keep the probability of the target object’s position always high when it comes to the blind spot area. The blind spot area is one of the most challenging aspects for a robot especially when it has only one type of perception sensor. Since the robot uses only the surveillance sensor as the perception sensor, the Target Object’s position may be blocked by other objects. As shown in Figure 20, it illustrates the use case scenario when the target object and other objects meet at the same waypoint and create a blind spot for the robot or when the target object’s position is blocked by some stationary objects. Referring now to Figure 21, the diagram of the Keeping Moving Object Module is revealed, to address the use case scenario when the target object moves into the blind spot area. This method uses SLAM and MOT as inputs and generates recommended actions for the robot. SLAM will provide information about the map and the robot’s position, while MOT will provide information about the possible position and momentum of the moving object. Based on the output of both methods, the robot can predict the target object’s movement trajectory and use it to check whether it will move into the blind spot area or not. When that is possible the robot will calculate where the next best robot position is to continue monitoring the target object’s position. Referring now to Figure 22, the step-by-step process of the Moving Object Guard Module is revealed. First, the system will continuously update the position and velocity of the target object and each moving object on the map. Then the robot will continuously update the movement trajectory of the target object over a period of time. The trajectory is based on the historical position of the object. When the robot potentially moves into a blind spot area, the robot will determine the next position of the robot based on certain environmental conditions. The following are some examples of use case scenarios that a robot might encounter: When the blind spot area has a temporary position and moves in the opposite direction to the target object, the robot's next position will be aligned with the predicted position of the target object. When the blind spot area is caused by a stationary object or an undetermined position on the map, the robot's next position is the position of the target object. The robot will build a trajectory to move and control its speed based on the robot's moving ability, speed, and the distance between the robot's current and next positions. The Keeping Moving Objects module will allow the robot to execute movement recommendations. Referring now to Figures 23.a. to 23.c, illustrates various use case scenarios of how the Keep Moving Object Module works. As shown in Figure 23.a, it depicts how the system defines the initial moving object properties. For the initial conditions, each moving object will have its own properties for position and velocity which are displayed as red text near the object. The initial autonomous robot position can be represented by [(Xobjectn, Yobject^, Vob / ect^. Where Xohjectn= The object's position on the X-axis at time n, Yobjectn= The object's position on the Y-axis at time n, and Vobjectn= The velocity value time n. Next, as shown in Figure 23.b, the system will continuously update the moving object's trajectory over a period of time, in this use case scenario, say at m intervals. Using the trajectory, it is possible that the target object moves into a blind spot at time n + m.Thus, the updated moving object properties will be represented by [(Xo / jyecftl+m, Fo / j / ecin+tJTo / j^ Where Xobjectn+m= Position of the object on the X-axis at time n + m, Yobjectn+m= Position of the object on the axis. Y at time n + m, and Vobjectn+m= Velocity value at time n + m. As the final step, as shown in Figure 23. c, the robot will set the next position to avoid losing the target object based on the internal robot properties (position, initial and maximum speed, etc.), target object properties (position, trajectory, etc.), other object properties (position, trajectory, etc.), blind spot areas, and other environmental conditions. In the present invention, we use Fpath(t) as a Function that constructs a trajectory path for the robot to the target point based on time and Motion(t) as a Function that controls the speed of the robot to the target point based on time. The system will calculate the moving path and method to move that satisfy the equation of motion as shown in Figure 23. c. ] Referring now to Figure 24, it illustrates the robot’s motion when the target object is blocked by another object while moving. When the target trajectory prediction may enter the blind spot area, this method will give the robot a recommendation to move to a certain point so that it will have a better perception to maintain the target position. Here are two common cases in the real world. As shown in Figure 24, we illustrate that object A is the target object and both objects C and D, from our trajectory system, potentially block the robot’s position perception of object A. In the first figure, the position of target object A is potentially blocked by the movement of both objects C and D. While in the second figure, the robot moves to a certain point to get a better perception and get a higher probability of position and momentum for the target object. Referring now to Figure 25, it illustrates a use case scenario of robot movement when the position of the target object is unknown. In this illustration, object A is the target object and object C is a stationary object that has space and the robot still does not know the condition behind it. From our predicted trajectory, Object A will move behind object C. In the first figure, the position of target object A is potentially blocked by object C and the robot still does not know the condition behind object C. While in the second figure, the robot moves to several points to get a better perception and get a higher probability of position and momentum for the target object. Although the present disclosure has been shown and described with reference to various embodiments thereof, it will be understood by those skilled in the art that various changes in form and detail may be made therein without departing from the essence and scope of the present disclosure as defined by the appended claims and their equivalents.

Claims

Claim 1. System and Method of an Intelligent Remote Sensing System for Autonomous Robot Navigation in Dynamic Environments for autonomously navigating a robot through a dynamic / unstructured environment with a reconnaissance sensor as the perception sensor and an inertial measurement unit sensor as the primary orientation sensing modality. The present invention will combine machine learning methods and computational model navigation by sensing, processing, and actuating all information captured by the sensors. The various modules in the advanced intelligent system are: Perception Sensing comprising a reconnaissance sensor as the perception sensor, a single or compact direct distance measurement sensor that uses electromagnetic or mechanical waves to measure distance; and an orientation sensor such as an inertial measurement unit as the measurement sensor;Simultaneous Localization and Mapping, a module to generate real-time mapping and localization of coordinate objects such as distance and degrees in an environment without prior knowledge of the location, which is activated based on inputs generated by perception sensors and orientation sensors; Multiple Object Tracking, the module serves as a prediction system to calculate a series of measurements observed over time to estimate the state of a process and / or object by calculating its momentum to provide recommendations for the direction of movement of the object and construct a trajectory based on its historical position; Keeping Object Moving, the module serves as a recommendation system to maintain control of the robot's position with a selected target object through various action recommendations to perform specific autonomous tasks such as tracking, following, and / or guiding the targeted object.; 2. The method of Claim 1, wherein the system shall implement a Multiple Object Tracking module, comprising the following entities: Moving Object Detection (MOD), a method utilizing a Recurrent Neural Network (RNN) to perform object detection in an environment by analyzing the object's movement pattern while preserving the privacy of user data collected by the method; Moving Object Prediction (MOP), a method used to calculate an estimate of the future process state and / or momentum of an object, wherein the output of the Multiple Object Detection module shall be used as an input of the predictor to generate subsequent estimates of the vector position and predict the target object's movement trajectory.

3. The method of Claim 1, wherein the system implements a Keeping Moving Objects module to continuously update the positions and velocities of both a target object and each identified object in an environment and provide a next point position for the autonomous robot, comprising the following steps: utilizing the output of the Simultaneous Localization and Mapping module to generate real-time mapping and localization coordinates of the autonomous robot in the environment and continuously updating the information; utilizing the output of the Multiple Object Tracking module to determine the position and momentum probabilities of all moving objects; performing automatic labeling for each moving object in the environment, wherein the object properties including position and velocity at the time of the initial state are captured; and identifying the position and velocity of the internal robot at the time of the initial state.The system will continuously measure the position and velocity of the observed moving objects and robots over time at certain intervals; generate a predicted trajectory path of the movement of the selected target object by calculating the historical data of the position, velocity, and movement of the target object to provide an estimate of the next position and velocity vector of the selected target object at certain time intervals; generate a predicted trajectory path of the next position of the robot, by calculating the internal robot properties including position, velocity, initial and maximum velocities, and all the properties of objects in an environment; to determine the trajectory path to control the direction and speed of the robot's movement to move to the selected target point at a certain internal time and at a certain distance between the current and next positions of the robot.process and learn all information continuously fed into the system to update the predictive trajectory path for the robot and the selected target object to maintain the robot's position with the selected target object.

4. The method of claim 3, wherein the selected target object moves into a blind spot area, wherein the predicted position of the selected target object cannot be determined, the system shall perform the following: Identify if the target object and other objects meet at the same waypoint, and / or the position of the target object is blocked by some stationary objects, and / or the map construction is not completely completed; wherein when the blind spot area has a temporary position and moves in the opposite direction to the target object, the next position of the robot shall be aligned with the predicted position of the target object; and when the blind spot area is caused by a stationary object or an undetermined position on the map, the next position of the robot shall be the position of the target object. Measuring and comparing the properties of the target object, the properties of other objects, and the internal robot properties in an environment to determine which space to cover; wherein the distance between the selected target object and the robot shall be set at a certain distance.

Images