An autonomous mobile system and method for enabling mobile devices to interact with mobile platforms

Abstract

The application relates to an autonomous mobile system and method for realizing interaction between a mobile device and a mobile platform, and belongs to the technical field of robot control. The system uses a mobile communication device as an intelligent interaction device with a person, uses a mobile robot with a specific purpose to expand the mobile ecology, and is composed of an autonomous mobile system with intelligent interaction, intelligent networking and intelligent driving capability in a specific scene. The autonomous mobile system comprises an intelligent mobile platform, a mobile communication device and a cloud server, so that the mobile robot with a specific purpose can reduce the cost by saving the intelligent interaction system, can support various intelligent devices, and has an ecological expansion capability, so that the system is more flexible and diverse in interaction with a person and interconnection with Iot devices, and service capability for a specific person in a specific occasion is realized.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of robot control, and relates to an autonomous mobile system and method for realizing interaction between a mobile device and a mobile platform. BACKGROUND

[0002] With the development of automatic driving technology, the technology for realizing automatic driving of a vehicle in a specific scene, in particular, in a closed scene (such as a mine, a port, a factory, a warehouse or the like) is relatively mature. With the development of perception and network connection technology, a vehicle that is autonomously operated through network scheduling can realize more complex applications, such as interaction and cooperation between vehicles in a mine or a port. However, such a vehicle can only be used in a dedicated field or a scene that does not need to interact with a person / driver on site, and cannot realize service to a person in any scene.

[0003] A vehicle or robot that can serve a person in a specific occasion is currently integrated with a man-machine interaction system, such as carplay, harmonyOs, linux or the like, which is directly installed to a robot system and realizes interconnection and cooperation of multiple machines based on an ecological system of the system, but can only execute a predetermined interaction mode of the robot, is not flexible enough, and needs to be randomly deployed, resulting in repeated investment and waste.

[0004] Patent No. CN108281143A discloses a robot system that is remotely controlled by using a mobile phone APP, and is interacted through a mobile phone or a mobile device, mainly through a wireless mode. Such a robot system that can only execute a relatively simple task is not suitable for an application that needs a large amount of perception ability interaction and fusion, in particular, a system that needs to fuse different perception sources, comprehensively understand a command and a task, and execute an autonomous complex mobile task. SUMMARY

[0005] Therefore, the application aims to provide an autonomous mobile system and method for realizing interaction between a mobile device and a mobile platform, using a mobile communication device such as a mobile phone to realize driving in a specific area through temporary connection of a mobile platform, and having an interaction ability with a person.

[0006] To achieve the above-mentioned purpose, the application provides the following technical scheme.

[0007] An autonomous mobile system for mobile device and mobile platform interaction, including a smart mobile platform P, a mobile communication device M and a cloud server S. Using a mobile communication (handheld / wearable) device M to connect with a smart mobile platform P, it can realize automatic driving in a specific situation, has the networking of the local area, environmental perception and mapping in the area, and mobile obstacle avoidance capability. Support through the cloud or the given planning trajectory of the smart mobile device, and feedback the real-time map of the environment to the cloud or the smart mobile device.

[0008] Mobile communication device M, after connecting with the robot mobile platform, wired and / or wireless, through the device capabilities of the smart mobile device, such as:

[0009] a) Camera for facial expression features, body movement features, and capture and recognition of environmental targets interacting with people;

[0010] b) Microphone for capturing and recognizing human speech and external sounds;

[0011] c) Touch screen for human operation and current running state interaction display;

[0012] d) Network (wifi, Bluetooth, mobile communication, USB interface) for data interaction and transmission;

[0013] Send the person's instructions, application settings, and the person's habits and other information to the cloud, and send the set trajectory to the smart mobile platform P, which is automatically executed by the mobile platform.

[0014] The cloud server S collects information from the personal mobile communication device M and the smart mobile platform P, as well as information from other smart device users in the area, and performs statistics and planning according to the business, thereby realizing targeted services.

[0015] Further, the smart mobile platform P includes an energy system P1, a mobile system P2, a perception system P3, and a first wireless communication system P4, for automatically and safely driving along the command trajectory in a specific scenario;

[0016] The mobile communication device M includes an image display and interaction system M1 (such as a touch screen), a sound interaction system M2, an image perception system M3, and a second wireless communication system M4, to realize interaction with the user, including interaction with the user through APP and other tasks and commands, and information sharing between mobile devices and mobile communication devices in the mobile ecosystem, such as location, time, schedule, preferences, etc., so that the mobile trajectory planning can be optimized by the server among multiple mobile communication devices;

[0017] The cloud server S includes but is not limited to a mobile application server S1 and an interactive task server S2, which are scheduled according to the requirements of the task, and the management of the task, including the mobile application service of the APP, the basic service of the mobile communication device, and the interactive task service, etc.

[0018] Further, on the basis of the basic mobile capability of the intelligent mobile platform P, the human-computer interaction capability of the mobile communication device M is added, and the process of the mobile service task is realized, which specifically includes:

[0019] 1) The intelligent mobile platform P has an environment perception system P3, including the sensors of the platform, such as cameras, laser radars, ultrasonic sensors, millimeter wave radars, etc., which can detect the running environment. The environment perception is performed by using the perception system P3 of the intelligent mobile platform P, the motion decision and the motion planning are performed by using the cloud server S, and the motion is executed by the mobile system P2, which includes the actuators of the platform, such as drive motors, steering motors, braking systems, and indication systems, etc., which can make the intelligent mobile platform P safely run according to the task of the cloud server S, i.e. the planned trajectory;

[0020] 2) After the mobile communication device M and the intelligent mobile platform P are connected through the support and the installation system P5, and the second wireless communication system M4 and the first wireless communication system P4, the interaction with the user needs to be considered for the specific task service, which specifically includes:

[0021] a) The perception system (M1, M2, M3) of the mobile communication device M, such as cameras, microphones, touch screens, and network, etc., perceives the character state and command of the user, such as language, action, emotion, etc., until the user obtains the character and the interactive command input by the mobile communication device M or the Iot device associated with the mobile communication device M;

[0022] b) The character and the command input are fused with the environment perception information, which can understand the character and the command in the current environment, so as to determine how to execute the task, and input the task decision to the motion decision module and the interactive decision module;

[0023] c) The motion request of the task decision module is combined with the environment information, and the intelligent mobile platform P decides how to move, and the planning control module controls the actuators of the intelligent mobile platform after considering the actions to be performed by the intelligent mobile platform P;

[0024] d) The interactive request of the task decision module is divided into three parts of action decision, expression and content interactive decision, and voice decision, wherein,

[0025] i. Action decision, such as the action of mechanical arm, head, etc. affecting the balance of movement, needs to be input to the movement control module for comprehensive stability control, and the action control of the mechanical arm and head is executed;

[0026] ii. Expression and content interaction decision, mainly to control the touch screen or state light display device of the interactive device;

[0027] iii. Speech decision, mainly for the management of continuous dialogue, output by the sound executor;

[0028] 3) Limited to the resources of intelligent mobile platform P and mobile communication device M, complex calculation needs to be calculated by cloud server S with large data, including environmental perception, character perception and fusion, complex task decision and planning, etc. Especially in the case of combining other information, comprehensive coordination control within the scene, such as passenger flow service in the site.

[0029] Further, the complex task decision and planning includes comprehensive task decision and planning based on scene understanding, wherein the basic map model for task understanding and decision planning is in the form of hierarchical map, as shown in Figure 5 Corresponding to the route planning, trajectory planning and movement decision content of executing task, and the tracking and following of interactive target. Among them, the hierarchical map includes basic topology layer map1, real-time dynamic layer map2, mobile map layer map3, interactive target layer map4 and operation map layer map5. The basic topology layer includes three types of information, as shown in Figure 6

[0030] 1) Node: the position / location associated with the current executing task in the map, and the corresponding attribute description. Within the scope of action area, the node can be used as the target place, the interest point with certain use, and the basic information of the map. The node will be given corresponding attribute information, such as:

[0031] a) Position information: used to define the position, find the node, and share the node position among multiple devices;

[0032] b) Use information: used for task and scene matching, and stage use related information;

[0033] c) State description of node content, including general public attribute information such as use, opening time, capacity, activity information, etc. Also including new attributes or marks specified by the user through the mobile communication device M, such as event occurrence place, restrictions, etc.

[0034] ​2) Topology: including the connection topology between nodes in the map, also including the rules that can describe the topology relationship and the characteristic information or attributes of the node position.

[0035] Topology is the driving route connecting the nodes in the defined range, and the characteristic objects on the route for identifying rules and positions, and also includes the line and the attribute information of the rules and positioning features. The attribute information specifically includes:

[0036] a) Route attributes: mainly describe the link between the route and the node and the available state information, etc., which can be temporarily assigned to congestion, maintenance, and other traffic information.

[0037] a) Rule objects and attributes; lane lines, edge lines, traffic rule description information such as traffic lights and signs on the route for describing and limiting rules, and temporary obstacle indicating rule understanding information, and also includes the marking of new rules specified by the user through the mobile communication device M, such as speed limit, detour, etc.

[0038] c) Positioning facility attributes: description of facilities that can assist in positioning on the road or road side, such as trees, road signs, buildings, etc., also including the attributes of the facilities specified by the user through the mobile communication device M.

[0039] 3) Association relationship: attribute association relationship between nodes and topology, such as replacement, same type, priority, etc. Association relationship for real-time planning or adjustment to get the optimal route, especially for the decision information of the optimal route under different personalized selection or limitation conditions. Also divided into public attribute association relationship and specified attribute association relationship.

[0040] The autonomous mobile system performs global route planning based on the global basic topology layer, and formulates a moving route from the current position to the target node, and can perform real-time following adjustment according to the actual movement and target node adjustment.

[0041] Further, under the guidance of route planning, the autonomous mobile system moves safely and stably in a specific scene, and the environmental information perceived by the mobile communication device M and the intelligent mobile platform P is fused in real time to generate a real-time dynamic layer map2, as shown in Figure 7 The expression system or user's surrounding real-time road and obstacle situation is used for real-time running trajectory planning; the perceived environmental information specifically includes:

[0042] 1) Node and position characteristics map2-1, used to assist in accurate positioning and match to the elements of the basic topology layer map1. Through the matching of such characteristics and positions, it can be ensured that the local running trajectory conforms to the overall planned route.

[0043] 2) Road and feature map2-2, to describe the traffic rules in the area where the system can pass, through the ground in the environment, the identification of the line / object, the identification and positioning of the reference facility, and the description of the attributes of the current road traffic, such as whether it can pass, the range of the passing area, the speed limit, etc. It also includes the association between the road and the rules, such as the traffic switching triggered by the traffic light, etc.

[0044] 3) Static target map2-3, static target around the vehicle / user, including content and relative position, and additional attribute information, generally divided into 2 categories:

[0045] a) Static obstacles, which can be used to describe the range boundary and the static objects that may collide;

[0046] b) Task-related operations or indicators, static objects associated with the current task, generally specified by the APP of the mobile communication device M, and need to interact with the task or the action of the user, such objects need to be identified in the real-time dynamic layer map2.

[0047] 4) Dynamic target map2-4: dynamic objects around the vehicle / user, including content and relative position, motion state, additional attribute information, and special target identification; dynamic objects are generally divided into 3 categories:

[0048] a) Moving objects, identify and judge / predict the moving trajectory, used to judge whether a collision will occur;

[0049] b) Moving people, identify and judge / predict the moving trajectory, and need to identify the identity of the additional person and the action information through the mobile communication device M, as matching corresponding information of the interactive target layer map4;

[0050] c) Objects interacting with people, identify and judge / predict the moving trajectory, and are associated with the person, assisting in judging the behavior of the person, as matching corresponding information of the interactive target layer map4;

[0051] 5) Risk target: based on the current position, route and motion state of the vehicle / user, etc. Information, predict and estimate the possible collision risk, predict and track static targets and dynamic targets, identify the mutual influence and association of the motion trajectory in the real-time dynamic layer map2, such as avoidance, etc., for real-time obstacle avoidance decision and planning;

[0052] Based on the information of real-time dynamic layer map2, including the current position, route and motion state of the vehicle / user, considering the current task and the spatial position of the interactive characters and operating objects, and meeting the set restrictions, such as prohibited passage, the optimal running track is obtained for execution according to the priorities of safety, interaction / following, stability and comfort.

[0053] Further, when the autonomous mobile system plans the moving track for a relatively long time, and needs to perform emergency actions of the mechanism, it needs to identify:

[0054] 1) Ground state map3-1: Generally a three-dimensional real-time map, planning the driving or landing position in a short distance;

[0055] 2) Risk target map3-2: In a short distance range, it provides higher accuracy and more real-time tracking of risk targets to perform emergency avoidance actions.

[0056] After considering the adjustment of the user following and the influence of the arm action on the overall balance, the autonomous mobile system adjusts the track execution based on the ground state and risk, and its planning control is mainly based on the information of the moving map layer map3, combined with the demand of interactive action and operation action to obtain the comprehensive result.

[0057] Further, the autonomous mobile system establishes the interactive target layer map4 by maintaining the interaction with the user, the connection relationship between the mobile communication device M and the mobile platform P, and the tracking of the user, and matches the positioning of the intelligent mobile platform P to the corresponding position of the real-time dynamic layer map2, as shown in Figure 8 Based on the identification, tracking, planning of the vehicle movement state and the understanding content of the interaction of the user by the interactive target layer, including:

[0058] 1) User map4-1, including the identification and confirmation of identity and characteristics, and the identification and tracking of associated people and objects;

[0059] 2) User associated object map4-2, including the identification and tracking of objects indicated by action and language;

[0060] Through the sensing device of the mobile communication device M, including the operation of the support and installation system P5, the tracking and following of the interactive target are maintained, the command and command state to be executed are comprehensively understood, so as to adjust the task, which also includes the direct operation input by the user through the operating mechanism of the mobile platform.

[0061] Further, when the autonomous mobile system is an operating robot with a mechanical arm, a corresponding operating map layer map5 is established for the robot. Based on the operating target, the static target map2-3 and dynamic target map2-4 corresponding to the real-time dynamic map of the real-time dynamic layer map2, and the execution capability of the mechanical arm, the operation is planned and the state of the progress is divided into 3 parts:

[0062] 1) Spatial obstacle map5-1, used for spatial obstacle avoidance of the mechanical arm during the movement of the system, so as to indicate the special space and action requirements when passing through the obstacle, and also as an input for movement and operation adjustment when passing through the obstacle;

[0063] 2) Operating target map5-2, used to indicate the characteristics of the target and the process and limitations of the operation, so as to indicate the operation requirements of pick-up, carry and drop-off at different stages, and also as an input for estimation and adjustment of the center of gravity and balance at different stages;

[0064] 3) Operating space map5-3, used to indicate the starting position and the posture of the mechanical arm operation, especially the association with the obstacle.

[0065] Further, the interaction of expressions, contents and languages of the interactive decision is a reaction according to the perception and understanding of the contents, and the action, including the head and operation, will have an impact on the balance of the whole machine; after considering the adjustment of following the user and the impact of the mechanical arm action on the overall balance, the autonomous mobile system adjusts the execution of the trajectory based on the ground state and the risk, so as to be able to adjust and maintain driving along the planned trajectory in real time.

[0066] Further, the planning control module mainly plans the execution based on the information of the moving map layer, combined with the requirements of the interactive action and the operation action for comprehensive stability control.

[0067] The application has the beneficial effects that: the application provides a framework method, which can use a mobile communication device such as a mobile phone as a smart interaction device with a person, expand a specific-purpose mobile robot by using a mobile ecology, and form an autonomous mobile system that has intelligent interaction, intelligent networking and intelligent driving capabilities in a specific scene, so that the specific-purpose mobile robot can reduce costs by saving an intelligent interaction system, and can support various intelligent devices and the ecological expansion capabilities of the devices, so that the system is more flexible and diverse in human interaction and Iot device networking, and realizes service capabilities for a specific person in a specific situation. Specifically, the following benefits are achieved: (1) the original mobile robot platform can meet the requirements of unmanned autonomous driving and support optimized scheduling; (2) the original robot platform no longer needs a human-machine interaction system, and can save costs; (3) the reuse of personal dedicated application settings can achieve more targeted services; and (4) the information networking of mobile applications can realize the sharing of user-based information and optimize personal services in a specific range.

[0068] This framework interaction method mainly integrates, layers, shares and transplants the sensing and processing capabilities of different devices, so that the combined autonomous mobile system can autonomously perform mobile tasks in a specific scene. The autonomous movement can be: (1) the same mobile device uses different robot mobile platforms to complete cross-platform tasks; (2) the same robot platform uses different mobile communication devices to achieve different user personalized tasks; and (3) the mobile communication device and the robot mobile platform are different, and the relay of different people and different stage tasks is realized.

[0069] Other advantages, objects and features of the application will be set forth in part in the following specification, and in part will become apparent to those skilled in the art from the following, or can be learned from the practice of the application. The objects and other advantages of the application can be realized and obtained by the following description. BRIEF DESCRIPTION OF DRAWINGS

[0070] In order to make the objects, technical solutions and advantages of the application clearer, the preferred detailed description of the application will be combined with the drawings, in which:

[0071] Figure 1 The autonomous mobile system structure schematic diagram provided by the application is shown in the figure;

[0072] Figure 2 The working mode of the system and the composition of the system are shown in the figure;

[0073] Figure 3 The system network and communication architecture are shown in the figure;

[0074] Figure 4 The logic for implementing the service;

[0075] Figure 5 The map layering relationships used by the system;

[0076] Figure 6 This describes the methods and content for map fusion tasks based on a basic topology layer.

[0077] Figure 7 To plan real-time running trajectories based on real-time dynamic layer maps;

[0078] Figure 8 For action decisions on the moving map layer;

[0079] Figure 9 To enable tracking and interaction with people on the interactive map layer;

[0080] Figure 10 For object manipulation in the operation map layer of a robotic arm system

[0081] Figure 11 A framework diagram for layered map fusion.

[0082] Figure 12 This is an example of cross-platform collaboration based on the system across different devices and platforms. Detailed Implementation

[0083] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0084] The accompanying drawings are for illustrative purposes only and are schematic diagrams, not actual pictures. They should not be construed as limiting the invention. To better illustrate the embodiments of the invention, some parts in the drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions. It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0085] The same or similar reference numerals in the drawings of the embodiments of the present application correspond to the same or similar components; in the description of the present application, it is understood that if the orientations or positional relationships indicated by the terms "upper", "lower", "left", "right", "front", "back", etc. are based on the orientations or positional relationships shown in the drawings, they are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the devices or elements referred to must have a particular orientation, be constructed and operated in a particular orientation, therefore the terms describing the positional relationships in the drawings are only used for exemplary illustration, and cannot be understood as a limitation on the present application, for those skilled in the art, the specific meanings of the above terms can be understood according to the specific circumstances.

[0086] Please refer to Figures 1-12 The present application discloses a kind of autonomous mobile system for realizing mobile device and mobile platform interaction, as shown in Figure 1 The system can utilize the ability of device, realize information-based sharing, by increasing intelligent mobile communication device-M, originally only in the field within the execution specific mobile task intelligent mobile platform P has human-computer interaction ability, can provide personalized and strong interactive experience task for tenant.

[0087] 1) Intelligent mobile platform P (platform Chassis), the typical system of platform includes energy system P1, mobile system P2, intelligent sensing system P3 and intelligent network system P4, realize in specific scene, along command track automatic safe travel;Possess support hand-held device installation support P5, realize with mobile communication device and signal connection of installation;And possible mechanical arm system P6, carry out special task such as grabbing operation.

[0088] Intelligent mobile platform P is optimally configured according to scene and use, including:

[0089] (1) artificial operation device rented by scanning code, such as shopping cart, need person to push, but can be shared by position, and statistics is carried out to passenger flow;

[0090] (2) manned mobile device, such as electric vehicle, need person to drive;

[0091] (3) unmanned mobile device, such as AGV electric vehicle.

[0092] 2) Mobile Device, M, is a personal mobile terminal, such as a mobile phone, a watch, etc., and can call the interconnection Iot device under the mobile ecology, such as smart glasses, smart earphones, etc. The typical system of the device includes: an image display and interaction system M1, typically a touch screen, a smart voice interaction system M2, an image perception system M3, and a second wireless communication system M4, which realizes interaction with the user, including interaction with the user through APPs, etc. and the sharing of information between applications and devices under the mobile ecology, such as location, time, schedule, preferences, etc., so as to be able to realize mobile trajectory planning while being able to optimize between multiple devices with a server.

[0093] Mobile communication device M, which is mainly used to interact with people and the environment, can be one or more of the following devices: (1) a mobile phone; (2) a tablet computer; (3) a smart wearable device, etc.

[0094] 3) Cloud Server S, which is used for scheduling and management of tasks, including mobile application services of APPs, basic services of devices, and interactive task services, etc.

[0095] Cloud Server S is a dedicated server for different applications and device management, and can intercommunicate and share data and information, such as location, state statistics, scheduling and management of devices and facilities in the field, etc.

[0096] Figure 2 The process of establishing a use relationship and mode switching of the system of the application at the mobile communication device M is shown, that is, on the basis of the smart mobile platform P and the basic services of the cloud server S, the user's command and service are realized by entering the mobile communication device M, and the process is as follows:

[0097] 1) Scheduling state: At this time, the smart mobile platform P is directly managed and scheduled by the cloud server S, including:

[0098] a) The cloud server S can respond to the use request of the mobile communication device M through the APP and confirm the location of the user;

[0099] b) The cloud server S configures the smart mobile platform P and sends the planned trajectory to the smart mobile platform P;

[0100] c) The smart mobile platform P moves according to the trajectory planned by the cloud server S and reaches the designated location, waiting for the user to establish a connection through the mobile communication device M.

[0101] 2) Service initialization; after reaching the designated location, the user rents through the App of the mobile communication device M, establishes wired and wireless connection, and re-performs usage-based initialization of the system, including:

[0102] a) Verification of user identity and device, such as ID of the mobile communication device M;

[0103] b) Installation and connection of the mobile communication device M, establishment of the connection network of the system, and initialization, system network and interaction content as shown in Figure 3 ;

[0104] c) Request and confirmation of tasks, interaction through the system of the mobile communication device M, confirmation of movement of the intelligent mobile platform P and service of the cloud server S;

[0105] 3) Service: complete initialization of the system and service, after confirming the task, the system interacts with the user through the device of the mobile communication device M, realizes the required service, including mobile service provided through the APP and service based on the mobile ecological of the mobile communication device M, and is dispatched by the management system of the mobile communication device M;

[0106] a) The mobile communication device M requests the cloud server S according to the content of the task, different tasks and capabilities, realizes the command and task of the user;

[0107] b) The special mobile application server S1 will integrate environmental, other device and other information, plan the task comprehensively, and interact and confirm with the user to ensure the optimization of the mobile application service and the driving safety;

[0108] c) Return: after completing the task, the user removes or ports the connection between the mobile communication device M and the intelligent mobile platform P, and returns the device through the APP of the mobile communication device M. In particular, the return and service initialization can be switched due to the operation of the user.

[0109] Figure 3 The system network architecture after establishing the connection is shown, and the channel and content of data interaction include:

[0110] a) Channel 1: wireless interaction channel of the intelligent mobile platform P and the cloud server S, mainly the mobile application server S1, through the connection established by the first wireless communication system P4.

[0111] i. The cloud server S sends the task information of trajectory scheduling and planning to the intelligent mobile platform P, and the intelligent mobile platform P judges the task information from the cloud server S and the mobile communication device P and executes safely.

[0112] ii. Intelligent mobile platform P feedbacks the current moving position and running state information to cloud server S.

[0113] b) Channel 2: Mobile communication device M and cloud server S, including interaction channel with mobile application server S1 and other interactive task servers S2, such as voice interaction, etc. Here, the interaction of mobile application server S1 is mainly explained.

[0114] i. Mobile communication device M sends the request of task and the command of execution to cloud server S through APP.

[0115] ii. Cloud server S feedbacks the planned task and the information of scheduling and planning related to the task to APP of mobile communication device M for the user to make decision and selection.

[0116] c) Channel 3: Mobile communication device M and intelligent mobile platform P interact with large amount of data, which is generally wired USB, used for interaction and synchronization of environmental map and command information.

[0117] i. Mobile communication device M sends the moving task decided by the user and the state information of the user to intelligent mobile platform P, which decides how to execute by combining the safety and its own state information.

[0118] ii. Intelligent mobile platform P feedbacks the execution state and the map and environmental information to mobile communication device M and displays through APP.

[0119] d) Channel 4: Mobile communication device M and intelligent mobile platform P wireless channel, mainly for identity verification and information interaction in the use return process, which also serves as the backup channel of channel 3.

[0120] e) Channel 5 and channel 6: When more computing resources are needed for environmental perception and fusion, command understanding and fusion, and comprehensive decision planning, the system will dynamically allocate the task execution among cloud server S, mobile communication device M and intelligent mobile platform P according to the actual resource and capability status of mobile communication device M and intelligent mobile platform P, and the corresponding transmission data will also be different, refer to Figure 4 Implementation process.

[0121] i. Intelligent mobile platform P sends the environmental information collected by the sensor to cloud server S through the first wireless communication system P4 of intelligent mobile platform P, which can use the same connection as channel 1.

[0122] ii.The mobile communication device M sends the sensor collected information of human interaction to the interactive task server S2 through the second wireless communication system M4 of the mobile communication device M, and the new information such as voice and image of the human is sent to the interactive task server S2 for understanding of voice and image interaction of the human, so that the command information is fed back to the mobile communication device M, or is directly sent to the mobile application server S1 for comprehensive understanding, decision and planning based on the task.

[0123] Figure 4 The process of realizing the mobile service task after adding the human-computer interaction capability of the mobile communication device M on the basis of the basic mobile capability of the intelligent mobile platform P is shown, and specifically includes:

[0124] 1) The intelligent mobile platform P has an environment perception system P3 including sensors of the platform such as a camera, a laser radar, an ultrasonic sensor, a millimeter wave radar and the like, which can detect the running environment, and can perform environment perception, motion decision and motion planning by using the capability of the platform P or the cloud server S; and the motion is executed by the mobile system P2 including actuators of the platform such as a drive motor, a steering motor, a brake system and an indication system, which can enable the intelligent mobile platform P to safely run according to the task of the cloud server S, i.e., the planned trajectory.

[0125] 2) After the mobile communication device M and the intelligent mobile platform P are connected through the support and mounting system P5 and the second wireless communication system M4 and the first wireless communication system P4, the interaction with the user is considered for realizing the specific task service.

[0126] a) The perception system (M1, M2, M3) of the mobile communication device M such as a camera, a microphone, a touch screen and a network perceives the language, action, emotion and the like of the user, and the command until the user inputs the character and the interaction command through the mobile communication device M or the M associated Iot device.

[0127] b) The character and the command input are fused with the environment perception information, which can understand the character and the command in the current environment, so as to be able to decide how to execute the task, and input the decision of the task to the motion decision and interaction decision module.

[0128] c) The motion request of the task decision is combined with the environment information, and the intelligent mobile platform P decides how to move, and the planning control module controls the actuators of the intelligent mobile platform P by considering the action to be executed by the intelligent mobile platform P.

[0129] d) The interaction request of the task decision is divided into three parts of action decision, expression and content interaction decision, and voice decision.

[0130] i. Action decision, such as the action of mechanical arm, head, etc. affecting the balance of motion, needs to be input to the motion control module for comprehensive stability control, and the action control of the mechanical arm and head is executed;

[0131] ii. Expression and content interaction decision, mainly to control the touch screen or state light display device of the interactive device;

[0132] iii. Speech decision, mainly for the management of continuous dialogue, output by the sound executor;

[0133] 3) Limited to the resources of intelligent mobile platform P and mobile communication device M, complex calculations may need to be calculated and processed by cloud server S, such as environmental perception, character perception and fusion, complex task decision and planning, etc., Especially in the case of integrated coordination control within the scene, such as passenger flow service in the venue.

[0134] In order to realize the comprehensive task decision and planning based on scene understanding, and can share and interwork in multiple scene areas, the task and mobile planning map is divided into five layers (basic topology layer, real-time dynamic layer, mobile map layer, interactive target layer and operation map layer), reference Figure 5 , based on the perception ability of intelligent mobile platform P and the App instruction of mobile communication device M to switch scenes and share information, the content, purpose and use of each layer of map are different. But it can be synchronized and connected through the location of mobile communication device M and intelligent mobile platform P, especially when renting intelligent mobile platform P through mobile communication device M remotely, refer to Figure 11 .

[0135] Figure 5 As shown in the figure, to realize multi-platform, scene and task, through the instruction of mobile communication device M, the location and map sharing between mobile communication device M, intelligent mobile platform P and server S are realized, and the corresponding task calculation and processing are realized based on this. The map and the corresponding function and task are as follows:

[0136] 1) Basic topology layer map1, similar to car navigation map, expressing the nodes and topological relationship between nodes in the global environment, used for global route planning and positioning based on features on the route. This layer also needs to describe the mutual relationship between nodes, topologies and nodes and topologies.

[0137] 2) Real-time dynamic layer map2, intelligent mobile platform P, including mobile communication device M and mobile system P2, high-precision positioning by real-time environment information perceived by the perception system P3, including nodes actually seen, features and rules of topology, and positioning features, and static and dynamic targets existing in the environment that affect actual movement.

[0138] 3) Mobile map layer map3, mainly a local short-range obstacle map established by the perception system P3 of the intelligent mobile platform P, which can support high-real-time movement commands.

[0139] 4) Interactive target layer map4, mainly targets that need to be interacted with by tracking and observation through the mobile communication device M, and can be positioned by matching the targets and features observed by the real-time dynamic layer map2, so as to maintain correct tracking, orientation, and output operation commands to the mounting bracket system P5 during movement.

[0140] 5) Operation map layer map5, mainly the relative positions of targets that need to be operated, observed by the perception system P3 of the intelligent mobile platform P, which includes corresponding sensing devices on the operation actuators, and can obtain operation mechanisms to perform corresponding operations.

[0141] The implementation of the mobile application task is based on the information related to the person and the command obtained through the mobile communication device M, and the environment information perceived by the intelligent mobile platform P. The information obtained by the two devices is fused in the specific device depending on the computing power and the demand for resource fusion.

[0142] For task understanding and planning, the basic topology layer map1 shown in Figure 5 is used to position and plan the global path using the features of the map and its own positioning system. The information on the map can be edited by the mobile communication device M, the intelligent mobile platform P, and the server S, and can be shared, divided into three categories of information:

[0143] 1) Node and attribute information map1-1: nodes with location, purpose, state, etc. on the global map, which can be searched and positioned on the map, including conditional node search, whose attributes include general public attribute information and system-edited specified information.

[0144] 2) Topology relationship information map1-2: various routes on the basic topology layer map1, and route rule and feature information, which at least need to include:

[0145] a) Topology rules: description of drivable routes, including general public attribute information such as traffic rules, number of lanes, speed limit, etc., and auxiliary indicators for traffic rule state indication such as traffic lights, signs, etc., and new attributes or markers specified by the user through the mobile communication device M such as special event markers.

[0146] b) Topology features, mainly features for assisting positioning on the topology route, which have attribute information describing use, state, usage rules, etc., and can be auxiliary indicators for describing topology rules, with relative position information, and new attributes or markers specified by the user through the mobile communication device M such as special event markers.

[0147] 3) Association map1-3: various association relationships between basic contents such as substitution, same type, priority, etc., for real-time planning or adjustment to obtain the optimal route. Also divided into public attribute association and specified attribute association.

[0148] Under the guidance of route planning, the autonomous mobile system moves safely and stably in a specific scene, and the environment information perceived by the mobile communication device M and the intelligent mobile platform P is fused in real time to generate a real-time dynamic target real-time dynamic layer map2, as shown in Figure 7 which expresses the real-time road and obstacle situation around the system or user, for real-time running trajectory planning, which is divided into the following contents:

[0149] 1) Node and position feature map2-1, which can assist in accurate positioning and match to the elements of the basic topology layer map1. Through the matching of such features and positions, it can be ensured that the local running trajectory conforms to the overall planned route.

[0150] 2) Road and feature map2-2, for describing the traffic rules in the area where the system can pass, through the identification and positioning of the ground, marking lines / objects, and reference facilities in the environment, to describe the attributes of the current road such as whether it can pass, the range of the passing area, speed limit, etc. It also includes the association between roads and rules such as red-green light triggering traffic switching, etc.

[0151] 3) Static target map2-3, static targets around the vehicle / user, including content and relative position, and additional attribute information, generally divided into two categories.

[0152] a) Static obstacles, which can be used to describe the range boundary and static objects that may collide;

[0153] b) Task related operation or indicator, static object associated with current task, generally specified by APP of mobile communication device M, and need to interact with task or user's action, such target needs to be identified in real-time dynamic layer map2;

[0154] 4) Dynamic target map2-4: dynamic target around vehicle / user, including content and relative position, and motion state, additional attribute information, generally divided into 3 categories and special target identification.

[0155] a) Moving object, identify and judge / predict the trajectory of movement, for judging whether collision will occur;

[0156] b) Moving person, identify and judge / predict the trajectory of movement, and need to identify additional character's identity and action information through mobile communication device M, as matching corresponding information of interactive target layer map4;

[0157] c) Object interacting with people, identify and judge / predict the trajectory of movement, and associated with people, assist in judging the behavior of people, as matching corresponding information of interactive target layer map4;

[0158] 5) Risk target: based on current position, route and motion state of vehicle / user, predict and estimate possible collision risk, predict and track static target and dynamic target, identify the mutual influence and correlation of motion trajectory in real-time dynamic layer map2, such as avoidance, for real-time obstacle avoidance decision and planning.

[0159] Based on the information of real-time dynamic layer map2, including current position, route and motion state of vehicle / user, consider the spatial position of currently executed task and interactive person, operation object, under the condition of meeting the set limit, such as prohibition of passage, get the optimal running trajectory for execution according to the priority of safety, interaction / follow, stability and comfort for the selectable running trajectory.

[0160] Autonomous mobile system plans movement trajectory in relatively long time, may need to execute emergency action of mechanism, for wheel type vehicle, generally braking and steering action, especially foot type robot's landing foot and gait adjustment action decision needs more and more real-time ground details, these details are expressed in movement map layer map3, as shown in Figure 8 Movement map layer map3 is generally the range of instantaneous movement, such as moving 5 meters in 2 seconds, mainly expresses the drivable state of road, such as road surface water, pothole, step and temporary obstacle, etc., which is not easy to accurately identify in long distance, for real-time adjustment of actuator action in short distance, divided into two categories:

[0161] 1) Ground state map3-1: Real-time map in general three-dimensional, planning for short-distance travel or landing position;

[0162] 2) Risk target map3-2: In the short-distance range, provide higher accuracy, more real-time tracking of risk targets, to perform emergency avoidance actions.

[0163] After considering the adjustment of following the user and the influence of the mechanical arm action on the overall balance, the autonomous mobile system adjusts the execution of the trajectory based on the ground state and the risk. Its planning control is mainly based on the information of the mobile map layer map3, combined with the demand of interactive action and operation action to obtain the comprehensive result. The autonomous mobile system needs to be able to maintain interaction with the user, needs to be able to maintain the connection relationship between the mobile communication device M and the mobile platform P, and can adjust the support and mounting system P5 to adjust the orientation of the sensor M3 to keep tracking the user, including dynamic target person map2-4, and dynamic or static target map2-3 and dynamic target person map2-4 operated by the person, establish the interactive target layer map4 based on the three-dimensional space, and match and fuse into the corresponding position of the real-time dynamic layer (map2) through the positioning of the intelligent mobile platform P, as shown in Figure 9 Based on the recognition, tracking, planning of the vehicle movement state and the understanding content of the interaction of the user by the interactive target layer, including:

[0164] 1) User map4-1: Including identity, feature recognition and confirmation, and associated person and object recognition and tracking.

[0165] 2) User associated object map4-2, including the recognition and tracking of objects indicated by action and language.

[0166] Through the perception device of the mobile communication device M, including through the operation of the support and mounting system P5, keep tracking and following the interactive target, comprehensively understand the command and command state that need to be executed, and real-time adjustment of the task, also including the direct operation input by the user through the operation mechanism of the mobile platform.

[0167] The autonomous mobile system can be an operating robot with a mechanical arm, and the general operation of the mechanical arm is for a space target, and the space target needs to have a spatial position, an operation attribute and a spatial environment restriction. For this purpose, for the system with a mechanical arm, the corresponding operation map layer map5 is established, as shown in Figure 10As shown, the operation of the robot arm is planned and controlled. Based on the target of the operation, the static and dynamic objects of the real-time dynamic map corresponding to the real-time dynamic layer map2, and the execution ability of the robot arm, the planning operation and the state of the operation are divided into 3 parts:

[0168] 1) Spatial obstacle map5-1 for the robot arm to avoid obstacles in space during movement of the system, so as to indicate the special space and action requirements when passing through the obstacle, and also as an input for movement and operation adjustment when passing through the obstacle.

[0169] 2) Operation target map5-2 for indicating the characteristics of the target and the process and limitations of the operation, so as to indicate the operation requirements of taking-delivering-placing at different stages, and also as an input for estimating and adjusting the center of gravity and balance at different stages.

[0170] 3) Operation space map5-3 for indicating the starting position and the posture of the robot arm operation, especially the association with the obstacle.

[0171] In summary, the framework example of the layered map fusion of the intelligent mobile platform P and the mobile communication device M is as shown in Figure 11 As shown, the perception ability of the mobile platform P is mainly to identify the environment and establish the corresponding position, topology and motion relationship, including 3d relationship, while the perception ability of the mobile communication device M provides the master's command and operation, and makes personalized and targeted adjustments and modifications to the information in the map, and marks. From Figure 11 It can be seen from the above that the perception ability of the intelligent mobile platform P is related to the scene of operation and the ability and application of the platform, while the perception and interaction ability of the mobile communication device M is related to the understanding of the task and the user, and the content in different scenes will be different.

[0172] Through the synchronization and sharing of the map by the server S, including the calling and sharing of public map resources, the task connection, interconnection and personalized service of different intelligent mobile platforms P and mobile communication devices M can be realized, such as the user initiating a procurement task through the App and specifying the procurement content and destination as Figure 12 As shown:

[0173] 1) The user books a taxi through the mobile phone, and the vehicle arrives at the specified location under the command of the server S, and then the user commands the vehicle to reach the shopping mall through the mobile phone, and realizes the movement to the shopping mall and the specified parking position using the road map and the mobile phone. At this time, the intelligent mobile platform P is a shared car;

[0174] 2) reach the mall, the user returns the vehicle, the vehicle can be rented by other people's mobile communication device M, and the user books an automatic shopping cart in the mall through the mobile phone, and follows the user to shop in the mall. At this time, the mobile intelligent platform P is the automatic shopping cart of the mall.

[0175] 3) complete shopping, the user returns the shopping cart, and again books a taxi to return to the community, and calls the home robot to wait at the designated position on the car, at this time the user controls the home robot to move autonomously through the mobile phone and the server S.

[0176] 4) arrive and return the vehicle, the goods are transferred to the home robot, and the mobile communication device M is placed to the home robot, so that it follows the user home. The home robot can have a mechanical arm to operate the goods, including moving, placing, etc., and can operate under the remote control of the mobile phone. At this time, the robot intelligent moving platform P is the home robot.

[0177] Finally, it should be pointed out that the above embodiments are only used to illustrate the technical solutions of the present application and are not limiting. Although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application can be modified or replaced by equivalents without departing from the purpose and scope of the technical solutions, which should be covered in the scope of the claims of the present application.

Claims

1. An autonomous mobile system that enables mobile device interaction with a mobile platform, the system comprising: The system comprises an intelligent mobile platform, a mobile communication device and a cloud server; The intelligent mobile platform comprises an energy system, a moving system, a sensing system and a first wireless communication system, and is used for automatically and safely moving along a command trajectory in a specific scene; The mobile communication device comprises an image display and interaction system, a sound interaction system, an image sensing system and a second wireless communication system, and is used for realizing interaction with a user, including interaction of a task and a command with the user through an APP, and information sharing between an application and the mobile communication device in a mobile ecology, so that the mobile trajectory planning can be optimized among a plurality of mobile communication devices through a server; The cloud server comprises a mobile application server and an interactive task server, and is used for scheduling and managing a task, including mobile application service of the APP, basic service of the mobile communication device and interactive task service; On the basis of the basic moving capability of the intelligent mobile platform, the process of the mobile service task is realized after the human-computer interaction capability of the mobile communication device is added, and specifically comprises: 1) the sensing system of the intelligent mobile platform is used for sensing an environment, the cloud server is used for making a motion decision and motion planning, and the moving system is used for executing, so that the intelligent mobile platform moves safely according to the task of the cloud server, i.e. the planned trajectory; 2) after the mobile communication device and the intelligent mobile platform are connected through the second wireless communication system and the first wireless communication system, the interaction with the user needs to be considered for a specific task service, and specifically comprises: a) the sensing system of the mobile communication device senses a character state and a command of the user until the user inputs a character and an interactive command through the mobile communication device or an Iot device associated with the mobile communication device; b) the character and the command input are fused with the sensed information, so that the character and the command can be understood in the current environment, and how to execute the task can be decided, and the task decision is input to a motion decision module and an interactive decision module; c) the motion request of the task decision module is combined with the environmental information, the intelligent mobile platform decides how to move, and a planning control module controls an actuator of the intelligent mobile platform after considering the action to be executed by the intelligent mobile platform; d) the interactive request of the task decision module is divided into three parts of action decision, expression and content interactive decision and voice decision; 3) complex calculation needs to be calculated by the cloud server in a large data volume, including sensing of an environment, sensing and fusion of a character, decision and planning of a complex task; The decision and planning of the complex task comprise comprehensive task decision and planning based on scene understanding, wherein a basic map model for task understanding and decision planning is in the form of a hierarchical map, corresponding to route planning, trajectory planning and moving decision content for executing a task, and tracking and following of an interactive target; The hierarchical map comprises a basic topology layer, a real-time dynamic layer, a moving map layer, an interactive target layer and an operation map layer; the basic topology layer comprises three types of information: 1) Node: the location / position associated with the current task in the map, and the corresponding attribute description; 2) Topology: the connection topology between nodes in the map, including the rules that can describe the topology relationship and the feature information or attributes of the node position; 3) Association relationship: the attribute association relationship between nodes and topologies, used for real-time planning or adjustment to obtain the optimal decision and planning scheme; Under the guidance of route planning, the autonomous mobile system moves safely and stably in a specific scene, and the environment information perceived by the mobile communication device and the intelligent mobile platform is fused in real time to generate a real-time dynamic layer, which expresses the real-time road and obstacle situation around the system or the user, for real-time running trajectory planning; the perceived environment information specifically includes: 1) Node and position characteristics, used to assist in accurate positioning and matching to the elements of the basic topology layer; 2) Road and features, used to describe the traffic rules in the area where the system passes through, identify the facilities by reference to the ground, marker lines / objects in the environment, and describe the attributes of the current road traffic, the association relationship between roads and rules; 3) Static target objects: static target objects around the vehicle / user, including content and relative position, and additional attribute information, divided into two categories: a) Static obstacles, which can be used to describe the range boundary and the static objects that may collide; b) Task-related operations or indicators, which are associated with the current task and specified by the APP of the mobile communication device, and need to interact with the task or the action of the user, such objects need to be identified in the real-time dynamic layer; 4) Dynamic target objects: dynamic objects around the vehicle / user, including content and relative position, motion state, additional attribute information, and special target identification; dynamic objects are divided into three categories: a) Moving objects, which are identified and judged / predicted to move along a trajectory, used to determine whether a collision will occur; b) Moving people, which are identified and judged / predicted to move along a trajectory, and need to identify the identity and action information of the person through the mobile communication device as matching corresponding information of the interactive target layer; c) Objects interacting with people, which are identified and judged / predicted to move along a trajectory, and are associated with the person to assist in judging the behavior of the person as matching corresponding information of the interactive target layer; 5) Risk target: based on the current position, route and motion state of the vehicle / user, the possible collision risk is predicted and estimated, the static target objects and dynamic target objects are predicted and tracked, and the real-time dynamic layer is identified to identify the mutual influence and association relationship of the motion trajectory, for real-time obstacle avoidance decision and planning; Based on the information of the real-time dynamic layer, including the current position, route and motion state of the vehicle / user, considering the current executed task and the spatial position of the interactive person and operating object, under the condition of meeting the set limit, the optimal running trajectory is obtained for execution according to the priority of safety, interaction / following, stability and comfort for the selectable running trajectory; When the autonomous mobile system needs to perform emergency actions of the mechanism during long-time planning of the moving trajectory, the ground state and the risk target need to be identified; The autonomous mobile system establishes the interactive target layer of the map by keeping the interaction with the user, the connection relationship between the mobile communication device and the mobile platform, and the tracking of the user, and fuses into the corresponding position of the real-time dynamic layer through the positioning matching of the intelligent mobile platform; Based on the identification of the user, the tracking, the planning of the state of the vehicle movement and the understanding content of the interaction of the interactive target layer, including: 1) The user, including the identification and confirmation of the identity and characteristics, and the identification and tracking of the associated people and objects; 2) The associated objects of the user, including the identification and tracking of the objects indicated by actions and language; Through the perception device of the mobile communication device, including the operation through the support and installation system, the tracking and following of the interactive target are kept, the command to be executed is comprehensively understood and the command state is adjusted in order to adjust the task, which also includes the direct operation input by the user through the operation mechanism of the mobile platform.

2. The autonomous mobile system of claim 1, wherein, When the autonomous mobile system is an operating robot with a mechanical arm, a corresponding operation map layer is established for the robot; based on the target of the operation, the static target objects and dynamic target objects corresponding to the real-time dynamic map of the real-time dynamic layer, and the execution ability of the mechanical arm, the planning of the operation and the state of the operation are divided into three parts: 1) Spatial obstacles, used for spatial obstacle avoidance of the mechanical arm during movement of the system, in order to indicate special space and action requirements when passing through obstacles, and also as input for movement and operation adjustment when passing through obstacles; 2) Operation target, used to indicate the characteristics of the target and the process and limitation of the operation, in order to indicate the operation requirements of taking-transporting-placing in different stages, and also as input for estimation and adjustment of the center of gravity and balance in different processes; 3) Operation space, used to indicate the starting position and the posture of the mechanical arm operation.

3. The autonomous mobile system of claim 1, wherein, The interaction of expressions, contents and languages of the interactive decision is a reaction according to the perception and understanding content, because the action, including the head and the operation, will have an impact on the balance of the whole machine; After considering the adjustment of following the user and the impact of the mechanical arm action on the overall balance, the autonomous mobile system adjusts the execution of the trajectory based on the ground state and the risk, so as to be able to adjust in real time and keep driving along the planned trajectory.

4. The autonomous mobile system of claim 1, wherein, The planning control module is based on the information of the moving map layer, combined with the demand of interactive action and operation action, and comprehensively stabilized to plan the execution.

Images