Human-riding robot dog travel path regulation system and method based on multi-modal perception

Abstract

The present application relates to the technical field of manned mechanical dog travel path regulation, in particular to a manned mechanical dog travel path regulation system and method based on multi-modal sensing, the system comprising a travel interference analysis module, the travel interference analysis module obtains a manned mechanical dog travel planning path stored in a current database, combines current time manned mechanical dog travel characteristic mapping information, generates a current time manned mechanical dog travel interference area, and obtains a current time manned mechanical dog to-be-updated obstacle avoidance mapping planning scheme. The present application not only considers the situation of obstacles, but also takes into account the state information and manned task execution capability of the manned mechanical dog, realizes planning of the manned mechanical dog obstacle avoidance path regulation scheme and dynamic determination of the manned mechanical dog cooperative demand, and according to the obstacle avoidance scheme planning result and the cooperative demand determination result, realizes dynamic regulation of the manned mechanical dog travel path.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of manned mechanical dog travel path regulation, in particular to a manned mechanical dog travel path regulation system and method based on multi-modal perception. BACKGROUND

[0002] With the rapid development of sensor technology and artificial intelligence, multi-modal perception technology has been gradually applied in various fields, especially in the field of robots. As a new type of mobile robot, the regulation of the travel path of a manned mechanical dog is crucial to its stability and safety. Multi-modal perception technology can improve the perception ability of the mechanical dog to the environment by fusing the data of multiple sensors, thereby achieving accurate regulation of the travel path.

[0003] Existing manned mechanical dog travel path regulation systems based on multi-modal perception only perceive the position of obstacles and achieve path planning and regulation. However, the state information of the manned mechanical dog and the ability to perform a manned task (judgment of whether the task can be completed and whether a surrounding manned mechanical dog needs to provide cooperation) are not considered in the path regulation process. Therefore, the existing technology has significant defects. SUMMARY

[0004] The present application aims to provide a manned mechanical dog travel path regulation system and method based on multi-modal perception to solve the problems raised in the background.

[0005] To solve the above technical problems, the present application provides the following technical solution: a manned mechanical dog travel path regulation method based on multi-modal perception, comprising the following steps:

[0006] S1. Real-time acquisition of state information and environmental information of the manned mechanical dog during travel by multi-modal sensors to generate travel feature mapping information of the manned mechanical dog;

[0007] S2. Acquisition of the travel planning path of the manned mechanical dog stored in the current database, combination of the travel feature mapping information of the manned mechanical dog at the current time, generation of the travel disturbance area of the manned mechanical dog at the current time, and obtaining of the updated obstacle mapping planning scheme of the manned mechanical dog at the current time;

[0008] S3. Calculation of the regulation fluctuation value of each updated obstacle mapping planning scheme of the manned mechanical dog at the current time based on the travel planning path of the manned mechanical dog stored in the current database, screening of the best obstacle mapping planning scheme, combination of the initial planning path corresponding to the travel planning path of the manned mechanical dog stored in the current database, obtaining of the path regulation integration fluctuation coefficient corresponding to the manned mechanical dog at the current time, and prediction of the travel range fluctuation interval of the subsequent travel planning path of the manned mechanical dog;

[0009] S4, extracting state information in the manned mechanical dog travel feature mapping information, combining the travel endurance mileage fluctuation interval prediction value of the subsequent travel planning path of the manned mechanical dog, and the empty collaborative object around the manned mechanical dog at the current time, analyzing the collaborative demand coefficient of the manned mechanical dog at the current time, and completing the collaborative demand judgment of the manned mechanical dog;

[0010] S5, according to the collaborative demand judgment result of the manned mechanical dog, updating the manned mechanical dog travel planning path stored in the current database, and feeding back the updated manned mechanical dog travel planning path and manned mechanical dog travel feature mapping information to the remote monitoring end in real time.

[0011] Further, the manned mechanical dog travel feature mapping information is composed of state information in the manned mechanical dog travel process and environment information at the same time; the state information in the manned mechanical dog travel process includes residual endurance mileage, travel speed and current time position; the environment information includes map model, static obstacle information and dynamic obstacle information within a preset unit radius distance around the manned mechanical dog; the map model within the preset unit radius distance around the manned mechanical dog is obtained by processing the collected images within the preset unit radius distance around the manned mechanical dog based on a map construction tool; the types of static obstacles and dynamic obstacles are obtained through image recognition results of the collected images within the preset unit radius distance around the manned mechanical dog; the static obstacle information includes the distribution position and size of each static obstacle in the corresponding image recognition result based on the current position of the manned mechanical dog; the dynamic obstacle information includes the distribution position, size, motion speed and motion direction of each dynamic obstacle in the corresponding image recognition result based on the current position of the manned mechanical dog, the motion speed represents the modulus of the vector formed by the first appearance position and the current position of the corresponding dynamic obstacle in the collected images within the preset unit time divided by the quotient of the interval time between the first appearance time and the current time of the corresponding dynamic obstacle; the motion direction represents the direction of the vector formed by the first appearance position and the current position of the corresponding dynamic obstacle in the collected images within the preset unit time; if the corresponding dynamic obstacle appears in the collected images within the preset unit time, it is judged that the motion speed in the corresponding dynamic obstacle information is 0 and the motion direction is the same as the forward direction of the manned mechanical dog at the current time.

[0012] The application realizes the collection of information of the manned mechanical dog from multiple aspects, which is used for analyzing and regulating the marching path of the manned mechanical dog in subsequent steps; wherein, the state information is convenient for controlling the condition of the mechanical dog itself, considering three factors of the remaining range (convenient for subsequent judgment of the collaborative demand of the manned mechanical dog), the marching speed and the current time position (two factors are convenient for predicting the behavior path route of the subsequent manned mechanical dog, and obtaining the marching interference area of the current time manned mechanical dog).

[0013] Further, the method for generating the marching interference area of the current time manned mechanical dog comprises the following steps:

[0014] S21, acquiring the marching planning path of the manned mechanical dog stored in the current database and the marching characteristic mapping information of the current time manned mechanical dog;

[0015] S22, acquiring the obstacle distance farthest from the current time position in the distance state information in the environment information in the marching characteristic mapping information of the current time manned mechanical dog, denoted as the first characteristic distance, wherein the obstacle includes static obstacles and dynamic obstacles; calculating the quotient of the first characteristic distance and the marching speed in the state information, denoted as T; and generating the dynamic reaction time interval [0, T];

[0016] S23, predicting the distribution positions of the manned mechanical dog, the static obstacles and the dynamic obstacles in the map model in the environment information in the marching characteristic mapping information of the current time manned mechanical dog at time t, wherein t∈[0, T]; the position of the manned mechanical dog in the map model in the environment information in the marching characteristic mapping information of the current time manned mechanical dog at time t is the position point in the marching planning path of the manned mechanical dog stored in the current database, which is farthest from the current time position of the manned mechanical dog and the product of the corresponding marching speed and t; the position of the static obstacle in the map model in the environment information in the marching characteristic mapping information of the current time manned mechanical dog at time t remains unchanged; and the position of the dynamic obstacle in the map model in the environment information in the marching characteristic mapping information of the current time manned mechanical dog at time t is the position point, which is farthest from the position of the corresponding dynamic obstacle and equal to the product of the corresponding dynamic obstacle information and t, and points to the same direction as the corresponding dynamic obstacle information;

[0017] S24, acquiring all the predicted position points of the manned mechanical dog in the map model in the environment information in the marching characteristic mapping information of the current time manned mechanical dog, which are less than or equal to the preset distance between the predicted position of the manned mechanical dog and each static obstacle and dynamic obstacle at the same time within [0, T], and marking the corresponding position in the marching planning path of the manned mechanical dog stored in the current database, wherein the set of the obtained marked position points is denoted as the marching interference area of the current time manned mechanical dog.

[0018] In the process of obtaining the to-be-updated obstacle avoidance mapping planning scheme of the current manned mechanical dog, the travel interference region of the current manned mechanical dog is obtained, if the travel interference region of the current manned mechanical dog is an empty set, then the current manned mechanical dog does not have a to-be-updated obstacle avoidance mapping planning scheme, if the travel interference region of the current manned mechanical dog is not an empty set, then the region formed by the distributed positions of the static obstacles and the dynamic obstacles corresponding to [0, T] is recorded as OB, the intersection region of the travel planning path of the manned mechanical dog stored in the current database and OB is taken as the to-be-updated obstacle avoidance planning region, recorded as Q, the path section corresponding to Q is planned by the Beidou navigation software based on the OB region, different obstacle avoidance path planning schemes are generated, and the binding result of each obstacle avoidance path planning scheme and Q is taken as a to-be-updated obstacle avoidance mapping planning scheme of the current manned mechanical dog.

[0019] In the process of generating the travel interference region of the manned mechanical dog and obtaining the to-be-updated obstacle avoidance mapping planning scheme in the application, the predicted positions of each static obstacle and dynamic obstacle in [0, T] are needed, but there is a big difference between the application modes of the two, when the travel interference region is obtained, all the predicted position points of the manned mechanical dog in [0, T] are obtained, which are less than or equal to the preset distance between the predicted position of the manned mechanical dog and the predicted position of each static obstacle and dynamic obstacle at the same time, while when the to-be-updated obstacle avoidance mapping planning scheme is obtained, OB is directly obtained, the reason is that in [0, T], once the travel interference region is not an empty set, it means that the travel path of the manned mechanical dog needs to be regulated, but the regulation and change of the path may also change the subsequent predicted position of the manned mechanical dog in [0, T], therefore, the analysis mode used when the travel interference region is obtained may cause a large deviation to the subsequent path planning result.

[0020] Further, the best obstacle avoidance mapping planning scheme is the to-be-updated obstacle avoidance mapping planning scheme with the minimum regulation fluctuation value based on the travel planning path of the manned mechanical dog stored in the current database at the current time, the formula for calculating the regulation fluctuation value of each to-be-updated obstacle avoidance mapping planning scheme of the manned mechanical dog at the current time based on the travel planning path of the manned mechanical dog stored in the current database is as follows: i = LG i / LB i Wherein, C i represents the regulation fluctuation value of the i-th to-be-updated obstacle avoidance mapping planning scheme of the manned mechanical dog at the current time based on the travel planning path of the manned mechanical dog stored in the current database, LG i represents the total length of the planning path of the corresponding obstacle avoidance path planning scheme in the i-th to-be-updated obstacle avoidance mapping planning scheme of the manned mechanical dog at the current time, and LBi represents the total length of the path of the corresponding updated obstacle avoidance planning region Q in the i-th updated obstacle avoidance mapping planning scheme of the current time manned mechanical dog;

[0021] The calculation formula of the path regulation integration fluctuation coefficient corresponding to the current time manned mechanical dog is as follows:

[0022] CP = (LS + LGZ - LY) / (LCY - LY + LB i )

[0023] Wherein, CP represents the path regulation integration fluctuation coefficient corresponding to the current time manned mechanical dog; LS represents the actual driving distance between the starting point of the initial planning path corresponding to the current database stored manned mechanical dog travel planning path and the position of the current time manned mechanical dog; LGZ represents the total length of the obstacle path planning scheme in the best obstacle avoidance mapping planning scheme; LCY represents the total length of the path between the starting point of the initial planning path corresponding to the current database stored manned mechanical dog travel planning path and the mapping point of the current time manned mechanical dog position; LY represents the path length of the intersection section between the path corresponding to LCY and the actual driving path corresponding to LS; The mapping point of the current time manned mechanical dog position in the initial planning path corresponding to the current database stored manned mechanical dog travel planning path is the same position point as the foot of the perpendicular line corresponding to the current time manned mechanical dog position in the respective initial planning path on the plane reference line of the respective initial planning path, and the plane reference line of the respective initial planning path represents the projection line segment of the connecting line between the initial point and the terminal point in the respective initial planning path on the horizontal plane;

[0024] The prediction result of the travel endurance mileage fluctuation interval of the subsequent travel planning path of the manned mechanical dog is denoted as [LEB, LEF],

[0025] LEB = LW + LGZ - LB i

[0026] LEF = LW·CP

[0027] Wherein, LEB represents the minimum value in the prediction result of the travel endurance mileage fluctuation interval of the subsequent travel planning path of the manned mechanical dog; LEF represents the maximum value in the prediction result of the travel endurance mileage fluctuation interval of the subsequent travel planning path of the manned mechanical dog; LW represents the path length between the terminal point and the position of the current time manned mechanical dog in the current database stored manned mechanical dog travel planning path.

[0028] Further, the method for analyzing the coordination demand coefficient of the current time manned mechanical dog comprises the following steps:

[0029] S41, obtaining state information in the manned mechanical dog travel characteristic mapping information, a travel endurance mileage fluctuation interval prediction value of a subsequent manned mechanical dog travel planning path of the manned mechanical dog, and an empty-carrying collaborative object around the manned mechanical dog at the current time, the empty-carrying collaborative object representing all manned mechanical dogs that do not perform manned tasks within a preset radius distance range from the manned mechanical dog at the current time;

[0030] S42, obtaining a collaborative demand coefficient of the manned mechanical dog at the current time, and the calculation formula is as follows:

[0031]

[0032] SC represents the collaborative demand coefficient of the manned mechanical dog at the current time; LR represents the remaining endurance mileage in the state information in the manned mechanical dog travel characteristic mapping information; LD represents the shortest distance between the empty-carrying collaborative object around the manned mechanical dog at the current time and the position of the manned mechanical dog at the current time; H{} represents a comparison and determination function; if if

[0033] In the collaborative demand determination process of the manned mechanical dog, if the SC is greater than or equal to a preset collaborative demand threshold value, it is determined that the manned mechanical dog at the current time has a collaborative demand; otherwise, it is determined that the manned mechanical dog at the current time does not have a collaborative demand.

[0034] Further, in the process of updating the manned mechanical dog travel planning path stored in the current database in S5, if the manned mechanical dog at the current time does not have a collaborative demand, the section corresponding to Q in the initial planning path corresponding to the manned mechanical dog travel planning path stored in the current database is replaced by the obstacle avoidance path planning scheme in the corresponding best obstacle avoidance mapping planning scheme; if the manned mechanical dog at the current time has a collaborative demand, the Beidou navigation software plans a travel path between the manned mechanical dog at the current time and the empty-carrying collaborative object closest to the manned mechanical dog at the current time based on the OB area, and replaces the obtained shortest planning path with the manned mechanical dog travel planning path stored in the current database. The intersection between the obtained planning path and the OB area is an empty set.

[0035] The manned mechanical dog travel path regulation system based on multi-modal perception includes the following modules:

[0036] A travel characteristic mapping information acquisition module, which acquires state information and environmental information in the travel process of the manned mechanical dog in real time through a multi-modal sensor, and generates manned mechanical dog travel characteristic mapping information;

[0037] ​​The travel interference analysis module obtains a travel planning path of the manned mechanical dog stored in the current database, combines travel characteristic mapping information of the manned mechanical dog at the current time, generates a travel interference area of the manned mechanical dog at the current time, and obtains an updated obstacle avoidance mapping planning scheme of the manned mechanical dog at the current time.

[0038] The obstacle avoidance planning scheme analysis module calculates a regulation fluctuation value of each updated obstacle avoidance mapping planning scheme of the manned mechanical dog at the current time based on the travel planning path of the manned mechanical dog stored in the current database, screens an optimal obstacle avoidance mapping planning scheme, combines an initial planning path corresponding to the travel planning path of the manned mechanical dog stored in the current database, obtains a path regulation integration fluctuation coefficient corresponding to the manned mechanical dog at the current time, and predicts a travel endurance fluctuation interval of a subsequent travel planning path of the manned mechanical dog.

[0039] The empty object cooperative judgment management module extracts state information in the travel characteristic mapping information of the manned mechanical dog, combines a prediction value of the travel endurance fluctuation interval of the subsequent travel planning path of the manned mechanical dog, and analyzes a cooperative demand coefficient of the manned mechanical dog at the current time based on an empty cooperative object around the manned mechanical dog at the current time, to complete cooperative demand judgment of the manned mechanical dog.

[0040] The travel path regulation management module updates the travel planning path of the manned mechanical dog stored in the current database according to the cooperative demand judgment result of the manned mechanical dog, and feeds back the updated travel planning path of the manned mechanical dog and the travel characteristic mapping information of the manned mechanical dog to the remote monitoring end in real time.

[0041] Further, the obstacle avoidance planning scheme analysis module includes an optimal obstacle avoidance mapping planning scheme screening unit, a path regulation integration analysis unit, and an endurance fluctuation analysis unit.

[0042] The optimal obstacle avoidance mapping planning scheme screening unit calculates a regulation fluctuation value of each updated obstacle avoidance mapping planning scheme of the manned mechanical dog at the current time based on the travel planning path of the manned mechanical dog stored in the current database, and screens an optimal obstacle avoidance mapping planning scheme.

[0043] The path regulation integration analysis unit combines an initial planning path corresponding to the travel planning path of the manned mechanical dog stored in the current database, and obtains a path regulation integration fluctuation coefficient corresponding to the manned mechanical dog at the current time.

[0044] The endurance fluctuation analysis unit predicts a travel endurance fluctuation interval of a subsequent travel planning path of the manned mechanical dog according to the obtained results of the optimal obstacle avoidance mapping planning scheme screening unit and the path regulation integration analysis unit.

[0045] Compared with the prior art, the present application has the beneficial effects that: in the process of regulating the walking path of the manned mechanical dog, not only the situation of the obstacle is considered, but also the state information of the manned mechanical dog itself and the carrying task execution ability are considered, the planning of the obstacle avoidance path regulation scheme of the manned mechanical dog and the dynamic determination of the cooperative demand of the manned mechanical dog are realized, and according to the obstacle avoidance scheme planning result and the cooperative demand determination result, the dynamic regulation of the walking path of the manned mechanical dog is realized. BRIEF DESCRIPTION OF DRAWINGS

[0046] The accompanying drawings are included to provide a further understanding of the present application, and constitute a part of the specification, illustrate the present application together with the embodiments thereof, and explain the technical solutions of the present application, but do not constitute a limitation on the present application. In the drawings:

[0047] Figure 1 is a structural schematic diagram of the manned mechanical dog walking path regulation system based on multi-modal perception of the present application;

[0048] Figure 2 is a flowchart of the manned mechanical dog walking path regulation method based on multi-modal perception of the present application. DETAILED DESCRIPTION

[0049] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0050] Please refer to Figure 1 The present application provides a technical solution: a manned mechanical dog walking path regulation system based on multi-modal perception, which comprises the following modules:

[0051] A walking feature mapping information acquisition module, which acquires state information and environmental information in the walking process of the manned mechanical dog in real time through a multi-modal sensor, and generates walking feature mapping information of the manned mechanical dog;

[0052] A walking interference analysis module, which acquires the walking planning path of the manned mechanical dog stored in the current database, combines the walking feature mapping information of the manned mechanical dog at the current time, generates the walking interference area of the manned mechanical dog at the current time, and obtains the to-be-updated obstacle avoidance mapping planning scheme of the manned mechanical dog at the current time;

[0053] An obstacle avoidance planning scheme analysis module, the obstacle avoidance planning scheme analysis module comprising an optimal obstacle avoidance mapping planning scheme screening unit, a path regulation integration analysis unit, and a cruising wave fluctuation analysis unit,

[0054] The optimal obstacle avoidance mapping planning scheme screening unit calculates each to-be-updated obstacle avoidance mapping planning scheme of the manned mechanical dog at the current time based on the regulation fluctuation value of the manned mechanical dog travel planning path stored in the current database, and screens an optimal obstacle avoidance mapping planning scheme.

[0055] The path regulation integration analysis unit obtains the path regulation integration fluctuation coefficient corresponding to the manned mechanical dog at the current time in combination with the initial planning path corresponding to the manned mechanical dog travel planning path stored in the current database.

[0056] The cruising wave fluctuation analysis unit predicts the travel cruising mileage fluctuation interval of the subsequent travel planning path of the manned mechanical dog according to the acquisition results of the optimal obstacle avoidance mapping planning scheme screening unit and the path regulation integration analysis unit.

[0057] An empty object cooperative judgment management module extracts state information in the manned mechanical dog travel characteristic mapping information, combines the travel cruising mileage fluctuation interval prediction value of the subsequent travel planning path of the manned mechanical dog, and analyzes the cooperative demand coefficient of the manned mechanical dog at the current time in combination with the empty cooperative object around the manned mechanical dog at the current time, to complete the cooperative demand judgment of the manned mechanical dog.

[0058] A travel path regulation management module updates the manned mechanical dog travel planning path stored in the current database according to the cooperative demand judgment result of the manned mechanical dog, and feeds back the updated manned mechanical dog travel planning path and the manned mechanical dog travel characteristic mapping information to the remote monitoring end in real time.

[0059] As shown in the figure, Figure 2 A manned mechanical dog travel path regulation method based on multi-modal perception, the method comprising the following steps:

[0060] S1, real-time acquisition of state information and environment information in the manned mechanical dog travel process through a multi-modal sensor, to generate manned mechanical dog travel characteristic mapping information;

[0061] The human-robot walking feature mapping information is composed of state information and environment information of the human-robot in the walking process at the same time; the state information of the human-robot in the walking process includes residual range, walking speed and current time position; the environment information includes a map model, static obstacle information and dynamic obstacle information within a preset unit radius distance around the human-robot; the map model within the preset unit radius distance around the human-robot is obtained by processing the collected images within the preset unit radius distance around the human-robot based on a map construction tool; the types of static obstacles and dynamic obstacles are obtained by image recognition results of the collected images within the preset unit radius distance around the human-robot; the static obstacle information includes the distribution position and size of each static obstacle in the corresponding image recognition result based on the current position of the human-robot; the dynamic obstacle information includes the distribution position, size, motion speed and motion direction of each dynamic obstacle in the corresponding image recognition result based on the current position of the human-robot, the motion speed represents the quotient of the module length of the vector formed by the first appearance position and the current position of the corresponding dynamic obstacle in the collected images within the preset unit time and the interval time length between the first appearance time and the current time of the corresponding dynamic obstacle; the motion direction represents the pointing direction of the vector formed by the first appearance position and the current position of the corresponding dynamic obstacle in the collected images within the preset unit time; if the corresponding dynamic obstacle appears for the first time in the collected images within the preset unit time, the motion speed in the corresponding dynamic obstacle information is determined as 0 and the motion direction is the same as the forward direction of the human-robot at the current time.

[0062] The map construction tool for constructing the map model according to the collected images in the embodiment can be selected from geospy software, ArcGIS software and the like;

[0063] The types of static obstacles and dynamic obstacles in the embodiment are pre-set in a database, the types of static obstacles are stationary objects, and the types include steps, buildings, trees, slopes and water areas; the types of dynamic obstacles are movable objects, and the types include pedestrians, animals, cars and snow scenes.

[0064] S2, obtaining the walking planning path of the human-robot stored in the current database, combining the walking feature mapping information of the human-robot at the current time, generating the walking interference area of the human-robot at the current time, and obtaining the updated obstacle avoidance mapping planning scheme of the human-robot at the current time;

[0065] The method for generating the walking interference area of the human-robot at the current time includes the following steps:

[0066] S21, obtaining the walking planning path of the human-robot stored in the current database, and the walking feature mapping information of the human-robot at the current time;

[0067] S22, obtain the distance of the obstacle farthest from the current position in the distance state information in the environment information in the current time manned mechanical dog travel characteristic mapping information, denoted as the first characteristic distance, the obstacle includes static obstacles and dynamic obstacles; calculate the quotient of the first characteristic distance and the travel speed in the state information, denoted as T; generate a dynamic reaction time interval [0, T];

[0068] S23, predict the distribution positions of the manned mechanical dog, static obstacles and dynamic obstacles in the map model in the environment information in the current time manned mechanical dog travel characteristic mapping information at time t, t∈[0, T]; the position of the manned mechanical dog in the map model in the environment information in the current time manned mechanical dog travel characteristic mapping information at time t is the position point in the manned mechanical dog travel planning path stored in the current database, which is a distance of the current time position of the manned mechanical dog and the product of the corresponding travel speed and t; the position of the static obstacle in the map model in the environment information in the current time manned mechanical dog travel characteristic mapping information at time t remains unchanged; the position of the dynamic obstacle in the map model in the environment information in the current time manned mechanical dog travel characteristic mapping information at time t is a position point which is a distance equal to the product of the corresponding dynamic obstacle information motion speed and t and points to the same direction as the corresponding dynamic obstacle information motion direction from the corresponding dynamic obstacle position;

[0069] S24, obtain all the manned mechanical dog prediction position points in the map model in the environment information in the current time manned mechanical dog travel characteristic mapping information, which are less than or equal to the preset distance between the prediction position of the manned mechanical dog and each static obstacle and dynamic obstacle at the same time within [0, T], and mark the corresponding positions in the manned mechanical dog travel planning path stored in the current database, and the set of the obtained marked position points is denoted as the travel interference area of the current time manned mechanical dog;

[0070] In the process of obtaining the to-be-updated obstacle avoidance mapping planning scheme of the current manned mechanical dog, the travel interference region of the current manned mechanical dog is obtained. If the travel interference region of the current manned mechanical dog is an empty set, the current manned mechanical dog does not have a to-be-updated obstacle avoidance mapping planning scheme. If the travel interference region of the current manned mechanical dog is not an empty set, the region formed by the distribution positions of the static obstacles and the dynamic obstacles corresponding to [0, T] is recorded as OB, the intersection region of the travel planning path of the manned mechanical dog stored in the current database and OB is recorded as Q, and the path segment corresponding to Q is planned by the Beidou navigation software based on the OB region to generate different obstacle avoidance path planning schemes. The result of binding each obstacle avoidance path planning scheme with Q is taken as a to-be-updated obstacle avoidance mapping planning scheme of the current manned mechanical dog.

[0071] In the embodiment, T in [0, T] is 30 seconds.

[0072] Based on the current time travel feature mapping information of the manned mechanical dog, the positions of the manned mechanical dog, the static obstacles and the dynamic obstacles are predicted,

[0073] If all the predicted positions of the manned mechanical dog, whose distance from the predicted positions of each static obstacle and dynamic obstacle at the same time is less than or equal to the preset distance, do not exist, it is determined that the travel interference region of the current manned mechanical dog is an empty set, and the travel path of the manned mechanical dog does not need to be planned for obstacle avoidance at this time.

[0074] If all the predicted positions of the manned mechanical dog, whose distance from the predicted positions of each static obstacle and dynamic obstacle at the same time is less than or equal to the preset distance, exist, it is determined that the travel interference region of the current manned mechanical dog is not an empty set, and the travel path of the manned mechanical dog needs to be planned for obstacle avoidance at this time. In the process of obstacle avoidance planning, the region formed by the predicted positions of each static obstacle and dynamic obstacle corresponding to 30 seconds needs to be avoided.

[0075] S3, calculate the control fluctuation value of each to-be-updated obstacle avoidance mapping planning scheme of the current manned mechanical dog based on the travel planning path of the manned mechanical dog stored in the current database, select the best obstacle avoidance mapping planning scheme, combine the initial planning path corresponding to the travel planning path of the manned mechanical dog stored in the current database, obtain the path control integration fluctuation coefficient corresponding to the current manned mechanical dog, and predict the travel endurance range of the subsequent travel planning path of the manned mechanical dog.

[0076] The optimal obstacle avoidance mapping planning scheme is a to-be-updated obstacle avoidance mapping planning scheme with the minimum regulation fluctuation value of the manned mechanical dog travel planning path stored in the current database based on the current time; the formula for calculating the regulation fluctuation value of each to-be-updated obstacle avoidance mapping planning scheme of the manned mechanical dog based on the manned mechanical dog travel planning path stored in the current database at the current time is as follows: C i = LG i / LB i , wherein C i represents the regulation fluctuation value of the i-th to-be-updated obstacle avoidance mapping planning scheme of the manned mechanical dog at the current time based on the manned mechanical dog travel planning path stored in the current database; LG i represents the total length of the planning path of the corresponding obstacle avoidance path planning scheme in the i-th to-be-updated obstacle avoidance mapping planning scheme of the manned mechanical dog at the current time; and LB i represents the total length of the distance of the corresponding obstacle planning to-be-updated region Q in the i-th to-be-updated obstacle avoidance mapping planning scheme of the manned mechanical dog at the current time.

[0077] The calculation formula of the path regulation integration fluctuation coefficient corresponding to the manned mechanical dog at the current time is as follows:

[0078] CP = (LS + LGZ - LY) / (LCY - LY + LB i )

[0079] , wherein CP represents the path regulation integration fluctuation coefficient corresponding to the manned mechanical dog at the current time; LS represents the actual driving distance between the starting point of the initial planning path corresponding to the manned mechanical dog travel planning path stored in the current database and the position of the manned mechanical dog at the current time; LGZ represents the total length of the planning path of the obstacle path planning scheme in the optimal obstacle avoidance mapping planning scheme; LCY represents the total length of the path between the starting point of the initial planning path corresponding to the manned mechanical dog travel planning path stored in the current database and the mapping point of the position of the manned mechanical dog at the current time; LY represents the path length of the intersection section between the path corresponding to LCY and the actual driving path corresponding to LS; the mapping point of the position of the manned mechanical dog at the current time in the initial planning path corresponding to the manned mechanical dog travel planning path stored in the current database is the same position point as the foot of the perpendicular line of the position of the manned mechanical dog at the current time in the corresponding initial planning path on the plane reference line of the corresponding initial planning path, and the plane reference line of the corresponding initial planning path represents the projection line segment of the connecting line between the initial point and the terminal point of the corresponding initial planning path on the horizontal plane;

[0080] The prediction result of the travel endurance mileage fluctuation interval of the subsequent travel planning path of the manned mechanical dog is denoted as [LEB, LEF],

[0081] LEB = LW + LGZ - LB i

[0082] LEF = LW CP

[0083] Wherein, LEB represents the minimum value in the prediction result of the travel endurance fluctuation interval of the subsequent travel planning path of the manned mechanical dog; LEF represents the maximum value in the prediction result of the travel endurance fluctuation interval of the subsequent travel planning path of the manned mechanical dog; and LW represents the path length between the end point of the travel planning path of the manned mechanical dog stored in the current database and the position of the manned mechanical dog at the current time.

[0084] S4, extracting state information in the travel characteristic mapping information of the manned mechanical dog, combining the travel endurance fluctuation interval prediction value of the subsequent travel planning path of the manned mechanical dog, and analyzing the cooperative demand coefficient of the manned mechanical dog at the current time, to complete the cooperative demand judgment of the manned mechanical dog;

[0085] The method for analyzing the cooperative demand coefficient of the manned mechanical dog at the current time comprises the following steps:

[0086] S41, obtaining state information in the travel characteristic mapping information of the manned mechanical dog, the travel endurance fluctuation interval prediction value of the subsequent travel planning path of the manned mechanical dog, and the empty-load cooperative objects around the manned mechanical dog at the current time, wherein the empty-load cooperative objects represent all manned mechanical dogs that do not perform manned tasks within a preset radius distance range from the manned mechanical dog at the current time;

[0087] S42, obtaining the cooperative demand coefficient of the manned mechanical dog at the current time, and the calculation formula is as follows:

[0088]

[0089] Wherein, SC represents the cooperative demand coefficient of the manned mechanical dog at the current time; LR represents the remaining endurance in the state information in the travel characteristic mapping information of the manned mechanical dog; LD represents the shortest distance between the empty-load cooperative objects around the manned mechanical dog at the current time and the position of the manned mechanical dog at the current time; H{} represents a comparison and judgment function; if If

[0090] In the cooperative demand judgment process of the manned mechanical dog, if SC is greater than or equal to a preset cooperative demand threshold value, it is judged that the manned mechanical dog at the current time has a cooperative demand; otherwise, it is judged that the manned mechanical dog at the current time does not have a cooperative demand.

[0091] ​​S5, updating the walking planning path of the manned mechanical dog stored in the current database according to the collaborative demand determination result of the manned mechanical dog, and feeding back the updated walking planning path of the manned mechanical dog and the walking feature mapping information of the manned mechanical dog to the remote monitoring end in real time.

[0092] In the process of updating the walking planning path of the manned mechanical dog stored in the current database in S5, if there is no collaborative demand of the manned mechanical dog at the current time, the road section corresponding to Q in the initial planning path corresponding to the walking planning path of the manned mechanical dog stored in the current database is replaced by the obstacle avoidance path planning scheme in the corresponding optimal obstacle avoidance mapping planning scheme; if there is a collaborative demand of the manned mechanical dog at the current time, the walking path between the manned mechanical dog at the current time and the closest empty collaborative object around the manned mechanical dog at the current time is planned based on the OB area by the Beidou navigation software, and the obtained shortest planning path is replaced by the walking planning path of the manned mechanical dog stored in the current database. The intersection between the obtained planning path and the OB area is an empty set.

[0093] It should be noted that in this document, the terms such as first and second are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or equipment including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or equipment.

[0094] Finally, it should be noted that: the above only describes the preferred embodiments of the present application, and does not limit the present application. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments, or make equivalent replacements to some technical features. Any modification, equivalent replacement, improvement, etc. within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for controlling the movement path of a manned robotic dog based on multimodal perception, characterized in that, The method includes the following steps: S1. Real-time acquisition of the state information and environmental information of the manned mechanical dog during its movement using multimodal sensors, and generation of manned mechanical dog movement feature mapping information; S2. Obtain the manned robot dog's travel planning path stored in the current database, combine it with the manned robot dog's travel feature mapping information at the current time, generate the manned robot dog's travel interference area at the current time, and obtain the obstacle avoidance mapping planning scheme to be updated for the manned robot dog at the current time. S3. Calculate each obstacle avoidance mapping plan scheme to be updated for the manned robot dog at the current time. Based on the adjustment fluctuation value of the manned robot dog's travel planning path stored in the current database, select the best obstacle avoidance mapping plan scheme. Combine the initial planning path corresponding to the manned robot dog's travel planning path stored in the current database to obtain the path adjustment integration fluctuation coefficient corresponding to the manned robot dog at the current time, and predict the travel range fluctuation range of the manned robot dog's subsequent travel planning path. The optimal obstacle avoidance mapping plan is the one that minimizes the adjustment fluctuation value of the manned robot dog's planned path stored in the current database at the current time. The formula for calculating the adjustment fluctuation value of each obstacle avoidance mapping plan to be updated for the manned robot dog at the current time based on the manned robot dog's planned path stored in the current database is as follows: , where C i This represents the adjustment fluctuation value of the i-th obstacle avoidance mapping plan of the manned robot dog to be updated at the current time, based on the manned robot dog's travel planning path stored in the current database; LG i This represents the total length of the planned path for the i-th obstacle avoidance mapping plan that needs updating in the current time for the manned robotic dog; LB i This represents the total length of the obstacle avoidance planning region Q to be updated in the i-th obstacle avoidance mapping planning scheme to be updated for the manned robotic dog at the current time. The formula for calculating the path control integration fluctuation coefficient of the manned robotic dog at the current time is as follows: ; Wherein, CP represents the path control integration fluctuation coefficient corresponding to the current time of the manned robot dog; LS represents the actual travel distance between the starting point of the initial planned path corresponding to the manned robot dog's travel planning path stored in the current database and the current time of the manned robot dog's position; LGZ represents the total length of the planned path of the obstacle avoidance path planning scheme in the optimal obstacle avoidance mapping planning scheme; LCY represents the total length of the path between the starting point of the initial planned path corresponding to the manned robot dog's travel planning path stored in the current database and the mapping point of the current time of the manned robot dog's position; LY represents the path length of the intersection segment between the path corresponding to LCY and the actual travel path corresponding to LS; the mapping point of the current time of the manned robot dog's position in the initial planned path corresponding to the manned robot dog's travel planning path stored in the current database is the position point in the corresponding initial planned path where the foot of the perpendicular line of the perpendicular line of the current time of the manned robot dog's position on the plane reference line of the corresponding initial planned path is the same as the foot of the perpendicular line of the perpendicular line of the corresponding initial planned path; the plane reference line of the corresponding initial planned path represents the projection line segment of the line connecting the initial point and the end point in the corresponding initial planned path on the horizontal plane; The predicted range of the manned robotic dog's subsequent travel planning path is denoted as [LEB, LEF]. ； Where LEB represents the minimum value in the predicted range of the manned robot dog's subsequent planned travel distance; LEF represents the maximum value in the predicted range of the manned robot dog's subsequent planned travel distance; and LW represents the path length between the endpoint of the manned robot dog's planned travel distance and the current position of the manned robot dog in the current database. S4. Extract the state information from the manned robot dog's travel feature mapping information, combine it with the predicted value of the travel range fluctuation range of the manned robot dog's subsequent travel planning path, and analyze the coordination demand coefficient of the manned robot dog at the current time, and complete the determination of the manned robot dog's coordination demand. The method for analyzing the collaborative demand coefficient of the manned robotic dog at the current time includes the following steps: S41. Obtain the status information in the manned robot dog's travel feature mapping information, the predicted value of the travel range fluctuation range of the manned robot dog's subsequent travel planning path, and the empty collaborative objects around the manned robot dog at the current time. The empty collaborative objects represent all manned robots that have not performed manned tasks within a preset radius distance from the manned robot dog at the current time. S42. Obtain the coordination requirement coefficient of the manned robotic dog at the current time. The calculation formula is as follows: ； Where SC represents the collaboration requirement coefficient of the manned robot dog at the current time; LR represents the remaining range in the status information of the manned robot dog's travel feature mapping information; LD represents the shortest distance between the current time's location and the location of the manned robot dog among the empty collaboration objects around it; H{} represents the comparison judgment function; if ,but ;like ,but ; During the process of determining the collaborative needs of the manned robot dog, if SC is greater than or equal to the preset collaborative needs threshold, it is determined that the manned robot dog has a collaborative needs at the current time; otherwise, it is determined that the manned robot dog does not have a collaborative needs at the current time. S5. Based on the determination result of the manned robot dog's collaborative needs, update the manned robot dog's travel planning path stored in the current database, and feed back the updated manned robot dog's travel planning path and manned robot dog's travel feature mapping information to the remote monitoring terminal in real time.

2. The method for controlling the movement path of a manned robotic dog based on multimodal perception according to claim 1, characterized in that... The manned robotic dog's movement feature mapping information consists of the manned robotic dog's state information and environmental information at the same time. The manned robotic dog's state information includes remaining range, speed, and current position. The environmental information includes a map model, static obstacle information, and dynamic obstacle information within a preset unit radius around the manned robotic dog. The map model within a preset unit radius around the manned robotic dog is obtained by processing images collected within that radius using a map building tool. The types of static and dynamic obstacles are obtained through image recognition results of images collected within a preset unit radius around the manned robotic dog. The static obstacle information includes the distribution position and size of each static obstacle in the corresponding image recognition results based on the manned robotic dog's current position. The dynamic obstacle information includes the distribution position, size, speed, and direction of movement of each dynamic obstacle in the corresponding image recognition results based on the manned robotic dog's current position. The speed of movement is represented by the magnitude of the vector formed by the first appearance position of the corresponding dynamic obstacle in the most recently collected image within a preset unit time period and its current position, divided by the time interval between the first appearance time of the corresponding dynamic obstacle and the current time. The direction of movement is represented by the direction of the vector formed by the first appearance position of the corresponding dynamic obstacle in the most recently collected image within a preset unit time period and its current position. If the corresponding dynamic obstacle appears for the first time in the image acquired in the most recent preset unit time, it is determined that the speed of the corresponding dynamic obstacle is 0 and the direction of movement is the same as the direction of movement of the manned mechanical dog at the current time.

3. The method for controlling the movement path of a manned robotic dog based on multimodal perception according to claim 1, characterized in that... The method for generating the travel interference area of ​​the manned robotic dog at the current time includes the following steps: S21. Obtain the manned robot dog's planned path stored in the current database, as well as the manned robot dog's movement feature mapping information at the current time; S22. Obtain the distance to the obstacle furthest from the current time position in the environmental information and the state information in the current time travel feature mapping information of the manned mechanical dog, and denot it as the first feature distance. The obstacle includes static obstacles and dynamic obstacles. Calculate the quotient of the first feature distance and the travel speed in the state information, and denot it as T. Generate the dynamic reaction time interval [0, T]. S23. In the map model within the environmental information of the predicted manned robot dog's movement feature mapping information at the current time, the distribution positions of the manned robot dog, static obstacles, and dynamic obstacles at time t, where t∈[0,T]; the position of the manned robot dog at time t in the map model within the environmental information of the predicted manned robot dog's movement feature mapping information at the current time is a point in the manned robot dog's movement planning path stored in the current database, whose distance from the manned robot dog's current position is the product of the corresponding movement speed and t; the position of the static obstacle in the map model within the environmental information of the predicted manned robot dog's movement feature mapping information at the current time remains unchanged at time t; the position of the dynamic obstacle in the map model within the environmental information of the predicted manned robot dog's movement feature mapping information at the current time is a point whose distance from the corresponding dynamic obstacle's position is equal to the product of the movement speed and t in the corresponding dynamic obstacle's information and points in the same direction as the movement in the corresponding dynamic obstacle's information; S24. In the map model within the environmental information of the manned robot dog's current time travel feature mapping information, within [0, T], for all manned robot dog predicted position points whose distance between the predicted position of the manned robot dog and the predicted positions of each static obstacle and dynamic obstacle at the same time is less than or equal to a preset distance, mark the corresponding positions in the manned robot dog's travel planning path stored in the current database, and denote the set of the obtained marked position points as the travel interference area of ​​the manned robot dog at the current time; In the process of obtaining the obstacle avoidance mapping plan scheme to be updated for the manned robot dog at the current time, the travel interference area of ​​the manned robot dog at the current time is obtained. If the travel interference area of ​​the manned robot dog at the current time is an empty set, then there is no obstacle avoidance mapping plan scheme to be updated for the manned robot dog at the current time. If the travel interference area of ​​the manned robot dog at the current time is not an empty set, then the area formed by the distribution positions of static obstacles and dynamic obstacles in [0, T] is denoted as OB, and the intersection area of ​​the manned robot dog's travel planning path stored in the current database and OB is taken as the obstacle avoidance planning update area, denoted as Q. The obstacle avoidance path is planned based on the path segment corresponding to Q by the Beidou navigation software based on the OB area, generating different obstacle avoidance path planning schemes. The binding result of each obstacle avoidance path planning scheme with Q is taken as an obstacle avoidance mapping plan scheme to be updated for the manned robot dog at the current time.

4. The method for controlling the movement path of a manned robotic dog based on multimodal perception according to claim 3, characterized in that: In step S5, during the process of updating the manned robot dog's planned travel path stored in the current database, if the manned robot dog does not have a coordination requirement at the current time, the road segment corresponding to Q in the initial planned path of the manned robot dog's planned travel path stored in the current database is replaced with the obstacle avoidance path planning scheme in the corresponding best obstacle avoidance mapping planning scheme; if the manned robot dog has a coordination requirement at the current time, the Beidou navigation software plans the travel path between the manned robot dog at the current time and the nearest empty coordination object around the manned robot dog at the current time based on the OB area, and replaces the manned robot dog's planned travel path stored in the current database with the shortest planned path obtained; the intersection between the obtained planned path and the OB area is an empty set.

5. A manned robotic dog path control system based on multimodal perception, employing the manned robotic dog path control method based on multimodal perception as described in any one of claims 1-4, characterized in that, The system includes the following modules: The traveling feature mapping information acquisition module acquires the state information and environmental information of the manned mechanical dog in real time through a multimodal sensor, and generates the traveling feature mapping information of the manned mechanical dog. The travel interference analysis module obtains the travel planning path of the manned robot dog stored in the current database, combines it with the travel feature mapping information of the manned robot dog at the current time, generates the travel interference area of ​​the manned robot dog at the current time, and obtains the obstacle avoidance mapping planning scheme to be updated for the manned robot dog at the current time. The obstacle avoidance planning scheme analysis module calculates the adjustment fluctuation value of each obstacle avoidance mapping planning scheme to be updated for the manned robot dog at the current time based on the manned robot dog's travel planning path stored in the current database, and selects the best obstacle avoidance mapping planning scheme. By combining the initial planned path corresponding to the manned robot dog's travel planning path stored in the current database, the path regulation and integration fluctuation coefficient corresponding to the manned robot dog at the current time is obtained, and the travel range fluctuation range of the manned robot dog's subsequent travel planning path is predicted. The empty object collaboration determination and management module extracts the state information from the manned robot dog's travel feature mapping information, combines the predicted value of the travel range fluctuation range of the manned robot dog's subsequent travel planning path, and the empty collaborative objects around the manned robot dog at the current time, analyzes the collaboration demand coefficient of the manned robot dog at the current time, and completes the collaboration demand determination of the manned robot dog. The travel path control and management module updates the travel planning path of the manned robot dog stored in the current database based on the collaborative requirement judgment result of the manned robot dog, and feeds back the updated travel planning path of the manned robot dog and the travel feature mapping information of the manned robot dog to the remote monitoring terminal in real time.

6. The manned mechanical dog path control system based on multimodal perception according to claim 5, characterized in that: The obstacle avoidance planning scheme analysis module includes an optimal obstacle avoidance mapping planning scheme screening unit, a path control integration analysis unit, and a range fluctuation analysis unit. The optimal obstacle avoidance mapping planning scheme screening unit calculates each obstacle avoidance mapping planning scheme to be updated for the manned robot dog at the current time based on the adjustment fluctuation value of the manned robot dog's travel planning path stored in the current database, and screens the optimal obstacle avoidance mapping planning scheme. The path control integration analysis unit combines the initial planning path corresponding to the manned mechanical dog's travel planning path stored in the current database to obtain the path control integration fluctuation coefficient corresponding to the manned mechanical dog at the current time. The range fluctuation analysis unit predicts the range fluctuation range of the manned mechanical dog's subsequent travel planning path based on the results obtained from the optimal obstacle avoidance mapping planning scheme screening unit and the path control integration analysis unit.

Images