Manned mechanical dog advancing path regulation and control system and method based on multi-modal perception

Abstract

The invention relates to the technical field of manned mechanical dog marching path regulation and control, in particular to a manned mechanical dog marching path regulation and control system and method based on multi-modal perception, and the system comprises a marching interference analysis module which obtains a manned mechanical dog marching planning path stored in a current database, and in combination with the marching feature mapping information of the manned mechanical dog at the current time, generating a marching interference area of the manned mechanical dog at the current time, and obtaining a to-be-updated obstacle avoidance mapping planning scheme of the manned mechanical dog at the current time. According to the method, the condition of the obstacle is considered, the state information of the manned mechanical dog and the manned task execution capability are also considered, planning of the regulation and control scheme of the obstacle avoidance path of the manned mechanical dog and dynamic judgment of the cooperation requirement of the manned mechanical dog are achieved, and the cooperation requirement of the manned mechanical dog is determined according to the obstacle avoidance scheme planning result and the cooperation requirement judgment result. And dynamic regulation and control of the advancing path of the manned mechanical dog are realized.

Description

Technical Field

[0001] This invention relates to the field of path control technology for manned robotic dogs, specifically to a path control system and method for manned robotic dogs based on multimodal perception. Background Technology

[0002] With the rapid development of sensor technology and artificial intelligence, multimodal perception technology is gradually being applied to various fields, especially robotics. For manned robotic dogs, a new type of mobile robot, the control of their movement path is crucial for their stability and safety; multimodal perception technology can improve the robot's ability to perceive its environment by fusing data from multiple sensors, thereby achieving precise control of its movement path.

[0003] Existing multimodal perception-based manned robotic dog path control systems only perceive the location of obstacles and plan obstacle avoidance paths and control the movement path. However, the path control process does not take into account the manned robotic dog's own state information and manned mission execution capabilities (judging whether it can complete the manned mission and whether surrounding manned robotic dogs need to provide coordination). Therefore, the existing technology has significant shortcomings. Summary of the Invention

[0004] The purpose of this invention is to provide a system and method for controlling the movement path of a manned robotic dog based on multimodal perception, so as to solve the problems mentioned in the background art.

[0005] To address the aforementioned technical problems, this invention provides the following technical solution: a method for controlling the movement path of a manned robotic dog based on multimodal perception, the method comprising the following steps:

[0006] S1. Real-time acquisition of the state information and environmental information of the manned mechanical dog during its movement using multimodal sensors, and generation of manned mechanical dog movement feature mapping information;

[0007] S2. Obtain the manned robot dog's travel planning path stored in the current database, combine it with the manned robot dog's travel feature mapping information at the current time, generate the manned robot dog's travel interference area at the current time, and obtain the obstacle avoidance mapping planning scheme to be updated for the manned robot dog at the current time.

[0008] S3. Calculate each obstacle avoidance mapping plan scheme to be updated for the manned robot dog at the current time. Based on the adjustment fluctuation value of the manned robot dog's travel planning path stored in the current database, select the best obstacle avoidance mapping plan scheme. Combine the initial planning path corresponding to the manned robot dog's travel planning path stored in the current database to obtain the path adjustment integration fluctuation coefficient corresponding to the manned robot dog at the current time, and predict the travel range fluctuation range of the manned robot dog's subsequent travel planning path.

[0009] S4. Extract the state information from the manned robot dog's travel feature mapping information, combine it with the predicted value of the travel range fluctuation range of the manned robot dog's subsequent travel planning path, and analyze the coordination demand coefficient of the manned robot dog at the current time, and complete the determination of the manned robot dog's coordination demand.

[0010] S5. Based on the determination result of the manned robot dog's collaborative needs, update the manned robot dog's travel planning path stored in the current database, and feed back the updated manned robot dog's travel planning path and manned robot dog's travel feature mapping information to the remote monitoring terminal in real time.

[0011] Furthermore, the manned robotic dog's movement feature mapping information consists of the manned robotic dog's state information and environmental information during its movement at the same time. The manned robotic dog's state information includes remaining range, speed, and current position. The environmental information includes a map model within a preset unit radius around the manned robotic dog, static obstacle information, and dynamic obstacle information. The map model within a preset unit radius around the manned robotic dog is obtained by processing images collected within that radius using a map building tool. The types of static and dynamic obstacles are obtained through image recognition results of the images collected within the preset unit radius around the manned robotic dog. The static obstacle information includes information on each static obstacle in the corresponding image recognition results based on the manned robotic dog's movement. The distribution and size of the manned robot dog at its current position; the dynamic obstacle information includes the distribution, size, speed, and direction of movement of each dynamic obstacle based on the current position of the manned robot dog in the corresponding image recognition results. The speed is represented by the magnitude of the vector formed by the first appearance position of the corresponding dynamic obstacle in the image acquired in the most recent preset unit time and the current position, divided by the time interval between the first appearance time of the corresponding dynamic obstacle and the current time. The direction of movement is represented by the direction of the vector formed by the first appearance position of the corresponding dynamic obstacle in the image acquired in the most recent preset unit time and the current position. If the corresponding dynamic obstacle appears for the first time in the image acquired in the most recent preset unit time, then the speed of movement in the corresponding dynamic obstacle information is determined to be 0 and the direction of movement is the same as the forward direction of the manned robot dog at the current time.

[0012] This invention collects information about the manned robotic dog from multiple angles and aspects, which is used to analyze and control the manned robotic dog's travel path in subsequent steps. Among them, the status information facilitates the control of the robotic dog's own situation, and is considered from three aspects: remaining driving range (facilitating the subsequent determination of the manned robotic dog's collaborative needs), travel speed, and current time and location (these two factors facilitate the prediction of the manned robotic dog's subsequent behavior path and obtain the current travel interference area of ​​the manned robotic dog).

[0013] Furthermore, the method for generating the current time's motion interference area for the manned robotic dog includes the following steps:

[0014] S21. Obtain the manned robot dog's planned path stored in the current database, as well as the manned robot dog's movement feature mapping information at the current time;

[0015] S22. Obtain the distance to the obstacle furthest from the current time position in the environmental information and the state information in the current time travel feature mapping information of the manned mechanical dog, and denot it as the first feature distance. The obstacle includes static obstacles and dynamic obstacles. Calculate the quotient of the first feature distance and the travel speed in the state information, and denot it as T. Generate the dynamic reaction time interval [0, T].

[0016] S23. In the map model within the environmental information of the predicted manned robot dog's movement feature mapping information at the current time, the distribution positions of the manned robot dog, static obstacles, and dynamic obstacles at time t, where t∈[0,T]; the position of the manned robot dog at time t in the map model within the environmental information of the predicted manned robot dog's movement feature mapping information at the current time is a point in the manned robot dog's movement planning path stored in the current database, whose distance from the manned robot dog's current position is the product of the corresponding movement speed and t; the position of the static obstacle in the map model within the environmental information of the predicted manned robot dog's movement feature mapping information at the current time remains unchanged at time t; the position of the dynamic obstacle in the map model within the environmental information of the predicted manned robot dog's movement feature mapping information at the current time is a point whose distance from the corresponding dynamic obstacle's position is equal to the product of the movement speed and t in the corresponding dynamic obstacle's information and points in the same direction as the movement in the corresponding dynamic obstacle's information;

[0017] S24. In the map model within the environmental information of the manned robot dog's current time travel feature mapping information, within [0, T], for all manned robot dog predicted position points whose distance between the predicted position of the manned robot dog and the predicted positions of each static obstacle and dynamic obstacle at the same time is less than or equal to a preset distance, mark the corresponding positions in the manned robot dog's travel planning path stored in the current database, and denote the set of the obtained marked position points as the travel interference area of ​​the manned robot dog at the current time;

[0018] In the process of obtaining the obstacle avoidance mapping plan scheme to be updated for the manned robot dog at the current time, the travel interference area of ​​the manned robot dog at the current time is obtained. If the travel interference area of ​​the manned robot dog at the current time is an empty set, then there is no obstacle avoidance mapping plan scheme to be updated for the manned robot dog at the current time. If the travel interference area of ​​the manned robot dog at the current time is not an empty set, then the area formed by the distribution positions of static obstacles and dynamic obstacles in [0, T] is denoted as OB, and the intersection area of ​​the manned robot dog's travel planning path stored in the current database and OB is taken as the obstacle avoidance planning update area, denoted as Q. The obstacle avoidance path is planned based on the path segment corresponding to Q by the Beidou navigation software based on the OB area, generating different obstacle avoidance path planning schemes. The binding result of each obstacle avoidance path planning scheme with Q is taken as an obstacle avoidance mapping plan scheme to be updated for the manned robot dog at the current time.

[0019] In the process of generating the travel interference area of ​​the manned robot dog and obtaining the obstacle avoidance mapping plan to be updated in this invention, the predicted positions of various static and dynamic obstacles within [0, T] are both required. However, there are significant differences in their application methods. When obtaining the travel interference area, the predicted positions of the manned robot dog within [0, T] are all the predicted positions of the manned robot dog whose distance from the predicted positions of the manned robot dog to the predicted positions of various static and dynamic obstacles at the same time is less than or equal to a preset distance. When obtaining the obstacle avoidance mapping plan to be updated, the OB is directly obtained. The reason for this is that if the travel interference area is not empty within [0, T], it means that the travel path of the manned robot dog needs to be adjusted. However, if the path adjustment is changed, the predicted position of the manned robot dog within [0, T] may also change. Therefore, using the above analysis method when obtaining the travel interference area may cause a large deviation in the subsequent path planning results.

[0020] Furthermore, the optimal obstacle avoidance mapping plan is the obstacle avoidance mapping plan to be updated that minimizes the control fluctuation value of the manned robot dog's planned path stored in the current database at the current time; the formula for calculating the control fluctuation value of each obstacle avoidance mapping plan to be updated for the manned robot dog at the current time based on the manned robot dog's planned path stored in the current database is as follows: C i =LG i / LB i , where C i This represents the adjustment fluctuation value of the i-th obstacle avoidance mapping plan of the manned robot dog to be updated at the current time, based on the manned robot dog's travel planning path stored in the current database; LG i This represents the total length of the planned path for the i-th obstacle avoidance mapping plan that needs updating in the current time for the manned robotic dog; LBi This represents the total length of the obstacle avoidance planning region Q to be updated in the i-th obstacle avoidance mapping planning scheme to be updated for the manned robotic dog at the current time.

[0021] The formula for calculating the path control integration fluctuation coefficient of the manned robotic dog at the current time is as follows:

[0022] CP = (LS + LGZ - LY) / (LCY - LY + LB) i )

[0023] Wherein, CP represents the path control integration fluctuation coefficient corresponding to the current time of the manned robot dog; LS represents the actual travel distance between the starting point of the initial planned path corresponding to the manned robot dog's travel planning path stored in the current database and the current time of the manned robot dog's position; LGZ represents the total length of the planned path of the obstacle avoidance path planning scheme in the optimal obstacle avoidance mapping planning scheme; LCY represents the total length of the path between the starting point of the initial planned path corresponding to the manned robot dog's travel planning path stored in the current database and the mapping point of the current time of the manned robot dog's position; LY represents the path length of the intersection segment between the path corresponding to LCY and the actual travel path corresponding to LS; the mapping point of the current time of the manned robot dog's position in the initial planned path corresponding to the manned robot dog's travel planning path stored in the current database is the position point in the corresponding initial planned path where the foot of the perpendicular line of the perpendicular line of the current time of the manned robot dog's position on the plane reference line of the corresponding initial planned path is the same as the foot of the perpendicular line of the perpendicular line of the corresponding initial planned path; the plane reference line of the corresponding initial planned path represents the projection line segment of the line connecting the initial point and the end point in the corresponding initial planned path on the horizontal plane;

[0024] The predicted range of the manned robotic dog's subsequent travel planning path is denoted as [LEB, LEF].

[0025] LEB = LW + LGZ - LB i

[0026] LEF = LW·CP

[0027] Where LEB represents the minimum value in the predicted range of the manned robot dog's subsequent planned travel distance; LEF represents the maximum value in the predicted range of the manned robot dog's subsequent planned travel distance; and LW represents the path length between the endpoint of the manned robot dog's planned travel distance and the current position of the manned robot dog in the current database.

[0028] Furthermore, the method for analyzing the collaborative demand coefficient of the manned robotic dog at the current time includes the following steps:

[0029] S41. Obtain the status information in the manned robot dog's travel feature mapping information, the predicted value of the travel range fluctuation range of the manned robot dog's subsequent travel planning path, and the empty collaborative objects around the manned robot dog at the current time. The empty collaborative objects represent all manned robots that have not performed manned tasks within a preset radius distance from the manned robot dog at the current time.

[0030] S42. Obtain the coordination requirement coefficient of the manned robotic dog at the current time. The calculation formula is as follows:

[0031]

[0032] Where SC represents the collaboration requirement coefficient of the manned robot dog at the current time; LR represents the remaining range in the status information of the manned robot dog's travel feature mapping information; LD represents the shortest distance between the current time's location and the location of the manned robot dog among the empty collaboration objects around it; H{} represents the comparison judgment function; if but like but

[0033] During the process of determining the collaborative needs of the manned robot dog, if SC is greater than or equal to the preset collaborative needs threshold, it is determined that the manned robot dog has a collaborative needs at the current time; otherwise, it is determined that the manned robot dog does not have a collaborative needs at the current time.

[0034] Furthermore, in step S5, during the process of updating the manned robot dog's planned travel path stored in the current database, if the manned robot dog does not have a coordination requirement at the current time, the road segment corresponding to Q in the initial planned path of the manned robot dog's planned travel path stored in the current database is replaced with the obstacle avoidance path planning scheme in the corresponding optimal obstacle avoidance mapping planning scheme; if the manned robot dog has a coordination requirement at the current time, the Beidou navigation software plans the travel path between the manned robot dog at the current time and the nearest empty coordination object around the manned robot dog based on the OB area, and replaces the manned robot dog's planned travel path stored in the current database with the shortest planned path obtained; the intersection between the obtained planned path and the OB area is an empty set.

[0035] A multimodal perception-based path control system for manned robotic dogs, comprising the following modules:

[0036] The traveling feature mapping information acquisition module acquires the state information and environmental information of the manned mechanical dog in real time through a multimodal sensor, and generates the traveling feature mapping information of the manned mechanical dog.

[0037] The travel interference analysis module obtains the travel planning path of the manned robot dog stored in the current database, combines it with the travel feature mapping information of the manned robot dog at the current time, generates the travel interference area of ​​the manned robot dog at the current time, and obtains the obstacle avoidance mapping planning scheme to be updated for the manned robot dog at the current time.

[0038] The obstacle avoidance planning scheme analysis module calculates the adjustment fluctuation value of each obstacle avoidance mapping planning scheme to be updated for the manned robot dog at the current time based on the manned robot dog's travel planning path stored in the current database, and selects the best obstacle avoidance mapping planning scheme; combined with the initial planning path corresponding to the manned robot dog's travel planning path stored in the current database, it obtains the path adjustment integration fluctuation coefficient corresponding to the manned robot dog at the current time, and predicts the travel range fluctuation range of the manned robot dog's subsequent travel planning path;

[0039] The empty object collaboration determination and management module extracts the state information from the manned robot dog's travel feature mapping information, combines the predicted value of the travel range fluctuation range of the manned robot dog's subsequent travel planning path, and the empty collaborative objects around the manned robot dog at the current time, analyzes the collaboration demand coefficient of the manned robot dog at the current time, and completes the collaboration demand determination of the manned robot dog.

[0040] The travel path control and management module updates the travel planning path of the manned robot dog stored in the current database based on the collaborative requirement judgment result of the manned robot dog, and feeds back the updated travel planning path of the manned robot dog and the travel feature mapping information of the manned robot dog to the remote monitoring terminal in real time.

[0041] Furthermore, the obstacle avoidance planning scheme analysis module includes an optimal obstacle avoidance mapping planning scheme screening unit, a path control integration analysis unit, and a range fluctuation analysis unit.

[0042] The optimal obstacle avoidance mapping planning scheme screening unit calculates each obstacle avoidance mapping planning scheme to be updated for the manned robot dog at the current time based on the adjustment fluctuation value of the manned robot dog's travel planning path stored in the current database, and screens the optimal obstacle avoidance mapping planning scheme.

[0043] The path control integration analysis unit combines the initial planning path corresponding to the manned mechanical dog's travel planning path stored in the current database to obtain the path control integration fluctuation coefficient corresponding to the manned mechanical dog at the current time.

[0044] The range fluctuation analysis unit predicts the range fluctuation range of the manned mechanical dog's subsequent travel planning path based on the results obtained from the optimal obstacle avoidance mapping planning scheme screening unit and the path control integration analysis unit.

[0045] Compared with the prior art, the beneficial effects achieved by the present invention are as follows: In the process of controlling the movement path of the manned mechanical dog, the present invention not only considers the situation of obstacles, but also takes into account the state information of the manned mechanical dog itself and the manned task execution capability, realizes the planning of the obstacle avoidance path control scheme of the manned mechanical dog and the dynamic determination of the collaborative needs of the manned mechanical dog, and realizes the dynamic control of the movement path of the manned mechanical dog based on the obstacle avoidance scheme planning results and collaborative need determination results. Attached Figure Description

[0046] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0047] Figure 1 This is a schematic diagram of the structure of the manned mechanical dog movement path control system based on multimodal perception of the present invention;

[0048] Figure 2 This is a flowchart illustrating the method for controlling the movement path of a manned mechanical dog based on multimodal perception, as proposed in this invention. Detailed Implementation

[0049] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0050] See also Figure 1 The present invention provides a technical solution: a manned mechanical dog movement path control system based on multimodal perception, the system comprising the following modules:

[0051] The traveling feature mapping information acquisition module acquires the state information and environmental information of the manned mechanical dog in real time through a multimodal sensor, and generates the traveling feature mapping information of the manned mechanical dog.

[0052] The travel interference analysis module obtains the travel planning path of the manned robot dog stored in the current database, combines it with the travel feature mapping information of the manned robot dog at the current time, generates the travel interference area of ​​the manned robot dog at the current time, and obtains the obstacle avoidance mapping planning scheme to be updated for the manned robot dog at the current time.

[0053] The obstacle avoidance planning scheme analysis module includes an optimal obstacle avoidance mapping planning scheme screening unit, a path control integration analysis unit, and a range fluctuation analysis unit.

[0054] The optimal obstacle avoidance mapping planning scheme screening unit calculates each obstacle avoidance mapping planning scheme to be updated for the manned robot dog at the current time based on the adjustment fluctuation value of the manned robot dog's travel planning path stored in the current database, and screens the optimal obstacle avoidance mapping planning scheme.

[0055] The path control integration analysis unit combines the initial planning path corresponding to the manned mechanical dog's travel planning path stored in the current database to obtain the path control integration fluctuation coefficient corresponding to the manned mechanical dog at the current time.

[0056] The range fluctuation analysis unit predicts the range fluctuation range of the manned mechanical dog's subsequent travel planning path based on the results obtained by the optimal obstacle avoidance mapping planning scheme screening unit and the path control integration analysis unit.

[0057] The empty object collaboration determination and management module extracts the state information from the manned robot dog's travel feature mapping information, combines the predicted value of the travel range fluctuation range of the manned robot dog's subsequent travel planning path, and the empty collaborative objects around the manned robot dog at the current time, analyzes the collaboration demand coefficient of the manned robot dog at the current time, and completes the collaboration demand determination of the manned robot dog.

[0058] The travel path control and management module updates the travel planning path of the manned robot dog stored in the current database based on the collaborative requirement judgment result of the manned robot dog, and feeds back the updated travel planning path of the manned robot dog and the travel feature mapping information of the manned robot dog to the remote monitoring terminal in real time.

[0059] like Figure 2 As shown, a method for controlling the movement path of a manned robotic dog based on multimodal perception includes the following steps:

[0060] S1. Real-time acquisition of the state information and environmental information of the manned mechanical dog during its movement using multimodal sensors, and generation of manned mechanical dog movement feature mapping information;

[0061] The manned robotic dog's movement feature mapping information consists of the manned robotic dog's state information and environmental information at the same time. The manned robotic dog's state information includes remaining range, speed, and current position. The environmental information includes a map model within a preset unit radius around the manned robotic dog, static obstacle information, and dynamic obstacle information. The map model within the preset unit radius around the manned robotic dog is obtained by processing images collected within that radius using a map building tool. The types of static and dynamic obstacles are obtained through image recognition results of the collected images within the preset unit radius around the manned robotic dog. The static obstacle information includes information on each static obstacle in the corresponding image recognition results based on the manned robotic dog's position. The distribution and size of the dog's current position; the dynamic obstacle information includes the distribution, size, speed, and direction of movement of each dynamic obstacle based on the current position of the manned mechanical dog in the corresponding image recognition results. The speed of movement is represented by the magnitude of the vector formed by the first appearance position of the corresponding dynamic obstacle in the image acquired in the most recent preset unit time and the current position, divided by the time interval between the first appearance time of the corresponding dynamic obstacle and the current time. The direction of movement is represented by the direction of the vector formed by the first appearance position of the corresponding dynamic obstacle in the image acquired in the most recent preset unit time and the current position. If the corresponding dynamic obstacle appears for the first time in the image acquired in the most recent preset unit time, then the speed of movement in the corresponding dynamic obstacle information is determined to be 0 and the direction of movement is the same as the forward direction of the manned mechanical dog at the current time.

[0062] In this embodiment, the map building tools that can be selected to build a map model based on the collected images include geospy software, ArcGIS software, etc.

[0063] In this embodiment, the types of static and dynamic obstacles are preset in the database. Static obstacles are stationary objects, including steps, buildings, trees, hillsides, and water bodies. Dynamic obstacles are movable objects, including pedestrians, animals, cars, and snow scenes.

[0064] S2. Obtain the manned robot dog's travel planning path stored in the current database, combine it with the manned robot dog's travel feature mapping information at the current time, generate the manned robot dog's travel interference area at the current time, and obtain the obstacle avoidance mapping planning scheme to be updated for the manned robot dog at the current time.

[0065] The method for generating the travel interference area of ​​the manned robotic dog at the current time includes the following steps:

[0066] S21. Obtain the manned robot dog's planned path stored in the current database, as well as the manned robot dog's movement feature mapping information at the current time;

[0067] S22. Obtain the distance to the obstacle furthest from the current time position in the environmental information and the state information in the current time travel feature mapping information of the manned mechanical dog, and denot it as the first feature distance. The obstacle includes static obstacles and dynamic obstacles. Calculate the quotient of the first feature distance and the travel speed in the state information, and denot it as T. Generate the dynamic reaction time interval [0, T].

[0068] S23. In the map model within the environmental information of the predicted manned robot dog's movement feature mapping information at the current time, the distribution positions of the manned robot dog, static obstacles, and dynamic obstacles at time t, where t∈[0,T]; the position of the manned robot dog at time t in the map model within the environmental information of the predicted manned robot dog's movement feature mapping information at the current time is a point in the manned robot dog's movement planning path stored in the current database, whose distance from the manned robot dog's current position is the product of the corresponding movement speed and t; the position of the static obstacle in the map model within the environmental information of the predicted manned robot dog's movement feature mapping information at the current time remains unchanged at time t; the position of the dynamic obstacle in the map model within the environmental information of the predicted manned robot dog's movement feature mapping information at the current time is a point whose distance from the corresponding dynamic obstacle's position is equal to the product of the movement speed and t in the corresponding dynamic obstacle's information and points in the same direction as the movement in the corresponding dynamic obstacle's information;

[0069] S24. In the map model within the environmental information of the manned robot dog's current time travel feature mapping information, within [0, T], for all manned robot dog predicted position points whose distance between the predicted position of the manned robot dog and the predicted positions of each static obstacle and dynamic obstacle at the same time is less than or equal to a preset distance, mark the corresponding positions in the manned robot dog's travel planning path stored in the current database, and denote the set of the obtained marked position points as the travel interference area of ​​the manned robot dog at the current time;

[0070] In the process of obtaining the obstacle avoidance mapping plan scheme to be updated for the manned robot dog at the current time, the travel interference area of ​​the manned robot dog at the current time is obtained. If the travel interference area of ​​the manned robot dog at the current time is an empty set, then there is no obstacle avoidance mapping plan scheme to be updated for the manned robot dog at the current time. If the travel interference area of ​​the manned robot dog at the current time is not an empty set, then the area formed by the distribution positions of static obstacles and dynamic obstacles in [0, T] is denoted as OB, and the intersection area of ​​the manned robot dog's travel planning path stored in the current database and OB is taken as the obstacle avoidance planning update area, denoted as Q. The obstacle avoidance path is planned based on the path segment corresponding to Q by the Beidou navigation software based on the OB area, generating different obstacle avoidance path planning schemes. The binding result of each obstacle avoidance path planning scheme with Q is taken as an obstacle avoidance mapping plan scheme to be updated for the manned robot dog at the current time.

[0071] In this embodiment, if T is 30 seconds in [0, T];

[0072] Based on the current time-based feature mapping information of the manned robotic dog, the positions of the manned robotic dog, static obstacles, and dynamic obstacles are predicted.

[0073] If there are no manned robot dog predicted positions whose distances to the predicted positions of static and dynamic obstacles at the same time are less than or equal to the preset distances, then the manned robot dog's travel interference area at the current time is determined to be an empty set, and obstacle avoidance planning for the manned robot dog's travel path is not required.

[0074] If there are all manned robot dog prediction positions whose distances to the predicted positions of static and dynamic obstacles at the same time are less than or equal to a preset distance, then it is determined that the manned robot dog's travel interference area at the current time is not an empty set. In this case, obstacle avoidance planning is required for the manned robot dog's travel path. During the obstacle avoidance planning process, the areas formed by the predicted positions of each static and dynamic obstacle within 30 seconds should be treated as obstacle areas for avoidance.

[0075] S3. Calculate each obstacle avoidance mapping plan scheme to be updated for the manned robot dog at the current time. Based on the adjustment fluctuation value of the manned robot dog's travel planning path stored in the current database, select the best obstacle avoidance mapping plan scheme. Combine the initial planning path corresponding to the manned robot dog's travel planning path stored in the current database to obtain the path adjustment integration fluctuation coefficient corresponding to the manned robot dog at the current time, and predict the travel range fluctuation range of the manned robot dog's subsequent travel planning path.

[0076] The optimal obstacle avoidance mapping plan is the one that minimizes the control fluctuation value of the manned robot dog's planned path stored in the current database at the current time. The formula for calculating the control fluctuation value of each obstacle avoidance mapping plan to be updated based on the manned robot dog's planned path stored in the current database at the current time is as follows: C i =LG i / LB i , where C i This represents the adjustment fluctuation value of the i-th obstacle avoidance mapping plan of the manned robot dog to be updated at the current time, based on the manned robot dog's travel planning path stored in the current database; LG i This represents the total length of the planned path for the i-th obstacle avoidance mapping plan that needs updating in the current time for the manned robotic dog; LB i This represents the total length of the obstacle avoidance planning region Q to be updated in the i-th obstacle avoidance mapping planning scheme to be updated for the manned robotic dog at the current time.

[0077] The formula for calculating the path control integration fluctuation coefficient of the manned robotic dog at the current time is as follows:

[0078] CP = (LS + LGZ - LY) / (LCY - LY + LB) i )

[0079] Wherein, CP represents the path control integration fluctuation coefficient corresponding to the current time of the manned robot dog; LS represents the actual travel distance between the starting point of the initial planned path corresponding to the manned robot dog's travel planning path stored in the current database and the current time of the manned robot dog's position; LGZ represents the total length of the planned path of the obstacle avoidance path planning scheme in the optimal obstacle avoidance mapping planning scheme; LCY represents the total length of the path between the starting point of the initial planned path corresponding to the manned robot dog's travel planning path stored in the current database and the mapping point of the current time of the manned robot dog's position; LY represents the path length of the intersection segment between the path corresponding to LCY and the actual travel path corresponding to LS; the mapping point of the current time of the manned robot dog's position in the initial planned path corresponding to the manned robot dog's travel planning path stored in the current database is the position point in the corresponding initial planned path where the foot of the perpendicular line of the perpendicular line of the current time of the manned robot dog's position on the plane reference line of the corresponding initial planned path is the same as the foot of the perpendicular line of the perpendicular line of the corresponding initial planned path; the plane reference line of the corresponding initial planned path represents the projection line segment of the line connecting the initial point and the end point in the corresponding initial planned path on the horizontal plane;

[0080] The predicted range of the manned robotic dog's subsequent travel planning path is denoted as [LEB, LEF].

[0081] LEB = LW + LGZ - LB i

[0082] LEF = LW·CP

[0083] Where LEB represents the minimum value in the predicted range of the manned robot dog's subsequent planned travel distance; LEF represents the maximum value in the predicted range of the manned robot dog's subsequent planned travel distance; and LW represents the path length between the endpoint of the manned robot dog's planned travel distance and the current position of the manned robot dog in the current database.

[0084] S4. Extract the state information from the manned robot dog's travel feature mapping information, combine it with the predicted value of the travel range fluctuation range of the manned robot dog's subsequent travel planning path, and analyze the coordination demand coefficient of the manned robot dog at the current time, and complete the determination of the manned robot dog's coordination demand.

[0085] The method for analyzing the collaborative demand coefficient of the manned robotic dog at the current time includes the following steps:

[0086] S41. Obtain the status information in the manned robot dog's travel feature mapping information, the predicted value of the travel range fluctuation range of the manned robot dog's subsequent travel planning path, and the empty collaborative objects around the manned robot dog at the current time. The empty collaborative objects represent all manned robots that have not performed manned tasks within a preset radius distance from the manned robot dog at the current time.

[0087] S42. Obtain the coordination requirement coefficient of the manned robotic dog at the current time. The calculation formula is as follows:

[0088]

[0089] Where SC represents the collaboration requirement coefficient of the manned robot dog at the current time; LR represents the remaining range in the status information of the manned robot dog's travel feature mapping information; LD represents the shortest distance between the current time's location and the location of the manned robot dog among the empty collaboration objects around it; H{} represents the comparison judgment function; if but like but

[0090] During the process of determining the collaborative needs of the manned robot dog, if SC is greater than or equal to the preset collaborative needs threshold, it is determined that the manned robot dog has a collaborative needs at the current time; otherwise, it is determined that the manned robot dog does not have a collaborative needs at the current time.

[0091] S5. Based on the determination result of the manned robot dog's collaborative needs, update the manned robot dog's travel planning path stored in the current database, and feed back the updated manned robot dog's travel planning path and manned robot dog's travel feature mapping information to the remote monitoring terminal in real time.

[0092] In step S5, during the process of updating the manned robot dog's planned travel path stored in the current database, if the manned robot dog does not have a coordination requirement at the current time, the road segment corresponding to Q in the initial planned path of the manned robot dog's planned travel path stored in the current database is replaced with the obstacle avoidance path planning scheme in the corresponding best obstacle avoidance mapping planning scheme; if the manned robot dog has a coordination requirement at the current time, the Beidou navigation software plans the travel path between the manned robot dog at the current time and the nearest empty coordination object around the manned robot dog at the current time based on the OB area, and replaces the manned robot dog's planned travel path stored in the current database with the shortest planned path obtained; the intersection between the obtained planned path and the OB area is an empty set.

[0093] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus.

[0094] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for controlling the travel path of a manned mechanical dog based on multimodal perception, characterized in that: The method comprises the following steps: S1. Using a multimodal sensor to obtain the state information and environment information of the manned mechanical dog during its movement in real time, and generate the manned mechanical dog's movement feature mapping information; S2, obtaining the manned mechanical dog's travel planning path stored in the current database, combining the manned mechanical dog's travel feature mapping information at the current time, generating the manned mechanical dog's travel interference area at the current time, and obtaining the manned mechanical dog's obstacle avoidance mapping planning scheme to be updated at the current time; S3, calculate each obstacle avoidance mapping planning scheme to be updated for the manned mechanical dog at the current time based on the control fluctuation value of the manned mechanical dog's travel planning path stored in the current database, and select the best obstacle avoidance mapping planning scheme; combine the initial planning path corresponding to the manned mechanical dog's travel planning path stored in the current database, obtain the path control integration fluctuation coefficient corresponding to the manned mechanical dog at the current time, and predict the travel range fluctuation range of the manned mechanical dog's subsequent travel planning path; S4, extracting the state information in the manned robot dog's travel feature mapping information, combining the predicted value of the travel range fluctuation interval of the manned robot dog's subsequent travel planning path, and the unloaded cooperative objects around the manned robot dog at the current time, analyzing the cooperation demand coefficient of the manned robot dog at the current time, and completing the manned robot dog's cooperation demand determination; S5. According to the result of the manned robot dog's collaborative demand determination, the manned robot dog's travel planning path stored in the current database is updated, and the updated manned robot dog's travel planning path and the manned robot dog's travel feature mapping information are fed back to the remote monitoring end in real time.

2. The method for controlling the travel path of a manned mechanical dog based on multimodal perception according to claim 1 is characterized in that The manned mechanical dog's travel characteristic mapping information is composed of the state information of the manned mechanical dog during its travel and the environment information corresponding to the same time; the state information of the manned mechanical dog during its travel includes the remaining cruising range, travel speed and current time position; the environment information includes the map model within a preset unit radius around the manned mechanical dog, static obstacle information and dynamic obstacle information; the map model within a preset unit radius around the manned mechanical dog is obtained by processing the collected images within a preset unit radius around the manned mechanical dog based on the map building tool; The types of static obstacles and dynamic obstacles are obtained through the image recognition results of the collected images within a preset unit radius around the manned robot dog; the static obstacle information includes the distribution position and size of each static obstacle in the corresponding image recognition results based on the current position of the manned robot dog; the dynamic obstacle information includes the distribution position, size, movement speed and movement direction of each dynamic obstacle in the corresponding image recognition results based on the current position of the manned robot dog, the movement speed represents the quotient of the modulus of the vector formed by the position of the first appearance of the corresponding dynamic obstacle in the image collected within the latest preset unit time and the current position divided by the interval between the first appearance time of the corresponding dynamic obstacle and the current time; the movement direction represents the direction of the vector formed by the position of the first appearance of the corresponding dynamic obstacle in the image collected within the latest preset unit time and the current position; If the corresponding dynamic obstacle appears for the first time in the image collected within the most recent preset unit time, it is determined that the movement speed in the corresponding dynamic obstacle information is 0 and the movement direction is the same as the forward direction of the manned mechanical dog at the current time.

3. The method for controlling the travel path of a manned mechanical dog based on multimodal perception according to claim 1 is characterized in that The method for generating the current time travel interference area of ​​the manned mechanical dog comprises the following steps: S21, obtaining the manned mechanical dog's travel planning path stored in the current database, as well as the manned mechanical dog's travel feature mapping information at the current time; S22, obtaining the distance of the obstacle farthest from the current time position in the state information in the environment information in the current time moving characteristic mapping information of the manned mechanical dog, recorded as the first characteristic distance, the obstacle includes static obstacles and dynamic obstacles; calculating the quotient of the first characteristic distance and the moving speed in the state information, recorded as T; generating a dynamic reaction time interval [0, T]; S23, predicting the distribution positions of the manned mechanical dog, static obstacles and dynamic obstacles at time t in the map model in the environment information of the manned mechanical dog's travel characteristic mapping information at the current time, wherein t∈[0, T]; the position of the manned mechanical dog at time t in the map model in the environment information of the manned mechanical dog's travel characteristic mapping information at the current time is a position point in the manned mechanical dog's travel planning path stored in the current database whose distance from the manned mechanical dog's current time position is the product of the corresponding travel speed and t; the position of the static obstacle at time t in the map model in the environment information of the manned mechanical dog's travel characteristic mapping information at the current time remains unchanged; the position of the dynamic obstacle at time t in the map model in the environment information of the manned mechanical dog's travel characteristic mapping information at the current time is a position point whose distance from the corresponding dynamic obstacle position is equal to the product of the movement speed in the corresponding dynamic obstacle information and t and points to the same movement direction as the corresponding dynamic obstacle information; S24, obtaining the map model in the environment information of the manned mechanical dog's travel feature mapping information at the current time, for all the manned mechanical dog's predicted position points whose distances between the manned mechanical dog's predicted position and the predicted positions of each static obstacle and dynamic obstacle at the same time are less than or equal to the preset distance within [0, T], and marking the corresponding positions in the manned mechanical dog's travel planning path stored in the current database, and recording the set of the obtained marked position points as the manned mechanical dog's travel interference area at the current time; In the process of obtaining the obstacle avoidance mapping planning scheme to be updated for the manned mechanical dog at the current time, the travel interference area of ​​the manned mechanical dog at the current time is obtained. If the travel interference area of ​​the manned mechanical dog at the current time is an empty set, then there is no obstacle avoidance mapping planning scheme to be updated for the manned mechanical dog at the current time; if the travel interference area of ​​the manned mechanical dog at the current time is not an empty set, then the area formed by the corresponding distribution positions of static obstacles and dynamic obstacles in [0, T] is recorded as OB, and the intersection area of ​​the manned mechanical dog travel planning path stored in the current database and OB is used as the obstacle avoidance planning to be updated area, recorded as Q; obstacle avoidance path planning is performed on the path segment corresponding to Q based on the OB area through the Beidou navigation software, and different obstacle avoidance path planning schemes are generated, and the result of binding each obstacle avoidance path planning scheme with Q is used as an obstacle avoidance mapping planning scheme to be updated for the manned mechanical dog at the current time.

4. The method for controlling the travel path of a manned mechanical dog based on multimodal perception according to claim 2 is characterized in that: The optimal obstacle avoidance mapping planning scheme is the obstacle avoidance mapping planning scheme to be updated with the minimum control fluctuation value of the manned mechanical dog's travel planning path stored in the current database at the current time; The formula for calculating the control fluctuation value of each obstacle avoidance mapping plan to be updated for the manned robot dog at the current time based on the manned robot dog's travel planning path stored in the current database is as follows: C i =LC i / LB i , where C i LG represents the control fluctuation value of the manned robot dog's planned travel path stored in the current database based on the i-th obstacle avoidance mapping planning scheme to be updated at the current time; i LB represents the total length of the planned path of the obstacle avoidance path planning scheme corresponding to the i-th obstacle avoidance mapping planning scheme to be updated at the current time; i Indicates the total length of the obstacle avoidance planning to be updated area Q corresponding to the i-th obstacle avoidance mapping planning scheme to be updated of the manned robot dog at the current time; The calculation formula for the path control integrated fluctuation coefficient corresponding to the manned robot dog at the current time is as follows: CP=(LS+LGZ-LY) / (LCY-LY+LB i ) Among them, CP represents the path control integrated fluctuation coefficient corresponding to the manned mechanical dog at the current time; LS represents the actual driving distance between the starting point of the initial planning path corresponding to the manned mechanical dog's travel planning path stored in the current database and the manned mechanical dog's position at the current time; LGZ represents the total length of the planned path of the obstacle avoidance path planning scheme in the optimal obstacle avoidance mapping planning scheme; LCY represents the total length of the path between the starting point of the initial planning path corresponding to the manned mechanical dog's travel planning path stored in the current database and the mapping point of the manned mechanical dog's position at the current time; LY represents the path length of the intersection section between the path corresponding to LCY and the actual driving path corresponding to LS; the mapping point of the manned mechanical dog's position at the current time in the initial planning path corresponding to the manned mechanical dog's travel planning path stored in the current database is the position point in the corresponding initial planning path that is the same as the foot of the perpendicular line of the manned mechanical dog's position at the current time on the plane reference line of the corresponding initial planning path, and the plane reference line of the corresponding initial planning path represents the projection line segment of the line connecting the initial point and the end point in the corresponding initial planning path on the horizontal plane; The predicted result of the range fluctuation range of the manned robot dog's subsequent planned path is recorded as [LEB, LEF]. <h2 style=";text-align:left;direction:ltr">LEB = LW + LGZ - LB<h2 style=";text-align:left;direction:ltr"> i LEF=LW·CP Among them, LEB represents the minimum value in the predicted results of the fluctuation range of the cruising range of the subsequent planned path of the manned robot dog; LEF represents the maximum value in the predicted results of the fluctuation range of the cruising range of the subsequent planned path of the manned robot dog; LW represents the path length between the end point of the manned robot dog's planned path stored in the current database and the current time position of the manned robot dog.

5. The method for controlling the travel path of a manned mechanical dog based on multimodal perception according to claim 4 is characterized in that The method for analyzing the coordination demand coefficient of the manned mechanical dog at the current time includes the following steps: S41, obtaining the state information in the manned mechanical dog's travel feature mapping information, the predicted value of the travel range fluctuation interval of the manned mechanical dog's subsequent travel planning path, and the unloaded cooperative objects around the manned mechanical dog at the current time, wherein the unloaded cooperative objects represent all manned mechanical dogs that are not performing manned tasks within a preset radius distance from the manned mechanical dog at the current time; S42. Obtain the coordination demand coefficient of the manned robot dog at the current time, and the calculation formula is as follows: Among them, SC represents the coordination demand coefficient of the manned robot dog at the current time; LR represents the remaining range in the state information in the manned robot dog's travel feature mapping information; LD represents the shortest distance between the unloaded coordination object around the manned robot dog at the current time and the manned robot dog's position at the current time; H{} represents the comparison judgment function; if but like but During the process of determining the coordination demand of the manned robot dog, if SC is greater than or equal to the preset coordination demand threshold, it is determined that there is a coordination demand for the manned robot dog at the current time; otherwise, it is determined that there is no coordination demand for the manned robot dog at the current time.

6. The method for controlling the travel path of a manned mechanical dog based on multimodal perception according to claim 3 is characterized in that: In the process of updating the manned mechanical dog's travel planning path stored in the current database in S5, if there is no coordination demand for the manned mechanical dog at the current time, the section corresponding to Q in the initial planning path corresponding to the manned mechanical dog's travel planning path stored in the current database is replaced with the obstacle avoidance path planning scheme in the corresponding optimal obstacle avoidance mapping planning scheme; if there is a coordination demand for the manned mechanical dog at the current time, the travel path between the manned mechanical dog at the current time and the unloaded coordination object closest to the manned mechanical dog at the current time is planned based on the OB area through the Beidou navigation software, and the planned path with the shortest distance is used to replace the manned mechanical dog's travel planning path stored in the current database; the intersection between the obtained planned path and the OB area is an empty set.

7. A manned robot dog travel path control system based on multimodal perception, characterized in that: The system includes the following modules: A traveling feature mapping information acquisition module, which acquires state information and environmental information of the manned mechanical dog in real time during its travel through a multimodal sensor, and generates traveling feature mapping information of the manned mechanical dog; A traveling interference analysis module, wherein the traveling interference analysis module obtains the travel planning path of the manned mechanical dog stored in the current database, combines the travel feature mapping information of the manned mechanical dog at the current time, generates the travel interference area of ​​the manned mechanical dog at the current time, and obtains the obstacle avoidance mapping planning scheme to be updated for the manned mechanical dog at the current time; An obstacle avoidance planning scheme analysis module, wherein the obstacle avoidance planning scheme analysis module calculates the control fluctuation value of each obstacle avoidance mapping planning scheme to be updated of the manned mechanical dog at the current time based on the manned mechanical dog's travel planning path stored in the current database, and selects the best obstacle avoidance mapping planning scheme; Combined with the initial planned path corresponding to the manned robot dog's planned travel path stored in the current database, the path control integration fluctuation coefficient corresponding to the manned robot dog at the current time is obtained, and the travel range fluctuation range of the manned robot dog's subsequent planned travel path is predicted; An unloaded object coordination determination management module, which extracts the state information in the manned robot dog's travel feature mapping information, combines the predicted value of the travel range fluctuation interval of the manned robot dog's subsequent travel planning path, and the unloaded coordination objects around the manned robot dog at the current time, analyzes the coordination demand coefficient of the manned robot dog at the current time, and completes the coordination demand determination of the manned robot dog; The travel path control and management module updates the manned robot dog's travel planning path stored in the current database according to the manned robot dog's collaborative demand determination result, and feeds back the updated manned robot dog's travel planning path and manned robot dog's travel feature mapping information to the remote monitoring end in real time.

8. The multimodal sensing-based manned mechanical dog travel path control system according to claim 7 is characterized in that: The obstacle avoidance planning scheme analysis module includes an optimal obstacle avoidance mapping planning scheme screening unit, a path control integration analysis unit and a cruising range fluctuation analysis unit. The optimal obstacle avoidance mapping planning scheme screening unit calculates each obstacle avoidance mapping planning scheme to be updated by the manned mechanical dog at the current time based on the control fluctuation value of the manned mechanical dog's travel planning path stored in the current database, and screens the optimal obstacle avoidance mapping planning scheme; The path control integration analysis unit combines the initial planned path corresponding to the manned mechanical dog's travel planning path stored in the current database to obtain the path control integration fluctuation coefficient corresponding to the manned mechanical dog at the current time; The cruising range fluctuation analysis unit predicts the cruising range fluctuation range of the manned mechanical dog's subsequent planned path according to the results obtained by the optimal obstacle avoidance mapping planning scheme screening unit and the path control integration analysis unit.

Images