A multi-modal information fusion robot motion planning method, device and medium

Abstract

The application discloses a kind of multi-modal information fusion's robot motion planning method, equipment and medium, it is related to robot motion control technical field, including in reference path and task coordinate system, in each control cycle, the robot body motion state and environment multi-modal forward observation data are collected and aligned time, calculate multi-modal motion process parameter, obtain multi-modal motion process observation;Based on multi-modal motion process observation and the forward speed and angular velocity of last execution, construct motion process state, predict the motion process state of next control cycle by discrete time update relationship, obtain motion process state prediction;According to motion process state prediction, generate candidate forward speed and candidate angular velocity in preset speed range, calculate process performance index and select the candidate forward speed and candidate angular velocity of process performance index minimum and forward obstacle meet safety requirement as motion planning instruction.The predictable description and adjustment of robot motion process are realized.

Description

Technical Field

[0001] This invention relates to the field of robot motion control technology, and in particular to a robot motion planning method, device and medium that integrates multimodal information. Background Technology

[0002] In mobile service robots, warehousing and logistics robots, and autonomous mobile platforms for inspection and search and rescue, motion planning is usually accomplished by combining environmental modeling and path planning. On the one hand, environmental information is acquired through laser rangefinders, cameras, depth cameras, proximity sensors, etc., to construct local or global environmental representations for identifying passable areas and potential obstacles. On the other hand, robot posture and displacement are estimated based on body sensors such as odometry, inertial measurement, and joint coding, and trajectory tracking and speed control are performed on this basis. In engineering practice, a reference path is often pre-generated, and then linear velocity and angular velocity commands are calculated within a discrete control cycle based on the current position, target pose, and obstacle distance. With the help of constraints such as forward safety distance and turning restrictions, basic obstacle avoidance, path tracking, and task arrival functions are achieved. Multi-sensor information often participates in the above decision-making process in a redundant or simple rule-based manner.

[0003] Conventional methods still focus on using multi-sensor data as the basis for geometric obstacle avoidance or simple threshold judgment, lacking systematic modeling and utilization of multimodal observations as a process state that can evolve over time and be predicted. When making decisions on linear velocity and angular velocity, they often prioritize path tracking error or local safety constraints, lacking coordinated consideration of the coupling relationship between path advancement efficiency, forward obstacle margin, attitude stability boundary and medium- and long-term energy consumption, making it difficult to reflect and comprehensively weigh the quality and cost of the overall motion process in a timely manner. Summary of the Invention

[0004] In view of the aforementioned existing problems, the present invention is proposed.

[0005] Therefore, this invention provides a robot motion planning method based on multimodal information fusion to solve the problems that multimodal observation makes it difficult to construct a predictable motion process state and that velocity planning is difficult to simultaneously consider path advancement, safety margin, attitude stability and coordinated adjustment of energy trends.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0007] In a first aspect, the present invention provides a robot motion planning method based on multimodal information fusion, comprising,

[0008] Under the reference path and task coordinate system, the robot body motion state and environmental multimodal forward observation data are collected in each control cycle and time is aligned to calculate the multimodal motion process parameters and obtain the multimodal motion process observation.

[0009] Based on multimodal motion process observations and the most recently executed forward velocity and angular velocity, the motion process state is constructed, and the motion process state of the next control cycle is predicted through discrete-time update relationships, thus obtaining the motion process state prediction.

[0010] Based on the motion process state prediction, candidate forward speed and candidate angular velocity are generated within the preset speed range. The process performance index is calculated and the candidate forward speed and candidate angular velocity with the smallest process performance index and the forward obstacle meeting the safety requirements are selected as motion planning instructions.

[0011] The robot executes motion planning instructions to control its movement. At the end of the control cycle, it re-acquires multimodal motion process observations and compares them with the motion process state prediction. Based on the deviation, it adjusts the reference path, discrete time update relationship, and multimodal motion process parameters and provides feedback.

[0012] As a preferred embodiment of the robot motion planning method based on multimodal information fusion described in this invention, the following includes: collecting robot motion state and environmental multimodal forward observation data and aligning the time.

[0013] At the beginning of each control cycle, the current motion state of the robot body is read from the robot control device. Within the same control cycle, multimodal forward observation data of the environment, including camera observation, laser observation and proximity observation, is collected and converted.

[0014] Based on the start time of the control cycle, time alignment is performed on the forward observation data of the body motion state and the environment multimodal.

[0015] As a preferred embodiment of the robot motion planning method for multimodal information fusion described in this invention, the calculation of multimodal motion process parameters and the acquisition of multimodal motion process observations include: in the task coordinate system, along the robot's current forward direction, pre-setting forward observation sectors, calculating the nearest forward obstacle distances for camera observation, laser observation, and proximity observation, and calculating the multimodal fusion forward obstacle distance;

[0016] The robot's geometric center is projected onto the reference path, and the path advance is obtained by normalizing the ratio of the reference path arc length to the total reference path length. The stability margin is determined based on the minimum distance from the robot's geometric center to the boundary of the supporting polygon. The cumulative energy consumption of the current control cycle is obtained by integrating the current and voltage of the previous control cycle. The multimodal fusion of forward obstacle distance, path advance, stability margin, and cumulative energy consumption is combined to form the multimodal motion process observation of the current control cycle.

[0017] As a preferred embodiment of the robot motion planning method based on multimodal information fusion described in this invention, the specific steps for constructing the motion process state are as follows:

[0018] The forward obstacle distance of the multimodal fusion is recorded as the forward obstacle distance of the current control cycle, the path advance is recorded as the path advance of the current control cycle, the stability margin is recorded as the attitude stability of the current control cycle, the cumulative energy consumption is recorded as the energy state of the current control cycle, the linear velocity of the machine is recorded as the most recently executed forward velocity, and the angular velocity of the machine is recorded as the most recently executed angular velocity.

[0019] The forward obstacle distance, path advance, attitude stability, energy state, forward speed, and angular velocity constitute the motion process state of the current control cycle.

[0020] As a preferred embodiment of the robot motion planning method based on multimodal information fusion described in this invention, the specific steps for predicting the motion process state of the next control cycle through discrete-time update relationships are as follows:

[0021] The product of the current control cycle time interval and the current forward speed is used as the forward distance along the reference path in the current control cycle. The ratio of the forward distance to the total length of the reference path is used as the propulsion increment. The propulsion increment is accumulated into the path propulsion amount of the current control cycle. The accumulated path propulsion amount is used as the path propulsion amount prediction for the next control cycle. The relationship of the path propulsion amount gradually accumulating with the control cycle constitutes the discrete-time update relationship of the path propulsion amount.

[0022] The difference between the multimodal fusion forward obstacle distance and the forward distance along the reference path in the current control cycle is used as the forward obstacle distance prediction for the next control cycle. The relationship between the difference and the forward obstacle distance prediction for the next control cycle constitutes the discrete-time update relationship of the forward obstacle distance.

[0023] The power increment is calculated by recording the current and voltage in the previous control cycle. When calculating the energy prediction for the next control cycle, the sum of the cumulative energy consumption at the end of the current control cycle and the power increment is used as the cumulative energy consumption prediction for the next control cycle. The relationship between the current cumulative energy consumption and the power increment to obtain the cumulative energy consumption prediction for the next control cycle constitutes the discrete-time update relationship of energy consumption.

[0024] The stability margin calculated in the current control cycle is directly used as the stability margin prediction for the next control cycle. The relationship between the current stability margin and the stability margin of the next cycle constitutes the discrete-time update relationship of the stability margin.

[0025] The path advance prediction, forward obstacle distance prediction, cumulative energy consumption prediction, and stability margin prediction are recombined in sequence to form the motion process state prediction for the next control cycle.

[0026] As a preferred embodiment of the robot motion planning method based on multimodal information fusion described in this invention, the specific steps for generating candidate forward velocities and candidate angular velocities within a preset speed range are as follows:

[0027] Before the robot's motion task begins, the total length of the reference path is divided into continuous path segments by equal intervals based on the total length of the reference path and the time interval of the control cycle. The ratio of the length of each path segment to the total length of the reference path is used as the reference propulsion amount for each control cycle, forming a reference propulsion amount sequence.

[0028] Set the lower limit of forward safety distance, the lower limit of stability margin, the energy reference value, the allowable range of forward speed and the allowable range of angular velocity. Divide the allowable range of forward speed into candidate forward speeds and the allowable range of angular velocity into candidate angular velocities. Combine them in pairs to generate a finite set of candidate forward speeds and candidate angular velocities.

[0029] As a preferred embodiment of the robot motion planning method based on multimodal information fusion described in this invention, the specific steps for calculating process performance indicators and selecting candidate forward velocities and candidate angular velocities with the smallest process performance indicators and meeting the safety requirements of forward obstacles as motion planning instructions are as follows:

[0030] For each group of candidate forward speed and candidate angular velocity, the path advance, forward obstacle distance, stability margin, and cumulative energy consumption are predicted based on the discrete-time update relationship. These are then compared with the reference advance, the lower limit of the forward safety distance, the lower limit of the stability margin, and the energy reference value. The ratio of each comparison difference to the preset value of the same dimension scale yields the dimensionless advance error, insufficient safety margin, insufficient stability margin, energy deviation, velocity change, and angular velocity change. The dimensionless quantity with the largest absolute value is selected as the process performance index of the current group of candidate forward speed and candidate angular velocity.

[0031] Candidate forward speeds and angular velocities that are less than the lower limit of the forward safe distance are eliminated. The candidates are sorted by process performance index from smallest to largest, and the optimal candidate forward speed and optimal candidate angular velocity with the smallest process performance index are selected as the motion planning instructions for the current control cycle.

[0032] As a preferred embodiment of the robot motion planning method based on multimodal information fusion described in this invention, the specific steps of adjusting the reference path, updating the discrete-time relationship, and feeding back the multimodal motion process parameters according to the deviation are as follows:

[0033] At the end of the control cycle, the multimodal fusion forward obstacle distance, path advance, stability margin and cumulative energy consumption are recalculated and compared with the motion process state prediction to obtain various deviations.

[0034] Long-term deviations are statistically analyzed within continuous control cycles. Based on the long-term deviations of path advance, forward obstacle distance, stability margin, and energy consumption, the reference advance sequence and discrete-time update relationship are fine-tuned, and the update results are used for the next control cycle.

[0035] In a second aspect, the present invention provides a computer device including a memory and a processor, wherein the memory stores a computer program, wherein when the computer program is executed by the processor, it implements any step of the robot motion planning method for multimodal information fusion as described in the first aspect of the present invention.

[0036] Thirdly, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, it implements any step of the robot motion planning method for multimodal information fusion as described in the first aspect of the present invention.

[0037] The beneficial effects of this invention are as follows: by constructing multimodal motion process observation and forming motion process state prediction, predictable description and adjustment of robot motion process can be achieved. By selecting motion planning instructions from candidate forward speed and candidate angular velocity based on process performance indicators, coordinated control of path advancement, safety margin, attitude stability and energy can be achieved. This solves the problems that multimodal observation is difficult to form predictable motion process state and that velocity planning is difficult to coordinate path advancement, safety margin, attitude stability and energy. Attached Figure Description

[0038] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0039] Figure 1 This is a flowchart of a robot motion planning method based on multimodal information fusion.

[0040] Figure 2 A flowchart for observing and acquiring multimodal motion processes.

[0041] Figure 3 This is a flowchart for predicting the state during motion.

[0042] Figure 4 Generate flowcharts for motion planning instructions. Detailed Implementation

[0043] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0044] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0045] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0046] Reference Figures 1-4 As one embodiment of the present invention, this embodiment provides a robot motion planning method based on multimodal information fusion, comprising the following steps:

[0047] S1. Under the reference path and task coordinate system, collect the robot body motion state and environmental multimodal forward observation data in each control cycle and align the time, calculate the multimodal motion process parameters, and obtain the multimodal motion process observation.

[0048] Before the robot's motion task begins, a reference path is planned in a map or simulation environment for the robot to move along. A task coordinate system is established based on the robot's geometric center, and all path points in the reference path are uniformly represented in the task coordinate system.

[0049] At the beginning of each control cycle, the current motion state of the robot body is read from the robot control device, including the joint state obtained by the joint angle encoding device, the attitude information obtained by the inertial measurement device, and the linear velocity and angular velocity of the body obtained by the odometry and drive device.

[0050] Within the same control cycle, the current image or depth map is acquired from the forward-facing camera or depth camera. The image pixels or depth points are converted into forward 3D environment points in the task coordinate system using pre-calibrated extrinsic parameters. The forward distance data obtained from the current scan is acquired from the forward-facing laser ranging device and converted into forward obstacle points in the task coordinate system. The current approach distance or contact event is acquired from the forward-facing approach device or forward-facing force-sensitive device and converted into forward obstacle information in the task coordinate system, thus obtaining three types of multimodal forward-facing environmental observation data: camera observation, laser observation, and proximity observation.

[0051] Pre-calibrated extrinsic parameters refer to the fixed position and orientation of each observation device relative to the task coordinate system (robot body).

[0052] Based on the start time of the control cycle, the forward observation data of the body motion state and three types of environments are time-aligned, and only the observation data falling within the current control cycle are retained.

[0053] In the task coordinate system, a forward observation sector is predefined along the robot's current direction of travel (for example, with the robot's geometric center as the center, the robot's current direction of travel as the sector's center line, extending 30° to the left and right in the horizontal plane, extending from 0.5 meters to 3 meters in the forward distance, and from 0.1 meters above the ground to 1.5 meters in the vertical direction). The effective obstacle points within the forward observation sector from camera observation, laser observation, and proximity observation are counted. The nearest forward obstacle distances corresponding to camera observation, laser observation, and proximity observation are calculated respectively, and the multimodal fusion forward obstacle distance is calculated, expressed as:

[0054] ;

[0055] in, This indicates the forward obstacle distance in the multimodal fusion during the current control cycle. This represents the nearest forward obstacle distance obtained from the forward camera or depth camera after coordinate transformation during the current control cycle. This represents the nearest forward obstacle distance obtained by coordinate transformation from the forward laser rangefinder during the current control cycle. This represents the nearest forward obstacle distance obtained by coordinate transformation from the forward approach device or forward force-sensitive device in the current control cycle. The dimensionless constant 3 represents the number of forward obstacle distances involved in the fusion.

[0056] Search for the path point on the reference path that is closest to the robot's geometric center in the task coordinate system. Use the path point as the projection point. Use the arc length of the path point on the reference path as the reference path arc length position for the current control cycle. Then, normalize the ratio of the arc length of the path point to the total length of the reference path to obtain the path advance amount for the current control cycle.

[0057] Based on the posture information in the robot's motion state and the range of the robot's support area, the minimum distance from the projection point of the geometric center in the direction of gravity to the boundary of the support area is calculated in the task coordinate system. The minimum distance is used as the stability margin of the current control cycle. The larger the stability margin, the more stable the posture.

[0058] The support area refers to a polygonal region defined on a horizontal surface based on the contact profile between the robot chassis and the ground. It represents the range of support that the robot can provide in its normal posture.

[0059] Based on the current and voltage records of the previous control cycle, the electrical power of the servo drive device in the current control cycle is discretely integrated, and the obtained electrical power is accumulated into the historical cumulative value to obtain the cumulative energy consumption at the end of the current control cycle.

[0060] The four types of multimodal motion process parameters—forward obstacle distance, path advance, stability margin, and cumulative energy consumption—are combined sequentially to form the multimodal motion process observation for the current control cycle.

[0061] S2. Based on multimodal motion process observations and the most recently executed forward velocity and angular velocity, construct the motion process state, and predict the motion process state of the next control cycle through discrete-time update relationships to obtain the motion process state prediction.

[0062] The forward obstacle distance fused by multimodal analysis is recorded as the forward obstacle distance of the current control cycle, the path advance is recorded as the path advance of the current control cycle, the stability margin is recorded as the attitude stability of the current control cycle, the cumulative energy consumption is recorded as the energy state of the current control cycle, the linear velocity of the body is recorded as the forward velocity of the most recently executed operation, and the angular velocity of the body is recorded as the angular velocity of the most recently executed operation, which constitutes the motion process state of the current control cycle.

[0063] The product of the current control cycle time interval and the current forward speed is used as the forward distance along the reference path in the current control cycle. The ratio of the forward distance to the total length of the reference path is used as the propulsion increment. The propulsion increment is accumulated into the path propulsion amount of the current control cycle. The accumulated path propulsion amount is used as the path propulsion amount prediction for the next control cycle. The relationship of the path propulsion amount gradually accumulating with the control cycle constitutes the discrete-time update relationship of the path propulsion amount.

[0064] The time interval of a control cycle refers to the length of time between the start of the current control cycle and the start of the next control cycle.

[0065] While predicting the path advance, the difference between the multimodal fusion forward obstacle distance and the forward distance along the reference path in the current control cycle is used as the forward obstacle distance prediction for the next control cycle. The relationship between the difference and the forward obstacle distance prediction for the next control cycle constitutes the discrete-time update relationship of the forward obstacle distance.

[0066] The power increment is calculated by recording the current and voltage in the previous control cycle. When calculating the energy prediction for the next control cycle, the sum of the cumulative energy consumption at the end of the current control cycle and the power increment is used as the cumulative energy consumption prediction for the next control cycle. The relationship between the current cumulative energy consumption and the power increment to obtain the cumulative energy consumption prediction for the next control cycle constitutes the discrete-time update relationship of energy consumption.

[0067] The stability margin calculated in the current control cycle is directly used as the stability margin prediction for the next control cycle. It is assumed that the stability margin remains approximately unchanged on a single control cycle scale. The relationship between using the current stability margin directly as the stability margin for the next cycle constitutes the discrete-time update relationship of the stability margin.

[0068] After completing the path advance prediction, forward obstacle distance prediction, cumulative energy consumption prediction, and stability margin prediction for the next control cycle, the path advance prediction, forward obstacle distance prediction, cumulative energy consumption prediction, and stability margin prediction are recombined in the same order as the current motion process state to form the motion process state prediction for the next control cycle.

[0069] S3. Based on the motion process state prediction, generate candidate forward speed and candidate angular velocity within the preset speed range, calculate the process performance index, and select the candidate forward speed and candidate angular velocity with the smallest process performance index and the forward obstacle meeting the safety requirements as the motion planning instruction.

[0070] After predicting the motion state for the next control cycle, the predicted path advance, forward obstacle distance, stability margin, and cumulative energy consumption from the motion state prediction are used as prediction quantities for process control. Simultaneously, combined with the path advance, forward obstacle distance, stability margin, and cumulative energy consumption of the current control cycle, before the robot's motion task begins, the total length of the reference path is divided into continuous path segments by equal intervals based on the total length of the reference path and the time interval of the control cycle. The ratio of the length of each path segment to the total length of the reference path is used as the reference advance for each control cycle, forming a reference advance sequence. In the current control cycle, the reference advance corresponding to the next control cycle is read from the reference advance sequence as the path advance target.

[0071] The lower limit of forward safety distance is set based on the effective detection distance of the forward camera, forward laser rangefinder, and forward approach device, and the braking capability of the robot at its maximum forward speed. The lower limit of stability margin is set based on the support area range and attitude safety requirements. The energy reference value is set based on the power capacity and task energy consumption requirements. The allowable range of forward speed and allowable range of angular velocity are set based on the robot's mechanical structure and driving capability. The allowable range of forward speed is divided into candidate forward speeds, and the allowable range of angular velocity is divided into candidate angular velocities. A finite set of candidate forward speeds and candidate angular velocities are generated by combining them in pairs.

[0072] The lower limit of the forward safe distance can be determined by calculating the braking distance, conducting multiple simulations, and statistically analyzing experiments. For example, in indoor service robot tasks, the lower limit of the forward safe distance can be set to 0.5 meters to 1.5 meters. The lower limit of the stability margin can be determined based on the geometric dimensions of the support area and experience with the attitude instability boundary. For example, it can be taken as one-tenth to one-fifth of the shortest side length of the support area. The energy reference value can be estimated based on the task duration and typical power consumption. For example, it can be taken as 70% to 80% of the rated power supply capacity. The allowable range of forward speed and angular velocity can be determined by combining the simulation results of driving capability and stability. For example, the allowable range of forward speed is limited to one-half to two-thirds of the nominal maximum forward speed, and the allowable range of angular velocity is limited to below the maximum angular velocity that meets the requirements for stable turning.

[0073] For each group of candidate forward velocities and candidate angular velocities, the discrete-time update relationships for path advance, forward obstacle distance, energy consumption, and stability margin are invoked. The forward velocity and angular velocity of the current control cycle are temporarily replaced with the candidate forward velocity and candidate angular velocity of the current group. The predicted path advance, forward obstacle distance, cumulative energy consumption, and stability margin under the influence of the candidate forward velocity and candidate angular velocity are calculated and compared with the reference advance, the lower limit of the forward safe distance, the lower limit of the stability margin, and the energy reference value. The differences between the predicted path advance and the reference advance, the differences between the predicted forward obstacle distance and the lower limit of the forward safe distance, and the stability margin are used to determine the stability margin. The differences between the predicted velocity and the lower limit of the stability margin, the difference between the predicted cumulative energy consumption and the energy reference value, the difference between the current group candidate forward velocity and the forward velocity of the previous control cycle, and the difference between the current group candidate angular velocity and the angular velocity of the previous control cycle are compared with preset values ​​of the same dimension to obtain dimensionless propulsion error, dimensionless safety margin deficiency, dimensionless stability margin deficiency, dimensionless energy deviation, dimensionless velocity change, and dimensionless angular velocity change. The index with the largest absolute value among the dimensionless propulsion error, dimensionless safety margin deficiency, dimensionless stability margin deficiency, dimensionless energy deviation, dimensionless velocity change, and dimensionless angular velocity change is used as the process performance index corresponding to the current group candidate forward velocity and candidate angular velocity.

[0074] The preset dimensionless scale value refers to a positive reference quantity with the same physical dimension as each difference value, which is used to normalize each difference value into a dimensionless quantity.

[0075] After obtaining the process performance indices corresponding to all candidate forward velocities and candidate angular velocities, candidate forward velocities and candidate angular velocities whose predicted forward obstacle distance is less than the lower limit of the forward safe distance are eliminated. Only candidate forward velocities and candidate angular velocities whose predicted forward obstacle distance meets the lower limit of the forward safe distance are retained. The retained candidate forward velocities and candidate angular velocities are sorted in ascending order of process performance indices. The candidate forward velocities and candidate angular velocities corresponding to the first position in the sort are selected as the optimal motion planning results for the current control cycle. The optimal candidate forward velocities and optimal candidate angular velocities are used as the motion planning instructions for the current control cycle.

[0076] S4. Execute motion planning instructions to control the robot's movement. At the end of the control cycle, re-acquire multimodal motion process observations and compare them with the motion process state prediction. Adjust the reference path, discrete time update relationship, and multimodal motion process parameters according to the deviation and provide feedback.

[0077] The optimal candidate forward speed and optimal candidate angular velocity are sent to the robot control device, so that the robot moves along the reference path according to the motion planning instructions within the current control cycle. During the execution of the motion planning instructions, the robot continuously records the motion status, current and voltage and other operating data.

[0078] At the end of the current control cycle, the forward observation data of the body motion state and the environment multimodal motion process are time-aligned and coordinate-transformed again. The forward obstacle distance, path advance, stability margin, and cumulative energy consumption of the multimodal fusion are recalculated. The obtained forward obstacle distance, path advance, stability margin, and cumulative energy consumption are used to form the actual multimodal motion process observation of the current control cycle. From the motion process state prediction obtained at the beginning of the current control cycle, the corresponding path advance prediction, forward obstacle distance prediction, stability margin prediction, and cumulative energy consumption prediction are extracted and compared with the actual multimodal motion process observation to obtain the path advance deviation, forward obstacle distance deviation, stability margin deviation, and energy consumption deviation of the current control cycle. The deviations of each actual multimodal motion process observation are statistically analyzed in continuous control cycles.

[0079] When the path advance amount deviates long-term, i.e. the predicted path advance amount is generally greater than the actual path advance amount, the reference advance amount increment corresponding to each control cycle is reduced, so that the rate at which the path advance amount accumulates with the control cycle decreases. When the predicted path advance amount is generally less than the actual path advance amount, fine-tuning is performed in the opposite direction to make the predicted path advance amount consistent with the actual path advance amount. When the forward obstacle distance deviates long-term, i.e. the predicted forward obstacle distance is generally too large, the forward obstacle distance is increased in the discrete-time update relationship of the forward obstacle distance in the current control cycle during the calculation, so that the predicted forward obstacle distance is closer to the actual change of the forward obstacle distance in the multimodal fusion. When the predicted forward obstacle distance is generally too small, the reverse adjustment is performed.

[0080] When the stability margin deviates over a long period, i.e., when the stability margin prediction and the actual stability margin differ significantly under the turning obstacle avoidance condition, an additional stability margin estimate based on the current attitude and support area range is added under the turning obstacle avoidance condition, and it is directly used under the straight and stable condition. When the energy consumption deviates over a long period, i.e., the cumulative energy consumption prediction is too large or too small over a long period, the superposition ratio of the electrical work increment obtained from the current and voltage records in the cumulative energy consumption is slowly and unidirectionally fine-tuned in the discrete time update relationship of energy consumption, so that the energy prediction gradually approaches the actual cumulative energy consumption with the control cycle.

[0081] The updated reference propulsion sequence, the discrete-time update relationship of path propulsion, the discrete-time update relationship of forward obstacle distance, the discrete-time update relationship of energy consumption, the discrete-time update relationship of stability margin, and the calculation methods of multimodal fusion forward obstacle distance, path propulsion, stability margin, and cumulative energy consumption are saved to the robot control device.

[0082] In subsequent control cycles, based on the updated reference propulsion sequence, the update relationship of each discrete time, and the calculation method of multimodal motion process parameters, multimodal motion process observation, motion process state construction, motion process state prediction, and motion planning instruction generation are executed sequentially.

[0083] This embodiment also provides a computer device applicable to the robot motion planning method of multimodal information fusion, including: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to realize the robot motion planning method of multimodal information fusion as proposed in the above embodiment.

[0084] The computer device can be a terminal, comprising a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0085] This embodiment also provides a storage medium storing a computer program, which, when executed by a processor, implements the robot motion planning method for multimodal information fusion as proposed in the above embodiments. The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Red-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0086] In summary, this invention achieves predictable description and regulation of robot motion processes by constructing multimodal motion process observations and forming motion process state predictions. By selecting motion planning instructions from candidate forward velocities and candidate angular velocities based on process performance indicators, it achieves coordinated control of path propulsion, safety margin, attitude stability, and energy. This solves the problems that multimodal observations cannot form predictable motion process states and that velocity planning cannot coordinate path propulsion, safety margin, attitude stability, and energy.

[0087] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A robot motion planning method based on multimodal information fusion, characterized in that: include, Under the reference path and task coordinate system, the robot body motion state and environmental multimodal forward observation data are collected in each control cycle and time is aligned to calculate the multimodal motion process parameters and obtain the multimodal motion process observation. Based on multimodal motion process observations and the most recently executed forward velocity and angular velocity, the motion process state is constructed, and the motion process state of the next control cycle is predicted through discrete-time update relationships, thus obtaining the motion process state prediction. Based on the motion process state prediction, candidate forward speed and candidate angular velocity are generated within the preset speed range. The process performance index is calculated and the candidate forward speed and candidate angular velocity with the smallest process performance index and the forward obstacle meeting the safety requirements are selected as motion planning instructions. The robot executes motion planning instructions to control its movement. At the end of the control cycle, it re-acquires multimodal motion process observations and compares them with motion process state predictions. Based on the deviation, it adjusts the reference path, discrete time update relationship, and multimodal motion process parameters and provides feedback. The specific steps for generating candidate forward speed and candidate angular velocity within the preset speed range are as follows: before the robot motion task begins, the total length of the reference path is divided into continuous path segments by equal intervals according to the total length of the reference path and the time interval of the control cycle. The ratio of the length of each path segment to the total length of the reference path is used as the reference propulsion amount for each control cycle to form a reference propulsion amount sequence. Set the lower limit of forward safety distance, the lower limit of stability margin, the energy reference value, the allowable range of forward speed and the allowable range of angular velocity, divide the allowable range of forward speed into candidate forward speeds, divide the allowable range of angular velocity into candidate angular velocities, and combine them in pairs to generate a finite set of candidate forward speeds and candidate angular velocities. The calculation process performance index selects the candidate forward speed and candidate angular velocity with the smallest process performance index and the forward obstacle meeting the safety requirements as motion planning instructions. The specific steps are as follows: For each group of candidate forward speed and candidate angular velocity, predict the path advance, forward obstacle distance, stability margin and cumulative energy consumption based on discrete time update relationship, and compare them with the reference advance, forward safety distance lower limit, stability margin lower limit and energy reference value. The ratio of each comparison difference to the preset value of the same dimension scale is used to obtain the dimensionless advance error, safety margin deficiency, stability margin deficiency, energy deviation, speed change and angular velocity change. The dimensionless quantity with the largest absolute value is selected as the process performance index of the current group of candidate forward speed and candidate angular velocity. Candidate forward speeds and angular velocities that are less than the lower limit of the forward safe distance are eliminated. The candidates are sorted by process performance index from smallest to largest, and the optimal candidate forward speed and optimal candidate angular velocity with the smallest process performance index are selected as the motion planning instructions for the current control cycle.

2. The robot motion planning method based on multimodal information fusion as described in claim 1, characterized in that: The acquisition of robot motion state and environmental multimodal forward observation data, and the alignment of the time, include... At the beginning of each control cycle, the current motion state of the robot body is read from the robot control device. Within the same control cycle, multimodal forward observation data of the environment, including camera observation, laser observation and proximity observation, is collected and converted. Based on the start time of the control cycle, time alignment is performed on the forward observation data of the body motion state and the environment multimodal.

3. The robot motion planning method based on multimodal information fusion as described in claim 2, characterized in that: The calculation of multimodal motion process parameters and the acquisition of multimodal motion process observations include: in the task coordinate system, along the robot's current forward direction, pre-setting forward observation sectors, calculating the nearest forward obstacle distances for camera observation, laser observation, and proximity observation, and calculating the multimodal fusion forward obstacle distance; The robot's geometric center is projected onto the reference path, and the path advance is obtained by normalizing the ratio of the reference path arc length to the total reference path length. The stability margin is determined based on the minimum distance from the robot's geometric center to the boundary of the supporting polygon. The cumulative energy consumption of the current control cycle is obtained by integrating the current and voltage of the previous control cycle. The multimodal fusion of forward obstacle distance, path advance, stability margin, and cumulative energy consumption is combined to form the multimodal motion process observation of the current control cycle.

4. The robot motion planning method based on multimodal information fusion as described in claim 3, characterized in that: The specific steps for constructing the motion process state are as follows: The forward obstacle distance of the multimodal fusion is recorded as the forward obstacle distance of the current control cycle, the path advance is recorded as the path advance of the current control cycle, the stability margin is recorded as the attitude stability of the current control cycle, the cumulative energy consumption is recorded as the energy state of the current control cycle, the linear velocity of the machine is recorded as the most recently executed forward velocity, and the angular velocity of the machine is recorded as the most recently executed angular velocity. The forward obstacle distance, path advance, attitude stability, energy state, forward speed, and angular velocity constitute the motion process state of the current control cycle.

5. The robot motion planning method based on multimodal information fusion as described in claim 4, characterized in that: The specific steps for predicting the motion process state of the next control cycle through discrete-time update relationships are as follows: The product of the current control cycle time interval and the current forward speed is used as the forward distance along the reference path in the current control cycle. The ratio of the forward distance to the total length of the reference path is used as the propulsion increment. The propulsion increment is accumulated into the path propulsion amount of the current control cycle. The accumulated path propulsion amount is used as the path propulsion amount prediction for the next control cycle. The relationship of the path propulsion amount gradually accumulating with the control cycle constitutes the discrete-time update relationship of the path propulsion amount. The difference between the multimodal fusion forward obstacle distance and the forward distance along the reference path in the current control cycle is used as the forward obstacle distance prediction for the next control cycle. The relationship between the difference and the forward obstacle distance prediction for the next control cycle constitutes the discrete-time update relationship of the forward obstacle distance. The power increment is calculated by recording the current and voltage in the previous control cycle. When calculating the energy prediction for the next control cycle, the sum of the cumulative energy consumption at the end of the current control cycle and the power increment is used as the cumulative energy consumption prediction for the next control cycle. The relationship between the current cumulative energy consumption and the power increment to obtain the cumulative energy consumption prediction for the next control cycle constitutes the discrete-time update relationship of energy consumption. The stability margin calculated in the current control cycle is directly used as the stability margin prediction for the next control cycle. The relationship between the current stability margin and the stability margin of the next cycle constitutes the discrete-time update relationship of the stability margin. The path advance prediction, forward obstacle distance prediction, cumulative energy consumption prediction, and stability margin prediction are recombined in sequence to form the motion process state prediction for the next control cycle.

6. The robot motion planning method based on multimodal information fusion as described in claim 5, characterized in that: The specific steps for adjusting the reference path, discrete-time update relationship, and multimodal motion process parameters based on the deviation and feeding them back are as follows: At the end of the control cycle, the multimodal fusion forward obstacle distance, path advance, stability margin and cumulative energy consumption are recalculated and compared with the motion process state prediction to obtain various deviations. Long-term deviations are statistically analyzed within continuous control cycles. Based on the long-term deviations of path advance, forward obstacle distance, stability margin, and energy consumption, the reference advance sequence and discrete-time update relationship are fine-tuned, and the update results are used for the next control cycle.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, it implements the steps of the robot motion planning method of multimodal information fusion as described in any one of claims 1 to 6.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, it implements the steps of the robot motion planning method of multimodal information fusion as described in any one of claims 1 to 6.

Images