Robot action control method, device, medium, control apparatus, and robot
By performing structured semantic parsing and motion parameter mapping on the speech text, combined with environmental perception and user history data, the problem of inaccurate voice control in existing technologies has been solved, achieving precise execution and safety of robot actions.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- CHONGQING PHOENIX TECHNOLOGY CO LTD
- Filing Date
- 2026-03-25
- Publication Date
- 2026-06-09
AI Technical Summary
In existing technologies, voice control methods cannot understand the vague expressions and implicit needs commonly found in spoken language, resulting in inaccurate execution of robot action commands in complex environments and posing safety risks.
A pre-defined large language model is used to perform structured semantic parsing of speech text to obtain action semantic units. Then, action control commands are generated through a mapping table from pre-defined semantic units to action parameters. Combined with environmental perception and user history data, action parameters are corrected to ensure the accuracy and security of the commands.
It achieves a complete closed loop from user natural language to precise robot execution, improves the ability to understand fuzzy expressions and personalized language, and ensures that motion control commands are feasible and safe in complex environments.
Smart Images
Figure CN122165403A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of control technology, and more specifically, to a robot motion control method, device, medium, control equipment, and robot. Background Technology
[0002] In practical applications such as service robots, industrial collaborative robots, and special-purpose robots, users often need to issue control commands to the robots using natural language. These interactions typically occur in complex environments with high background noise and dynamically changing spatial layouts, posing significant challenges to speech recognition and comprehension capabilities.
[0003] Currently, common voice control methods typically employ a step-by-step processing approach. Automatic speech recognition technology converts the user's speech into text. After the text conversion is completed, keyword matching, fixed sentence pattern recognition, or simple text classification methods are used to determine the operation that most closely matches the text content from a pre-set limited set of instructions. At the same time, basic contextual information is combined to assist in understanding the current instruction.
[0004] However, current technologies for converting text into actions rely on manually preset rules or limited instruction templates, making it unable to understand the ambiguous expressions and implicit needs commonly found in spoken language. This means that even if the text is correctly understood, the generated action instructions may not be executable in reality, posing a security risk. Summary of the Invention
[0005] The purpose of this application is to address the shortcomings of the prior art by providing a robot motion control method, device, medium, control equipment, and robot, thereby improving the accuracy of robot control and preventing robot safety issues.
[0006] To achieve the above objectives, the technical solutions adopted in the embodiments of this application are as follows: In a first aspect, one embodiment of this application provides a robot motion control method, the method comprising: Acquire user-input control voice signals for the target robot; The control voice signal is subjected to speech recognition to obtain the corresponding voice text. A pre-defined large language model is used to perform structured semantic parsing on the speech text to obtain structured semantic information, which includes: action semantic units; Based on the action semantic unit, the corresponding target action parameters are obtained using a preset semantic unit to action parameter mapping table; Based on the target motion parameters, motion control instructions are generated for the target robot; the motion control instructions are used to control the motion of the target robot.
[0007] Optionally, before obtaining the corresponding target action parameters based on the action semantic unit using a preset semantic unit-to-action parameter mapping table, the method further includes: Obtain the semantic combination corresponding to the preset task; Obtain successful action parameters for the preset task collected based on multiple semantic units in the semantic combination, and obtain the action parameters corresponding to the multiple semantic units; Cluster the action parameters corresponding to each semantic unit in the same semantic unit to obtain the clustering action parameters corresponding to the same semantic unit. Based on the semantic units of the same type and the corresponding clustering action parameters, a mapping table from the preset semantic units to the action parameters is constructed.
[0008] Optionally, before generating motion control commands for the target robot based on the target motion parameters, the method further includes: Based on the environmental perception data of the current environment, the target robot's most recent historical execution state, and the user's historical operation records, the motion correction amount of the target robot is obtained; The motion correction amount and the target motion parameters are weighted to obtain the corrected motion parameters; The step of generating motion control commands for the target robot based on the target motion parameters includes: Based on the corrected motion parameters, motion control commands are generated for the target robot.
[0009] Optionally, before generating motion control commands for the target robot based on the target motion parameters, the method further includes: Perform multi-dimensional execution verification on the target action parameters; If the execution verification fails in any of the multiple dimensions, the preset action parameter correction algorithm corresponding to that dimension is used to correct the target action parameter to obtain the corrected action parameter.
[0010] Optionally, if the execution verification fails in any of the multiple dimensions, a preset action parameter correction algorithm corresponding to that dimension is used to correct the target action parameters, resulting in corrected action parameters including: If the joint limits in multiple dimensions exceed the execution verification of the dimension, the range of motion of each joint corresponding to the target motion parameters is obtained; the range of motion of the joints exceeding the limits is limited and trimmed, and the corrected motion parameters are obtained based on the trimmed range of motion. Alternatively, if the execution verification of the collision risk dimension among multiple dimensions fails, the pose points with collision risk in the predicted trajectory corresponding to the target action parameters are obtained, and the corrected action parameters are obtained based on the pose points with collision risk. Alternatively, if the execution verification of the workspace constraint dimension fails in multiple dimensions, the end target corresponding to the target action parameter is projected to the nearest workspace boundary, and the corrected action parameter is obtained based on the projected end target. Alternatively, if the execution verification of the dimension with excessive safety parameters fails in multiple dimensions, the safety parameters corresponding to the target action parameters are obtained, and the safety parameters are scaled to a preset safety range. Based on the scaled-down parameters, the corrected action parameters are obtained.
[0011] Optionally, before performing speech recognition on the control speech signal to obtain the corresponding speech text, the method further includes: Based on the estimated noise components of the current environment, obtain the noise suppression weighting coefficient of the current environment; Based on the estimated noise components and the noise suppression weighting coefficients, the control speech signal is subjected to noise suppression processing to obtain the processed speech signal; The step of performing speech recognition on the control speech signal to obtain the speech text corresponding to the control speech signal includes: The processed speech signal is subjected to speech recognition to obtain the speech text.
[0012] Optionally, before performing speech recognition on the control speech signal to obtain the corresponding speech text, the method further includes: Obtain the user's historical voice features; Based on the historical speech characteristics, the control speech signal is personalized and repaired to obtain the repaired speech signal; The step of performing speech recognition on the control speech signal to obtain the speech text corresponding to the control speech signal includes: The repaired speech signal is subjected to speech recognition to obtain the speech text corresponding to the control speech signal.
[0013] Secondly, another embodiment of this application provides a robot motion control device, the device comprising: The acquisition module is used to acquire the control voice signals input by the user for the target robot; The recognition module is used to perform speech recognition on the control voice signal to obtain the voice text corresponding to the control voice signal; The parsing module is used to perform structured semantic parsing on the speech text using a preset large language model to obtain structured semantic information, which includes: action semantic units; The mapping module is used to obtain the corresponding target action parameters based on the action semantic unit and a preset mapping table from semantic unit to action parameters. The generation module is used to generate motion control instructions for the target robot based on the target motion parameters; the motion control instructions are used to control the motion of the target robot.
[0014] Thirdly, another embodiment of this application provides a control device, including: a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the control device is running, the processor communicates with the memory via the bus, and the processor executes the machine-readable instructions to perform the steps of any of the robot motion control methods described in the first aspect above.
[0015] Fourthly, another embodiment of this application provides a robot, the robot comprising: a robot body, and a controller within the robot body, the controller being configured to execute the steps of any of the robot motion control methods described in the first aspect above.
[0016] Fifthly, another embodiment of this application provides a storage medium storing a computer program, which, when executed by a processor, performs the steps of the robot motion control method as described in any of the first aspects above.
[0017] The beneficial effects of this application are: This application provides a robot motion control method, device, medium, control equipment, and robot. The method involves acquiring a user-input control voice signal for a target robot; performing speech recognition on the control voice signal to obtain the corresponding speech text; using a pre-set large language model to perform structured semantic parsing on the speech text to obtain structured semantic information; obtaining the corresponding target action parameters based on action semantic units using a pre-set semantic unit-to-action parameter mapping table; generating motion control instructions for the target robot based on the target action parameters; and using the motion control instructions to control the target robot's actions. This application decomposes the voice control process into five constrained stages: voice signal acquisition, speech recognition, structured semantic parsing, semantic-to-action parameter mapping, and action instruction generation, achieving a complete closed loop from the user's natural expression to the robot's precise execution. By introducing structured semantic units as intermediate representations, the system can accurately capture key information such as action intent, operation object, attribute requirements, and execution constraints, improving its understanding of fuzzy expressions, contextual relationships, and personalized language. The semantic-to-action parameter mapping table, continuously optimized based on actual execution feedback, ensures that language descriptions can be stably converted into action parameters that meet task requirements, balancing versatility and adaptability. The generated motion control commands are directly adapted to the target robot, ensuring that the commands are not only grammatically correct, but also practically feasible and safe in the physical world. This achieves both natural voice interaction and accurate motion execution in complex environments and under diverse user conditions. Attached Figure Description
[0018] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.
[0019] Figure 1 This application provides a flowchart illustrating a robot motion control method, as shown below. Figure 1 As shown; Figure 2 A flowchart illustrating the processing of voice signals in a robot motion control method provided in this application embodiment; Figure 3 A flowchart illustrating the processing of voice signals in another robot motion control method provided in this application embodiment; Figure 4 A schematic diagram of the process for constructing a mapping table in a robot motion control method provided in an embodiment of this application; Figure 5A schematic flowchart illustrating the correction of target motion parameters in a robot motion control method provided in this application embodiment; Figure 6 A flowchart illustrating the execution verification process in a robot motion control method provided in this application embodiment; Figure 7 This is a schematic diagram of the structure of a robot motion control device provided in an embodiment of this application; Figure 8 This is a schematic diagram of the structure of a control device provided in an embodiment of this application; Figure 9 This is a schematic diagram of the structure of a robot provided in an embodiment of this application. Detailed Implementation
[0020] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the accompanying drawings in this application are for illustrative and descriptive purposes only and are not intended to limit the scope of protection of this application. Furthermore, it should be understood that the schematic drawings are not drawn to scale. The flowcharts used in this application illustrate operations implemented according to some embodiments of this application. It should be understood that the operations in the flowcharts may not be implemented in sequence, and steps without logical contextual relationships may be reversed or implemented simultaneously. In addition, those skilled in the art, guided by the content of this application, may add one or more other operations to the flowcharts, or remove one or more operations from the flowcharts.
[0021] Furthermore, the described embodiments are merely some, not all, of the embodiments of this application. The components of the embodiments of this application described and illustrated herein can typically be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.
[0022] It should be noted that the term "comprising" will be used in the embodiments of this application to indicate the presence of the features declared thereafter, but does not exclude the addition of other features.
[0023] Currently, in interactive robot applications, users need to issue corresponding control voice commands to the robot. The robot then generates a corresponding action sequence based on the control voice and performs movement based on the action sequence. Existing technologies can only recognize the corresponding control text from the control voice and match the text with the corresponding control instructions. However, matching only the text yields the surface meaning of the control voice and cannot recognize its implicit content. This results in control instructions that cannot accurately control the robot, posing safety risks. Therefore, this application provides a robot motion control method. By employing a pre-set large language model, structured semantic parsing of the speech text is performed to obtain structured semantic information. Based on the action semantic units, a pre-set mapping table from semantic units to action parameters is used to obtain the corresponding target action parameters. Based on the target action parameters, action control instructions for the target robot are generated. This application comprehensively recognizes both the surface and deep meanings of the speech text, improving the accuracy of determining the target action parameters. This results in more accurate action control instructions, thereby improving the precision and safety of robot task execution.
[0024] The robot motion control method provided in the embodiments of this application will be described below with reference to several accompanying drawings. Figure 1 This application provides a flowchart illustrating a robot motion control method, as shown below. Figure 1 As shown, the method includes: Step 101: Obtain the control voice signal input by the user for the target robot.
[0025] The voice control signal is the voice signal input by the user through the voice acquisition device of the target robot, which can be a microphone. The user can be any person with control authority over the target robot, and the system can identify the user based on the voice control signal to determine whether to accept the user's control signal. The target robot can be a bipedal robot, humanoid robot, quadrupedal robot, hand-held robot, etc., and this application embodiment does not impose any limitations on this.
[0026] Optionally, the user's input control voice signal can be acquired through the target robot's voice acquisition device.
[0027] Step 102: Perform speech recognition on the control speech signal to obtain the speech text corresponding to the control speech signal.
[0028] Speech recognition is the process of converting speech signals into corresponding text content. This can be achieved through a speech recognition module (Automatic Speech Recognition, or ASR for short).
[0029] Optionally, a preset speech recognition module is used to perform speech recognition on the control speech signal to obtain the speech text corresponding to the control speech signal.
[0030] Optionally, before performing speech recognition on the control speech signal to obtain the corresponding speech text, the initial speech signal is filtered and normalized to obtain the processed initial speech signal as the control speech signal.
[0031] Step 103: Using a pre-set large language model, perform structured semantic parsing on the speech text to obtain structured semantic information.
[0032] The structured semantic information includes: action semantic units. Each action semantic unit includes: action intention, target object, attribute constraints, and executive constraints. Action intention refers to an operation category with clear robot kinematics, distinguished from voice commands; for example, it may include: grasping, placing, moving, and aligning. The target object is a physical entity or spatial region identified within the robot's current perception space, serving as the carrier of the action intention; for example, it may include: a cup, a box, or a part. Attribute constraints are a set of static or semi-static features that modify the target object; for example, it may include: color, size, material, and positional relationships. Executive constraints are a set of restrictive conditions coupled with the environmental state that must be satisfied when the action intention is executed; for example, it may include: gently picking up, avoiding, moving to the left, and keeping horizontal.
[0033] The pre-set large language model can be a general-purpose Large Language Model (LLM), which can be pre-trained using multiple text contents and corresponding structured semantic information, or it can be untrained and use prompt words to perform structured semantic parsing of speech text. Structured semantic parsing means parsing speech text into structured content. Structured semantic information is the structured text containing multiple action semantic units obtained from the parsing of speech text.
[0034] Optionally, a pre-set large language model is used to perform structured semantic parsing on the speech text based on pre-set prompt words to obtain structured semantic information.
[0035] Step 104: Based on the action semantic unit, use the preset semantic unit to action parameter mapping table to obtain the corresponding target action parameters.
[0036] The target motion parameters are the motion trajectories of each joint of the target robot. For example, if the target robot is a humanoid robot, its joints are typically divided into lower limbs (legs / feet), upper limbs (arms / hands), torso (waist), and head. The target motion parameters include the joint angle sequence and joint acceleration sequence for each joint. They may also include motion-related parameters such as end-effector pose, grasping mode, grasping force level, and velocity scaling factor. A pre-defined semantic unit to motion parameter mapping table includes the relationships between multiple semantic units and motion parameters, used to match motion semantic units to obtain the corresponding motion parameters.
[0037] Optionally, based on the action semantic unit, a preset semantic unit to action parameter mapping table is used. The action semantic unit is semantically matched with the preset semantic units in the preset semantic unit to action parameter mapping table to obtain a semantic matching value. If the semantic matching value is greater than the preset semantic matching value threshold, the action parameter corresponding to the matched preset semantic unit is used as the target action parameter.
[0038] For example, based on the action semantic unit, a pre-defined mapping table from semantic units to action parameters is used to map the action semantic unit to obtain the corresponding target action parameters. ,in, For the target action parameters, This is a mapping table from predefined semantic units to action parameters. This is an action semantic unit.
[0039] Step 105: Generate motion control commands for the target robot based on the target motion parameters.
[0040] Among them, motion control commands are used to control the motion of the target robot. Motion control commands are data that the target robot's control interface can recognize. They are text-formatted data packets that directly drive the target robot's underlying controller to perform physical movements and conform to its native communication protocol.
[0041] Optionally, the target motion parameters are converted into data recognizable by the target robot control interface to obtain motion control instructions. The motion control instructions are a sequence of target pose at the end of the trajectory point sequence and execution time parameters, thereby controlling the target robot according to the motion control instructions.
[0042] In this embodiment, the system acquires a user-input control voice signal for a target robot; performs speech recognition on the control voice signal to obtain the corresponding speech text; uses a pre-defined large language model to perform structured semantic parsing on the speech text to obtain structured semantic information; based on the action semantic unit, uses a pre-defined semantic unit-to-action parameter mapping table to obtain the corresponding target action parameters; and generates action control instructions for the target robot based on the target action parameters. These action control instructions are used to control the target robot's actions. This application decomposes the voice control process into five constrained stages: voice signal acquisition, speech recognition, structured semantic parsing, semantic-to-action parameter mapping, and action instruction generation, achieving a complete closed loop from the user's natural expression to the robot's precise execution. By introducing structured semantic units as intermediate representations, the system can accurately capture key information such as action intent, operation object, attribute requirements, and execution constraints, improving its understanding of fuzzy expressions, contextual relationships, and personalized language. The semantic-to-action parameter mapping table, continuously optimized based on actual execution feedback, ensures that language descriptions can be stably converted into action parameters that meet task requirements, balancing versatility and adaptability. The generated motion control commands are directly adapted to the target robot, ensuring that the commands are not only grammatically correct, but also practically feasible and safe in the physical world. This achieves both natural voice interaction and accurate motion execution in complex environments and under diverse user conditions.
[0043] Based on the above embodiments, this application also provides a process for processing voice signals in a robot motion control method. Figure 2 This is a flowchart illustrating the processing of voice signals in a robot motion control method provided in an embodiment of this application, as shown below. Figure 2 As shown, before performing speech recognition on the control speech signal in step 102 above to obtain the corresponding speech text, the method further includes: Step 201: Obtain the noise suppression weighting coefficient of the current environment based on the estimated noise components of the current environment.
[0044] The estimated noise component of the current environment is a background sound signal segment, devoid of the target user's speech components, acquired in real-time by the multi-channel microphone array built into the voice acquisition module at the moment of user voice acquisition. The noise suppression weight coefficient is calculated from the estimated noise component of the current environment using a preset nonlinear mapping function. This function takes the root mean square energy of the noise component as input, and its output value monotonically increases with the noise energy, used to dynamically adjust the trade-off between noise residue and speech distortion in subsequent noise suppression algorithms.
[0045] Optionally, ambient sound signals are collected during gaps when the target user is not present. The estimated noise component of the current environment is determined based on the ambient sound signal. Based on the estimated noise component of the current environment and the correspondence between the estimated noise component and the noise suppression weight coefficient, the noise suppression weight coefficient of the current environment is determined.
[0046] Step 202: Based on the estimated noise components and noise suppression weight coefficients, perform noise suppression processing on the control speech signal to obtain the processed speech signal.
[0047] Optionally, noise is suppressed from the controlled speech signal based on the estimated noise components and noise suppression weighting coefficients, thereby obtaining the processed speech signal. Specifically, ,in, To process the speech signal, To control the voice signal, For noise suppression weighting coefficients, To estimate the noise components.
[0048] In step 102 above, speech recognition is performed on the control speech signal to obtain the speech text corresponding to the control speech signal, including: Step 203: Perform speech recognition on the processed speech signal to obtain the speech text.
[0049] Optionally, a preset speech recognition module is used to perform speech recognition on the processed speech signal to obtain the speech text corresponding to the processed speech signal.
[0050] In this embodiment, by introducing an environment-adaptive noise modeling and dynamic weighting mechanism before speech recognition, the accuracy of speech recognition in high-noise and complex environments is improved, the details and rhythmic features of the main frequency band of speech are preserved, and the processed speech signal maintains high intelligibility while improving the signal-to-noise ratio, thus solving the problem that existing speech recognition is susceptible to environmental interference.
[0051] Based on the above embodiments, this application also provides another process for processing voice signals in a robot motion control method. Figure 3 This is a flowchart illustrating the processing of voice signals in another robot motion control method provided in this application embodiment, as shown below. Figure 3 As shown, before performing speech recognition on the control speech signal in step 102 above to obtain the corresponding speech text, the method further includes: Step 301: Obtain the user's historical voice features.
[0052] Among them, historical voice features are a set of voice parameters used to characterize the user, obtained by analyzing the user's historical control voice.
[0053] Optionally, multiple historical control voice recordings of the user are acquired, and feature extraction is performed on the multiple historical control voice recordings to determine the user's acoustic parameter set as the user's historical voice features.
[0054] Step 302: Based on historical speech characteristics, perform personalized repair on the control speech signal to obtain the repaired speech signal.
[0055] Optionally, the current control voice signal is divided into frames, and the real-time fundamental frequency and the first formant frequency are calculated frame by frame. For each frame, the deviation ratio between its fundamental frequency and the user's historical average fundamental frequency, and the deviation ratio between the first formant and the historical average are calculated. If the deviation ratio exceeds a preset deviation threshold, the frame is determined to have pronunciation distortion and is repaired to obtain the repaired voice signal. The preset deviation threshold can be 15% for the fundamental frequency and 8% for the formant. Specific compensation can be performed by compensating for the energy of the corresponding fundamental frequency harmonic components in the frame's spectrum according to the historical standard deviation ratio; or by linearly recalibrating the amplitude spectrum within the first formant frequency band according to the historical formant average.
[0056] In step 102 above, speech recognition is performed on the control speech signal to obtain the speech text corresponding to the control speech signal, including: Step 303: Perform speech recognition on the repaired speech signal to obtain the speech text corresponding to the control speech signal.
[0057] Optionally, a preset speech recognition module is used to perform speech recognition on the repaired speech signal to obtain the speech text corresponding to the repaired speech signal.
[0058] In one possible implementation, noise suppression weighting coefficients for the current environment are obtained based on the estimated noise components of the current environment. Then, noise suppression processing is applied to the control speech signal based on the estimated noise components and the noise suppression weighting coefficients to obtain a processed speech signal. Next, the user's historical speech features are obtained, and the processed speech signal is repaired based on these features to obtain a repaired processed speech signal.
[0059] In one possible implementation, the user's historical speech features are obtained; based on these features, the control speech signal is repaired to obtain a repaired speech signal. Then, based on the estimated noise components of the current environment, noise suppression weight coefficients for the current environment are obtained; and based on the estimated noise components and the noise suppression weight coefficients, noise suppression processing is applied to the repaired speech signal to obtain a processed repaired speech signal.
[0060] In this embodiment, the control voice signal is repaired by using historical voice features, avoiding distortion caused by over-processing, and accurately restoring key pronunciation information weakened by environmental or physiological factors. It adapts to the user's speaking habits rather than forcibly changing the voice itself, improving the accuracy of voice command recognition and the continuity of interaction when multiple users share the same robot system. It also enhances the system's ability to adapt to the voice features of different users and solves the problem of insufficient interaction stability caused by user differences.
[0061] Based on the above embodiments, this application also provides a process for constructing a mapping table in a robot motion control method. Figure 4 This is a flowchart illustrating the process of constructing a mapping table in a robot motion control method provided in an embodiment of this application, as shown below. Figure 4 As shown, before obtaining the corresponding target action parameters in step 104 above based on the action semantic unit and using a preset semantic unit to action parameter mapping table, the method further includes: Step 401: Obtain the semantic combination corresponding to the preset task.
[0062] The preset task is a task that has been planned and verified to be feasible before the robot is deployed. For example, it can be grasping a specified object, placing an object in a specified area, moving along a specified path to a target pose, or aligning the relative poses of two objects. This application embodiment does not limit this. Semantic combination refers to a series of consecutive semantic units. The semantic unit includes the action intention, the target object, and the execution constraints. The semantic unit may also include attribute constraints.
[0063] Optionally, based on a preset task list, a set of standard natural language instructions is associated with each task in the preset task list. For each standard instruction, structured semantic parsing is performed using a large language model to output its corresponding semantic unit. The semantic units parsed from all instructions under the same preset task are merged and deduplicated according to the logical order of actions to form the semantic combination for that task.
[0064] Step 402: Obtain the successful action parameters collected by executing a preset task based on multiple semantic units in the semantic combination, and obtain the action parameters corresponding to multiple semantic units.
[0065] Among them, the successful action parameters refer to the set of control parameters that the robot actually completes the action and is judged as successful by the system during the execution of the preset task.
[0066] Optionally, in a simulation platform or real machine environment, multiple teaching or teleoperation operations are performed for each type of semantic unit. After execution, it is verified whether all success criteria are met. If successful, the control parameters of this operation are recorded as a sample of successful action parameters for that semantic unit. Each semantic unit is verified sequentially to obtain action parameters corresponding to multiple semantic units.
[0067] Step 403: Cluster the action parameters corresponding to each semantic unit in the same semantic unit to obtain the clustering action parameters corresponding to the same semantic unit.
[0068] Among them, semantic units of the same kind refer to semantic units in semantic combination that have the same action intention, the same target object category, and whose execution constraints can be classified into the same category; for example, "grab a cup", "grab a glass", and "grab a cup with a handle" all belong to the same category of grasping container targets.
[0069] Optionally, the action parameter samples corresponding to multiple semantic units belonging to the same class are merged into a total sample set, and the sample set is clustered to obtain the action parameters corresponding to the cluster centers as the clustering action parameters of the semantic units of the same class.
[0070] Step 404: Construct a mapping table from the semantic units of the same type to the action parameters based on the corresponding clustering action parameters.
[0071] Optionally, a mapping table from the preset semantic units to the action parameters can be constructed by using the string identifier of each semantic unit of the same type as the key and the clustering action parameter as the value.
[0072] In this embodiment, action parameters are collected based on the actual successful execution process, ensuring the physical authenticity and engineering usability of the data. Semantic units with common characteristics are grouped into the same category and clustered for analysis. The action patterns under different expressions are integrated, which improves the coverage of users' diverse language expressions and avoids the data sparsity and overfitting problems caused by modeling each subtle semantic difference separately.
[0073] Based on the above embodiments, this application also provides a process for correcting target motion parameters in a robot motion control method. Figure 5 This is a flowchart illustrating the correction of target motion parameters in a robot motion control method provided in an embodiment of this application, as shown below. Figure 5 As shown, before generating motion control commands for the target robot based on the target motion parameters in step 105 above, the method further includes: Step 501: Based on the environmental perception data of the current environment, the target robot's most recent historical execution state, and the user's historical operation records, obtain the target robot's action correction amount.
[0074] Optionally, based on the environmental perception data of the current environment, an environmental feature vector of the current environment is obtained; based on the historical execution state of the target robot most recently, historical state features of the target robot are obtained; based on the user's historical operation records, user operation features are obtained; and based on the environmental feature vector, historical state features, and operation features, the action correction amount of the target robot is obtained.
[0075] Optionally, the environmental feature vector of the current environment can be obtained based on the environmental perception data of the current environment.
[0076] The environmental perception data refers to the current scene state acquired by the target robot through vision or sensors. The current scene state can include the target's position, obstacle distribution, reachable space, and relative relationships. The environmental feature vector is a fixed-length numerical vector obtained by deterministically transforming the current environmental perception data. It can include: the geometric center coordinates of the target object's point cloud, bounding box size, surface normal vector distribution, distance to the nearest obstacle, azimuth and distance of the obstacle on the horizontal plane, chassis tilt angle, normalized illumination intensity value, and corresponding semantic labels.
[0077] Optionally, the target robot acquires the current scene state through vision or sensors. The current scene state may include environmental data such as target position, obstacle distribution, reachable space and relative relationships. By performing vector transformation on the environmental data, corresponding binary labels are obtained, and environmental feature vectors are generated based on the binary transformation of the environmental data.
[0078] Optionally, the historical state characteristics of the target robot can be obtained based on the target robot's most recent historical execution state.
[0079] The historical execution state refers to the most recent one or more execution states of the target robot before the current task, including the end effector pose, motion trajectory, and success or failure flags. Historical state characteristics refer to a snapshot of the target robot's state during its last complete action task before the current voice command was triggered. This may include: the final pose of the end effector, execution time, actual peak grasping force, whether a collision alarm occurred, whether the target was successfully placed, and task type encoding, etc. This application embodiment does not impose limitations on these aspects.
[0080] Optionally, feature extraction methods are used to extract features from the target robot's most recent historical execution state to obtain the target robot's historical state features. The feature extraction methods can include finite difference methods, sliding window statistics, etc., and this application embodiment does not limit the specific methods used.
[0081] Optionally, user operation characteristics can be obtained based on the user's historical operation records.
[0082] Among them, historical operation records are speed preferences, force preferences, and path preferences obtained from the user's past operation records. Operation features refer to the features that reflect the user's operation preferences extracted from the user's historical operation records.
[0083] Optionally, the user's historical operation records can be obtained, and feature recognition can be performed on these records to obtain the user's operation characteristics. Specifically, for speed and force, the user's typical preference values can be extracted by calculating the mean, median, or Gaussian distribution fitting; for path preferences, clustering algorithms can be used to perform cluster analysis on the starting point, ending point, and waypoints of the historical trajectory to identify the user's frequently used fixed routes or habitual obstacle avoidance strategies.
[0084] Optionally, the motion correction amount of the target robot can be obtained based on environmental feature vectors, historical state features, and operational features.
[0085] Optionally, the environmental feature vector and historical state features are fused and concatenated using a pre-defined context fusion network to obtain the action correction amount of the target robot. The pre-defined context fusion network can be a pre-trained neural network for feature fusion and mapping. The pre-defined context fusion network may include a fusion module and a transformation module. Specifically, the fusion module in the pre-defined context fusion network fuses the environmental feature vector, historical state features, and operational features to obtain context fusion features, and the transformation module converts the context fusion features into action vectors to obtain context action parameters, which are then used as the action correction amount.
[0086] Step 502: Weight the motion correction amount and the target motion parameters to obtain the corrected motion parameters.
[0087] Optionally, the motion correction amount and the target motion parameters are weighted and summed based on the first weighting coefficient corresponding to the motion correction amount and the second weighting coefficient corresponding to the target motion parameters to obtain the corrected motion parameters. The first and second weighting coefficients are used to generate different parameters based on different motion correction amounts. The weighting coefficients are determined according to actual needs, and this embodiment does not impose any restrictions on them.
[0088] For example, the motion correction amount and the target motion parameters are weighted and summed according to the first weighting coefficient and the second weighting coefficient to obtain the final control motion parameters. ,in, For the target action parameters, For motion correction amount, The first weighting coefficient for the target action parameter. This is the second weighting coefficient for the motion correction.
[0089] In step 105 above, motion control commands for the target robot are generated based on the target motion parameters, including: Step 503: Generate motion control commands for the target robot based on the corrected motion parameters.
[0090] Optionally, the corrected motion parameters are converted into data that can be recognized by the target robot's control interface to obtain motion control commands.
[0091] In this embodiment, by integrating environmental perception, the robot's own state, and user operating habits to dynamically generate action correction amounts, the robot can achieve environmentally adaptable and personalized intelligent control, ensuring the stability and safety of robot operation. Furthermore, by analyzing the historical operating habits of specific users, the robot's response can be made more in line with the individual's operating style, thereby providing more natural and accurate assistance in human-machine collaboration, improving overall operating efficiency and user experience.
[0092] Based on the above embodiments, this application also provides a process for execution verification in a robot motion control method. Figure 6 This is a flowchart illustrating the execution verification process in a robot motion control method provided in this application embodiment, as shown below. Figure 6 As shown, before generating motion control commands for the target robot based on the target motion parameters in step 105 above, the method further includes: Step 601: Perform multi-dimensional execution verification on the target action parameters.
[0093] The multiple dimensions can include joint limit exceedance dimension, joint limit exceedance dimension, joint limit exceedance dimension, and joint limit exceedance dimension. Executability verification is used to check whether the target motion parameters can be executed.
[0094] Optionally, the executableness of the target action parameters can be verified by judging whether the target action parameters can be executed from multiple dimensions.
[0095] Step 602: If the execution verification of any dimension fails, the preset action parameter correction algorithm corresponding to any dimension is used to correct the target action parameter to obtain the corrected action parameter.
[0096] Among them, the execution failure indicates that the target motion parameters do not meet the corresponding dimensional judgment criteria in any dimension. In other words, executing the target motion parameters in this dimension may cause damage to the robot or result in task failure. The preset motion parameter correction algorithm is a correction algorithm based on the features of the corresponding dimension.
[0097] Optionally, the target action parameters are executed using multiple dimensions. If the execution verification fails in any of the multiple dimensions, the target action parameters are corrected using the preset action parameter correction algorithm corresponding to that dimension to obtain the corrected action parameters.
[0098] In the embodiments of this application, the safety and robustness of robot action execution are significantly improved through multi-dimensional pre-verification and intelligent fault-tolerant correction mechanisms, avoiding execution failure or equipment damage caused by exceeding the limits of a single dimension, and generating safe and feasible new action instructions, thereby ensuring that the robot can continue to operate stably and continuously in complex or unexpected environments.
[0099] Based on the above embodiments, this application also provides a process for correcting motion parameters in a robot motion control method. In step 602 above, if the execution verification of any dimension among multiple dimensions fails, a preset motion parameter correction algorithm corresponding to any dimension is used to correct the target motion parameters, and the corrected motion parameters include: If the execution verification fails because the joint limits in multiple dimensions exceed the limits of the dimensions, the range of motion of each joint corresponding to the target motion parameters is obtained; the range of motion of the joints that exceed the limits is limited and clipped, and the corrected motion parameters are obtained based on the clipped range of motion.
[0100] Among them, the joint limit out-of-dimension is used to limit the joint angles of each joint in the target robot.
[0101] Optionally, if the execution verification fails for joints exceeding their limits in multiple dimensions, it indicates that some joint angles in the target motion parameters exceed the joint limits and thus the safe range of motion for that joint in the dimension. Then, based on the target robot, the range of motion for each joint corresponding to the target motion parameters is obtained. Based on the target motion parameters, the corresponding motion angles for each joint are obtained. Joints whose motion angles exceed their corresponding physical limits are subject to limit clipping, restricting their motion range to the safe range of motion for that joint. This results in the clipped motion range, which is then mapped to the target control parameters to obtain the corrected motion parameters. The safe range of motion is the range of motion within the robot's safe range obtained through prior testing of the target robot.
[0102] If the execution verification of the collision risk dimension fails among multiple dimensions, the pose points with collision risk in the predicted trajectory corresponding to the target action parameters are obtained, and the corrected action parameters are obtained based on the pose points with collision risk.
[0103] Among them, the collision risk dimension is used to restrict the movement trajectory of the target robot.
[0104] Optionally, if the execution verification of the collision risk dimension fails among multiple dimensions, it indicates that the target robot may face collision risk during movement. Then, the pose points with collision risk in the predicted trajectory corresponding to the target motion parameters are obtained. Based on the starting point and intermediate points of the pose points with collision risk, and the trajectories of the pose points with collision risk and obstacles in the current scene, new pose points are replanned to obtain new pose points. These new pose points are then mapped to the target control parameters to obtain the corrected motion parameters.
[0105] If the execution verification of the workspace constraint dimension fails in multiple dimensions, the end target corresponding to the target action parameter is projected to the nearest workspace boundary, and the corrected action parameter is obtained based on the projected end target.
[0106] The workspace constraint dimension is used to limit the working range of the target robot.
[0107] Optionally, if the execution verification of the workspace constraint dimension fails among multiple dimensions, it indicates that the target robot will exceed the workspace boundary during its movement. Then, based on the workspace boundary, the end-effector corresponding to the target motion parameters is projected onto the nearest workspace boundary to obtain the projected end-effector of the target robot. The projected end-effector is then mapped to the target control parameters to obtain the corrected motion parameters.
[0108] If the execution verification of multiple dimensions fails due to the safety parameters exceeding the limits, the safety parameters corresponding to the target action parameters are obtained, and the safety parameters are scaled to a preset safety range. Based on the scaled-down parameters, the corrected action parameters are obtained.
[0109] The safety parameter limit dimension is used to restrict the target robot's speed, acceleration, or gripping force. The preset safety range is the range obtained by testing the target robot. Safety parameters can include speed parameters, acceleration parameters, or gripping force parameters.
[0110] Optionally, if the execution verification of the dimension with out-of-limit safety parameters fails in multiple dimensions, it indicates that at least one parameter of the target robot—speed, acceleration, or grasping force—exceeds the corresponding preset safety range during its movement. In this case, the safety parameters corresponding to the target motion parameters are obtained, scaled to the preset safety range, and mapped to the target control parameters based on the scaled-down parameters to obtain the corrected motion parameters.
[0111] In this embodiment, any non-compliance is addressed and corrected accordingly. This ensures that the original intention of the action is preserved to the greatest extent possible while mitigating execution risks with minimal adjustment costs. This guarantees the continuity and safety of the robot's actions, ensures the absolute feasibility of action execution, and preserves the original intention and operational precision of the user's voice commands to the greatest extent possible.
[0112] Based on the same inventive concept, this application also provides a robot motion control device corresponding to the robot motion control method. Since the principle of the device in this application is similar to that of the robot motion control method described above, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be described again.
[0113] Figure 7 This is a schematic diagram of the structure of a robot motion control device provided in an embodiment of this application, as shown below. Figure 7 As shown, the device includes: an acquisition module 701, a recognition module 702, a parsing module 703, a mapping module 704, and a generation module 705; wherein, the acquisition module 701 is used to acquire the control voice signal for the target robot input by the user; The recognition module 702 is used to perform speech recognition on the control speech signal to obtain the speech text corresponding to the control speech signal; The parsing module 703 is used to perform structured semantic parsing on speech text using a preset large language model to obtain structured semantic information, which includes: action semantic units; The mapping module 704 is used to obtain the corresponding target action parameters based on the action semantic unit and using a preset semantic unit to action parameter mapping table; The generation module 705 is used to generate motion control instructions for the target robot based on the target motion parameters; the motion control instructions are used to control the motion of the target robot.
[0114] In one possible implementation, the identification module 702 is further configured to: obtain the noise suppression weighting coefficient of the current environment based on the estimated noise components of the current environment; Based on the estimated noise components and noise suppression weighting coefficients, noise suppression processing is performed on the control speech signal to obtain the processed speech signal. In one possible implementation, the recognition module 702 is specifically used to: perform speech recognition on the processed speech signal to obtain speech text.
[0115] In one possible implementation, the recognition module 702 is further configured to: acquire the user's historical voice features; Based on historical speech characteristics, the control speech signal is individually repaired to obtain the repaired speech signal; In one possible implementation, the recognition module 702 is specifically used to: perform speech recognition on the repaired speech signal to obtain the speech text corresponding to the control speech signal.
[0116] In one possible implementation, the mapping module 704 is further configured to: obtain the semantic combination corresponding to the preset task; Obtain successful action parameters collected by executing a preset task based on multiple semantic units in semantic combination, and obtain the action parameters corresponding to multiple semantic units; Cluster the action parameters corresponding to each semantic unit in the same semantic unit to obtain the clustering action parameters corresponding to the same semantic unit. Based on the semantic units of the same type and their corresponding clustering action parameters, a mapping table from preset semantic units to action parameters is constructed.
[0117] In one possible implementation, the generation module 705 is further configured to: obtain the environmental feature vector of the current environment based on the environmental perception data of the current environment; Based on the target robot's most recent historical execution state, obtain the target robot's historical state characteristics; Based on environmental feature vectors, historical state features, and operational features, the action correction amount of the target robot is obtained; The motion correction amount and the target motion parameters are weighted to obtain the corrected motion parameters; In one possible implementation, the generation module 705 is specifically used for: Based on the corrected motion parameters, motion control commands are generated for the target robot.
[0118] In one possible implementation, the generation module 705 is further configured to: perform multi-dimensional execution verification on the target action parameters; If the execution verification fails in any of the multiple dimensions, the preset action parameter correction algorithm corresponding to that dimension is used to correct the target action parameter and obtain the corrected action parameter.
[0119] In one possible implementation, the generation module 705 is specifically used to: if the joint limits in multiple dimensions exceed the execution verification of the dimension and fail, obtain the range of motion of each joint corresponding to the target motion parameters; limit and trim the range of motion of the joints that exceed the limits, and obtain the corrected motion parameters based on the trimmed range of motion. Alternatively, if the joint limits in multiple dimensions exceed the execution verification of the dimension, then obtain the pose points in the predicted trajectory corresponding to the target motion parameters that have a collision risk, and obtain the corrected motion parameters based on the pose points that have a collision risk. Alternatively, if the joint limits in multiple dimensions exceed the dimensional execution verification, the end target corresponding to the target motion parameters is projected to the nearest workspace boundary, and the corrected motion parameters are obtained based on the projected end target. Alternatively, if the execution verification of multiple dimensions fails due to the safety parameters exceeding the limits, the safety parameters corresponding to the target action parameters are obtained, and the safety parameters are scaled to a preset safety range. Based on the scaled-down parameters, the corrected action parameters are obtained.
[0120] The processing flow of each module in the device and the interaction flow between each module can be referred to the relevant descriptions in the above method embodiments, and will not be detailed here.
[0121] This application also provides a control device. Figure 8 This is a schematic diagram of the structure of a control device provided in an embodiment of this application, such as... Figure 8 As shown, the system includes a processor 801 and a memory 802, and optionally, a bus 803. The memory 802 stores machine-readable instructions executable by the processor 801. When the control device is running, the processor 801 and the memory 802 communicate via the bus 803. When the machine-readable instructions are executed by the processor 801, the steps of the robot motion control method described above are performed. The control device can be an external control device independent of the robot, or it can be an internal controller integrated into the robot. If it is an external control device, it can control the robot wirelessly or via a wired connection.
[0122] This application also provides a robot. Figure 9 This application provides a schematic diagram of the structure of a robot, as shown in the embodiment of the present application. Figure 9 As shown, it includes: a robot body 901, and a controller 902 within the robot body, the controller 902 being used to execute the steps of the above-described robot motion control method.
[0123] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the robot motion control method described above.
[0124] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems and devices described above can be referred to the corresponding processes in the method embodiments, and will not be repeated here. In the several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection can be through some communication interfaces; the indirect coupling or communication connection of devices or modules can be electrical, mechanical, or other forms.
[0125] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. If the functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes: USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, optical disks, and other media capable of storing program code.
[0126] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.
Claims
1. A robot motion control method, characterized in that, The method includes: Acquire user-input control voice signals for the target robot; The control voice signal is subjected to speech recognition to obtain the corresponding voice text. A pre-defined large language model is used to perform structured semantic parsing on the speech text to obtain structured semantic information, which includes: action semantic units; Based on the action semantic unit, the corresponding target action parameters are obtained using a preset semantic unit to action parameter mapping table; Based on the target motion parameters, motion control instructions are generated for the target robot; the motion control instructions are used to control the motion of the target robot.
2. The method according to claim 1, characterized in that, Before obtaining the corresponding target action parameters based on the action semantic unit using a preset semantic unit-to-action parameter mapping table, the method further includes: Obtain the semantic combination corresponding to the preset task; Obtain successful action parameters for the preset task collected based on multiple semantic units in the semantic combination, and obtain the action parameters corresponding to the multiple semantic units; Cluster the action parameters corresponding to each semantic unit in the same semantic unit to obtain the clustering action parameters corresponding to the same semantic unit. Based on the semantic units of the same type and the corresponding clustering action parameters, a mapping table from the preset semantic units to the action parameters is constructed.
3. The method according to claim 1, characterized in that, Before generating motion control commands for the target robot based on the target motion parameters, the method further includes: Based on the environmental perception data of the current environment, the target robot's most recent historical execution state, and the user's historical operation records, the motion correction amount of the target robot is obtained; The motion correction amount and the target motion parameters are weighted to obtain the corrected motion parameters; The step of generating motion control commands for the target robot based on the target motion parameters includes: Based on the corrected motion parameters, motion control commands are generated for the target robot.
4. The method according to claim 1, characterized in that, Before generating motion control commands for the target robot based on the target motion parameters, the method further includes: Perform multi-dimensional execution verification on the target action parameters; If the execution verification fails in any of the multiple dimensions, the preset action parameter correction algorithm corresponding to that dimension is used to correct the target action parameter to obtain the corrected action parameter.
5. The method according to claim 4, characterized in that, If the execution verification fails in any of the multiple dimensions, a preset action parameter correction algorithm corresponding to that dimension is used to correct the target action parameters, resulting in corrected action parameters including: If the execution verification fails due to joint limits exceeding the limits in multiple dimensions, then the range of motion of each joint corresponding to the target motion parameters is obtained; the range of motion of the joints exceeding the limits is limited and trimmed, and the corrected motion parameters are obtained based on the trimmed range of motion; or, If the joint limits exceed the dimensional execution verification in multiple dimensions, then obtain the pose points in the predicted trajectory corresponding to the target motion parameters that have a collision risk, and obtain the corrected motion parameters based on the pose points with collision risks; or, If the joint limits in multiple dimensions exceed the dimensional execution verification and fail, then the end target corresponding to the target motion parameters is projected to the nearest workspace boundary, and the corrected motion parameters are obtained based on the projected end target; or, If the execution verification of the dimension with excessive safety parameters fails in multiple dimensions, the safety parameters corresponding to the target action parameters are obtained, and the safety parameters are scaled to a preset safety range. Based on the scaled-down parameters, the corrected action parameters are obtained.
6. The method according to claim 1, characterized in that, Before performing speech recognition on the control speech signal to obtain the corresponding speech text, the method further includes: Based on the estimated noise components of the current environment, obtain the noise suppression weighting coefficient of the current environment; Based on the estimated noise components and the noise suppression weighting coefficients, the control speech signal is subjected to noise suppression processing to obtain the processed speech signal; The step of performing speech recognition on the control speech signal to obtain the speech text corresponding to the control speech signal includes: The processed speech signal is subjected to speech recognition to obtain the speech text.
7. The method according to claim 1, characterized in that, Before performing speech recognition on the control speech signal to obtain the corresponding speech text, the method further includes: Obtain the user's historical voice features; Based on the historical speech characteristics, the control speech signal is personalized and repaired to obtain the repaired speech signal; The step of performing speech recognition on the control speech signal to obtain the speech text corresponding to the control speech signal includes: The repaired speech signal is subjected to speech recognition to obtain the speech text corresponding to the control speech signal.
8. A control device, characterized in that, The control device includes a processor and a memory, the memory storing machine-readable instructions executable by the processor. When the computer device is running, the processor executes the machine-readable instructions to perform the steps of the robot motion control method as described in any one of claims 1 to 7.
9. A robot, characterized in that, The robot includes: a robot body, and a controller within the robot body, the controller being used to execute the steps of the robot motion control method according to any one of claims 1 to 7.
10. A robot motion control device, characterized in that, The device includes: The acquisition module is used to acquire the control voice signals input by the user for the target robot; The recognition module is used to perform speech recognition on the control voice signal to obtain the voice text corresponding to the control voice signal; The parsing module is used to perform structured semantic parsing on the speech text using a preset large language model to obtain structured semantic information, which includes: action semantic units; The mapping module is used to obtain the corresponding target action parameters based on the action semantic unit and a preset mapping table from semantic unit to action parameters. The generation module is used to generate motion control instructions for the target robot based on the target motion parameters; the motion control instructions are used to control the motion of the target robot.