Quadruped robot cargo placement correction method fusing visual identification and voice interaction

Abstract

The invention discloses a quadruped robot cargo placement correction method fusing visual identification and voice interaction, and the method comprises the following steps: S1, collecting and preprocessing cargo images, and generating a standardized image input set; s2, constructing a spatial state sensing model, extracting image features and generating error vectors; s3, recognizing user voice and extracting semantic keywords such as direction and action instructions; s4, fusing the error vector and the semantic keyword, generating a multi-modal fusion input representation, and encoding the multi-modal fusion input representation into a fusion embedding representation; s5, generating a gait control parameter and an action control sequence based on the fusion embedding representation, and constructing an action plan structure; s6, executing the action plan structure to complete cargo position and posture correction, and recording state feedback data; and S7, in combination with state feedback and additional voice, constructing an interactive correction closed-loop control process, and updating control information. According to the method, the multi-modal understanding and action adjusting capability of the robot is enhanced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of multi-modal perception and fusion, and in particular to a four-legged robot goods placement correction method fusing visual recognition and voice interaction. BACKGROUND

[0002] The existing four-legged robot mainly relies on a single visual recognition mechanism for environment perception and action planning when performing tasks such as goods placement, lacks real-time understanding and response ability to human voice intentions, and has weak correction ability, poor interactivity, and low task completion rate in complex environments or non-predefined scenarios.

[0003] In terms of voice interaction, traditional systems mostly use independent voice recognition and control modules, which cannot effectively fuse with the visual perception system of the robot. Especially in the case of ambiguous user instructions or inconsistent spatial descriptions, the system has difficulty accurately matching target actions, which can easily cause misoperation or response delay.

[0004] In addition, existing robots mostly use a one-time execution of preset actions, lack of closed-loop feedback and multi-round interaction ability. The action execution result often cannot be corrected in time, especially in tasks where the goods posture needs to be accurately adjusted, often resulting in repeated trial and error, substandard precision, and other problems, which seriously limits the autonomous adaptation and task robustness of the system.

[0005] Based on the above background problems, we need to build an intelligent correction method with semantic understanding ability, visual guidance precision, and human-computer interaction closed-loop control ability to improve the work efficiency and intelligence level of the four-legged robot in dynamic task environment.

[0006] Therefore, how to provide a four-legged robot goods placement correction method fusing visual recognition and voice interaction is a problem that those skilled in the art need to solve. SUMMARY

[0007] One purpose of the present application is to provide a four-legged robot goods placement correction method fusing visual recognition and voice interaction, which fully integrates image perception, semantic understanding, and multi-modal encoding technology, builds a fusion encoding model based on Transformer and an interactive correction closed-loop control mechanism, and realizes accurate identification of goods pose error and semantic-driven dynamic adjustment control. The method has the advantages of strong multi-modal collaborative perception, high semantic guidance precision, and high degree of human-computer interaction intelligence, breaking through the technical bottlenecks of existing robots such as fragmented perception, rigid control, and lack of semantic understanding ability.

[0008] According to the four-legged robot goods placement correction method fusing visual recognition and voice interaction of the present application, the method comprises the following steps:

[0009] S1. Based on the embedded camera system mounted on the quadruped robot, the original image data sequence containing the goods to be placed is acquired. Image preprocessing is performed on the original image data sequence, including image enhancement, resolution unification and noise suppression, to generate a standardized image input set to support subsequent spatial state modeling and error perception.

[0010] S2. Based on the standardized image input set, input it into the cargo spatial state perception model containing a convolutional neural network structure, extract edge features, spatial texture and geometric contours in the image, and calculate the difference with the target pose to form an error vector;

[0011] S3. The voice acquisition module carried by the quadruped robot receives the original voice signal issued by the user and transcribes it into voice text content through the voice recognition model. Then, based on the semantic parsing rules, it extracts a set of semantic keywords that include directional prompts, corrective action commands and spatial location descriptions.

[0012] S4. Based on the error vector and the set of semantic keywords, the two are respectively input into the preset embedding encoding module to generate visual embedding vector and semantic embedding vector. Vector concatenation and cross-attention mechanism are performed in the embedding space to construct a joint representation structure, which is then fused to form a multimodal fusion input representation. This multimodal fusion input representation is used as the input of the multimodal fusion encoding model based on the Transformer architecture. Joint encoding is performed in the model to output the fusion embedding representation, which is used to represent the contextual correlation features between visual errors and semantic guidance, and provides input for the subsequent generation of correction strategies.

[0013] S5. Based on the fusion embedding representation, the action decoder module maps it to a standardized set of gait control parameters and constructs an action control sequence consistent with the target action to drive the quadruped robot to generate an action plan structure including position fine-tuning, posture adjustment and movement trajectory correction.

[0014] S6. Based on the action plan structure, the behavior cloning algorithm, which incorporates posture constraints and gait stability optimization mechanisms, is invoked to drive the quadruped robot to complete the physical adjustment of the target cargo's position and posture. During execution, the strategy generalization capability is enhanced through multimodal state input, and a dynamic training set is constructed by combining real-time state feedback data and expert imitation samples to achieve adaptive optimization of action control and improve execution accuracy.

[0015] S7. Based on state feedback data and user-added voice input, construct a correction and optimization mechanism that integrates imitation learning and semantic control. Input the above data into an interactive correction closed-loop control system that introduces a behavior cloning reinforcement model, dynamically update expert behavior strategies and fusion embedded representations, and drive iterative optimization of the action generation model.

[0016] Optionally, the step S1 specifically comprises:

[0017] S11, acquiring a sequence of original image data composed of continuous frames by a quadruped robot-mounted embedded camera system at multiple angles, wherein each frame contains the current spatial pose of the goods and environmental background information;

[0018] S12, performing image enhancement processing on the sequence of original image data, including improving image contrast, saturation and edge definition, to enhance the separability of the target outline and the background;

[0019] S13, performing resolution uniform processing on the enhanced image, uniformly adjusting all images to preset size parameters, including image height pixel value and image width pixel value, to ensure the consistency of image input dimensions;

[0020] S14, performing noise suppression processing on the size-unified image sequence, using a spatial filtering algorithm to remove high-frequency random noise and illumination artifacts introduced by sensor interference in the image, and retaining the main feature distribution of the goods edge and surface texture;

[0021] S15, packaging the image sequence processed by the above processing into a structured image sample set to form a standardized image input set, which is used to support the subsequent step S2 of goods space state perception model input processing.

[0022] Optionally, the step S2 specifically comprises:

[0023] S21, constructing a goods space state perception model based on the standardized image input set, the model adopts a convolutional neural network structure containing a multi-scale convolution feature extraction unit and a spatial attention module, which is used to extract edge features and local geometric structure information in the image;

[0024] S22, inputting the standardized image input set into the convolution feature extraction unit to extract edge outlines, texture directions and spatial structure features in the image, forming a multi-level image feature map set;

[0025] S23, inputting the multi-level image feature map set into the spatial attention module, the spatial attention module includes a channel compression unit, an attention weight mapping unit and a weighted fusion unit, wherein: the channel compression unit compresses the feature map along the channel dimension by average pooling and maximum pooling to generate two single-channel spatial response maps; the attention weight mapping unit inputs the two response maps after splicing into a convolution layer to generate an attention weight map consistent with the input spatial dimension; the weighted fusion unit performs element-wise multiplication operation on the original feature map and the attention weight map, and outputs a spatial weighted feature representation, which is used to enhance the feature expression of the goods main region and suppress the environmental interference features;

[0026] S24, based on the spatial weighted feature representation, predicting the three-dimensional position vector and the attitude angle vector of the goods respectively by using a regression module, the regression module comprising a global pooling layer, a fully connected mapping layer and an output prediction layer, wherein: the global pooling layer is used to compress the spatial features into a one-dimensional feature vector; the fully connected mapping layer performs nonlinear transformation on the one-dimensional vector and forms two output branches; the output prediction layer is used to output the three-dimensional position vector comprising three position components and the attitude angle vector comprising three attitude angle components respectively, the position vector is used to represent the current coordinate position of the goods in space, and the attitude angle vector is used to represent the Euler angle direction information of the goods, which is used to represent the current pose information of the goods in space;

[0027] S25, performing element-by-element difference calculation on the predicted position vector and the preset target position vector to obtain a position error vector; performing element-by-element difference calculation on the predicted attitude angle vector and the target attitude angle vector to obtain an attitude error vector; and splicing the position error vector and the attitude error vector to form a complete error vector, which is used to support the multi-modal fusion processing in the subsequent step S4.

[0028] Optionally, the step S3 specifically comprises:

[0029] S31, receiving, by a voice collection module carried by the quadruped robot, an original voice signal issued by a user, and inputting the original voice signal into a voice recognition model, the voice recognition model comprising an audio feature extractor, a position encoding module, an attention mechanism encoder and a self-recurrent decoder, wherein: the audio feature extractor converts the voice signal into a frame-level acoustic feature vector, the position encoding module embeds time information to preserve the pronunciation order structure, the attention mechanism encoder extracts global context voice representation based on the attention mechanism, and the decoder generates corresponding standardized voice text content based on the acoustic representation as input for subsequent semantic analysis.

[0030] S32, inputting the recognized standardized voice text content into a semantic analysis module, and performing semantic analysis processing on the text content by the semantic analysis module based on semantic analysis rules and structured semantic extraction rules;

[0031] The semantic analysis rules comprise three types of directional prompt recognition rules, action trigger instruction extraction rules and spatial position description analysis rules, which are used to identify directional guide words, adjust action verb phrases and target reference position phrases from the text, and output a preliminary semantic keyword set;

[0032] The structured semantic extraction rule performs syntax fragment matching and semantic label mapping processing on the voice text based on a preset semantic template library, the semantic template library includes three types of templates, namely action instruction templates, direction correction templates and target position templates, which are respectively used to extract action keywords, direction phrases and spatial reference expression contents, and finally a unified semantic keyword set is constructed for subsequent multi-modal fusion processing.

[0033] Optionally, the step S4 specifically comprises:

[0034] S41, based on the error vector and the semantic keyword set, input them into an embedding coding module respectively, the embedding coding module includes a visual coding channel and a semantic coding channel, the visual coding channel maps the error vector into a unified dimension visual embedding vector based on a multi-layer full connection network, and the semantic coding channel jointly constructs a semantic embedding vector based on a word vector embedder and a context window modeler, and performs position coding and context sequence maintaining processing on the semantic keyword;

[0035] S42, based on the visual embedding vector and the semantic embedding vector, a cross-attention mechanism is performed in the embedding space to construct a joint representation structure, wherein the semantic embedding vector is taken as a query vector, and the visual embedding vector is taken as a key-value pair, a visual regulation and control dependency relationship under semantic guidance is established, so that a multi-modal fusion input representation is generated, which is used to capture the cross-modal context interaction association between the visual error and the semantic guidance; wherein the joint representation structure explicitly encodes the response relationship between the visual error information and the semantic instruction, and is taken as an input of a multi-modal fusion coding model, which is used to drive a subsequent action correction strategy generation process.

[0036] S43, taking the multi-modal fusion input representation as an input of a multi-modal fusion coding model, the multi-modal fusion coding model is constructed based on a Transformer architecture, and includes the following three key sub-modules:

[0037] The self-attention module is used to establish a full connection dependency relationship between the visual error information and the semantic keyword, the cross-modal context coupling features are captured by introducing a multi-head attention mechanism, the response intensity of the semantic instruction in different visual perception regions is highlighted, and the visual feature re-weighting under semantic guidance is realized; the position coding module is used to maintain the sequence sensitivity in the semantic keyword sequence, a learnable position vector embedding method is adopted, the action guidance sequence and the spatial direction description sequence in the semantic are explicitly coded into the fusion structure, and the understanding ability of the model to the semantic progressive relationship in the action sequence is enhanced; the feedforward update module is used to perform nonlinear conversion and compression expression on the fused multi-modal features, and utilizes an activation function and a residual connection mechanism to perform semantic compression, redundancy suppression and instruction feature enhancement on the fusion structure, and outputs a fusion embedding representation for supporting a subsequent correction strategy generation module;

[0038] S44, the fusion embedding representation represents the joint coding of visual error information, semantic instruction information and structural spatial context features, forms a semantic driven feature representation structure representing the association relationship among vision, language and action, and provides an input basis for subsequent deviation correction strategy generation and gait control parameter prediction.

[0039] Optionally, the step S5 specifically includes:

[0040] S51, based on the fusion embedding representation, the fusion embedding representation is accepted as an input feature vector and input into an action decoder module, the action decoder module includes three parts of an instruction mapping layer, a gait generation network and an action plan construction unit;

[0041] S52, in the instruction mapping layer, a full connection network is used to perform mapping conversion from a semantic space to a motion space on the fusion embedding representation, and a control semantic vector for driving an action is extracted;

[0042] S53, based on the control semantic vector, the control semantic vector is input into the gait generation network, the gait generation network constructs a network structure based on an adjustable motion control parameter template, and outputs a standardized gait control parameter set including rhythm frequency, step length, joint angle and posture control dimensions;

[0043] S54, based on the standardized gait control parameter set, the standardized gait control parameter set is input into the action plan construction unit, and an action control sequence including position fine-tuning, posture correction and path planning is constructed by combining a target pose requirement of the goods and physical execution constraints of the quadruped robot, for generating an action plan structure meeting the correction requirement, and driving the quadruped robot to complete the deviation correction task.

[0044] Optionally, the step S6 specifically includes:

[0045] S61, the control instruction sequence in the action plan structure is analyzed, information of each action unit is extracted, including joint control parameters, execution timing information, posture adjustment requirements and gait switching constraints, and parameterized encoding is performed to construct a structured execution parameter set, which is used as an input data source of the behavior cloning algorithm;

[0046] S62, the behavior cloning algorithm introducing a posture constraint objective function and a gait stability objective function is called, the structured execution parameter set is accepted as an input, and a low-order gait control instruction sequence is generated, which is used to drive the quadruped robot to perform a physical adjustment action of the target goods, wherein:

[0047] The attitude constraint target function is based on the attitude state space modeling of the robot body, limits the real-time change range of the pitch angle, roll angle and yaw angle in the control instruction generation process, constructs an angle keeping mechanism, and guarantees the attitude stability and angle accuracy of the goods in the pose adjustment process; the gait stability target function combines the robot dynamics model and the ground contact state information, dynamically evaluates the stability margin at different gait stages, suppresses the dynamic disturbance caused by gait switching by optimizing the switching rhythm of the support leg and the swing leg in the instruction sequence, and improves the continuity and stability of the robot action execution; the behavior cloning algorithm minimizes the trajectory deviation loss, the attitude stability loss and the gait disturbance loss function, guides the low-order control instruction sequence to imitate the expert demonstration trajectory while meeting the physical execution constraints, and realizes the adaptive response to the adjustment task in a complex environment;

[0048] S63, in the behavior cloning algorithm training stage, the behavior policy function is initialized based on the historical expert demonstration sample, then a dynamic state-action sample pool is constructed, the sensor state and action response sample in the current execution are fused with the expert demonstration sample, and are updated continuously, forming an incremental imitation learning training set supporting online optimization;

[0049] S64, the behavior policy function is updated, the generalization ability of the model in the disturbance scene is improved, and the generated low-order gait control instruction sequence is input into the robot bottom execution system to drive the robot to complete the physical adjustment of the target goods;

[0050] S65, the motion state feedback data is collected in real time during the execution process, including attitude change, ground stability and execution trajectory, and key indicators, providing a sensing basis for the subsequent correction process.

[0051] Optionally, the step S7 specifically comprises:

[0052] S71, based on the behavior policy function and the motion state feedback data, an interactive closed-loop correction control system integrating imitation learning and semantic control is constructed, the state feedback data and the user's additional voice input in the current round of execution are collected, the behavior deviation information and the instruction correction intention are extracted through the error detection module and the semantic analysis module respectively, and the fused correction input representation is generated;

[0053] S72, based on the correction input representation, the behavior cloning reinforcement model is input, the historical expert action sample and the abnormal trajectory information in the current execution state are combined, the existing behavior policy function is updated locally, the updated control action instruction is generated, which is used to replace and adjust the deviated or unstable instruction in the current action plan structure, and the behavior cloning reinforcement model also introduces the attitude constraint mechanism and the gait stability evaluation mechanism in the process of generating the updated action, wherein:

[0054] The posture constraint mechanism is used for maintaining the posture stability of the robot body in the pitch angle, roll angle and yaw angle dimensions during the updating of the control action, and by limiting the influence of the control action on the dynamic change range of the posture angle, it is ensured that the target goods do not tilt, deviate or rotate abnormally during the correction.

[0055] S73, a contrast imitation loss function is constructed, by integrating the behavior difference between the current control action and the expert demonstration action, the contrast distance between the state embedding, and the consistency relationship between the semantic instruction and the control response, a multi-level and multi-target optimization signal is formed, which is used to train and fine-tune the behavior strategy function, improve its adaptability to dynamic disturbance and semantic deviation, and enable the quadruped robot to maintain the imitation effect and response accuracy in continuous and multiple rounds of tasks.

[0056] S74, the updated control action is injected into the action plan structure and submitted to the bottom control system for execution, real-time collection of new round of state feedback data and voice input, writing into the correction database and imitation training sample pool, continuous optimization of behavior strategy function, realizing dynamic self-adaptation of closed-loop correction control system in complex interactive environment.

[0057] The beneficial effects of the present application are:

[0058] Firstly, the present application constructs a "visual-language-action" nested feature expression method with context cooperation capability by fusing the multi-modal modeling mechanism of visual recognition and voice interaction, breaks through the problem of fragmented processing between the perception information and semantic instructions of the existing robot, realizes the direct guidance and re-weighting modeling of visual behavior by semantic instructions, and effectively enhances the semantic consistency and behavior expression ability under multi-modal information fusion.

[0059] Secondly, by introducing the multi-modal fusion coding model and cross attention mechanism based on the Transformer architecture, the present application can capture the dependency relationship between semantic keywords and visual errors, strengthen the guiding role of semantic hints in the action decision-making process, and further improve the contribution of semantics to the understanding of complex spatial state and action generation, thereby realizing accurate and efficient action plan construction and execution parameter labeling.

[0060] In addition, the application innovatively designs an interactive deviation correction closed-loop control mechanism, which integrates state feedback and user voice additional instructions into the iterative process, and combines error evaluation and semantic consistency judgment logic to form an intelligent regulation and control system with a full-process closed loop of perception-fusion-execution-evaluation. The system not only supports the dynamic adjustment capability of "executing and correcting at the same time", but also has good self-adaptability and continuous learning capability, significantly improving the ability of the robot to process fuzzy instructions and dynamic feedback in unstructured scenes.

[0061] The method effectively improves the response accuracy and control flexibility of the quadruped robot in complex scenes to perform the goods placement correction task, and provides an innovative solution that is structured, deployable and engineering feasible for multi-modal human-robot collaborative control technology. BRIEF DESCRIPTION OF DRAWINGS

[0062] The accompanying drawings are included to provide a further understanding of the application, and constitute a part of the specification, together with the embodiments of the application, to explain the application, and do not constitute a limitation on the application. In the drawings:

[0063] Fig. 1 A whole flow chart of a quadruped robot goods placement correction method fusing visual recognition and voice interaction is provided for the application;

[0064] Fig. 2 A semantic guided visual feature reweighting structure based on a cross-attention mechanism is provided for the application;

[0065] Fig. 3 A structure diagram of an interactive deviation correction closed-loop control mechanism is provided for the application. DETAILED DESCRIPTION

[0066] The application will now be described in further detail with reference to the drawings. These drawings are simplified schematic diagrams, and only illustrate the basic structure of the application in a schematic manner, and therefore only show the components related to the application.

[0067] REFERENCE Figs. 1-3 A quadruped robot goods placement correction method fusing visual recognition and voice interaction includes the following steps:

[0068] S1, based on the embedded camera system carried by the quadruped robot, collect the original image data sequence containing the goods to be placed, perform image preprocessing on the original image data sequence, including image enhancement, resolution unification and noise suppression processing, generate a standardized image input set, which is used to support subsequent spatial state modeling and error perception;

[0069] S2, based on the standardized image input set, input it into the cargo space state perception model containing the convolutional neural network structure, extract the edge features, spatial texture and geometric contour in the image, and calculate the difference with the target pose to form an error vector;

[0070] S3, the voice collection module carried by the quadruped robot receives the original voice signal issued by the user, and converts it into voice text content through the voice recognition model, and then extracts a set of semantic keywords containing direction prompts, correction action instructions and spatial position descriptions based on semantic analysis rules;

[0071] S4, based on the error vector and the semantic keyword set, input them into the preset embedding coding module respectively, generate visual embedding vectors and semantic embedding vectors, and perform vector splicing and introduce cross attention mechanism in the embedding space to build a joint representation structure, fusion to form a multi-modal fusion input representation, and take this multi-modal fusion input representation as the input of the multi-modal fusion coding model based on the Transformer architecture, perform joint coding in the model, and output the fusion embedding representation, which is used to represent the context association features between visual errors and semantic guidance. Provide input for subsequent error correction strategy generation;

[0072] S5, based on the fusion embedding representation, map it to a set of standardized gait control parameters through the action decoder module, and construct an action control sequence consistent with the correction target action, which is used to drive the quadruped robot to generate an action plan structure including position fine-tuning, pose adjustment and movement trajectory correction;

[0073] S6, execute the correction action control, the quadruped robot completes the physical adjustment operation of the target cargo position and pose according to the action plan structure, and records the state feedback data in the execution process in real time, including whether the action is successfully executed, the balance state of the robot itself and the response time delay of the control signal;

[0074] S7, based on the state feedback data and the user's additional voice instructions, build an interactive error correction closed loop control module, input them into the interactive error correction closed loop control module, update the error vector and the semantic keyword set; if it is judged that the current pose does not meet the target requirement, repeat steps S4 to S6 until the task is completed, forming a complete perception-fusion-execution-evaluation closed loop control process.

[0075] The application proposes a four-legged robot goods placement correction method combining visual recognition and voice interaction, and establishes a complete closed loop from image and voice input to multi-round physical adjustment execution. Through image acquisition and preprocessing, a standardized image input set is constructed, the goods pose error is extracted in combination with a space state perception model, and a semantic keyword set is generated by synchronously recognizing user voice, forming a multi-modal fusion input representation. A multi-modal fusion encoding model based on the Transformer architecture is used to extract a fusion embedding representation, guiding the action generation module to construct a structured action plan structure. Further, a behavior cloning algorithm is introduced, combining posture constraint and gait stability optimization mechanism to generate low-order control instructions to drive the robot to complete preliminary physical adjustment. In the execution process, combined with multi-modal state input enhancement strategy and feedback data, a dynamic training set is continuously constructed and the expert behavior strategy is updated. Finally, an interactive correction closed loop control mechanism is introduced, combining additional voice input and action feedback to realize multi-round iterative optimization, significantly improving the action generation accuracy, semantic understanding ability and posture control stability in complex environments.

[0076] In the embodiment, the step S1 specifically includes:

[0077] S11, a multi-angle image acquisition of a work scene containing a to-be-placed goods is performed by an embedded camera system carried by a four-legged robot, to obtain a sequence of original image data composed of continuous frames, wherein each frame of image contains the current spatial pose of the goods and environmental background information;

[0078] S12, image enhancement processing is performed on the sequence of original image data, including improving image contrast, saturation and edge definition, to enhance the separability of the target outline and the background;

[0079] S13, resolution uniformity processing is performed on the enhanced image, to uniformly adjust all images to preset size parameters, including image height pixel value and image width pixel value, to ensure the consistency of image input dimensions;

[0080] S14, noise suppression processing is performed on the image sequence after size uniformity, and a spatial filtering algorithm is used to remove high-frequency random noise and illumination artifacts introduced by sensor interference in the image, and to retain the main feature distribution of the goods edge and surface texture;

[0081] S15, the image sequence after the above processing is packaged as a structured image sample set, to form a standardized image input set, for supporting the goods space state perception model input processing in the subsequent step S2.

[0082] The step ensures that the generated standardized image input set has stable input quality and structural consistency by multi-level image preprocessing operations including image enhancement, size unification, noise suppression and structured packaging, provides a high-robustness image basis for subsequent spatial perception and multi-modal fusion, and enhances the adaptability and structural expression of visual input in complex environments.

[0083] In the embodiment, the step S2 specifically includes:

[0084] S21, based on the standardized image input set, a goods space state perception model is constructed, the model adopts a convolutional neural network structure including a multi-scale convolution feature extraction unit and a spatial attention module, and is used for extracting edge features and local geometric structure information in the image;

[0085] S22, the standardized image input set is input into the convolution feature extraction unit, edge contours, texture directions and spatial structure features in the image are extracted, and a multi-level image feature map set is formed;

[0086] S23, the multi-level image feature map set is input into the spatial attention module, the spatial attention module includes a channel compression unit, an attention weight mapping unit and a weighted fusion unit, wherein: the channel compression unit compresses the feature map along the channel dimension by average pooling and maximum pooling, and generates two single-channel spatial response maps; the attention weight mapping unit inputs the two response maps into a convolution layer after splicing, and generates an attention weight map consistent with the input spatial dimension; the weighted fusion unit performs element-wise multiplication operation on the original feature map and the attention weight map, and outputs a spatial weighted feature representation, which is used to enhance the feature expression of the goods main body area and suppress the environmental interference features;

[0087] S24, based on the spatial weighted feature representation, a regression module is used to predict a three-dimensional position vector and an attitude angle vector of the goods, the regression module includes a global pooling layer, a full connection mapping layer and an output prediction layer, wherein: the global pooling layer is used to compress the spatial features into a one-dimensional feature vector; the full connection mapping layer performs nonlinear transformation on the one-dimensional vector and forms two output branches; the output prediction layer is used to output a three-dimensional position vector including three position components and an attitude angle vector including three attitude angle components, the position vector is used to represent the current coordinate position of the goods in space, the attitude angle vector is used to represent the Euler angle direction information of the goods, and is used to represent the current pose information of the goods in space;

[0088] S25, element-by-element difference calculation is performed on the predicted position vector and the preset target position vector to obtain a position error vector; element-by-element difference calculation is performed on the predicted attitude angle vector and the target attitude angle vector to obtain an attitude error vector; the position error vector and the attitude error vector are spliced to form a complete error vector, which is used to support the multi-modal fusion processing in the subsequent step S4.

[0089] In order to realize high-precision perception of the space position and attitude of the goods, the goods space state perception model adopts a deep convolutional neural network structure that fuses multi-scale feature extraction and spatial attention mechanism. Specifically, the model is composed of multi-level convolution units, a feature fusion module and a spatial attention module. The multi-scale convolution unit is used to extract local detail features and global structure features in the image in parallel, enhancing the modeling capability of different scale target contours; the feature fusion module integrates multi-layer feature maps, improves the semantic expression level, and ensures that fine-grained information is not lost in the deep semantic abstraction process; the spatial attention module dynamically gives different weight responses to feature regions according to context information, highlights the edge, corner and attitude features of the goods, and improves the robustness of the perception model under complex background and occlusion conditions. In the training stage, the model uses a joint loss function to optimize the position regression and attitude estimation error at the same time, ensuring the stability and accuracy of the space state estimation. The model has strong generalization ability and is suitable for goods state perception tasks of different sizes, shapes and materials, providing structural precision guarantee and data stability support for subsequent error vector generation and multi-modal fusion.

[0090] In the embodiment, the step S3 specifically includes:

[0091] S31, the voice acquisition module carried by the quadruped robot receives the original voice signal issued by the user, and inputs the original voice signal into a voice recognition model, the voice recognition model including an audio feature extractor, a position encoding module, an attention mechanism encoder and a self-recurrent decoder, wherein: the audio feature extractor converts the voice signal into a frame-level acoustic feature vector, the position encoding module embeds time information to preserve the pronunciation order structure, the attention mechanism encoder extracts global context voice representation based on the attention mechanism, and the decoder generates corresponding standardized voice text content based on the acoustic representation as input for subsequent semantic analysis.

[0092] S32, input the recognized standardized voice text content into a semantic analysis module, and the semantic analysis module performs semantic analysis processing on the text content based on semantic analysis rules and structured semantic extraction rules;

[0093] The semantic analysis rules include a directionality hint recognition rule, an action trigger instruction extraction rule, and a spatial position description analysis rule, and are used to identify direction guide words, adjust action verb phrases, and target reference position phrases from the text, and output a preliminary semantic keyword set.

[0094] The structured semantic extraction rule performs grammar fragment matching and semantic tag mapping processing on the voice text based on a preset semantic template library. The semantic template library includes three types of templates, namely, an action instruction template, a direction correction template, and a target position template, which are respectively used to extract action keywords, direction phrases, and spatial reference expression content, and finally construct a unified semantic keyword set for subsequent multi-modal fusion processing.

[0095] The semantic analysis rule is the core knowledge structure in the semantic analysis module, and is mainly used to guide the identification and extraction of action instructions, direction hints, and spatial descriptions in the voice text. The rule set is composed of a general semantic template library and a task scenario specific rule library. The general semantic template library is used to cover common command structures, such as "please put … in …", "move towards …", "adjust the direction of …", etc. By defining the combination mode of verb type, direction preposition, and spatial object noun, the core intention of the command is quickly extracted. The task scenario specific rule library is extended for common spatial expressions and correction instructions in the goods placement correction scene, such as "a little to the right", "rotate ninety degrees", "align", "back up a little", etc. Through the position label mapping and action classification structure in the rule, the embedding form in the semantic keyword set is analyzed. The rule matching process combines part-of-speech tagging, dependency syntax structure, and context window information to locate the component structure that meets the rule template in the input sentence, and outputs the corresponding semantic label and keyword vector according to the definition. The semantic analysis rule system has a flexible and extensible structure, can adapt to the diversity of user language habits, significantly improves the parsing accuracy and semantic integrity of voice instructions in robot interaction tasks, and provides stable and high-quality semantic input for subsequent multi-modal fusion.

[0096] In the embodiment, the step S4 specifically includes:

[0097] S41, based on the error vector and the semantic keyword set, inputting the two into an embedding coding module, the embedding coding module including a visual coding channel and a semantic coding channel, the visual coding channel mapping the error vector into a unified dimension visual embedding vector based on a multi-layer fully connected network, and the semantic coding channel constructing a semantic embedding vector based on a word vector embedder and a context window modeler, and performing position coding and context sequence maintaining processing on the semantic keyword;

[0098] S42, based on the visual embedding vector and the semantic embedding vector, performing a cross-attention mechanism in an embedding space to construct a joint representation structure, wherein the semantic embedding vector is taken as a query vector and the visual embedding vector is taken as a key-value pair, to establish a visual regulation and control dependency relationship under semantic guidance, thereby generating a multi-modal fusion input representation for capturing cross-modal context interaction association between visual error and semantic guidance; wherein the joint representation structure explicitly encodes the response relationship between visual error information and semantic instructions, and serves as an input of a multi-modal fusion encoding model for driving a subsequent action correction strategy generation process.

[0099] S43, taking the multi-modal fusion input representation as an input of a multi-modal fusion encoding model, the multi-modal fusion encoding model being constructed based on a Transformer architecture and comprising the following three key sub-modules:

[0100] A self-attention module is used to establish a full connection dependency relationship between visual error information and semantic keywords, to capture context coupling features between cross-modalities by introducing a multi-head attention mechanism, to highlight the response intensity of semantic instructions in different visual perception regions, and to realize re-weighting of visual features under semantic guidance; a position encoding module is used to maintain order sensitivity in a semantic keyword sequence, to explicitly encode action guidance order and spatial direction description order in the semantic into the fusion structure by using a learnable position vector embedding method, and to enhance the understanding ability of the model for the semantic progressive relationship in the action sequence; a feedforward update module is used to perform nonlinear conversion and compressed expression on the fused multi-modal features, to perform semantic compression, redundancy suppression and instruction feature enhancement on the fusion structure by using an activation function and a residual connection mechanism, and to output a fusion embedding representation for supporting a subsequent correction strategy generation module.

[0101] S44, the fusion embedding representation jointly encodes visual error information, semantic instruction information and structural spatial context features to form a semantic driven feature representation structure representing the association relationship between vision, language and action, thereby providing an input basis for subsequent correction strategy generation and gait control parameter prediction.

[0102] The generation process of the multi-modal fusion input representation aims to achieve efficient alignment and joint modeling of visual perception information and speech semantic information, and improve the understanding ability of the quadruped robot to the "graph-sentence" coordination graph in the cargo deviation correction task. The process first extracts a visual embedding vector corresponding to the error vector and a semantic embedding vector corresponding to the semantic keyword set through an embedding coding module. The two embedding vectors are standardized in dimension to ensure the structural compatibility of the subsequent fusion operation. The fusion operation adopts a joint modeling strategy based on a cross-attention mechanism, taking the semantic embedding vector as the query vector to guide the context-dependent modeling of the visual embedding vector (Key / Value), thereby identifying the perception area or instruction part in the visual error feature that is highly associated with the current semantic keyword. This attention mechanism can highlight the response position of the key semantic instruction (such as "left" and "forward adjustment") in the visual error information, and strengthen the regulatory role of semantic guidance in the feature space. Subsequently, the context vector output by the attention mechanism is fused with the original embedding vector through a weighted fusion strategy, and normalization and nonlinear mapping operations are introduced, and finally a fused multi-modal fusion input representation is generated. This fusion representation not only retains the spatial perception attributes of the visual error information, but also fuses the action guidance features of the speech instructions, laying a high semantic consistency multi-modal semantic foundation for subsequent action decoding and plan generation. The whole process realizes the fine coupling of cross-modal information, effectively solving the bottleneck problem of the difficulty of image and speech information to be aligned in traditional methods.

[0103] In the embodiment, the step S5 specifically includes:

[0104] S51, based on the fusion embedding representation, accepting it as an input feature vector, inputting it into an action decoder module, which includes three parts of an instruction mapping layer, a gait generation network, and an action plan construction unit;

[0105] S52, in the instruction mapping layer, a full connection network is used to perform mapping conversion from semantic space to motion space on the fusion embedding representation, and a control semantic vector for driving action is extracted;

[0106] S53, based on the control semantic vector, inputting it into the gait generation network, the gait generation network constructs a network structure based on an adjustable motion control parameter template, and outputs a standardized gait control parameter set containing rhythm frequency, step length, joint angle, and posture control dimensions;

[0107] S54, based on the standardized gait control parameter set, inputting it into the action plan construction unit, combining the cargo target pose requirement and the physical execution constraint of the quadruped robot, constructing an action control sequence containing position fine-tuning, posture correction, and path planning, for generating an action plan structure meeting the correction requirement, to drive the quadruped robot to complete the deviation correction task.

[0108] In the present application, the action plan construction unit serves as a key bridge for multi-modal semantic fusion to action execution, and undertakes the task of structured mapping from gait control parameters to complete physical execution sequence. This unit takes the standardized gait control parameter set as input, fuses the target pose information of the goods and the kinematic model characteristics of the robot, and completes the organization, combination and optimization of multi-dimensional action instructions in the posture adjustment process. The construction process includes the following core steps: first, the input gait control parameters are time-coded to form a primary control frame with a clear time advancement structure; second, combined with the spatial orientation in the semantic keywords and the meaning of the action instructions, the posture change amplitude and turning strategy in the control frame are dynamically adjusted to ensure that the action sequence is consistent with the user's semantic intention; in addition, this unit introduces an action constraint filtering mechanism based on task targets to eliminate control schemes that may cause posture deviation or out-of-bound operations, and according to the target displacement and posture transformation, a reasonable execution trajectory is matched to finally output an action plan structure containing continuous posture change instructions, stable support strategies and environment adaptation parameters. This construction unit establishes a stable mapping relationship between high-level semantics and low-level control, and is a key component of the present application to realize multi-modal action execution capability under semantic driving.

[0109] In the present embodiment, the step S6 specifically includes:

[0110] S61, analyze the control instruction sequence in the action plan structure, extract the information of each action unit, including joint control parameters, execution timing information, posture adjustment requirements and gait switching constraints, and perform parameterized coding to construct a structured execution parameter set, which is used as an input data source for the behavior cloning algorithm;

[0111] S62, call the behavior cloning algorithm introducing the posture constraint objective function and the gait stability objective function, accept the structured execution parameter set as input, generate a low-order gait control instruction sequence, and use it to drive the quadruped robot to execute the physical adjustment action of the target goods, wherein:

[0112] The pose constraint target function is based on the pose state space modeling of the robot body, limits the real-time change range of the pitch angle, roll angle and yaw angle in the control instruction generation process, constructs an angle keeping mechanism, and guarantees the pose stability and angle accuracy of the goods in the pose adjustment process; the gait stability target function combines the robot dynamics model and the ground contact state information, dynamically evaluates the stability margin at different gait stages, suppresses the dynamic disturbance caused by gait switching by optimizing the switching rhythm of the support leg and the swing leg in the instruction sequence, and improves the continuity and stability of the robot action execution; the behavior cloning algorithm minimizes the trajectory deviation loss, the pose stability loss and the gait disturbance loss function, guides the low-order control instruction sequence to imitate the expert demonstration trajectory while meeting the physical execution constraints, and realizes the adaptive response to the adjustment task in the complex environment;

[0113] S63, in the behavior cloning algorithm training stage, the behavior policy function is initialized based on the historical expert demonstration sample, then the dynamic state-action sample pool is constructed, the sensor state and action response sample in the current execution are fused with the expert demonstration sample, and are continuously updated to form an incremental imitation learning training set supporting online optimization;

[0114] S64, update the behavior policy function, improve the generalization ability of the model in the disturbance scene, and input the generated low-order gait control instruction sequence into the robot bottom execution system to drive the robot to complete the physical adjustment of the target goods;

[0115] S65, real-time acquisition of motion state feedback data in the execution process, including key indicators such as pose variation, ground stability and execution trajectory, to provide a sensing basis for the subsequent correction process.

[0116] During the execution of the behavior cloning algorithm, the behavior policy function is used to establish the mapping relationship between the current environment state and the output action, and is the core decision mechanism for generating low-order gait control instruction sequences. The function takes a structured set of execution parameters as input, encodes and learns joint control parameters, execution timing information, posture adjustment requirements, and gait switching constraints through a multi-layer perception network, and fuses historical expert demonstration samples to build an initial policy weight. During the training phase, the behavior policy function continuously iteratively optimizes data in the dynamic state-action sample pool, and the optimization objectives include key indicators such as trajectory deviation loss, posture stability loss, and gait disturbance loss. Specifically, the trajectory deviation loss measures the degree of deviation of the current control instruction from the expert demonstration trajectory in the spatial path, the posture stability loss reflects the maintenance of the cargo posture angle during execution, and the gait disturbance loss evaluates the inertial disturbance caused by the switching of the supporting leg and the swinging leg. By jointly minimizing the above loss terms, the behavior policy function gradually improves its ability to generate actions in complex scenarios, and balances imitation and stability in subsequent low-order control instruction reasoning, ensuring the accuracy and stability of the quadruped robot during physical adjustment.

[0117] In the embodiment, the step S7 specifically includes:

[0118] S71, based on the behavior policy function and the motion state feedback data, an interactive closed-loop correction control system integrating imitation learning and semantic control is constructed, the state feedback data and the user's additional voice input during the current wheel execution are collected, the behavior deviation information and the instruction correction intention are extracted through the error detection module and the semantic analysis module respectively, and a fused correction input representation is generated;

[0119] S72, based on the correction input representation, input it to the behavior cloning reinforcement model, combine the historical expert action samples and the abnormal trajectory information in the current execution state, update the existing behavior policy function locally based on it, generate updated control action instructions, and use them to replace and adjust the deviated or unstable instructions in the current action plan structure. The behavior cloning reinforcement model also introduces a posture constraint mechanism and a gait stability evaluation mechanism during the generation of updated actions, wherein:

[0120] The posture constraint mechanism is used to maintain the stability of the robot body in the pitch angle, roll angle and yaw angle dimensions during the updating of the control action, by limiting the influence of the control action on the dynamic change range of the posture angle, to ensure that the target goods do not tilt, deviate or rotate abnormally in space during the correction; the gait stability evaluation mechanism dynamically evaluates the switching rhythm of the supporting leg and the swinging leg and the position of the gravity center projection based on the robot dynamics parameters and the ground contact state information, and analyzes the execution timing and landing position of the action switching during the correction to ensure the stability and continuity of the robot during the correction action, and to avoid action failure or secondary error caused by unstable gait;

[0121] S73, a contrast imitation loss function is constructed, by integrating the behavior difference between the current control action and the expert demonstration action, the contrast distance between the state embeddings, and the consistency relationship between the semantic instructions and the control responses, a multi-level and multi-objective optimization signal is formed, which is used to train and fine-tune the behavior policy function, improve its adaptability to dynamic disturbance and semantic deviation, and enable the quadruped robot to maintain the imitation effect and response accuracy in continuous and multiple rounds of tasks;

[0122] S74, the updated control action is injected into the action plan structure and submitted to the underlying control system for execution, real-time collection of new round of state feedback data and voice input, writing into the correction database and imitation training sample pool, continuous optimization of behavior policy function, realizing dynamic self-adaptation of closed-loop correction control system in complex interactive environment.

[0123] The closed-loop correction control system is constructed on the execution framework of the fusion of the imitation learning mechanism and the semantic control mechanism, based on the multi-round perception-reasoning-execution-feedback cycle, dynamically connecting the state perception module, the voice analysis module, the behavior cloning reinforcement model and the action execution system. In each execution cycle, the closed-loop correction control system receives the motion state feedback data from the robot bottom and the user's additional semantic input, forms the current round of multi-modal correction input representation, and outputs the updated control action through the behavior cloning reinforcement model, which replaces or adjusts the control instruction in the action plan structure in real time. The system records the execution result of each round, updates the correction database, and constructs the imitation loss signal based on the new round of state and behavior data, realizes the online fine-tuning of the behavior policy function, and has the ability of continuous self-optimization. The system can ensure that the quadruped robot can efficiently, continuously and stably respond and correct the action deviation or execution anomaly in the dynamic environment, and improve the robustness and accuracy of the overall task execution.

[0124] Embodiment 1:

[0125] To verify the feasibility of the application in implementation, the application is applied to a certain city intelligent warehousing handling scene, and the spatial pose correction task of the quadruped robot to the stacked goods is completed. In this scene, the target goods are logistics boxes with a size of 320mmx220mmx180mm, which are initially randomly placed on the ground in front of the goods shelf by manual, and there are various types of offset errors, including X-axis, Y-axis direction offset, Z-axis flip tilt, angle error, etc. The quadruped robot needs to adjust its standard pose according to the visual recognition result and the manual voice instruction: parallel to the edge of the goods shelf and the center aligned with the positioning mark set on the ground.

[0126] The perception module carried by the robot collects multi-angle images of the working site, collects 18 frames of image sequences with a resolution of 1280x720, and generates a standardized image input set after enhancement, resolution normalization and noise suppression processing, and inputs it into the trained goods spatial state perception model. The model outputs the current spatial coordinates and attitude angle of the goods. Combined with the preset target pose, the position error vector and the attitude error vector are calculated.

[0127] The user's voice input is: move a little to the left, and straighten the box. The system automatically identifies the keywords, including move to the left, straighten, and box, to form a semantic keyword set, and encodes the error vectors into visual embedding vectors and semantic embedding vectors, respectively, to form a multi-modal fusion input representation after fusion. After inputting into the Transformer fusion encoding module, the fusion embedding representation is output. The system action decoder converts the fusion representation into a set of gait control parameters, generating an action control sequence containing translation control, steering control and leg lifting action. After the robot executes the control sequence, the goods move to the left and correct the angle, so that the final attitude error is reduced to within ±1.5°.

[0128] During the action execution process, the robot collects real-time execution feedback data, records position and attitude information at 10Hz per second, and generates standardized state feedback data after combining IMU and visual odometry data fusion. After one adjustment, the user issues the instruction again: turn right a little more; the system automatically triggers the interactive correction control module, identifies the "right turn" in the additional instruction, re-builds the semantic keyword set and updates the new fusion input combined with the feedback error, generates a new round of action plan, and completes the second adjustment operation. Finally, the goods angle is corrected to within 0.3° of yaw error, and the position error is less than ±1cm.

[0129] In the comparative test, the traditional deviation correction method based on visual error feedback is used for 10 experiments, and the average completion time is 28.4s, and the success alignment rate is 80%; while the method of the application is used for 10 experiments, the average completion time is 18.9s, and the alignment success rate reaches 98%. In addition, in the response time of the user's additional instruction, the traditional scheme needs additional initialization adjustment process, and the average response delay is 4.6s, while the additional voice response delay in the application scheme is controlled within 1.8s.

[0130] The above examples fully demonstrate that the application has good processing capability and robustness when facing multi-modal fuzzy semantics, unstructured scenes and dynamic error feedback, and has higher deviation correction accuracy, lower response delay and better human-computer interaction experience.

[0131] The above is only a preferred specific embodiment of the application, but the protection scope of the application is not limited thereto, any person skilled in the art can make equivalent replacement or change according to the technical scheme and the inventive concept of the application within the technical range disclosed by the application, which should be covered within the protection scope of the application.

Claims

1. A four-legged robot goods placement correction method fusing visual recognition and voice interaction, characterized in that, Comprise the following steps: S1, collect the original image data sequence containing the goods to be placed, and perform image preprocessing to generate a standardized image input set; S2, based on the standardized image input set, input it into the goods space state perception model, extract the image information, and calculate the difference with the target pose to form an error vector; S3, receive the original voice signal issued by the user, and transcribe it into voice text content, and then extract a set of semantic keywords containing direction prompts, correction action instructions and spatial position descriptions; S4, based on the error vector and the semantic keyword set, generate visual embedding vectors and semantic embedding vectors, and fuse them to form a multi-modal fusion input representation, and input the multi-modal fusion input representation as the input of the multi-modal fusion coding model, and encode it in the model. Joint coding, output fusion embedding representation; S5, based on the fusion embedding representation, map it to a standardized gait control parameter set through the action decoder module, and construct an action control sequence consistent with the correction target action to drive the quadruped robot execution system to generate an action plan structure; S6, based on the action plan structure, call the behavior cloning algorithm with posture constraint and gait stability optimization mechanism to drive the quadruped robot to complete the physical adjustment operation of the target goods position and posture; In the execution process, the generalization ability is enhanced through the multi-modal state input enhancement strategy, and the dynamic training set is constructed combining the real-time state feedback data and the expert imitation sample; S7, based on the state feedback data and the user's additional voice input, construct a deviation correction optimization mechanism that combines imitation learning and semantic control, input them into the interactive deviation correction closed-loop control system of the behavior cloning reinforcement model, dynamically update the expert behavior strategy and the fusion embedding representation, and drive the action generation model to iterate and optimize.

2. The four-legged robot goods placement correction method fusing visual recognition and voice interaction according to claim 1, characterized in that, The step S1 specifically comprises: S11, acquire the original image data sequence composed of continuous frames by collecting the work scene image containing the goods to be placed at multiple angles through the quadruped robot; S12, perform image enhancement processing on the original image data sequence; S13, perform resolution uniform processing on the enhanced image to uniformly adjust all images to the preset size parameter, ensuring the consistency of the image input dimension; S14, perform noise suppression processing on the image sequence after size uniformization to retain the main feature distribution of the goods edge and surface texture; S15, encapsulate the image sequence processed by the above processing into a structured image sample set to form a standardized image input set.

3. The method according to claim 1, wherein, The step S2 specifically comprises: S21, based on the standardized image input set, build a goods space state perception model to extract edge features and local geometric structure information in the image; S22, input the standardized image input set into the convolution feature extraction unit to extract edge contours, texture directions and spatial structure features in the image to form a multi-level image feature map set; S23, input the multi-level image feature map set into the spatial attention module to output spatial weighted feature representation; S24, based on the spatial weighted feature representation, predicting the three-dimensional position vector and the attitude angle vector of the goods respectively by using a regression module, the position vector is used to represent the current coordinate position of the goods in space, and the attitude angle vector is used to represent the Euler angle direction information of the goods; S25, calculating the difference between the predicted position vector and the preset target position vector element by element to obtain a position error vector; calculating the difference between the predicted attitude angle vector and the target attitude angle vector element by element to obtain an attitude error vector; and concatenating the position error vector and the attitude error vector to form a complete error vector.

4. The method of claim 1, wherein the method further comprises: The step S3 specifically comprises: S31, receiving the original voice signal sent by the user by the voice acquisition module carried by the quadruped robot, and inputting the original voice signal into the voice recognition model to generate corresponding standardized voice text content; S32, inputting the recognized standardized voice text content into the semantic analysis module, and performing semantic analysis processing on the text content based on the semantic analysis rule and the structured semantic extraction rule, and finally constructing a unified semantic keyword set.

5. The method of claim 1, wherein the method further comprises: The step S4 specifically comprises: S41, based on the error vector and the semantic keyword set, inputting them into the embedding coding module respectively, outputting the visual embedding vector and the semantic embedding vector, and performing position coding and context order maintaining processing on the semantic keyword; S42, based on the visual embedding vector and the semantic embedding vector, performing cross-attention mechanism in the embedding space to construct a joint representation structure, wherein the semantic embedding vector is used as a query vector, and the visual embedding vector is used as a key-value pair, to establish a visual regulation and control dependency relationship under semantic guidance, thereby generating a multi-modal fusion input representation; S43, inputting the multi-modal fusion input representation into the multi-modal fusion coding model constructed based on the Transformer architecture to output a fusion embedding representation.

6. The method of claim 1, wherein the method further comprises: The step S5 specifically comprises: S51, based on the fusion embedding representation, accepting it as an input feature vector, and inputting it into the action decoder module, which includes an instruction mapping layer, a gait generation network, and an action plan construction unit; S52, in the instruction mapping layer, using a fully connected network to perform mapping conversion from semantic space to motion space on the fusion embedding representation to extract a control semantic vector for driving actions; S53, based on the control semantic vector, inputting it into the gait generation network, which constructs a network structure based on an adjustable motion control parameter template to output a standardized gait control parameter set; S54, based on the standardized gait control parameter set, inputting it into the action plan construction unit, combining the target pose demand of the goods and the physical execution constraint of the quadruped robot to construct an action control sequence, which is used to generate an action plan structure that meets the correction requirements to drive the quadruped robot to complete the deviation correction task.

7. The method of claim 1, wherein the method further comprises: The step S6 specifically comprises: S61, analyzing the control instruction sequence in the action plan structure, extracting the information of each action unit, and performing parameterized coding to construct a structured execution parameter set; S62, call the behavior cloning algorithm of introducing the gait stability objective function and the posture constraint objective function, accept the structured execution parameter set as input, generate the low-order gait control instruction sequence, wherein: The posture constraint objective function is based on the posture state space modeling of the robot body, and limits the real-time change range of the cargo pitch angle, roll angle and yaw angle in the control instruction generation process; The gait stability objective function combines the robot dynamics model and the ground contact state information, dynamically evaluates the stability margin of the cargo in different gait stages, and suppresses the dynamic disturbance caused by gait switching by optimizing the switching rhythm of the support leg and the swing leg in the instruction sequence; The behavior cloning algorithm guides the low-order control instruction sequence to imitate the expert demonstration trajectory while meeting the physical execution constraints by jointly minimizing the trajectory deviation loss, the posture stability loss and the gait disturbance loss function; S63, in the behavior cloning algorithm training stage, initialize the behavior policy function based on the historical expert demonstration sample, then build a dynamic state-action sample pool, fuse the current execution sensor state and action response sample with the expert demonstration sample, and continuously update to form an incremental imitation learning training set supporting online optimization; S64, update the behavior policy function to improve the generalization ability of the model under disturbance scenarios, and input the generated low-order gait control instruction sequence into the robot bottom execution system to drive the robot to complete the physical adjustment of the target cargo; S65, collect motion state feedback data in real time during execution.

8. The method of claim 1, wherein the method further comprises: The step S7 specifically includes: S71, based on the behavior policy function and the motion state feedback data, build an interactive closed-loop correction control system that integrates imitation learning and semantic control, collect state feedback data and user additional voice input during current wheel execution, extract behavior deviation information and instruction correction intention through error detection module and semantic analysis module respectively, and generate fused correction input representation; S72, based on the correction input representation, input it into the behavior cloning reinforcement model, combine the historical expert action sample with the abnormal trajectory information in the current execution state, update the existing behavior policy function based on the existing behavior policy function, generate updated control action instructions, which are used to replace and adjust the deviated or unstable instructions in the current action plan structure, and the behavior cloning reinforcement model also introduces the posture constraint mechanism and the gait stability evaluation mechanism in the process of generating updated actions, wherein: The posture constraint mechanism is used to maintain the posture stability of the robot body in the pitch angle, roll angle and yaw angle dimensions during the update of the control action, by limiting the influence of the control action on the dynamic change range of the posture angle, to ensure that the target cargo maintains the spatial posture without inclination, deviation or rotation abnormality during correction; The gait stability evaluation mechanism dynamically evaluates the switching rhythm of the support leg and the swing leg and the position of the gravity center projection based on the robot dynamics parameters and the ground contact state information, and constrains the execution timing and landing position of the action switching in the correction process through stability margin analysis, to ensure the stability and continuity of the robot in the correction action, and avoid action failure or secondary error caused by unstable gait; S73, construct a contrast imitation loss function to update the difference between the control action and the expert standard action in the trajectory, semantics and physical constraints as a signal, fine-tune the current behavior policy function, and improve its adaptability to dynamic disturbance and semantic deviation; S74, inject the updated control action into the action planning structure, and submit it to the underlying control system for execution, real-time collection of new round of state feedback data and voice input, writing into the correction database and imitation training sample pool, continuous optimization of behavior policy function, and realization of dynamic self-adaptation of closed-loop correction control system in complex interactive environment.

Images