Robot action control method, device and electronic equipment

Abstract

The application provides a robot action control method and device and electronic equipment, the action control method comprises: encoding natural language operation instructions and image data to obtain image feature sequence, target embedding vector and text embedding vector; performing dot product operation based on embedding retrieval on the image feature sequence and the target embedding vector to obtain a similarity score matrix, and performing mask reconstruction on the image feature sequence to obtain a fine-grained perception feature sequence; performing feature vector linear modulation aggregation on the image feature sequence and the text embedding vector, and performing pruning routing on the fine-grained perception feature sequence to obtain a target visual feature sequence; based on the target visual feature sequence, outputting an action trajectory sequence, and controlling a target robot to perform a corresponding action by an action expert layer. Through the above method, the accuracy of image representation and the processing capability of complex graphics are improved, and the logical close coupling of the action trajectory sequence and the operation instructions is ensured.

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence and robot control technology, and in particular to a method, device and electronic device for controlling the motion of a robot. Background Technology

[0002] Currently, the methods for controlling embodied intelligent robots typically employ an architecture based on "OpenPi 0.5" and utilize a visual-language-action model to directly map visual and verbal commands into robot actions.

[0003] However, existing technologies (e.g., control models based on "OpenPi0.5") have the following problems in practical applications: weak multilingual interaction capabilities; for example, the "PaliGemma" visual language model used by the native multimodal base of the "OpenPi0.5" architecture has insufficient Chinese understanding capabilities and is difficult to process Chinese commands; poor fine-grained perception accuracy and semantic alignment; when handling fine tasks, visual perception is redundant, resulting in visual language features and action features not being accurately aligned during the alignment process, leading to information loss and semantic degradation, and reducing the robot's operational accuracy; wasted computational resources; a large number of task-irrelevant redundant units are generated during the visual encoding process, and these redundant units participate in the computation flow of the entire system, significantly increasing computational costs and inference latency; and a lack of complex reasoning capabilities; when facing long sequences or multi-step complex tasks, it is easy for action sequences to become disconnected from high-level command intentions. Summary of the Invention

[0004] In view of this, the purpose of this application is to provide a robot motion control method, device, and electronic device. By using a visual language model as the robot's universal visual language base, a decoder-free embedding retrieval mechanism is introduced to achieve high-precision fine-grained perception of image features, improving the accuracy of image representation and the robot's ability to process complex graphics. The fine-grained perception results are used to guide feature aggregation to linear modulation and dynamic pruning routing, thereby reducing the computational load of the visual language model and reducing perceptual redundancy, accelerating the reasoning of the visual language model and focusing on key information. Finally, combined with thought chain reasoning, highly robust motion trajectory generation is achieved, ensuring that the motion trajectory sequence is logically tightly coupled with the input operation command.

[0005] This application provides a robot motion control method, the motion control method including:

[0006] In response to the target robot receiving an externally input natural language operation command, the image data collected by the target robot is acquired;

[0007] The natural language operation instructions and the image data are input into the encoder of a preset visual language model. The encoder encodes the natural language operation instructions and the image data respectively to obtain the image feature sequence, target embedding vector and text embedding vector.

[0008] The image feature sequence and the target embedding vector are subjected to a dot product operation based on embedding retrieval to obtain a similarity score matrix. Based on the similarity score matrix and the target embedding vector, the image feature sequence is reconstructed by masking to obtain a fine-grained perceptual feature sequence.

[0009] The image feature sequence and the text embedding vector are subjected to feature-directed linear modulation aggregation to obtain an aggregation result. Based on the aggregation result and the similarity score matrix, the fine-grained perceptual feature sequence is pruned and routed to obtain the target visual feature sequence output by the encoder.

[0010] The target visual feature sequence is input into the decoder of the visual language model. The decoder performs thought chain reasoning to output a motion trajectory sequence. The motion expert layer generates motion vectors based on the motion trajectory sequence and the state information of the target robot. Based on the motion vectors, the target robot is controlled to perform corresponding actions.

[0011] Furthermore, the process of encoding the natural language operation instructions and the image data by the encoder to obtain image feature sequences, target embedding vectors, and text embedding vectors includes:

[0012] Based on a preset extended instruction vocabulary, the encoder encodes the image data to obtain an image feature sequence; wherein, the extended instruction vocabulary includes at least extended object segmentation instructions and object detection instructions; and the image feature sequence includes at least object segmentation features and object detection features.

[0013] The encoder encodes the text information corresponding to the natural language operation instruction to generate a text response vector, and extracts the text embedding vector corresponding to the natural language operation instruction from the text response vector;

[0014] Based on the object segmentation instruction and the object detection instruction, the target embedding vector is extracted from the text embedding vector.

[0015] Furthermore, the step of performing masked reconstruction on the image feature sequence based on the similarity score matrix and the target embedding vector to obtain a fine-grained perceptual feature sequence includes:

[0016] The target region mask is obtained by splicing and reshaping multiple object segmentation features in the image feature sequence;

[0017] The similarity score matrix is ​​binarized using a preset score threshold to obtain a binarized score matrix. Based on the binarized score matrix, the target region mask is upsampled and interpolated to obtain a fine-grained mask.

[0018] Based on the fine-grained mask, a fine-grained perceptual feature sequence corresponding to the image feature sequence is generated.

[0019] Furthermore, the step of binarizing the similarity score matrix using a preset score threshold to obtain a binarized score matrix includes:

[0020] Each similarity score in the similarity score matrix is ​​compared with a preset score threshold to obtain a comparison result for each similarity score.

[0021] Based on the comparison results, the similarity scores that are greater than the score threshold are converted into a first preset value, so as to use the first preset value to characterize the image region corresponding to the similarity score as a foreground task-related region;

[0022] Based on the comparison results, the similarity scores that are less than or equal to the score threshold are converted into a second preset value, so as to use the second preset value to characterize the image region corresponding to the similarity score as a background task-independent region;

[0023] A binarized score matrix is ​​constructed based on the first preset value and the second preset value.

[0024] Furthermore, the step of performing feature-directed linear modulation aggregation on the image feature sequence and the text embedding vector to obtain the aggregation result includes:

[0025] Based on the text embedding vector, scaling and translation coefficients are generated using a preset linear activation function;

[0026] Based on the scaling factor and the translation factor, an affine transformation is performed on the image feature sequence to obtain an aggregation result by aligning the semantic space of the image feature sequence.

[0027] Furthermore, based on the aggregation result and the similarity score matrix, the fine-grained perceptual feature sequence is pruned and routed to obtain the target visual feature sequence output by the encoder, including:

[0028] Based on the aggregation result and the similarity score matrix, the retention probability corresponding to each fine-grained perceptual feature in the fine-grained perceptual feature sequence is determined using a preset router;

[0029] Each retention probability is compared with a preset probability threshold, and the fine-grained perceptual features with retention probabilities less than the probability threshold are pruned to obtain a target visual feature sequence, thereby eliminating background task-irrelevant regions.

[0030] The encoder outputs the target visual feature sequence.

[0031] Furthermore, the step of inputting the target visual feature sequence into the decoder of the visual language model, whereby the decoder performs thought chain reasoning to output a motion trajectory sequence, and the motion expert layer generates motion vectors based on the motion trajectory sequence and the state information of the target robot, and controls the target robot to perform corresponding actions based on the motion vectors, includes:

[0032] The target visual feature sequence is input into the decoder of the visual language model, and the decoder performs thought chain reasoning based on the target visual feature sequence to obtain thought chain reasoning text information.

[0033] Based on the thought chain reasoning text information, a motion trajectory sequence corresponding to the target robot is generated; wherein, the motion trajectory sequence includes the Cartesian coordinates of the motion trajectory points corresponding to the end effector of the target robot;

[0034] Obtain the state information of the target robot, and input the motion trajectory sequence and the state information into the motion expert layer of the target robot;

[0035] The motion expert layer generates motion vectors based on the motion trajectory sequence and the state information, and controls the target robot to perform corresponding actions based on the motion vectors.

[0036] Furthermore, the motion control method also includes:

[0037] Based on the action trajectory sequence, the fine-grained perception feature sequence, and the actions performed by the target robot, a target reward value is determined, and based on the target reward, the visual language model is adjusted through reinforcement learning to obtain an updated visual language model; wherein, the target reward value includes a task success reward value, a thought chain reasoning consistency reward value, a fine-grained perception feature accuracy reward value, and an action trajectory sequence smoothness reward value.

[0038] This application embodiment also provides a robot motion control device, the motion control device comprising:

[0039] The operation command response module is used to respond to the natural language operation command received by the target robot from the outside and to acquire the image data collected by the target robot;

[0040] The multimodal base enhancement module is used to input the natural language operation instructions and the image data into the encoder of the preset visual language model, and the encoder encodes the natural language operation instructions and the image data respectively to obtain the image feature sequence, the target embedding vector and the text embedding vector.

[0041] The fine-grained perception enhancement module is used to perform a dot product operation based on embedding retrieval on the image feature sequence and the target embedding vector to obtain a similarity score matrix, and to perform mask reconstruction on the image feature sequence based on the similarity score matrix and the target embedding vector to obtain a fine-grained perception feature sequence.

[0042] An adaptive task pruning module is used to perform feature-directed linear modulation aggregation on the image feature sequence and the text embedding vector to obtain the aggregation result, and based on the aggregation result and the similarity score matrix, to perform pruning routing on the fine-grained perceptual feature sequence to obtain the target visual feature sequence output by the encoder.

[0043] The thought chain reasoning enhancement module is used to input the target visual feature sequence into the decoder of the visual language model. The decoder performs thought chain reasoning to output a motion trajectory sequence. The motion expert layer generates motion vectors based on the motion trajectory sequence and the state information of the target robot. Based on the motion vectors, the module controls the target robot to perform corresponding actions.

[0044] Furthermore, when the multimodal base enhancement module is used to encode the natural language operation instructions and the image data by the encoder to obtain the corresponding image feature sequence, target embedding vector, and text embedding vector, the multimodal base enhancement module is used to:

[0045] Based on a preset extended instruction vocabulary, the encoder encodes the image data to obtain an image feature sequence; wherein, the extended instruction vocabulary includes at least extended object segmentation instructions and object detection instructions; and the image feature sequence includes at least object segmentation features and object detection features.

[0046] The encoder encodes the text information corresponding to the natural language operation instruction to generate a text response vector, and extracts the text embedding vector corresponding to the natural language operation instruction from the text response vector;

[0047] Based on the object segmentation instruction and the object detection instruction, the target embedding vector is extracted from the text embedding vector.

[0048] Furthermore, when the fine-grained perception enhancement module performs mask reconstruction on the image feature sequence based on the similarity score matrix and the target embedding vector to obtain a fine-grained perception feature sequence, the fine-grained perception enhancement module is used to:

[0049] The target region mask is obtained by splicing and reshaping multiple object segmentation features in the image feature sequence;

[0050] The similarity score matrix is ​​binarized using a preset score threshold to obtain a binarized score matrix. Based on the binarized score matrix, the target region mask is upsampled and interpolated to obtain a fine-grained mask.

[0051] Based on the fine-grained mask, a fine-grained perceptual feature sequence corresponding to the image feature sequence is generated.

[0052] Furthermore, when the fine-grained perception enhancement module performs binarization processing on the similarity score matrix using a preset score threshold to obtain a binarized score matrix, the fine-grained perception enhancement module is used to:

[0053] Each similarity score in the similarity score matrix is ​​compared with a preset score threshold to obtain a comparison result for each similarity score.

[0054] Based on the comparison results, the similarity scores that are greater than the score threshold are converted into a first preset value, so as to use the first preset value to characterize the image region corresponding to the similarity score as a foreground task-related region;

[0055] Based on the comparison results, the similarity scores that are less than or equal to the score threshold are converted into a second preset value, so as to use the second preset value to characterize the image region corresponding to the similarity score as a background task-independent region;

[0056] A binarized score matrix is ​​constructed based on the first preset value and the second preset value.

[0057] Furthermore, when the adaptive task pruning module performs feature-directed linear modulation aggregation on the image feature sequence and the text embedding vector to obtain the aggregation result, the adaptive task pruning module is used to:

[0058] Based on the text embedding vector, scaling and translation coefficients are generated using a preset linear activation function;

[0059] Based on the scaling factor and the translation factor, an affine transformation is performed on the image feature sequence to obtain an aggregation result by aligning the semantic space of the image feature sequence.

[0060] Furthermore, when the adaptive task pruning module is used to prune and route the fine-grained perceptual feature sequence based on the aggregation result and the similarity score matrix to obtain the target visual feature sequence output by the encoder, the adaptive task pruning module is used to:

[0061] Based on the aggregation result and the similarity score matrix, the retention probability corresponding to each fine-grained perceptual feature in the fine-grained perceptual feature sequence is determined using a preset router;

[0062] Each retention probability is compared with a preset probability threshold, and the fine-grained perceptual features with retention probabilities less than the probability threshold are pruned to obtain a target visual feature sequence, thereby eliminating background task-irrelevant regions.

[0063] The encoder outputs the target visual feature sequence.

[0064] Furthermore, when the thought chain reasoning enhancement module inputs the target visual feature sequence into the decoder of the visual language model, whereby the decoder performs thought chain reasoning to output a motion trajectory sequence, and the motion expert layer generates motion vectors based on the motion trajectory sequence and the state information of the target robot, and controls the target robot to perform corresponding actions based on the motion vectors, the thought chain reasoning enhancement module is used for:

[0065] The target visual feature sequence is input into the decoder of the visual language model, and the decoder performs thought chain reasoning based on the target visual feature sequence to obtain thought chain reasoning text information.

[0066] Based on the thought chain reasoning text information, a motion trajectory sequence corresponding to the target robot is generated; wherein, the motion trajectory sequence includes the Cartesian coordinates of the motion trajectory points corresponding to the end effector of the target robot;

[0067] Obtain the state information of the target robot, and input the motion trajectory sequence and the state information into the motion expert layer of the target robot;

[0068] The motion expert layer generates motion vectors based on the motion trajectory sequence and the state information, and controls the target robot to perform corresponding actions based on the motion vectors.

[0069] Furthermore, the motion control device also includes a reinforcement learning fine-tuning module, which is used for:

[0070] Based on the action trajectory sequence, the fine-grained perception feature sequence, and the actions performed by the target robot, a target reward value is determined, and based on the target reward, the visual language model is adjusted through reinforcement learning to obtain an updated visual language model; wherein, the target reward value includes a task success reward value, a thought chain reasoning consistency reward value, a fine-grained perception feature accuracy reward value, and an action trajectory sequence smoothness reward value.

[0071] This application also provides an electronic device, including: a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the memory via the bus. When the machine-readable instructions are executed by the processor, the steps of the robot motion control method described above are performed.

[0072] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the robot motion control method described above.

[0073] The robot motion control method, device, and electronic device provided in this application include: in response to a target robot receiving an externally input natural language operation command, acquiring image data collected by the target robot; inputting the natural language operation command and the image data into an encoder of a preset visual language model, whereby the encoder encodes the natural language operation command and the image data respectively to obtain an image feature sequence, a target embedding vector, and a text embedding vector; performing a dot product operation based on embedding retrieval on the image feature sequence and the target embedding vector to obtain a similarity score matrix, and then performing a similarity score matrix and the target embedding vector... The image feature sequence is masked and reconstructed to obtain a fine-grained perceptual feature sequence. Feature-directed linear modulation aggregation is performed on the image feature sequence and the text embedding vector to obtain an aggregation result. Based on the aggregation result and the similarity score matrix, the fine-grained perceptual feature sequence is pruned and routed to obtain a target visual feature sequence output by the encoder. The target visual feature sequence is input into the decoder of the visual language model, where the decoder performs thought chain reasoning to output a motion trajectory sequence. A motion expert layer generates motion vectors based on the motion trajectory sequence and the target robot's state information, and controls the target robot to perform corresponding actions based on the motion vectors.

[0074] Compared to existing technologies that use an architecture based on "OpenPi 0.5" and directly map visual and linguistic commands to robot actions using a visual language action model, this method uses a visual language model as the robot's universal visual language foundation. It introduces a decoder-free embedding retrieval mechanism to achieve high-precision, fine-grained perception of image features, improving the accuracy of image representation and the robot's ability to process complex graphics. Furthermore, the fine-grained perception results guide feature aggregation to linear modulation and dynamic pruning routing, reducing the computational load of the visual language model and minimizing perceptual redundancy. This accelerates the visual language model's reasoning and focuses on key information. Finally, it combines thought chain reasoning to achieve highly robust motion trajectory generation, ensuring a tight logical coupling between the motion trajectory sequence and the input operation commands.

[0075] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0076] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0077] Figure 1 One of the flowcharts for a robot motion control method provided in this application embodiment;

[0078] Figure 2 A second flowchart illustrating a robot motion control method provided in this application embodiment;

[0079] Figure 3 This is one of the structural schematic diagrams of a robot motion control device provided in an embodiment of this application;

[0080] Figure 4 This is a second schematic diagram of the structure of a robot motion control device provided in an embodiment of this application;

[0081] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0082] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of this application. Based on the embodiments of this application, every other embodiment obtained by those skilled in the art without inventive effort falls within the scope of protection of this application.

[0083] Research has found that current methods for controlling embodied intelligent robots typically employ an architecture based on "OpenPi 0.5" and utilize a visual-language-action model to directly map visual and verbal commands into robot actions.

[0084] Existing technologies (e.g., control models based on "OpenPi0.5") suffer from the following problems in practical applications: weak multilingual interaction capabilities; for example, the "PaliGemma" visual language model used by the native multimodal base of the "OpenPi0.5" architecture has insufficient Chinese understanding capabilities and struggles to process Chinese commands; poor fine-grained perception accuracy and semantic alignment; when handling fine tasks, visual perception is redundant, leading to inaccurate alignment between visual language features and action features during the alignment process, resulting in information loss and semantic degradation, and reducing the robot's operational accuracy; wasted computational resources; a large number of task-irrelevant redundant units are generated during visual encoding, and these redundant units participate in the computational flow of the entire system, significantly increasing computational costs and inference latency; and a lack of complex reasoning capabilities; when facing long sequences or multi-step complex tasks, it is easy for action sequences to become disconnected from high-level command intentions.

[0085] Based on this, this application provides a robot motion control method. By using a visual language model as the robot's universal visual language base, a decoder-free embedding retrieval mechanism is introduced to achieve high-precision fine-grained perception of image features, improving the accuracy of image representation and the robot's ability to process complex graphics. The fine-grained perception results are used to guide feature-to-linear modulation aggregation and dynamic pruning routing to reduce the computational load of the visual language model and reduce perceptual redundancy, accelerate the reasoning of the visual language model and focus on key information. Finally, combined with thought chain reasoning, highly robust motion trajectory generation is achieved, ensuring that the motion trajectory sequence is logically tightly coupled with the input operation command.

[0086] Please see Figure 1 , Figure 1This is one of the flowcharts for a robot motion control method provided in an embodiment of this application. For example... Figure 1 As shown in the figure, the robot motion control method provided in this application embodiment includes:

[0087] S101. In response to the target robot receiving an externally input natural language operation command, acquire the image data collected by the target robot.

[0088] It should be noted that the target robot is a robot for which motion control is desired to be performed using the method described in the embodiments of this application.

[0089] Here, the robot may include a robot capable of performing actions, including but not limited to wheeled robots, humanoid robots, and robotic arm robots, and the robot may also include an embodied intelligent robot.

[0090] In this application embodiment, Natural Language Robot Commands refer to commands issued by humans to robots in everyday language (such as Chinese and English) in the expectation that the robots will perform specific physical actions. That is, in a specific environment, for a specific object, the planning and control signals for performing a series of low-level physical actions. Natural language commands are not direct control signals, but task-level semantic descriptions, which need to be converted into machine-executable action sequences through multimodal perception, reasoning and control links.

[0091] The natural language operation instructions include, but are not limited to, action verbs, operation objects, source location / starting state, target location / target state, and constraints.

[0092] In this embodiment of the application, the image data is an image captured by a multi-view camera device set up by the target robot for the scene within its field of view.

[0093] In this step, when the target robot receives a natural language operation command input from the user, the multi-view camera device set up on the target robot is controlled to collect image data and acquire the image data.

[0094] S102. Input the natural language operation instructions and the image data into the encoder of the preset visual language model. The encoder encodes the natural language operation instructions and the image data respectively to obtain the image feature sequence, target embedding vector and text embedding vector.

[0095] In this embodiment of the application, the visual language model may include a “Qwen3-VL” series model, such as the “Qwen3-VL-2B” model, and the visual language model includes, but is not limited to, an encoder and a decoder.

[0096] In this way, the lightweight model from the "Qwen3-VL" series is used to replace the original PaliGemma visual language model in the existing "OpenPi0.5" model as the visual encoder and basic language model. This solves the problem of weak Chinese interactive capabilities of the PaliGemma visual language model, improves the model's visual language feature processing capabilities, and provides identifiers for subsequent embedding retrieval.

[0097] In one possible implementation of this application, step S102 may include:

[0098] S1021. Based on a preset extended instruction vocabulary, the encoder encodes the image data to obtain an image feature sequence.

[0099] The extended instruction vocabulary includes at least extended object segmentation instructions and object detection instructions; the image feature sequence includes at least object segmentation features and object detection features.

[0100] Here, the instruction vocabulary of the visual language model is expanded, that is, "..." is added to the instruction vocabulary. <mask>Special token instructions and <box>Special token instructions represent object segmentation instructions and object detection instructions, respectively, resulting in an extended instruction vocabulary.

[0101] The visual language model outputs one or more values ​​when segmenting target objects. <mask>The label, or object segmentation feature, corresponds to an embedding vector. When detecting a target object, the starting label of the bounding box coordinates is generated.

[0102] S1022. The encoder encodes the text information corresponding to the natural language operation instruction to generate a text response vector, and extracts the text embedding vector corresponding to the natural language operation instruction from the text response vector.

[0103] In this step, the encoder restores the text information corresponding to the natural language operation instructions into a natural language response that humans can understand, that is, generates a text response vector.

[0104] For example, enter "divide the red cup on the table" <mask>Output the text response vector y. t The instruction "will perform a splitting operation on the red cup on the table" is used to verify that the model understands the meaning of the instruction and the special token instruction.

[0105] Furthermore, the text embedding vectors corresponding to the natural language operation instructions are extracted from the text response vector, which is the final vector sequence.

[0106] S1023. Based on the object segmentation instruction and the object detection instruction, extract the target embedding vector from the text embedding vector.

[0107] In this step, firstly, the positions corresponding to the object segmentation features and object detection features in the text embedding vector are found; then, the vector at this position is directly extracted, which is the target embedding vector; if there are multiple object segmentation instructions, multiple vectors are extracted to form a set of target embedding vectors.

[0108] In this way, by accurately extracting vector features (segmentation / detection) that are directly related to the visual task, a one-to-one correspondence is established between "special tokens in language instructions" and "visual tasks", which prepares for subsequent fusion of visual features.

[0109] S103. Perform a dot product operation based on embedding retrieval on the image feature sequence and the target embedding vector to obtain a similarity score matrix. Based on the similarity score matrix and the target embedding vector, perform mask reconstruction on the image feature sequence to obtain a fine-grained perceptual feature sequence.

[0110] In this embodiment of the application, the dot product operation based on embedding retrieval is performed using the following formula.

[0111] .

[0112] in, This represents the similarity scores in the similarity score matrix; This represents each target embedding feature in the target embedding vector; Represents a sequence of image features; Indicates the dimension of the embedding vector.

[0113] Here, the dot product operation is performed in the feature space of the encoder, eliminating the need for an additional convolutional decoding layer and greatly reducing the number of parameters.

[0114] Thus, the embodiments of this application aim to achieve pixel-level semantic perception without introducing an additional decoder module, align visual and linguistic features, and adopt an embedding retrieval mechanism based on the introduced target embedding vector. Without adding an additional decoder, high-resolution fine-grained masks are directly generated by performing dot product operations on text embeddings of object segmentation features and object detection features with image features, thereby achieving accurate perception "without a decoder", solving the problems of perception accuracy and inaccurate representation, and improving the ability to process complex graphics.

[0115] In one possible implementation of this application, in specific implementation, the step S103 of performing mask reconstruction on the image feature sequence based on the similarity score matrix and the target embedding vector to obtain a fine-grained perceptual feature sequence may include:

[0116] S1031. The multiple object segmentation features in the image feature sequence are spliced ​​and reconstructed to obtain the target region mask.

[0117] In this step, through a multi-feature representation mechanism, when generating object segmentation features, in order to solve the problem of blurry low-resolution masks, the encoder is configured to output multiple object segmentation features (e.g., 16). Each object segmentation feature corresponds to the local details of the segmented region. Then, the matching results of all object segmentation features are spliced ​​and reshaped to obtain a high-definition target region mask.

[0118] S1032. The similarity score matrix is ​​binarized using a preset score threshold to obtain a binarized score matrix. Based on the binarized score matrix, the target region mask is upsampled and interpolated to obtain a fine-grained mask.

[0119] Here, the initially generated target region mask is often low-resolution (for example, the model generates a mask on a low-resolution image first for computational efficiency), and the edges have problems such as jagged edges and holes; the binarized fractional matrix contains pixel-level target confidence information, and by upsampling to enlarge the size and interpolating to fill in the details, and by using the binarized fractional matrix as a constraint, a fine-grained mask with the same resolution as the original image and fine edges is finally obtained.

[0120] Upsampling is an operation that enlarges the size of a low-resolution mask, with the goal of restoring the mask size to a resolution consistent with the original image or binary fractional matrix. Interpolation is an algorithm that fills in the newly added blank pixels after upsampling, with the aim of making the mask edges smooth and the details continuous, avoiding the "mosaic" effect.

[0121] In one possible implementation of this application, in specific implementation, the step S1032, which involves binarizing the similarity score matrix using a preset score threshold to obtain a binarized score matrix, may include:

[0122] S10321. Compare each similarity score in the similarity score matrix with a preset score threshold to obtain a comparison result corresponding to each similarity score.

[0123] For example, the score threshold can be set to 0.5.

[0124] S10322. Based on the comparison result, the similarity score that is greater than the score threshold is converted into a first preset value, so as to use the first preset value to characterize the image region corresponding to the similarity score as the foreground task-related region.

[0125] Here, the first preset value is generally set to 1; the foreground task-related region is the set of pixels in the image that are directly related to the current task target, and is the core region that the model needs to focus on identifying, segmenting and tracking.

[0126] S10323. Based on the comparison result, the similarity score that is less than or equal to the score threshold is converted into a second preset value, so as to use the second preset value to characterize the image region corresponding to the similarity score as a background task-irrelevant region.

[0127] Here, the second preset value is generally set to 0; the background task-irrelevant region is a set of pixels in the image that are not related to the current task target, and belongs to the region that needs to be excluded or suppressed, such as non-target objects in the scene, environmental background, etc. (such as tables, floors, walls).

[0128] S10324. Construct a binarized fraction matrix based on the first preset value and the second preset value.

[0129] Here, the binarized score matrix is ​​a pixel-level matrix with the same size as the original image data. Each element in the matrix takes a value of 0 or 1, representing the confidence score of the corresponding pixel belonging to the target region.

[0130] In this process, the binary fractional matrix is ​​different from unconstrained blind upsampling. The fractional matrix is ​​a "ruler" that can constrain the interpolation direction of the mask in the target edge region, avoid background pixels being misjudged as targets, and improve the accuracy of the mask.

[0131] S1033. Based on the fine-grained mask, generate a fine-grained perceptual feature sequence corresponding to the image feature sequence.

[0132] In this embodiment of the application, a perceptual enhancement feature with explicit semantic weights (i.e., a fine-grained perceptual feature sequence) is generated. At this time, the features of the target object in the fine-grained perceptual feature sequence are significantly activated, while background noise and non-task-related objects are suppressed.

[0133] Here, the fine-grained perceptual feature sequence refers to an ordered set of features extracted from an image using a fine-grained mask as a spatial constraint. This set includes pixel-level detail features of the foreground task-related region and language task instruction-related features. The fine-grained perceptual feature sequence may include a binary reconstruction mask feature sequence or a strong semantic feature sequence.

[0134] S104. Perform feature-directed linear modulation aggregation on the image feature sequence and the text embedding vector to obtain the aggregation result. Based on the aggregation result and the similarity score matrix, perform pruning routing on the fine-grained perceptual feature sequence to obtain the target visual feature sequence output by the encoder.

[0135] Here, the feature-wise linear modulation aggregation may include a FiLM aggregation (Feature-wise LinearModulation Aggregation), which is a technique used in deep learning for conditional feature modulation and multimodal information fusion. FiLM aggregation does not directly splice or add information from different modalities, but dynamically adjusts the representation of another modality using a "conditional signal" (such as a language instruction).

[0136] Among them, Pruned Routing is a key technique that has emerged in hybrid expert models and efficient neural network architectures. It aims to dynamically select the most relevant subnetworks while suppressing or "pruning" irrelevant computational paths, thereby significantly reducing computational overhead while maintaining model performance.

[0137] Specifically, pruning routing uses a router at each layer of the model to distribute input features to multiple expert layers. However, even if only the "Top-K" expert layers are activated, the other expert layers still occupy memory, and when K is fixed, redundant computation may be introduced. The improvement of pruning routing is that it not only selects the "Top-K" expert layers, but also sparsifies or thresholds the gate weights to further reduce the number of activated experts and achieve dynamic pruning routing.

[0138] In this way, by introducing a FiLM-based aggregation and pruning routing mechanism, image features are dynamically selected according to instructions to reduce perceptual redundancy and optimize computation, thereby solving the problem of high computational cost caused by intermediate redundant features, accelerating model inference and focusing on key information.

[0139] In one possible implementation of this application, in specific implementation, the step S104 of performing feature-directed linear modulation aggregation on the image feature sequence and the text embedding vector to obtain the aggregation result may include:

[0140] S1041. Based on the text embedding vector, a scaling factor and a translation factor are generated using a preset linear activation function.

[0141] In this embodiment of the application, the scaling factor and translation factor are determined by the following expression.

[0142] .

[0143] in, Indicates the scaling factor; Indicates the translation coefficient; Represents a text embedding vector; This represents the preset linear activation function.

[0144] S1042. Based on the scaling factor and the translation factor, perform an affine transformation on the image feature sequence to obtain an aggregation result by aligning the semantic space of the image feature sequence.

[0145] In this embodiment of the application, the image feature sequence is subjected to affine transformation using the following expression.

[0146] .

[0147] in, Indicates the aggregation result; Represents a sequence of image features; Indicates the scaling factor; This represents the translation coefficient.

[0148] In one possible implementation of this application, in specific implementation, step S104, based on the aggregation result and the similarity score matrix, performs pruning routing on the fine-grained perceptual feature sequence to obtain the target visual feature sequence output by the encoder, may include:

[0149] S1041. Based on the aggregation result and the similarity score matrix, the retention probability corresponding to each fine-grained perceptual feature in the fine-grained perceptual feature sequence is determined using a preset router.

[0150] In this embodiment, traditional pruning is often based on attention weights, which can easily lead to the accidental deletion of small targets. In contrast, this embodiment directly uses a preset router to calculate the retention probability of each fine-grained perceptual feature based on the fine-grained perceptual feature sequence, aggregation results, and similarity score matrix.

[0151] S1042. Each retention probability is compared with a preset probability threshold, and the fine-grained perceptual features with retention probabilities less than the probability threshold are pruned to obtain a target visual feature sequence, thereby eliminating background task-irrelevant regions.

[0152] In this step, since the foreground task-related region and background task-independent region in the image feature sequence are clearly identified through the dot product operation, the pruning process here will be extremely precise, that is, directly removing those background task-independent regions that score "0" in the similarity score matrix.

[0153] S1043, The encoder outputs the target visual feature sequence.

[0154] Here, the target visual feature sequence is a sparse visual feature sequence with a significantly reduced length. The visual features in the target visual feature sequence include the robotic arm, the object being manipulated, and related obstacles. The target visual feature sequence is the final output of the encoder of the visual language model.

[0155] S105. The target visual feature sequence is input into the decoder of the visual language model. The decoder performs thought chain reasoning to output the action trajectory sequence. The action expert layer generates an action vector based on the action trajectory sequence and the state information of the target robot. Based on the action vector, the target robot is controlled to perform the corresponding action.

[0156] In one possible implementation of this application, step S105 may include:

[0157] S1051. The target visual feature sequence is input into the decoder of the visual language model, and the decoder performs thought chain reasoning based on the target visual feature sequence to obtain thought chain reasoning text information.

[0158] In this step, a large language model is configured in the decoder of the visual language model. The large language model performs thought chain reasoning based on the target visual feature sequence through autoregression to generate thought chain reasoning text information, such as "First I need to move to the top of the cup, and then align with the handle..." to explicitly show the planning process.

[0159] S1052. Based on the thought chain reasoning text information, generate the action trajectory sequence corresponding to the target robot.

[0160] The motion trajectory sequence includes the Cartesian coordinates of the motion trajectory points corresponding to the end effector of the target robot.

[0161] S1053. Obtain the state information of the target robot, and input the motion trajectory sequence and the state information into the motion expert layer of the target robot.

[0162] Here, Action Expert is a key concept in Embodied Intelligence, Robotics Learning, and Multi-Agent Systems. It typically refers to a strategy module that is specifically responsible for performing a certain type of action or skill, used to decompose and specialize the control of complex tasks.

[0163] The state information includes, but is not limited to, the current state parameters of each motion-related component in the target robot.

[0164] S1054. The motion expert layer generates motion vectors based on the motion trajectory sequence and the state information, and controls the target robot to perform corresponding actions based on the motion vectors.

[0165] Here, the motion expert layer, as a functional module specifically responsible for motion planning, integrates historical motion data and real-time status information to output executable motion commands, driving the robot to complete the target task.

[0166] In this step, the network model set by the motion expert layer (such as a temporal convolutional network TCN or a reinforcement learning Actor network) reads the motion trajectory sequence and the state information of the target robot. Based on the patterns learned from historical motion trajectories, and combined with the motion trajectory sequence and the state information of the target robot, it outputs motion vectors that conform to the robot's execution capabilities.

[0167] The motion vector includes, but is not limited to, joint angle, movement speed, and execution priority.

[0168] Furthermore, the motion expert layer decomposes the high-dimensional motion vector into independent control commands for each actuator (e.g., robotic arm joint commands and gripper opening and closing commands), and converts these independent control commands into electrical signals that the robot hardware can recognize (e.g., PWM signals and bus commands), thereby controlling the target robot to perform corresponding actions.

[0169] Optional, please refer to Figure 2 , Figure 2 This is a second flowchart illustrating a robot motion control method provided in an embodiment of this application. Figure 2 As shown in the figure, the robot motion control method provided in this application embodiment includes step S106 in addition to steps S101 to S105. Specifically, step S106 is used to explain the method of fine-tuning the visual language model through reinforcement learning.

[0170] S106. Based on the action trajectory sequence, the fine-grained perception feature sequence, and the action performed by the target robot, determine the target reward value, and based on the target reward, perform reinforcement learning alignment adjustment on the visual language model to obtain an updated visual language model.

[0171] The target reward values ​​include task success reward value, thought chain reasoning consistency reward value, fine-grained perception feature accuracy reward value, and action trajectory sequence smoothness reward value.

[0172] In this step, the task success reward value is determined based on whether the final action has been completed; and the intent of the thought chain reasoning text information is consistent with the intent of the operation instruction. Figure 1 The intention behind reasoning about textual information and whether it aligns with the intention behind the generated action trajectory. Figure 1 To determine the consistency reward value for reasoning in the thought chain; to determine the smoothness reward value for the action trajectory sequence based on the smoothness of the action trajectory sequence; and to determine the accuracy reward value for the fine-grained perceptual feature sequence based on the overlap between the fine-grained perceptual feature sequence and the preset real reference sequence.

[0173] Furthermore, based on the task success reward value, the reasoning consistency reward value, the fine-grained perception feature accuracy reward value, and the action trajectory sequence smoothness reward value, combined with preset weight coefficients, a preset comprehensive reward function is used to determine the target reward value.

[0174] Furthermore, based on the target reward value, the visual language model is trained and optimized using the Gated Relative Policy Optimization (GRPO) method to achieve alignment adjustment, thereby obtaining an updated visual language model.

[0175] Here, the gating relative policy optimization method is based on the target reward value and uses a relative advantage estimation and gating update mechanism to perform reinforcement learning alignment adjustment on the visual language model.

[0176] In this way, the action generation mechanism that aligns with the thought chain reasoning and reinforcement learning is introduced to enhance the reasoning ability of long-term complex tasks, prevent semantic degradation during forward propagation from causing changes in the final operation accuracy, solve the problem of weak reasoning ability and disconnection of action instructions in complex tasks caused by the lack of thought chain in the existing technology, and ensure the tight coupling between action sequence and instruction logic.

[0177] The robot motion control method provided in this application uses a visual language model as the robot's universal visual language base and introduces a decoder-free embedding retrieval mechanism to achieve high-precision fine-grained perception of image features. This improves the accuracy of image representation and the robot's ability to process complex graphics. Furthermore, the fine-grained perception results guide feature-to-linear modulation aggregation and dynamic pruning routing to reduce the computational load of the visual language model and reduce perceptual redundancy, accelerate the reasoning of the visual language model and focus on key information. Finally, it combines thought chain reasoning to achieve highly robust motion trajectory generation, ensuring that the motion trajectory sequence is logically tightly coupled with the input operation instructions.

[0178] Please see Figure 3 , Figure 4 , Figure 3 This is one of the structural schematic diagrams of a robot motion control device provided in the embodiments of this application. Figure 4 This is a second schematic diagram of a robot motion control device provided in an embodiment of this application. Figure 3 As shown, the motion control device 300 includes:

[0179] The operation command response module 310 is used to respond to the natural language operation command received by the target robot from the outside and to acquire the image data collected by the target robot.

[0180] The multimodal base enhancement module 320 is used to input the natural language operation instructions and the image data into the encoder of the preset visual language model, and the encoder encodes the natural language operation instructions and the image data respectively to obtain the image feature sequence, the target embedding vector and the text embedding vector.

[0181] The fine-grained perception enhancement module 330 is used to perform a dot product operation based on embedding retrieval on the image feature sequence and the target embedding vector to obtain a similarity score matrix, and to perform mask reconstruction on the image feature sequence based on the similarity score matrix and the target embedding vector to obtain a fine-grained perception feature sequence.

[0182] The adaptive task pruning module 340 is used to perform feature-directed linear modulation aggregation on the image feature sequence and the text embedding vector to obtain the aggregation result, and based on the aggregation result and the similarity score matrix, to perform pruning routing on the fine-grained perceptual feature sequence to obtain the target visual feature sequence output by the encoder.

[0183] The thought chain reasoning enhancement module 350 is used to input the target visual feature sequence into the decoder of the visual language model, and the decoder performs thought chain reasoning to output the action trajectory sequence. The action expert layer generates action vectors based on the action trajectory sequence and the state information of the target robot, and controls the target robot to perform corresponding actions based on the action vectors.

[0184] Furthermore, when the multimodal base enhancement module 320 is used to encode the natural language operation instructions and the image data by the encoder respectively to obtain the image feature sequence, target embedding vector, and text embedding vector, the multimodal base enhancement module 320 is used to:

[0185] Based on a preset extended instruction vocabulary, the encoder encodes the image data to obtain an image feature sequence; wherein, the extended instruction vocabulary includes at least extended object segmentation instructions and object detection instructions; and the image feature sequence includes at least object segmentation features and object detection features.

[0186] The encoder encodes the text information corresponding to the natural language operation instruction to generate a text response vector, and extracts the text embedding vector corresponding to the natural language operation instruction from the text response vector;

[0187] Based on the object segmentation instruction and the object detection instruction, the target embedding vector is extracted from the text embedding vector.

[0188] Furthermore, when the fine-grained perception enhancement module 330 performs mask reconstruction on the image feature sequence based on the similarity score matrix and the target embedding vector to obtain a fine-grained perception feature sequence, the fine-grained perception enhancement module 330 is used to:

[0189] The target region mask is obtained by splicing and reshaping multiple object segmentation features in the image feature sequence;

[0190] The similarity score matrix is ​​binarized using a preset score threshold to obtain a binarized score matrix. Based on the binarized score matrix, the target region mask is upsampled and interpolated to obtain a fine-grained mask.

[0191] Based on the fine-grained mask, a fine-grained perceptual feature sequence corresponding to the image feature sequence is generated.

[0192] Furthermore, when the fine-grained perception enhancement module 330 performs binarization processing on the similarity score matrix using a preset score threshold to obtain a binarized score matrix, the fine-grained perception enhancement module 330 is used to:

[0193] Each similarity score in the similarity score matrix is ​​compared with a preset score threshold to obtain a comparison result for each similarity score.

[0194] Based on the comparison results, the similarity scores that are greater than the score threshold are converted into a first preset value, so as to use the first preset value to characterize the image region corresponding to the similarity score as a foreground task-related region;

[0195] Based on the comparison results, the similarity scores that are less than or equal to the score threshold are converted into a second preset value, so as to use the second preset value to characterize the image region corresponding to the similarity score as a background task-independent region;

[0196] A binarized score matrix is ​​constructed based on the first preset value and the second preset value.

[0197] Furthermore, when the adaptive task pruning module 340 performs feature-directed linear modulation aggregation on the image feature sequence and the text embedding vector to obtain the aggregation result, the adaptive task pruning module 340 is used to:

[0198] Based on the text embedding vector, scaling and translation coefficients are generated using a preset linear activation function;

[0199] Based on the scaling factor and the translation factor, an affine transformation is performed on the image feature sequence to obtain an aggregation result by aligning the semantic space of the image feature sequence.

[0200] Furthermore, when the adaptive task pruning module 340 performs pruning routing on the fine-grained perceptual feature sequence based on the aggregation result and the similarity score matrix to obtain the target visual feature sequence output by the encoder, the adaptive task pruning module 340 is used to:

[0201] Based on the aggregation result and the similarity score matrix, the retention probability corresponding to each fine-grained perceptual feature in the fine-grained perceptual feature sequence is determined using a preset router;

[0202] Each retention probability is compared with a preset probability threshold, and the fine-grained perceptual features with retention probabilities less than the probability threshold are pruned to obtain a target visual feature sequence, thereby eliminating background task-irrelevant regions.

[0203] The encoder outputs the target visual feature sequence.

[0204] Furthermore, when the thought chain reasoning enhancement module 350 inputs the target visual feature sequence into the decoder of the visual language model, whereby the decoder performs thought chain reasoning to output a motion trajectory sequence, and the motion expert layer generates motion vectors based on the motion trajectory sequence and the state information of the target robot, and controls the target robot to perform corresponding actions based on the motion vectors, the thought chain reasoning enhancement module 350 is used for:

[0205] The target visual feature sequence is input into the decoder of the visual language model, and the decoder performs thought chain reasoning based on the target visual feature sequence to obtain thought chain reasoning text information.

[0206] Based on the thought chain reasoning text information, a motion trajectory sequence corresponding to the target robot is generated; wherein, the motion trajectory sequence includes the Cartesian coordinates of the motion trajectory points corresponding to the end effector of the target robot;

[0207] Obtain the state information of the target robot, and input the motion trajectory sequence and the state information into the motion expert layer of the target robot;

[0208] The motion expert layer generates motion vectors based on the motion trajectory sequence and the state information, and controls the target robot to perform corresponding actions based on the motion vectors.

[0209] Furthermore, such as Figure 4 As shown, the motion control device 300 further includes a reinforcement learning fine-tuning module 360, which is used for:

[0210] Based on the action trajectory sequence, the fine-grained perception feature sequence, and the actions performed by the target robot, a target reward value is determined, and based on the target reward, the visual language model is adjusted through reinforcement learning to obtain an updated visual language model; wherein, the target reward value includes a task success reward value, a thought chain reasoning consistency reward value, a fine-grained perception feature accuracy reward value, and an action trajectory sequence smoothness reward value.

[0211] The robot motion control device provided in this application uses a visual language model as the robot's universal visual language base and introduces a decoder-free embedding retrieval mechanism to achieve high-precision fine-grained perception of image features. This improves the accuracy of image representation and the robot's ability to process complex graphics. Furthermore, the fine-grained perception results guide feature-to-linear modulation aggregation and dynamic pruning routing to reduce the computational load of the visual language model and reduce perceptual redundancy, accelerate the reasoning of the visual language model and focus on key information. Finally, it combines thought chain reasoning to achieve highly robust motion trajectory generation, ensuring that the motion trajectory sequence is logically tightly coupled with the input operation instructions.

[0212] Please see Figure 5 , Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 5 As shown, the electronic device 500 includes a processor 510, a memory 520, and a bus 530.

[0213] The memory 520 stores machine-readable instructions executable by the processor 510. When the electronic device 500 is running, the processor 510 and the memory 520 communicate via the bus 530. When the machine-readable instructions are executed by the processor 510, they can perform the operations described above. Figure 1 as well as Figure 2 The steps of the robot motion control method in the illustrated method embodiment can be found in the method embodiment for specific implementation, and will not be repeated here.

[0214] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, can perform the above-described actions. Figure 1 as well as Figure 2 The steps of the robot motion control method in the illustrated method embodiment can be found in the method embodiment for specific implementation, and will not be repeated here.

[0215] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0216] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the shown or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0217] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0218] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0219] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0220] Finally, it should be noted that the above-described embodiments are merely specific implementations of this application, used to illustrate the technical solutions of this application, and not to limit them. The scope of protection of this application is not limited thereto. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this application. Such modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.< / mask> < / mask> < / box> < / mask>

Claims

1. A method for controlling the motion of a robot, characterized in that, The motion control method includes: In response to the target robot receiving an externally input natural language operation command, the image data collected by the target robot is acquired; The natural language operation instructions and the image data are input into the encoder of a preset visual language model. The encoder encodes the natural language operation instructions and the image data respectively to obtain the image feature sequence, target embedding vector and text embedding vector. The image feature sequence and the target embedding vector are subjected to a dot product operation based on embedding retrieval to obtain a similarity score matrix. Based on the similarity score matrix and the target embedding vector, the image feature sequence is reconstructed by masking to obtain a fine-grained perceptual feature sequence. The image feature sequence and the text embedding vector are subjected to feature-directed linear modulation aggregation to obtain an aggregation result. Based on the aggregation result and the similarity score matrix, the fine-grained perceptual feature sequence is pruned and routed to obtain the target visual feature sequence output by the encoder. The target visual feature sequence is input into the decoder of the visual language model. The decoder performs thought chain reasoning to output a motion trajectory sequence. The motion expert layer generates motion vectors based on the motion trajectory sequence and the state information of the target robot. Based on the motion vectors, the target robot is controlled to perform corresponding actions.

2. The method according to claim 1, characterized in that, The encoder encodes the natural language operation instructions and the image data respectively to obtain image feature sequences, target embedding vectors, and text embedding vectors, including: Based on a preset extended instruction vocabulary, the encoder encodes the image data to obtain an image feature sequence; wherein, the extended instruction vocabulary includes at least extended object segmentation instructions and object detection instructions; and the image feature sequence includes at least object segmentation features and object detection features. The encoder encodes the text information corresponding to the natural language operation instruction to generate a text response vector, and extracts the text embedding vector corresponding to the natural language operation instruction from the text response vector; Based on the object segmentation instruction and the object detection instruction, the target embedding vector is extracted from the text embedding vector.

3. The method according to claim 1, characterized in that, The step of masking and reconstructing the image feature sequence based on the similarity score matrix and the target embedding vector to obtain a fine-grained perceptual feature sequence includes: The target region mask is obtained by splicing and reshaping multiple object segmentation features in the image feature sequence; The similarity score matrix is ​​binarized using a preset score threshold to obtain a binarized score matrix. Based on the binarized score matrix, the target region mask is upsampled and interpolated to obtain a fine-grained mask. Based on the fine-grained mask, a fine-grained perceptual feature sequence corresponding to the image feature sequence is generated.

4. The method according to claim 3, characterized in that, The step of binarizing the similarity score matrix using a preset score threshold to obtain a binarized score matrix includes: Each similarity score in the similarity score matrix is ​​compared with a preset score threshold to obtain a comparison result for each similarity score. Based on the comparison results, the similarity scores that are greater than the score threshold are converted into a first preset value, so as to use the first preset value to characterize the image region corresponding to the similarity score as a foreground task-related region; Based on the comparison results, the similarity scores that are less than or equal to the score threshold are converted into a second preset value, so as to use the second preset value to characterize the image region corresponding to the similarity score as a background task-independent region; A binarized score matrix is ​​constructed based on the first preset value and the second preset value.

5. The method according to claim 1, characterized in that, The step of performing feature-directed linear modulation aggregation on the image feature sequence and the text embedding vector to obtain the aggregation result includes: Based on the text embedding vector, scaling and translation coefficients are generated using a preset linear activation function; Based on the scaling factor and the translation factor, an affine transformation is performed on the image feature sequence to obtain an aggregation result by aligning the semantic space of the image feature sequence.

6. The method according to claim 4, characterized in that, The step of pruning and routing the fine-grained perceptual feature sequence based on the aggregation result and the similarity score matrix to obtain the target visual feature sequence output by the encoder includes: Based on the aggregation result and the similarity score matrix, the retention probability corresponding to each fine-grained perceptual feature in the fine-grained perceptual feature sequence is determined using a preset router; Each retention probability is compared with a preset probability threshold, and the fine-grained perceptual features with retention probabilities less than the probability threshold are pruned to obtain a target visual feature sequence, thereby eliminating background task-irrelevant regions. The encoder outputs the target visual feature sequence.

7. The method according to claim 1, characterized in that, The process involves inputting the target visual feature sequence into the decoder of the visual language model, where the decoder performs thought chain reasoning to output a motion trajectory sequence. A motion expert layer then generates motion vectors based on the motion trajectory sequence and the target robot's state information. Based on these motion vectors, the robot is controlled to perform corresponding actions. This includes: The target visual feature sequence is input into the decoder of the visual language model, and the decoder performs thought chain reasoning based on the target visual feature sequence to obtain thought chain reasoning text information. Based on the thought chain reasoning text information, a motion trajectory sequence corresponding to the target robot is generated; wherein, the motion trajectory sequence includes the Cartesian coordinates of the motion trajectory points corresponding to the end effector of the target robot; Obtain the state information of the target robot, and input the motion trajectory sequence and the state information into the motion expert layer of the target robot; The motion expert layer generates motion vectors based on the motion trajectory sequence and the state information, and controls the target robot to perform corresponding actions based on the motion vectors.

8. The method according to claim 1, characterized in that, The motion control method further includes: Based on the action trajectory sequence, the fine-grained perception feature sequence, and the actions performed by the target robot, a target reward value is determined, and based on the target reward, the visual language model is adjusted through reinforcement learning to obtain an updated visual language model; wherein, the target reward value includes a task success reward value, a thought chain reasoning consistency reward value, a fine-grained perception feature accuracy reward value, and an action trajectory sequence smoothness reward value.

9. A motion control device for a robot, characterized in that, The motion control device includes: The operation command response module is used to respond to the natural language operation command received by the target robot from the outside and to acquire the image data collected by the target robot; The multimodal base enhancement module is used to input the natural language operation instructions and the image data into the encoder of the preset visual language model, and the encoder encodes the natural language operation instructions and the image data respectively to obtain the image feature sequence, the target embedding vector and the text embedding vector. The fine-grained perception enhancement module is used to perform a dot product operation based on embedding retrieval on the image feature sequence and the target embedding vector to obtain a similarity score matrix, and to perform mask reconstruction on the image feature sequence based on the similarity score matrix and the target embedding vector to obtain a fine-grained perception feature sequence. An adaptive task pruning module is used to perform feature-directed linear modulation aggregation on the image feature sequence and the text embedding vector to obtain the aggregation result, and based on the aggregation result and the similarity score matrix, to perform pruning routing on the fine-grained perceptual feature sequence to obtain the target visual feature sequence output by the encoder. The thought chain reasoning enhancement module is used to input the target visual feature sequence into the decoder of the visual language model. The decoder performs thought chain reasoning to output a motion trajectory sequence. The motion expert layer generates motion vectors based on the motion trajectory sequence and the state information of the target robot. Based on the motion vectors, the module controls the target robot to perform corresponding actions.

10. An electronic device, characterized in that, include: The device includes a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the memory via the bus. The machine-readable instructions are executed by the processor to perform the steps of the robot motion control method as described in any one of claims 1 to 8.

Images