Adaptive control method and system for autonomous mobile platform

Abstract

The application discloses a kind of fusion multimodal feedback and visual thinking chain reasoning's autonomous mobile platform adaptive control method and system, by expert abstract unit, the environment data that multiple sensors and sensor are abstracted into structured expression in real time with user voice instruction information together input large language model with real-time collection.Multiple candidate paths and their corresponding sub-target image sequences are generated by the large language model using a hybrid attention mechanism based on visual thinking chain reasoning, and visualized through a human-machine interface. Users can modify instructions through voice feedback and modify targets through visual feedback. Finally, the multimodal feedback of the user is aligned, the dense reward function is updated, and the path is replanned. The system can also record user feedback results and train personalized user preference models. The application has the advantages of strong generalization ability and high feedback efficiency.

Description

Technical Field

[0001] This invention relates to the field of intelligent device control technology, and more specifically, to an adaptive control method and system for an autonomous mobile platform. Background Technology

[0002] With the aging population and increasing demand for rehabilitation services, autonomous mobility platforms such as intelligent wheelchairs, as an important direction for assistive mobile devices, are developing towards higher autonomy and stronger adaptability. Currently, most mainstream autonomous mobility platform control systems on the market employ local obstacle avoidance algorithms based on LiDAR or vision, and achieve path planning in simple scenarios through preset rules or single sensor data. These algorithms improve the user experience to some extent. However, existing path planning and obstacle avoidance technologies suffer from low efficiency in human-computer interaction feedback, a single feedback modality, a lack of explicit action sequences and path planning, and insufficient personalized adaptation capabilities, making it difficult to cope with complex usage environments and diverse user groups.

[0003] Traditional methods rely on local sensor data to generate obstacle avoidance strategies, lacking an abstract representation of the semantic information of the global environment and explicit intermediate reasoning steps. This makes them ill-suited for complex and dynamic scenarios, such as densely populated areas in residential communities, unstructured roads, and corridors in nursing homes, potentially leading to misunderstandings of user goals and inappropriate path planning. Furthermore, existing systems often employ single-modal interaction, failing to effectively integrate multimodal feedback such as user visual annotations and verbal commands. For elderly individuals with limited expressive abilities or upper limb mobility, feedback efficiency is low, and the lack of dynamic weight allocation mechanisms results in misunderstandings of the agent's control intentions, reducing human-machine collaboration efficiency.

[0004] Given the above market demands and the shortcomings of existing technologies, there is an urgent need for an adaptive control method and system for autonomous mobile platforms that integrates multimodal interactive feedback and explicit intermediate reasoning capabilities. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention proposes an adaptive control method and system for autonomous mobile platforms based on the integration of multimodal feedback and visual reasoning. This method systematically solves the problems of traditional autonomous mobile platforms in handling complex tasks, providing limited user feedback modalities, low feedback efficiency, and difficulty in understanding feedback through deep collaboration between multimodal feedback and explicit reasoning decision-making, thereby improving the accuracy of autonomous mobile platform control.

[0006] To achieve the above objectives, in a first aspect, the present invention provides an adaptive control method for an autonomous mobile platform, comprising the following steps:

[0007] Step S1: Collect surrounding environmental data in real time through the environmental state sensing device and generate a structured representation using expert abstraction units.

[0008] Step S2: After the structured representation of the environment and the user's voice command are jointly encoded, they are used as input to a large language model, which generates a segmented dynamic reinforcement learning dense reward function.

[0009] Step S3: Using a visual thinking chain reasoning model, candidate paths and corresponding sub-target image sequences are generated through a causal attention mechanism, so that image generation follows a temporal order.

[0010] Step S4: Using a full attention mechanism, based on the current observation and the generated sub-target images, predict and generate action sequences in blocks, allowing all action tokens to interact with each other and capturing the complex dependencies between action dimensions; display the sub-target image sequence and action sequence on the human-computer interaction interface.

[0011] Step S5: The human-computer interaction interface receives multimodal correction feedback from the user, and integrates the decision model through multimodal interaction feedback to update the dense reward function and replan the path.

[0012] The aforementioned technical solution facilitates real-time dynamic updates to complex environments by abstracting and representing the semantic information of the global environment. It generates sub-target image sequences through action segmentation and a hybrid attention mechanism in the visual thought chain reasoning module model in steps S3 and S4, making intermediate reasoning steps explicit and reducing the likelihood of user misunderstanding leading to unreasonable path planning. Furthermore, an adaptive algorithm is designed based on multimodal user feedback, along with a corresponding pattern decision-making mechanism, to ensure control reliability and feedback efficiency under different complex scenarios.

[0013] Furthermore, after step S5, the method further includes: step S6, which involves constructing a triplet dataset containing historical environmental states, user feedback types, and path selection results, and training a preference prediction model using a reinforcement learning framework to minimize the KL divergence between the dense reward function parameters output by the model and the user's actual feedback, thereby achieving personalized user preference modeling.

[0014] Furthermore, in step S1, environmental data is collected through multi-source vehicle-mounted cameras and sensors. The expert abstraction unit uses a semantic segmentation network and an instance detector to extract obstacle positions, channel widths, and semantic labels of target objects, generating a structured environmental representation containing geometric constraints and semantic information.

[0015] Furthermore, in step S2, the dynamic reinforcement learning dense reward function generation process is as follows: the structured environment representation and user instructions are jointly encoded to generate a segmented reward function containing security constraints, task completion, and comfort weights; the segmented reward function is implemented through coded conditional branches and includes security constraints, task completion, and comfort weights, wherein the weight coefficients are dynamically adjusted in priority by the large language model parsing user instructions.

[0016] Furthermore, in step S3, the visual thinking chain reasoning module decomposes the user's goal into multiple executable intermediate sub-goals, and generates sub-goal images corresponding to at least three candidate paths based on the intermediate sub-goals and the current environmental state through a causal attention mechanism.

[0017] Furthermore, step S4 includes discretizing the continuous action space into action blocks, each action block containing action instructions for multiple time steps; using a full attention mechanism to predict all action tokens in the action block in parallel, and optimizing the probability distribution of the action sequence through a cross-entropy loss function; the length of the action block is dynamically adjusted according to the task complexity and determined by sampling a task-specific context window range.

[0018] Furthermore, in step S5, the multimodal interactive fusion feedback includes two feedback modes: verbal feedback and visual feedback. The verbal feedback parsing unit identifies keywords in the user's correction commands and generates semantic vectors. The visual feedback processing unit parses the user's touch coordinates or marked areas on the sub-target image and generates spatial position correction vectors. A Transformer architecture is used to align the semantic vectors and spatial position correction vectors across modal features. Confidence scores for each feedback mode are calculated based on attention weights. Reward parameters are continuously updated through an online learning mechanism according to the score ratio. Based on the updated dense reward function, the process returns to step S3 to re-predict the action sequence and generate candidate paths.

[0019] Furthermore, step S6 includes:

[0020] Step S601: Drive dense reward function iteration using online learning mechanism: Record the environment state for each interaction. User feedback types and path selection results Organize and construct the triplet dataset ;

[0021] Step S602: The preference prediction model adopts a Transformer-based Seq2Seq architecture, with input triplet data. Output the distribution of dense reward function parameters ), and generate executable reward code snippets through softmax normalization; define the distribution of the prediction-dense reward function. Distribution with actual user feedback The goal is to minimize the KL divergence.

[0022] On the other hand, the present invention provides an adaptive control system based on an autonomous mobile platform for implementing the adaptive control method based on an autonomous mobile platform as described above, comprising:

[0023] The environmental state perception and abstraction module includes an expert abstraction unit, which collects surrounding environmental data in real time and uses expert abstraction to generate structured representations.

[0024] The dynamic reward function generation module uses a large language model to input jointly encoded structured representations of the environment and user voice commands to generate segmented dynamic reinforcement learning dense reward functions.

[0025] The visual thinking chain reasoning module includes a sub-target decomposition sub-module, a path planning sub-module, and a human-computer interaction interface module. It generates candidate paths and corresponding sub-target image sequences through a causal attention mechanism, ensuring that image generation follows a chronological order. It employs a full attention mechanism to predict and segment action sequences based on the current observation and already generated sub-target images, allowing all action tokens to interact with each other and capturing complex dependencies between action dimensions. The sub-target image sequences and action sequences are then displayed on the human-computer interaction interface.

[0026] The multimodal interactive feedback fusion decision module includes a voice feedback parsing unit, a visual feedback processing unit, and a feature alignment unit. It receives multimodal correction feedback from users, updates the dense reward function, and replans the path.

[0027] Compared with the prior art, the present invention has the following technical effects:

[0028] (1) This invention generates sub-target image sequences by using action segmentation technology and hybrid attention mechanism in visual thinking chain reasoning, making intermediate reasoning steps explicit, and discretizing the continuous action space, which significantly improves the prediction accuracy of long sequence complex actions and enhances the interpretability and reliability of path planning. Compared with the prior art, this invention has stronger generalization ability.

[0029] (2) This invention proposes an adaptive algorithm for an autonomous mobile platform that includes two feedback modes: language feedback and human-computer interface touch visual feedback. It also designs a corresponding mode decision mechanism to meet the control reliability and feedback efficiency under different complex situations.

[0030] (3) This invention constructs a triplet dataset, updates the dense reward function parameters online, minimizes the KL divergence between the predicted reward and the user's actual feedback, and realizes personalized user preference modeling in order to support autonomous driving and closed-loop control. Attached Figure Description

[0031] The invention, its features and advantages will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings.

[0032] Figure 1 This is an adaptive control architecture diagram of an autonomous mobile platform in one embodiment of the present invention;

[0033] Figure 2 This is a framework diagram of an autonomous mobile platform adaptive control system according to an embodiment of the present invention. Detailed Implementation

[0034] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, but these are not intended to limit the scope of the invention.

[0035] In the following detailed description, numerous specific details are set forth to provide a more thorough understanding of the invention. However, it will be apparent to those skilled in the art that well-known algorithms and models are not shown in detail to avoid obscuring the spirit of the invention.

[0036] Furthermore, the execution order of actions, steps, etc. in the apparatus and methods shown in the claims, specification, and drawings can be implemented in any order, as long as there is no specific express limitation on the order and the output of the preceding processing is not used in the subsequent processing.

[0037] Example

[0038] See Figure 1 This embodiment provides an adaptive control method for an autonomous mobile platform, including the following steps:

[0039] S1: Real-time environmental data is acquired using multiple sensors (LiDAR, RGB-D camera, ultrasonic sensors, etc.). Expert abstraction units are jointly trained on the Open X-Embodiment dataset and the Action-less Video Dataset. The Expert Abstraction unit uses a semantic segmentation network and an instance detector to extract obstacle bounding boxes, channel widths, and semantic labels for target objects. It then generates a high-level abstract description of the environment containing geometric constraints and semantic information using JSON (JavaScript Object Notation, a lightweight data exchange format). More specifically, this may include the following sub-steps:

[0040] S101: Multi-source sensor data acquisition and preprocessing. Sensor data is aligned using ROS (Robot Operating System) timestamps. The Kalibr toolchain (a toolbox for vision-inertial calibration) is used to calibrate the extrinsic parameters of the LiDAR and camera, ensuring pixel-level alignment between the point cloud and the image. The point cloud data is downsampled to 0.05-meter resolution using a voxel grid filter, and the depth map is aligned with the RGB image using bilinear interpolation.

[0041] S102: Employs a lightweight Mask R-CNN architecture (ResNet-18 backbone + FPN head), pre-trained and fine-tuned on the Open X-Embodiment dataset, to identify 10 semantic labels including roads, pedestrians, and obstacles. Simultaneously, it generates corresponding geometric constraints such as obstacle distance calculation and channel width estimation.

[0042] (1) Obstacle distance calculation: The minimum lateral distance between the wheelchair and the obstacle is calculated by fusing the point cloud and the depth map. :

[0043]

[0044] in Point cloud coordinates of the obstacle. (2) Channel width estimation: (The x and y coordinates are the boundary coordinates.)

[0045] Based on semantic segmentation results Extract road boundary lines and calculate the width of the passable area. :

[0046] in and The x-coordinates of the left and right boundary lines in the wheelchair coordinate system.

[0047] Finally, the extracted geometric constraints and semantic information are encoded into a structured environment description using JSON format.

[0048] S2: Jointly encode user speech commands and structured environment descriptions, generating segmented dynamic reinforcement learning (RL) dense reward functions through Large Language Models (LLMs), including safety constraints, task completion terms, and comfort weights. More specifically, this may include the following sub-steps:

[0049] S201: Based on the high-level abstract description of the environment obtained from S1, joint encoding and structured input are performed. The JSON-formatted environment description from the expert abstraction unit is encoded into a vector representation. The environment feature vector is mapped to the Transformer input space through a positional encoder. The position encoder is defined as follows:

[0050]

[0051] Where i is the location index and d is the dimension of the location encoding.

[0052] User voice commands are transcribed into text by the Whisper model (a general speech recognition model developed by OpenAI that uses an encoder-decoder Transformer model architecture), and then the BERT encoder generates corresponding semantic vectors. This captures the key intent within the instruction. The semantic vector is adapted to the Transformer input space through a linear projection layer: in and These are the learnable bias term and the learnable weight matrix, respectively. The semantic vectors are then used. Finally, the structured representation of the environment and the vector representation of the speech instructions are concatenated into a joint embedding vector using a Transformer architecture, which serves as the input to the large language model.

[0053] The Transformer consists of 6 encoder layers, each containing a multi-head attention mechanism (8 heads) and a feedforward network (FFN), and finally takes the output labeled [CLS] as the joint embedding.

[0054]

[0055] S202: The large language model, based on joint embedding vectors, generates Pythonic reward function code with multiple conditional branches, including safety constraints, task completion, comfort, and corresponding reward or penalty incentives. Simultaneously, the large language model dynamically adjusts the reward coefficients using attention weights, optimizing the reward function based on priority ranking. For example, when a user says "slow down," the comfort weight... promote:

[0056]

[0057] in The confidence score for voice feedback is (0–1). , The dynamic weights are adjusted using the following optimization objective function:

[0058]

[0059] in To predict the KL divergence between rewards and actual user feedback, The dense reward function distribution generated for the model, The distribution of rewards is based on actual user feedback. For the loss of action sequence prediction error, The weighting coefficients adjust the importance between the two loss terms.

[0060] S3: Employs a visual chain-of-thought reasoning module to generate candidate paths and corresponding sub-target image sequences through a causal attention mechanism, ensuring that image generation follows a temporal order. Based on the CoT-VLA visual chain-of-thought reasoning (CoT) framework, it generates candidate paths and corresponding sub-target image sequences through a causal attention mechanism, ensuring the consistency of explicit intermediate reasoning steps in path planning with temporal order. The user goal is decomposed into multiple executable intermediate sub-targets, and corresponding sub-target images are generated through a causal attention mechanism. The image content includes pixel-level descriptions of the target object, obstacles, and path directions. More specifically, this may include the following sub-steps:

[0061] S301: Based on the user's objective, the language model parses it into semantic vectors and combines this with a structured abstract description of the environment to generate a logically coherent sequence of sub-objectives, thus decomposing complex tasks. Each sub-objective generates a corresponding sub-objective image sequence through an autoregressive causal attention mechanism, ensuring the consistency of explicit intermediate inference steps in path planning with temporal order. The image content includes pixel-level descriptions of target objects, obstacles, and path directions.

[0062] The causal attention mechanism is implemented using the Transformer architecture, and its core formula is:

[0063]

[0064] Where Q, K, and V are the query matrix, key matrix, and value matrix, respectively. The scaling factor is set to 64. During the sub-target image generation stage, the causal attention mechanism masks information from future pixels (such as the first pixel). line, number (Column), ensuring that image generation follows a chronological order:

[0065]

[0066] in These are the elements of the attention weight matrix. Finally, a Stable Diffusion model is used to perform super-resolution reconstruction on the generated image, improving the clarity of pixel-level description. The loss function of the Stable Diffusion model is:

[0067]

[0068] in For real images, For adding noise to the image, For noise prediction networks, For time steps, This refers to the user's command text.

[0069] S4: Based on the CoT action chunk generation and full attention mechanism, the complex dependencies between action dimensions are captured by discretizing the action space and allowing all action tokens to interact with each other. The continuous action space is discretized into action chunks, each containing action instructions at multiple time steps. The full attention mechanism is used to predict all action tokens in the action chunks in parallel, and the probability distribution of the action sequence is optimized using the cross-entropy loss function. The length of the action chunk is dynamically adjusted according to the task complexity and determined by sampling a task-specific context window range. Finally, the sub-target image sequence and action sequence are displayed on the human-computer interface. More specifically, this may include the following sub-steps:

[0070] S401: Utilizing action block technology, the continuous action space is discretized into action blocks, each containing action instructions over N time steps. Each action dimension (e.g., steering angle, acceleration) is mapped to 256 discrete bins, with bin widths evenly divided by the 1%-99th percentile of the training data. For example, continuous values ​​of steering angular velocity are discretized into 256 bins, each bin corresponding to an action token. The action block length is determined by sampling a task-specific context window range.

[0071] S402: The full attention mechanism allows all action tokens to interact, capturing complex dependencies between action dimensions (such as coordinated control of steering and speed). Its core formula is:

[0072]

[0073] Where Q, K, and V are the query matrix, key matrix, and value matrix, respectively. The scaling factor is set to 64. An 8-head attention computation is used, with each head independently generating an attention weight matrix. Finally, the attention weights are generated through splicing and linear projection:

[0074]

[0075] in For learnable parameter matrix, The dimension of the value vector for each attention head is defined. The action sequence is then optimized using the cross-entropy loss function to predict the probability distribution.

[0076] in Let be the i-th action, and l be the user instruction. , The current environmental state and sub-target image. Let be the probability distribution function.

[0077] S403: Visualization output of sub-target images and action sequences. It employs the Stable Diffusion v2.1 architecture, including the UNet backbone, VAE encoder, and CLIP text encoder. UNet predicts noise residuals through multi-layer convolutions and attention modules.

[0078]

[0079] in For adding noise to the image, For time steps, This refers to the user command text. The sub-target image sequence is generated through multi-step diffusion sampling, using the following formula:

[0080]

[0081] in For diffusion scheduling parameters, Noise figure The sub-target image sequence and motion sequence are displayed on the human-computer interface.

[0082] S5: The human-computer interaction interface receives multimodal correction feedback from the user and updates the dense reward function and replans the path by fusing the decision model through multimodal interactive feedback. The interactive reward optimization framework and cross-modal alignment technology generate a dynamic weight allocation strategy by fusing linguistic and visual feedback, and continuously optimize the dense reward function parameters and action sequences by combining online learning mechanisms. More specifically, it may include the following sub-steps:

[0083] S501: The user's speech feedback is converted into a high-dimensional semantic vector through a speech feedback parsing unit (such as the BERT-Whisper joint encoder). The user's speech commands are transcribed into text by the Whisper model, and then the corresponding semantic vectors are generated by the BERT-base encoder. This captures the key intent within the instruction. The semantic vector is adapted to the Transformer input space through a linear projection layer: in This is a learnable weight matrix.

[0084] S502: Through the camera intrinsic parameter matrix The pixel coordinates of the touch point on the upper image of the human-computer interface Convert to three-dimensional spatial coordinates in the navigation coordinate system .

[0085]

[0086] in This represents the inverse of the camera intrinsic parameter matrix. , The rotation matrix and translation vector represent the wheelchair pose. User-annotated regions are used to extract local feature vectors through a convolutional neural network (e.g., ResNet-18), which are then combined with the semantic labels of the target object (e.g., "steps", "door") to generate spatially corrected vectors.

[0087] S503: Based on a cross-modal Transformer architecture, it aligns semantic vectors and spatially modified vectors for feature mapping. First, it uses a multi-head attention mechanism to calculate the similarity matrix between the semantic vectors and the spatially modified vectors. Where n is the number of language tokens and m is the visual feature dimension. The similarity calculation formula is as follows:

[0088]

[0089] This represents the similarity between the nth and nth tokens. Let i be the query vector corresponding to the i-th token. Let be the key vector corresponding to the j-th token. This is the scaling factor, i.e., the dimension of the key vector.

[0090] Subsequently, a self-attention mechanism is used to fuse spatial contextual information (such as the semantic association between "decelerate" and "labeled region") from language instructions and visual annotations. Furthermore, based on attention weights, a sigmoid function is used to predict the confidence scores of both language and visual feedback. and :

[0091]

[0092] in, For the Sigmoid function, For the weight matrix of language feedback, This is the weight matrix for visual feedback.

[0093] Integrate verbal and visual feedback into the reward function update based on confidence score proportions:

[0094]

[0095] in, For parameters of a dense reward function driven by language feedback, The parameters are for the dense reward function driven by visual feedback. Based on the updated dense reward function, the action sequence is re-predicted and candidate paths are generated using a full attention mechanism:

[0096]

[0097] Where l represents the user instruction. , The current environmental state and sub-target image. For the i-th action, Let be the probability distribution function.

[0098] S6: After implementing multimodal feedback and adjusting the reinforcement function parameters, a preference prediction model is trained using a reinforcement learning framework by constructing a triplet dataset containing historical environmental states, user feedback types, and path selection results. This ensures that the dense reward function parameters output by the model minimize the KL divergence with the user's actual feedback, thus achieving the training of a personalized user preference model. More specifically, this may include the following sub-steps:

[0099] S601: Utilizes an online learning mechanism to drive iterative dense reward functions. The system records the environmental state of each interaction. User feedback types (Language / Visual) and Path Selection Results Organize and construct the triplet dataset The triplet dataset is encoded, and environmental states (obstacle distance, aisle width, wheelchair pose, etc.) are encoded as follows: Language feedback is encoded into semantic vectors Visual feedback is encoded as a spatial correction vector. The path results and action sequences are encoded into time-series vectors. Finally, the joint embedding vector is concatenated using a Transformer architecture containing six encoder layers:

[0100]

[0101] S602: The preference prediction model adopts a Transformer-based Seq2Seq architecture, with input triplet data. Corresponding joint embedding vector Output the distribution of dense reward function parameters The algorithm generates executable reward code snippets by normalizing the reward function using softmax. It defines the distribution of the prediction-dense reward function. Distribution with actual user feedback The goal is to minimize the KL divergence.

[0102]

[0103] in, The probability quality function for the actual reward distribution based on user feedback. The KL divergence value between the distribution of rewards based on actual user feedback and the distribution of dense reward function generated by the model is calculated using the following formula: .

[0104] Introducing action sequence to predict total loss Ensure that the updated dense reward function-driven strategy generates the user adoption path:

[0105]

[0106] in The balancing coefficients are obtained through optimization using the validation dataset. To predict the loss for the action sequence, ensure that the updated dense reward function drives the policy to generate the user adoption path. After each user interaction, the newly generated triples are... Add to historical dataset The preference prediction model is fine-tuned using the PRO algorithm.

[0107]

[0108] in To measure the long-term benefits of path selection, we need to consider the discount return. For given parameters and state Feedback types Next, execute the action. The probability is calculated. Simultaneously, the parameters of the dense reward function are updated using a weighted average based on the confidence scores provided by users.

[0109]

[0110] in, For parameters of a dense reward function driven by language feedback, Parameters for the dense reward function driven by visual feedback.

[0111] See Figure 2 A control system that implements the adaptive control method for the autonomous mobile platform as described above includes:

[0112] The environmental state perception and abstraction module includes an expert abstraction unit, which collects surrounding environmental data in real time and uses expert abstraction to generate structured representations.

[0113] The dynamic reward function generation module uses a large language model to input jointly encoded structured representations of the environment and user voice commands to generate segmented dynamic reinforcement learning dense reward functions.

[0114] The visual thinking chain reasoning module includes a sub-target decomposition sub-module, a path planning sub-module, and a human-computer interaction interface module. It generates candidate paths and corresponding sub-target image sequences through a causal attention mechanism, ensuring that image generation follows a chronological order. It employs a full attention mechanism to predict and segment action sequences based on the current observation and already generated sub-target images, allowing all action tokens to interact with each other and capturing complex dependencies between action dimensions. The sub-target image sequences and action sequences are then displayed on the human-computer interaction interface.

[0115] The multimodal interactive feedback fusion decision module includes a voice feedback parsing unit, a visual feedback processing unit, and a feature alignment unit. It receives multimodal correction feedback from users, updates the dense reward function, and replans the path.

[0116] The adaptive control method of the aforementioned autonomous mobile platform can be embodied in the form of a computer program product or a software functional unit. If the adaptive control method of the aforementioned autonomous mobile platform is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Therefore, the essence of this technical solution, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause an electronic system (which may be a personal computer, server, or network system, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0117] Those skilled in the art will recognize that the units, i.e., algorithm steps, of the various examples described in connection with this embodiment can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0118] The preferred embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above. Systems and structures not described in detail should be understood as being implemented in a conventional manner in the art. Any person skilled in the art can make many possible variations and modifications to the technical solutions of the present invention using the methods and techniques disclosed above, or modify them into equivalent embodiments with equivalent changes, without departing from the scope of the present invention. This does not affect the essential content of the present invention. Therefore, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the present invention are still within the scope of protection of the present invention.

Claims

1. An adaptive control method for an autonomous mobile platform, characterized in that, Includes the following steps: Step S1: Collect surrounding environmental data in real time through the environmental state sensing device and generate a structured representation using expert abstraction units. Step S2: After the structured representation of the environment and the user's voice command are jointly encoded, they are used as input to a large language model, which generates a segmented dynamic reinforcement learning dense reward function. Step S3: Using a visual thinking chain reasoning model, candidate paths and corresponding sub-target image sequences are generated through a causal attention mechanism, so that image generation follows a temporal order. Step S4: Using a full attention mechanism, based on the current observation and the generated sub-target images, predict and generate action sequences in blocks, allowing all action tokens to interact with each other and capturing the complex dependencies between action dimensions; display the sub-target image sequence and action sequence on the human-computer interaction interface. Step S5: The human-computer interaction interface receives multimodal correction feedback from the user, updates the dense reward function, and replans the path through the multimodal interactive feedback fusion decision model. In step S5, the multimodal interactive feedback includes two feedback modes: verbal feedback and visual feedback. The verbal feedback parsing unit identifies keywords in the user's correction instructions and generates semantic vectors. The visual feedback processing unit parses the user's touch coordinates or marked areas on the sub-target image and generates spatial position correction vectors. The Transformer architecture is used to perform cross-modal feature alignment between the semantic vectors and the spatial position correction vectors, and the confidence scores of each feedback mode are calculated based on attention weights. The reward parameters are continuously updated through an online learning mechanism according to the score ratio. Based on the updated dense reward function, the process returns to step S3 to re-predict the action sequence and generate candidate paths. Step S6: By constructing a triplet dataset containing historical environment states, user feedback types, and path selection results, a preference prediction model is trained using a reinforcement learning framework to minimize the KL divergence between the dense reward function parameters output by the model and the user's actual feedback; Step S6 includes: Step S601: Drive dense reward function iteration using online learning mechanism: Record the environment state for each interaction. User feedback types and path selection results Organize and construct the triplet dataset ; Step S602: The preference prediction model adopts a Transformer-based Seq2Seq architecture, with input triplet data. Output the distribution of dense reward function parameters ), and generate executable reward code snippets through softmax normalization; define the distribution of the prediction-dense reward function. Distribution with actual user feedback The goal is to minimize the KL divergence.

2. The adaptive control method for an autonomous mobile platform according to claim 1, characterized in that, In step S1, environmental data is collected through multi-source vehicle-mounted cameras and sensors. The expert abstraction unit uses a semantic segmentation network and an instance detector to extract obstacle positions, channel widths, and semantic labels of target objects, generating a structured environmental representation containing geometric constraints and semantic information.

3. The adaptive control method for an autonomous mobile platform according to claim 1, characterized in that, In step S2, the dynamic reinforcement learning dense reward function generation process is as follows: the structured environment representation and user instructions are jointly encoded to generate a segmented reward function containing security constraints, task completion, and comfort weights; the segmented reward function is implemented through coded conditional branches and includes security constraints, task completion, and comfort weights, wherein the weight coefficients are dynamically adjusted in priority by the large language model parsing user instructions.

4. The adaptive control method for an autonomous mobile platform according to claim 1, characterized in that, In step S3, the visual thinking chain reasoning module decomposes the user's goal into multiple executable intermediate sub-goals, and generates sub-goal images corresponding to at least three candidate paths based on the intermediate sub-goals and the current environmental state through a causal attention mechanism.

5. The adaptive control method for an autonomous mobile platform according to claim 1, characterized in that, Step S4 includes discretizing the continuous action space into action blocks, each action block containing action instructions at multiple time steps; using a full attention mechanism to predict all action tokens in the action block in parallel, and optimizing the probability distribution of the action sequence through a cross-entropy loss function; the length of the action block is dynamically adjusted according to the task complexity and determined by sampling a task-specific context window range.

6. An adaptive control system based on an autonomous mobile platform, characterized in that, An adaptive control method for implementing an autonomous mobile platform as described in any one of claims 1 to 5, comprising: The environmental state perception and abstraction module includes an expert abstraction unit, which collects surrounding environmental data in real time and uses the expert abstraction unit to generate structured representations. The dynamic reward function generation module uses a large language model to input jointly encoded structured representations of the environment and user voice commands to generate segmented dynamic reinforcement learning dense reward functions. The visual thinking chain reasoning module includes a sub-target decomposition sub-module, a path planning sub-module, and a human-computer interaction interface module. It generates candidate paths and corresponding sub-target image sequences through a causal attention mechanism, ensuring that image generation follows a chronological order. It employs a full attention mechanism to predict and segment action sequences based on the current observation and already generated sub-target images, allowing all action tokens to interact with each other and capturing complex dependencies between action dimensions. The sub-target image sequences and action sequences are then displayed on the human-computer interaction interface. The multimodal interactive feedback fusion decision module includes a voice feedback parsing unit, a visual feedback processing unit, and a feature alignment unit. It receives multimodal correction feedback from users, updates the dense reward function, and replans the path.

Images