Motion control methods, devices and mobile equipment
By fusing and decomposing multi-source sensing data, precise voice control of mobile devices is achieved, solving the problem of limited voice control functionality in existing technologies and improving the intelligence and control accuracy of mobile devices.
Patent Information
- Application Number
- CN202611233178.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-13
- Publication Date
- 2026-11-17
AI Technical Summary
In existing technologies, voice-controlled mobile devices have relatively limited functionality and cannot accurately recognize complex driving intentions, especially in cases of ambiguity or implicit spatial preferences, thus failing to meet the user's control needs.
By acquiring multi-source perception data, including voice data, environmental perception data, and device status data, the system uses a visual-speech model for intent recognition, anchors physical space preferences to the physical environment of the environmental perception data, and breaks down driving intentions into multiple executable sub-tasks to generate action planning sequences to drive the movement of the mobile device.
It has enriched the application scenarios of voice interaction control, improved the accuracy and flexibility of controlling mobile devices, and enhanced the ability to execute complex driving intentions.
Smart Images

Figure CN123104671A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of intelligent driving technology, specifically to a motion control method, device, and mobile device. Background Technology
[0002] With the development of artificial intelligence technology, more and more mobile devices (such as vehicles and robots) are equipped with intelligent voice assistants. Users can interact with mobile devices via voice, and the mobile devices execute corresponding operations by recognizing the user's voice commands. However, the functions that can be achieved by controlling mobile devices using voice at present are relatively limited, such as simple functions like controlling car windows, adjusting air conditioning temperature, changing music, and navigating to a destination, thus restricting their application. Summary of the Invention
[0003] This application discloses a motion control method, device, and mobile device, which can control the motion of the mobile device based on user-input voice data, enriching the application scenarios of voice interaction control functions and improving the accuracy of controlling the mobile device.
[0004] This application discloses a motion control method applied to a mobile device, the method comprising: Acquire multi-source sensing data; wherein, the multi-source sensing data includes: voice data, environmental sensing data around the mobile device, and device status data; The voice data is subjected to intent recognition to obtain the driving intent, which includes physical space preferences and action types; The physical space preference is anchored to the physical environment of the environmental perception data to obtain the anchoring result; Based on the anchoring results, the environmental perception data, the device status data, and the action type, an action planning sequence consisting of multiple sub-tasks is planned and generated. The mobile device is driven to move based on the motion planning sequence.
[0005] This application discloses a motion control device applied to a mobile device, the device comprising: The data acquisition module is used for multi-source sensing data; wherein, the multi-source sensing data includes: voice data, environmental sensing data around the mobile device, and device status data; An intent recognition module is used to perform intent recognition on the voice data to obtain driving intent, which includes physical space preferences and action types; An anchoring module is used to anchor the physical space preference to the physical environment of the environmental perception data to obtain the anchoring result; The planning module is used to plan and generate an action planning sequence consisting of multiple sub-tasks based on the anchoring result, the environmental perception data, the device status data, and the action type. A driving module is used to drive the mobile device to move based on the action planning sequence.
[0006] This application discloses an electronic device, including a memory and a processor. The memory stores a computer program, and when the computer program is executed by the processor, the processor performs the method described above.
[0007] This application discloses a mobile device, including a memory and a processor. The memory stores a computer program, and when the computer program is executed by the processor, the processor performs the method described above.
[0008] This application discloses a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the method described above.
[0009] This application discloses a computer program product, including a computer program, which, when executed by a processor, implements the method described above.
[0010] The motion control method, device, mobile device, and program product disclosed in this application involve a mobile device acquiring multi-source perception data, including voice data, environmental perception data surrounding the mobile device, and device status data. Intent recognition is performed on the voice data to obtain a driving intent, which includes physical space preferences and action types. The physical space preferences are anchored to the physical environment of the environmental perception data to obtain an anchoring result. Then, based on the anchoring result, environmental perception data, device status data, and the action types, an action planning sequence consisting of multiple sub-tasks is planned and generated. Based on this action planning sequence, the mobile device is driven to move.
[0011] In this embodiment, the mobile device can accurately identify the physical space preferences and action types in the user's input voice data, anchoring the physical space preferences to the physical environment of the environmental perception data. This allows for the mapping of ambiguous voice content to the actual physical environment, thereby enabling the control of the mobile device using the user's input voice data and enriching the application scenarios of voice interaction control. Furthermore, the driving intent corresponding to the voice data can be combined with environmental perception data, device status data, etc., breaking down the driving intent into multiple executable sub-tasks, improving the accuracy of controlling the mobile device. In addition, breaking down the driving intent into multiple executable sub-tasks allows each sub-task to be interpreted, monitored, and rolled back, improving the flexibility of controlling the mobile device and increasing the success rate of completing the driving intent. Attached Figure Description
[0012] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0013] Figure 1 This is a diagram illustrating an application scenario of the motion control method in one embodiment; Figure 2 Here is a flowchart of a motion control method in one embodiment; Figure 3 This is a flowchart for determining whether a constraint condition is met in one embodiment; Figure 4 Here is a flowchart of a motion control method in another embodiment; Figure 5 This is a flowchart illustrating a motion control method in one embodiment; Figure 6 This is a flowchart illustrating the execution of each subtask in one embodiment; Figure 7 This is a system architecture diagram of a motion control method in one embodiment; Figure 8 This is a block diagram of a motion control device in one embodiment; Figure 9 This is a structural block diagram of a mobile device in one embodiment. Detailed Implementation
[0014] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0015] It should be noted that the terms "comprising" and "having," and any variations thereof, in the embodiments and accompanying drawings of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the steps or units listed, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices.
[0016] It is understood that the terms "first," "second," etc., used in this application may be used to describe various elements herein, but these elements are not limited by these terms. These terms are only used to distinguish one element / signal from another element / signal. For example, without departing from the scope of this application, a first trigger operation may be referred to as a second trigger operation, and similarly, a second trigger operation may be referred to as a first trigger operation. Both the first trigger operation and the second trigger operation are user-input trigger operations, but they are not the same trigger operation. The term "multiple" as used in this application refers to two or more. The term "and / or" as used in this application refers to one of the schemes, or any combination of multiple schemes.
[0017] It should be noted that the data related to the user obtained in the embodiments of this application are obtained with the user's knowledge and full authorization. For example, image data involving the user must be collected with the user's knowledge and full authorization, and is only used to implement the embodiments of this application and not for other purposes.
[0018] In related technologies, the functions achievable by voice control of mobile devices are relatively limited, such as simple functions like controlling car windows, adjusting air conditioning temperature, changing music, and navigating to a destination. This control is an atomic operation, and mobile devices cannot accurately recognize and perform corresponding functions for ambiguous voices or those with implicit spatial preferences. For example, if a user wants to park their car in a parking space near an elevator in a building within a park, they can only achieve this by controlling the vehicle step by step. For instance, they need to first use voice control to guide the vehicle to the parking lot entrance of the building, then manually drive the vehicle to find the elevator in the parking lot, and finally manually or automatically park the vehicle in a parking space near the elevator.
[0019] Therefore, in complex control scenarios, voice control functions often cannot achieve the functions required by users, thus limiting their application.
[0020] This application provides a motion control method, device, mobile device, and program product that can control the motion of a mobile device based on user-input voice data, enriching the application scenarios of voice interaction control functions and improving the accuracy of controlling mobile devices.
[0021] Figure 1 This is a diagram illustrating an application scenario of the motion control method in one embodiment. For example... Figure 1 As shown, the motion control method can be applied to the mobile device 100. It should be noted that the mobile device 100 may include, but is not limited to, vehicles, drones, airplanes, ships, robots, etc., and this application embodiment does not limit it.
[0022] In some embodiments, the mobile device 100 may be equipped with a terminal, which can be used to implement the motion control method provided in the embodiments of this application. Taking a vehicle as an example, the terminal may be a central control display screen, a central control console, an in-vehicle terminal, etc. installed inside the vehicle.
[0023] In some embodiments, the terminal may be an external device that establishes a communication connection with the mobile device 100. The communication connection may include, but is not limited to, wireless communication connections such as Bluetooth and Wi-Fi, or wired communication connections via cables.
[0024] The terminal may have a display screen, through which various interactive interfaces related to the mobile device 100 can be displayed, such as navigation interface, control interface of mobile device 100, audio and video playback interface, etc., but not limited to these.
[0025] In some embodiments, the terminal may have the function of controlling the mobile device 100. For example, the electronic device may control the driving-related functions of the mobile device 100 (such as controlling the driving speed and driving direction of the mobile device 100); for another example, the electronic device may also control various components in the mobile device 100 (such as air conditioning, windows, doors, lights, etc.).
[0026] In some embodiments, the mobile device 100 may be equipped with a variety of sensors, including but not limited to cameras, proximity sensors, lidar, microphones, etc. These various sensors may use different data, and the mobile device 100 may combine the data collected by the various sensors to control the actions performed by the mobile device 100.
[0027] like Figure 2 As shown, in one embodiment, a motion control method is provided, which can be applied to the aforementioned mobile device. The method may include the following steps: Step 210: Acquire multi-source sensing data; wherein, the multi-source sensing data includes: voice data, environmental sensing data around the mobile device, and device status data.
[0028] In some embodiments, the mobile device may be equipped with multiple sensors, and multi-source sensing data can be obtained through these multiple sensors. Optionally, the mobile device may be equipped with a communication module (such as a cellular radio frequency module, Bluetooth communication module, or Wi-Fi communication module, etc.), and the multi-source sensing data can be obtained through the network connected to the communication module. For example, the mobile device can connect to a server or Internet of Things (IoT) through the communication module to obtain multi-source sensing data through the server or IoT.
[0029] Optionally, the mobile device may be equipped with a voice acquisition device (such as a microphone), which can be used to acquire voice data input by the user.
[0030] Optionally, the mobile device may be equipped with one or more cameras to collect visual data, which may include, but is not limited to, images and / or videos. In some embodiments, the visual data may include visual data from outside the mobile device and / or visual data from inside the mobile device. For example, a camera may be installed inside a vehicle to collect images of the vehicle's interior, obtaining visual data of the vehicle's interior (such as images of the driver's face, the driver's gestures, and images of other passengers). Alternatively, a camera may be installed outside the vehicle to collect images of the exterior, obtaining visual data of the exterior (such as images of the surrounding environment and road).
[0031] Environmental perception data around mobile devices can be used to describe the environment surrounding the mobile device, such as the scene in which the mobile device is located and the surrounding road conditions (such as road information, pedestrian information, and surrounding vehicle information). Furthermore, the environmental perception data of the mobile device can include visual data from outside the mobile device, and optionally, it can also include data collected by other sensors, such as environmental perception data collected by distance sensors and LiDAR.
[0032] In some implementations, the environmental perception data around the mobile device may include real-time environmental perception data or environmental context data. Real-time environmental perception data may refer to environmental perception data collected in real time (i.e., current environmental perception data). Environmental context data can be used to describe changes in the environment around the mobile device, and may include historically collected environmental perception data and current environmental perception data, thereby determining environmental changes.
[0033] Device status data can be used to describe the device status of the mobile device itself. In some embodiments, device status information may include, but is not limited to, one or more of the following: driving speed, driving direction, location information, map information stored on the mobile device, driving permission information obtained by the mobile device, etc.
[0034] The map information stored on the mobile device can refer to the map stored in the mobile device's memory, such as a high-definition map (HD-Map). HD-Map can include structured geographic information such as road topology, lane lines, traffic signs, and guardrails. HD-Map can achieve more refined positioning, such as locating a specific elevator or parking space in a building.
[0035] Step 220: Perform intent recognition on the voice data to obtain the driving intent, which includes physical space preferences and action types.
[0036] After acquiring voice data input by the user, the mobile device can perform intent recognition on the voice data to obtain the user's driving intention. In some embodiments, the mobile device can input the voice data into a Vision-Language Model (VLM), and the VLM can perform intent recognition on the voice data to obtain the driving intention.
[0037] Driving intent can include physical space preferences and action types. Physical space preferences refer to objects in the physical space that the user expects the mobile device to reach or follow. Optionally, these preferences can be physical entities or locations in real space, such as a building, a car, a parking space, an elevator in a building, a charging station, or an exit in an underground parking garage, but are not limited to these. Action types refer to the categories of actions that the user expects the mobile device to perform, such as parking, following another vehicle, navigation, or avoiding obstacles, but are not limited to these.
[0038] As one implementation method, driving intentions can be represented by intention categories and corresponding slots. The intention category corresponds to the identified action type, and the slots can include key parameters identified from the voice data, such as specific parameters of physical space preferences (e.g., specific location, floor, objects in the physical space, etc.). For example, if the user inputs the voice data "Find me a parking space near the elevator in B2", the intention category (i.e., action type) can be identified as parking, and the slots can include: floor = 2, space preference = near the elevator, parking space type = any.
[0039] Optionally, the slots corresponding to different intent categories can be the same or different, and can be preset according to actual needs.
[0040] Optionally, the mobile device can perform automatic speech recognition (ASR) on the voice data, identify the voice content (which can be represented by text) corresponding to the voice data, and input the voice content into the VLM. The VLM then performs semantic analysis on the voice content to obtain the driving intention.
[0041] For example, the VLM may include an intent recognition module, which may adopt model architectures such as CNN (Convolutional Neural Network), RNN (Recurrent Neural Network), BERT (Bidirectional Encoder Representations from Transformers), LLM (Large Language Model), or a model based on cross-attention mechanism, without limitation.
[0042] Optionally, the VLM can be trained using supervised, unsupervised, or semi-supervised training methods. For example, the VLM can be trained using supervised training, which can acquire a large amount of training data. This training data may include ASR text content and corresponding intent labels and slot labels. The intent labels can be used to represent the intent category corresponding to the ASR text content. The slot labels can be used to mark key parameters contained in the ASR text content.
[0043] In some embodiments, visual data from within the mobile device can be acquired, and intent recognition can be performed based on the internal visual data and voice data to obtain the driving intent. Optionally, the internal visual data may include user gesture images and / or facial images, etc. The gesture images can be recognized to obtain user gesture information, such as the spatial coordinates of the gesture and the gesture type (e.g., pointing, waving, drawing a circle, etc.), and the facial images can be recognized to obtain information such as head posture and / or gaze area. Auxiliary information obtained based on the internal visual data (such as user gesture information, head posture and / or gaze area information, etc.) can be input into the VLM, and the VLM can combine this information with voice data to perform intent recognition to obtain the driving intent.
[0044] For example, if a user inputs the voice data "follow the car in front" and points to the target vehicle they want to follow, the mobile device can recognize the gesture type as pointing based on internal visual data and identify the spatial coordinates of the gesture. Combined with the user's voice input data, the device can determine the physical spatial preference of the driving intention as the car the user is pointing to, and the action type as following the car.
[0045] In the above implementation, intention recognition can be performed by combining natural language with visual data inside the mobile device, which improves the accuracy of intention recognition. Users can issue accurate vehicle control commands through natural language and actions (such as gestures, head turning, and eye contact) inside the mobile device without physical contact, thus improving the naturalness and accuracy of human-computer interaction.
[0046] Step 230: Anchor the physical space preference to the physical environment of the environmental perception data to obtain the anchoring result.
[0047] After the mobile device recognizes the user's driving intention, the physical space preference in the user's driving intention can be anchored to the physical environment of the environmental perception data to obtain the anchoring result. The anchoring result can include the target anchor point. The target anchor point can be the target object of the physical space preference in the actual physical environment (such as the location information of the physical space preference on the map), or the target anchor point can be the target data of the physical space preference in the environmental perception data. For example, the target anchor point can be the visual anchor point of the physical space preference in the visual data (such as the image region of the physical space preference in the image).
[0048] In some embodiments, the mobile device can be anchored in the environmental perception data surrounding the mobile device based on driving intention. If the target anchor point is not found in the environmental perception data, the target anchor point can be determined based on the device status data. During the control of the mobile device, environmental perception data can be acquired in real time, and physical space preferences can be anchored to the physical environment of the acquired environmental perception data in real time until the visual anchor point corresponding to the physical space preference is found in the environmental perception data.
[0049] For example, if a user inputs the voice message "Find me a parking space near the elevator in B2," but the vehicle is not near B2 when the user inputs this message, the vehicle cannot find the target anchor point (such as the B2 elevator lobby sign) corresponding to the physical space preference based on the surrounding environmental perception data. The vehicle can first use the location information of the B2 elevator lobby as the target anchor point based on the stored map information, and can obtain the surrounding environmental perception data in real time during the vehicle's journey until it finds the target anchor point (such as the B2 elevator lobby sign) corresponding to the physical space preference based on the surrounding environmental perception data.
[0050] In some embodiments, the VLM can be a model with multimodal data processing capabilities. The mobile device can input environmental perception data and device status data into the VLM. After obtaining the driving intention, the VLM can anchor the physical space preference to the physical environment of the environmental perception data based on the environmental perception data and device status data to obtain the anchoring result.
[0051] In some implementations, before inputting the voice data, environmental perception data, and device status data into the VLM, the voice content of the voice data, environmental perception data, and device status data can be encoded separately, and the voice data, environmental perception data, and device status data can be projected onto the same feature space to obtain feature vectors corresponding to the voice data, environmental perception data, and device status data, respectively.
[0052] Optionally, the mobile device may be equipped with an image encoder, a text encoder, and a state encoder. The image encoder can be used to encode visual data in environmental perception data to obtain image feature vectors. For example, the image encoder may include, but is not limited to, encoders such as VIT (Vision Transformer) and CLIP-Style (Contrastive Language-Image Pre-training Style). The text encoder can be used to encode the speech content obtained from speech data recognition to obtain text feature vectors. For example, the text encoder may include, but is not limited to, an encoder combining BPE (BytePair Encoding) and Transformer.
[0053] A state encoder can be used to encode device state data to obtain a state feature vector. For example, a state encoder may include, but is not limited to, an encoder combining MLP (Multi-Layer Perceptron) and PE (Positional Encoding).
[0054] In the above implementation, multi-source perception data can be uniformly encoded during the data input stage to prevent the loss of semantic data, thereby improving the accuracy of subsequent scene anchoring (or target anchoring) and task planning.
[0055] VLM can perform multimodal fusion on the feature vectors corresponding to the input voice data, environmental perception data, and device status data respectively. Based on the feature vectors corresponding to the voice data, it can perform intent recognition to obtain the driving intent. Based on the feature vectors corresponding to the environmental perception data and device status data respectively, it can perform scene anchoring or target anchoring of the driving intent to map physical space preferences to specific target objects in the actual physical environment.
[0056] Optionally, the VLM may include an intent recognition module and an anchoring module. The anchoring module can be used to anchor the driving intent to a scene or target based on the feature vectors corresponding to the environmental perception data and the device status data, respectively.
[0057] For example, if a user inputs the voice message "Help me find a parking space near the elevator in B2", the VLM can recognize the driving intention and, in conjunction with a high-precision map, anchor the physical space preference (B2, near the elevator) to the location information of the B2 elevator lobby. Furthermore, it can combine the parking space status obtained from the Internet of Things (such as whether the parking space is vacant) to anchor the physical space preference (B2, near the elevator) to a specific parking space.
[0058] For example, if the user inputs the voice data "help me follow the red car in front", the VLM can recognize the driving intention and combine it with the visual data of the vehicle's exterior to anchor the physical space preference (the red car in front) to the image area (or matrix box) in the visual data that corresponds to the target vehicle to be followed.
[0059] In some embodiments, VLM can also incorporate auxiliary information (such as user gesture information, head posture, and / or gaze area) obtained from real-time internal visual data of the mobile device to anchor physical spatial preferences to the physical environment of the environmental perception data. For example, the location of the target object that the user is focusing on can be determined based on user gesture information and / or gaze area, and mapped to the physical environment of the environmental perception data. This allows physical spatial preferences to be accurately mapped to target objects in the physical environment of the environmental perception data, such as a parking space the user is pointing to or a ground area the user is looking at. This improves the accuracy of mapping physical spatial preferences to specific target objects in the real environment.
[0060] In the above implementation, VLM can perform multimodal fusion of voice data, environmental perception data, and device status data. It has the ability to understand complex language and external scenes. After accurately identifying the driving intention, it anchors the physical space preference in the driving intention to the physical environment of the environmental perception data. Even if the voice data input by the user is vague or abstract, the vague or abstract description can be mapped to a specific target object in the actual physical environment. Thus, the driving intention can be transformed into a task that the mobile device can perform. The user does not need to input voice data at every step, but can control the mobile device to achieve complex functions or scenarios by inputting a short sentence, thereby improving the intelligence of the mobile device.
[0061] In some embodiments, a multi-source bidding and scoring method can be used to anchor physical space preferences to the physical environment of the environmental perception data. After the VLM identifies the driving intention, it can determine multiple candidate anchor points based on the environmental perception data and device status data, and score each candidate anchor point based on the physical space preference to obtain a score for each candidate anchor point. The candidate anchor point with the highest score can be selected as the target anchor point.
[0062] Optionally, the scoring rules can be preset or dynamically generated by the VLM based on physical space preferences. For example, if the user inputs voice data as "help me find a parking space near the elevator in B2", the candidate anchor point can be a candidate parking space. The corresponding scoring rules can include the distance between the candidate parking space and the B2 elevator lobby, the parking status of the candidate parking space (whether it is vacant), the width of the candidate parking space, the accessibility of the candidate parking space (whether it is reachable), etc. Each candidate parking space can be scored based on these multiple scoring rules, and then the candidate parking space with the highest score can be selected as the target anchor point.
[0063] In the above implementation, the use of multi-source bidding and scoring methods to anchor physical space preferences to the physical environment of environmental perception data can improve the accuracy of mapping vague or abstract language to target objects in the real environment.
[0064] Step 240: Based on the anchoring results, environmental perception data, equipment status data, and action type, plan and generate an action planning sequence consisting of multiple sub-tasks.
[0065] Based on driving intent, mobile devices can combine environmental perception data and device status data to convert anchoring results and action types into multiple executable sub-tasks. In other words, driving intent is decomposed into multiple executable sub-tasks, generating an action planning sequence composed of multiple sub-tasks.
[0066] Optionally, the action planning sequence may include task information corresponding to multiple subtasks. This task information may include execution parameters and task objectives for each subtask. Execution parameters can be used to represent the actions to be performed, such as driving, searching for a parking space, parking, or shifting gears. The task objective refers to the goal expected to be achieved by performing the subtask, such as reaching a certain location, recognizing a certain landmark, parking in a certain parking space, or entering a certain road.
[0067] In some embodiments, after obtaining the anchoring result, the VLM can perform task planning based on driving intent, combined with environmental perception data and device status data, to obtain an action planning sequence consisting of multiple sub-tasks. Optionally, these multiple sub-tasks can be chained, sequentially executed sub-tasks with a temporal relationship. For example, the VLM can use a Chain-of-Thought (CoT) task planning method to generate an action planning sequence consisting of multiple sub-tasks. The VLM may include a task planning module that can input the anchoring result and action type as prompts. The task planning module can infer multiple sequentially executed sub-tasks based on the input prompts and the feature vectors corresponding to the environmental perception data and device status data, respectively.
[0068] The task planning module can be trained based on a task template library, which can include task sample data corresponding to various different scenarios. The task sample data can describe each action or step performed by the mobile device in the corresponding scenario. For example, the task template library can include task sample data corresponding to complex scenarios such as parking, dropping off passengers, picking up passengers, finding a highway server, following another vehicle, avoiding another vehicle, and stopping at multiple destinations.
[0069] The task planning module can be trained based on task sample data in the task template library, enabling it to learn the ability to break down complex tasks in different scenarios into multiple executable sub-tasks. For example, if the user inputs the voice message "Help me find a parking space near the elevator in B2", it can be broken down into 7 sub-tasks: 1. Pass through the gate; 2. Drive along the main passage; 3. Enter B2 via the off-ramp; 4. Search for candidate parking spaces in the alleyway; 5. Locate a parking space near the elevator; 6. Plan the route and park; 7. Engage P gear (parking gear) and provide a voice feedback.
[0070] In some embodiments, the mobile device can determine initial scene information based on environmental perception data and device status data, and plan and generate an action planning sequence consisting of multiple sub-tasks based on the initial scene information, anchoring results and action types.
[0071] As one implementation method, VLM performs intent recognition on voice data. Upon obtaining the driving intent, it can identify the scene in which the mobile device is located based on environmental perception data and device status data, thus obtaining initial scene information. This initial scene information describes the scene in which the mobile device is located before generating the action planning sequence (such as its location and surrounding environment). The initial scene information, anchoring results, and action type can be used as prompts, which are then input into the task planning module. This allows the task planning module to combine the feature vectors corresponding to the environmental perception data and device status data to infer multiple sequentially executed subtasks.
[0072] For example, if the user inputs the voice data "help me find a parking space near the elevator in B2", the VLM can determine that the vehicle is at the park gate based on the visual data outside the vehicle (such as recognizing the sign or gate of the park gate) and the vehicle's location information (GPS positioning to the park gate). Based on the corresponding initial scene information and driving intention, it can plan and generate a sequence of action planning consisting of multiple sub-tasks.
[0073] For example, the user inputs voice data as "Help me follow that red car in front." Based on visual data from outside the vehicle (such as recognizing the red car and the situation of vehicles in the adjacent lane) and information such as the vehicle's speed and direction of travel, VLM determines the initial following scenario for the vehicle. Based on the corresponding initial scenario information and driving intention, it can plan and generate a sequence of action planning consisting of multiple sub-tasks.
[0074] In the above implementation, the scene in which the mobile device is located before task planning can be determined by combining environmental perception data and device status data. By combining the corresponding initial scene information and driving intention, task planning and decomposition can be carried out, which can further improve the accuracy of the generated multiple sub-tasks.
[0075] In some implementations, when the VLM recognizes the user's driving intention, it can perform task rules to obtain a sequence of action planning consisting of multiple sub-tasks. During the execution of these multiple sub-tasks, the task rules can continue to be performed to adjust unexecuted sub-tasks or re-execute executed sub-tasks.
[0076] It should be noted that the embodiments of this application can also be applied to a variety of complex scenarios. For example, drop-off scenarios in office buildings, pick-up and drop-off in shopping mall underground parking lots, highway service areas, following other vehicles, avoiding other vehicles, and multi-destination pick-up and drop-off scenarios. Taking the drop-off scenario in office buildings as an example, the user inputs voice data "Take me to the coffee shop in Building 12, and then find a parking space myself." Through VLM (Visual Management Model), the intent recognition can identify the action type, including navigation and parking, and the physical space preference, including Building 12, the coffee shop entrance, and parking spaces. It can also be broken down into multiple executable sub-tasks, such as: 1. Drive to the park entrance; 2. Enter the main road of the park and drive to the coffee shop in Building 12; 3. Park on the side of the road in front of the coffee shop in Building 12; 4. After the user gets out of the car, drive to the underground parking lot; 5. Search for available parking spaces in the underground parking lot; 6. Automatically park in the parking space; 8. Engage the P gear and remotely send a prompt message to the user's mobile phone. Task sample data for different scenarios can also be added to the task template library according to actual needs, so that VLM has the ability to plan and decompose tasks in that scenario. Different scenarios share the same model, which improves the applicability of the solution.
[0077] Step 250: Drive the movement of the mobile device based on the motion planning sequence.
[0078] The mobile device can control its movement based on the task information corresponding to the multiple sub-tasks contained in the generated motion planning sequence, so as to execute the multiple sub-tasks in sequence and finally complete the driving intention.
[0079] In this embodiment, the mobile device can accurately identify the physical space preferences and action types in the user's input voice data, anchoring the physical space preferences to the physical environment of the environmental perception data. This allows for the mapping of ambiguous voice content to the actual physical environment, thereby enabling the control of the mobile device using the user's input voice data and enriching the application scenarios of voice interaction control. Furthermore, the driving intent corresponding to the voice data can be combined with environmental perception data, device status data, etc., breaking down the driving intent into multiple executable sub-tasks, improving the accuracy of controlling the mobile device. In addition, breaking down the driving intent into multiple executable sub-tasks allows each sub-task to be interpreted, monitored, and rolled back, improving the flexibility of controlling the mobile device and increasing the success rate of completing the driving intent.
[0080] In some embodiments, such as Figure 3 As shown, after generating an action planning sequence consisting of multiple subtasks, the method may further include the following steps: Step 302: Determine whether the constraints corresponding to the action planning sequence are met. If yes, proceed to step 304; otherwise, proceed to step 306.
[0081] Constraints refer to the execution boundaries of a mobile device's action planning sequence. They are restrictions on the action planning sequence and can be used to define the execution criteria for the sequence, such as under what circumstances the sequence is not allowed to execute, what operations are allowed in subtasks, and what operations are not allowed in subtasks. The mobile device can determine whether the constraints corresponding to the action planning sequence are met, thereby determining whether each subtask within the sequence can be executed normally.
[0082] The constraints may include, but are not limited to, one or more of the following: (1) The current device capabilities of the mobile device meet the requirements for executing the action planning sequence.
[0083] When executing the various subtasks contained in the action planning sequence, the mobile device must have the ability to execute each subtask. For example, if the subtask is to search for available parking spaces, the mobile device must have the ability to search for available parking spaces; if the subtask is to automatically park in a parking space, the mobile device must have the ability to automatically park (e.g., it needs to enable the automatic parking mode), and so on.
[0084] If the current device capabilities of the mobile device do not meet the requirements for executing the action planning sequence, i.e., the constraints are not met, it can be determined that there are subtasks in the action planning sequence that cannot be executed.
[0085] (2) All sub-tasks comply with the preset driving regulations or preset safety regulations.
[0086] Optionally, preset driving rules can be used to define the conditions for automatic driving, such as road type, speed range, weather, lighting conditions, and traffic scenarios. Preset safety rules can be used to define the conditions for driving safety, such as traffic rules and driving operation procedures.
[0087] If any subtask in the motion planning sequence does not comply with the preset driving rules or preset safety rules, i.e. does not meet the constraints, then it can be determined that there is a subtask in the motion planning sequence that cannot be executed.
[0088] (3) The total expected execution time corresponding to the action planning sequence is less than or equal to the time threshold.
[0089] When performing task planning in VLM, the estimated execution time for each subtask can be predicted, and the estimated execution times for each subtask within the action planning sequence can be summed to obtain the total estimated execution time. It can then be determined whether the total estimated execution time is less than or equal to a time threshold. If the total estimated execution time is less than or equal to the time threshold, it indicates sufficient time budget, and the action planning sequence can execute normally. If the total estimated execution time is greater than the time threshold, it indicates insufficient time budget, failure to meet constraints, and that some subtasks in the action planning sequence cannot be completed.
[0090] Optionally, the time threshold can be a preset time; alternatively, if the user's input voice data contains expected time information (which can be used to characterize the time the user expects to complete the driving intention), the time threshold can be determined based on the expected time information.
[0091] (4) The driving permission information obtained by the mobile device meets the requirements of the execution action planning sequence.
[0092] Driving permission information can refer to the permissions that a mobile device has to drive in a certain geographical area, such as driving permission in a certain city (e.g., whether the vehicle is subject to traffic restrictions in the city), driving permission in a certain park (e.g., whether the vehicle is an outside vehicle in the park and whether it has driving rights), driving permission on a certain road, etc.
[0093] After VLM performs task planning, it can determine the permissions required to execute the action planning sequence and check whether the driving permission information obtained by the mobile device satisfies the permission requirements for executing the action planning sequence. For example, in the scenario of "finding a parking space near the elevator in B2", the rules have 7 sub-tasks, and the required permissions are the permission for the vehicle to enter the park and the permission for the vehicle to park in the underground garage. If the driving permission information obtained by the mobile device does not satisfy the permission requirements for executing the action planning sequence, that is, the constraint conditions are not met, it can be determined that the action planning sequence cannot be executed.
[0094] Step 304: Drive the movement of the mobile device based on the motion planning sequence.
[0095] Step 306: Re-plan and generate an action planning sequence consisting of multiple sub-tasks, or output target prompt information, which is used to indicate that the current driving intention cannot be completed.
[0096] If the constraints corresponding to the action planning sequence are not met, it means that the action planning sequence generated by the current rule cannot be executed successfully. In this case, the action planning sequence consisting of multiple sub-tasks can be re-planned and generated. Then, it is determined whether the re-generated action planning sequence meets the constraints. Then, steps 302 to 306 are repeated until the generated action planning sequence meets the constraints.
[0097] In some implementations, a maximum number of rule attempts can be set. If the number of times the action planning sequence is generated reaches the maximum number of rule attempts and the constraint conditions are still not met, the re-planning and generation of the action planning sequence can be stopped. This avoids endless task planning for driving intentions that cannot be completed and reduces resource consumption.
[0098] Optionally, if the generated action planning sequence does not meet the constraints, or if the constraints are still not met even after generating the maximum number of action planning sequences, a target prompt message can be output to inform the user that the current driving intention cannot be completed. For example, the target prompt message can be output via voice, interface, etc., and is not limited here. The user can re-enter voice data to adjust the driving intention.
[0099] In the above implementation, before executing the action planning sequence, it can be determined whether the action planning sequence meets the constraints. The action planning sequence is only executed if the constraints are met, so as to ensure that multiple sub-tasks in the action planning sequence can be successfully executed, thereby improving the success rate of fulfilling the user's driving intentions.
[0100] like Figure 4 As shown, in some embodiments, another motion control method is provided, which may include the following steps: Step 402: Acquire multi-source sensing data; wherein, the multi-source sensing data includes: voice data, environmental sensing data around the mobile device, and device status data.
[0101] Step 404: Input the voice data, environmental perception data, and device status data into the VLM respectively.
[0102] Step 406: Perform intent recognition on the voice data using VLM to obtain the driving intent, which includes physical space preferences and action types.
[0103] Step 408: Using VLM based on environmental perception data and device status data, the physical space preference is anchored to the physical environment of the environmental perception data to obtain the anchoring result.
[0104] Step 410: Based on the anchoring results, environmental perception data, device status data and action type, VLM plans and generates an action planning sequence consisting of multiple sub-tasks.
[0105] It should be noted that the descriptions of steps 402 to 410 can be found in the relevant descriptions in the above embodiments, and will not be repeated here.
[0106] In some embodiments, during the execution of multiple subtasks, the mobile device can input real-time environmental awareness data and real-time device status data into the VLM, and the VLM can adjust the unexecuted subtasks based on the real-time environmental awareness data and real-time device status data.
[0107] During the execution of multiple subtasks, the mobile device can acquire environmental perception data and device status data in real time, and input these data into the VLM (Virtual Machine Library). Optionally, the real-time environmental perception data and device status data can be encoded separately to obtain corresponding feature vectors, which are then input into the VLM. The VLM can analyze the feature vectors corresponding to the real-time environmental perception data and device status data, and determine whether adjustments are needed for unexecuted subtasks.
[0108] Adjusting unexecuted subtasks may include, but is not limited to, deleting one or more unexecuted subtasks, adding one or more unexecuted subtasks to the action planning sequence, or modifying the task information of one or more subtasks.
[0109] For example, if the user inputs the voice data "Help me find a parking space near the elevator in B2", and the VLM determines the target of the 5th subtask as parking space A near the B2 elevator hall through the parking space status obtained by the Internet of Things when initially planning the task, but when the vehicle drives near parking space A, the VLM detects that parking space A is occupied based on visual data, then it can search for the nearest available parking space to parking space A as the new task target and execute the 5th subtask.
[0110] For example, if the user inputs the voice data "Take me to the coffee shop in Building 12, and then find a parking space myself", VLM initially has a fourth subtask (after the user gets out of the car, drive to the underground parking lot) and a fifth subtask (search for a parking space in the underground parking lot). However, when the vehicle arrives at the coffee shop, visual data detects that there is a parking space available at the coffee shop entrance, so the fourth and fifth subtasks can be deleted, and the sixth subtask can be executed directly.
[0111] In the above implementation, during the execution of multiple sub-tasks, unexecuted sub-tasks can be dynamically adjusted based on real-time environmental perception data and real-time device status data, thereby improving the flexibility and accuracy of task planning.
[0112] Step 412: Input the task execution instructions corresponding to the target subtask to be executed into the VLA model.
[0113] Step 414: Generate a driving action sequence based on the task execution instructions using the VLA model. The driving action sequence includes one or more driving actions.
[0114] The VLM generates a sequence of action plans consisting of multiple subtasks, which can then be driven by the VLA (Vision-Language-Action) model to move the mobile device. VLA can be a model that directly maps visual data and language commands to executable physical actions. In this embodiment, most of the multi-source perception data analysis can be performed within the VLM, while VLA is responsible for mapping each subtask to specific driving actions of the mobile device.
[0115] The VLA model can input task execution instructions corresponding to the target subtask to be executed. These instructions are structured instructions that the VLA model can recognize. The VLA model can recognize the input task execution instructions and transform them into a series of driving actions that the mobile device can execute, generating a driving action sequence. It should be noted that the driving actions transformed by VLA are more granular than the execution actions indicated by the execution parameters of the subtask. For example, the execution actions indicated by the execution parameters of the subtask are driving, while the driving actions that the mobile device can execute can be specific actions performed by various components / functional modules of the mobile device (such as acceleration, deceleration, left turn, right turn, changing lanes to the left, etc.).
[0116] Step 416: Based on the various driving actions included in the driving action sequence, control the movement of the mobile device sequentially.
[0117] VLA can control the movement of a mobile device sequentially based on the various driving actions included in the driving action sequence. Furthermore, VLA can sequentially output control commands corresponding to each driving action, and control the movement of the mobile device according to the control commands.
[0118] Optionally, during the execution of any subtask corresponding to the driving action sequence by the VLA, real-time environmental perception data and equipment status data can also be input into the VLA. The VLA can adjust the driving actions in the driving action sequence based on the real-time environmental perception data and equipment status data to improve the accuracy of executing the subtask.
[0119] In some embodiments, auxiliary information (such as user gesture information, head posture, and / or gaze area information) obtained based on real-time internal visual data of the mobile device can be input into the VLA. The VLA can generate a sequence of driving actions based on the task execution instructions and the auxiliary information, or it can adjust the driving actions in the sequence. For example, a set of basic gestures can be predefined, such as a palm for forward movement, a fist for stop, and a circle for U-turn. The VLA can generate corresponding control instructions for driving actions based on the gestures performed by the user, and control the mobile device to perform the driving actions according to the control instructions. Optionally, the basic gestures can also be customized by the user, which is not limited here. This approach can improve the flexibility of interaction.
[0120] In some embodiments, the method further includes: upon completion of the first subtask, obtaining the execution result of the first subtask; if the execution result of the first subtask deviates from the task objective of the first subtask, then re-planning and generating an action planning sequence based on the anchoring result, current environmental perception data, current device status data, and action type. The first subtask is any one of the plurality of subtasks.
[0121] After each subtask is executed, the VLA model can report the execution result to the VLM. The VLM can then use the VLA's feedback to determine if the result deviates from the subtask's objective. If a deviation exists, it indicates that the subtask was not executed successfully, and task planning can be performed again.
[0122] Optionally, the execution result may include information indicating whether the execution was successful or failed, and may also include reference information used to determine the execution result, such as the location information of the mobile device and surrounding environmental information (e.g., lane information, road condition information, etc.). For example, if the task objective of a subtask is to park a vehicle in parking space A, but the VLA detects that parking space A is occupied when executing the subtask, it can report the execution failure result to the VLM. As another example, if the task objective of a subtask is to pass through the gate's turnstile, the VLA can send its relative position information to the gate to the VLM after completing the subtask, and the VLM can use this information to determine whether the execution was successful.
[0123] For example, Figure 5 This is a flowchart illustrating a motion control method in one embodiment. Figure 5 As shown, mobile devices can encode and input multi-source perception data into the VLM. After the VLM performs intent recognition on the voice data, it combines environmental perception data and device status data to perform scene anchoring, and decomposes the driving intent into multiple sub-tasks based on the anchoring results. After the VLM performs constraint verification on multiple sub-tasks, it can send each sub-task to the VLA step by step. The VLA executes each sub-task and sends the execution results back to the VLM. The VLM further determines whether the task needs to be replanned based on the results sent back by the VLA.
[0124] In some implementations, VLM replanning can be partial, such as replanning subtasks whose execution results deviate significantly from the task objective, or replanning one or more unexecuted subtasks. Optionally, VLM replanning can also be global, whereby, upon detecting a deviation between the execution result of any subtask and its task objective, task planning can be recalculated based on scene information and anchoring results of the mobile device's current environment, generating an action planning sequence that includes multiple subtasks.
[0125] Optionally, if the execution result of a subtask deviates significantly from the task objective, the VLM can also choose to roll back the subtask. In this case, the VLM will re-enter the task execution instruction for the subtask into the VLA, and the VLA will re-execute the subtask.
[0126] In the above implementation, if the execution result of the subtask deviates significantly from the task objective, the task can be replanned, which improves the accuracy of task planning and the accuracy of controlling the mobile device, ensuring that the user's driving intentions can be fulfilled.
[0127] In the embodiments of this application, compared with the end-to-end VLA model, most of the analysis and processing of multi-source sensing data can be placed in VLM, which can improve the accuracy of task planning. Furthermore, VLM can be used to decompose tasks, which not only retains the execution capability of VLA, but also retains the interpretability and safety supervision of sub-tasks.
[0128] like Figure 6 As shown, after the VLM generates an action planning sequence consisting of multiple subtasks, the above method may further include the following steps: Step 602: Send task information for one or more subtasks to the task planner via VLM.
[0129] Since the VLM lacks the ability to execute subtasks, in this embodiment, a task planner can be used to execute multiple subtasks planned by the VLM. The task planner can be located between the VLM and VLA, and plays a role in managing and executing the subtasks planned by the VLM. For example, the task planner can be responsible for confirming the execution time, execution method, and whether the subtask is executable, and can also monitor the execution status of each subtask.
[0130] After the VLM generates the action planning sequence, it can send task information for one or more subtasks to the task planner. Optionally, the VLM can send the task information for each subtask to the task planner one by one. For example, the VLM can send the task information for the next subtask to the task planner only after obtaining the execution result of the previously completed subtask. Optionally, the VLM can also send the task information for all subtasks to the task planner at once. For example, after the VLM generates the action planning sequence, it can directly send the action planning sequence to the task planner.
[0131] Step 604: The task planner determines the target subtask to be executed based on the task information of one or more subtasks.
[0132] Step 606: Send the task execution instructions corresponding to the target subtask to the VLA model through the task planner.
[0133] The task planner can determine the target subtask to be executed based on the task information of each subtask issued by the VLM. For example, if the VLM sends the task information of each subtask to the task planner one by one, the subtask corresponding to the task information sent this time is the target subtask to be executed. Or, if the VLM sends the task information of all subtasks to the task planner at once, the task planner can determine the target subtask to be executed based on the order of each subtask in the action planning sequence. After determining the target subtask to be executed, the task planner can generate the task execution instruction corresponding to the target subtask and send the task execution instruction to the VLA.
[0134] In some embodiments, the task information of a subtask may include preconditions corresponding to the subtask, which may refer to the conditions that trigger the start of execution of the subtask. If the task planner determines that the preconditions corresponding to the target subtask are met, it sends the task execution instruction corresponding to the target subtask to the VLA model.
[0135] The prerequisites include, but are not limited to, one or more of the following: The mobile device reaches the starting position corresponding to the target subtask; The target markers corresponding to the target sub-tasks were detected based on environmental perception data.
[0136] For example, the subtask of exiting the ramp requires the prerequisite of identifying the underground parking garage entrance sign based on environmental perception data; the subtask of searching for parking spaces requires the prerequisite of identifying parking space lines based on environmental perception data.
[0137] Optionally, the VLM can determine whether the preconditions of the target subtask are met based on real-time environmental perception data and / or real-time device status data. If they are met, it sends an acknowledgment signal to the task planner. When the task planner receives the acknowledgment signal from the VLM, it can determine that the preconditions of the target subtask are met and send the task execution instruction corresponding to the target subtask to the VLA model.
[0138] Optionally, the VLM can analyze real-time environmental perception data and / or real-time device status data to obtain analysis results, and then send these results to the task planner. For example, the analysis results may include the location information of the mobile device, markers identified based on the environmental perception data, etc. The task planner can determine whether the preconditions for the target sub-task are met based on the analysis results sent by the VLM. If so, the task planner sends the task execution instructions corresponding to the target sub-task to the VLA model.
[0139] In the above implementation, before the task planner sends the task execution instruction of the subtask to the VLA, it can first determine whether the preconditions of the subtask are met. Only when the preconditions of the subtask are met will the task planner send the task execution instruction of the subtask to the VLA, which can ensure the orderly and correct execution of each subtask.
[0140] In some embodiments, task information may include the confidence level and degradation strategy corresponding to the subtask. The confidence level characterizes the probability of the subtask being executed correctly and safely; a higher confidence level indicates a high probability of correct and safe execution, while a lower confidence level indicates a low probability of correct and safe execution. The task planner can determine whether the confidence level of the target subtask is lower than a preset confidence threshold. If the confidence level of the target subtask is lower than the preset confidence threshold, the task planner executes a degradation strategy.
[0141] A degradation strategy can refer to a strategy that ensures mobile device security and / or improves the success rate of subtask execution. In some embodiments, a degradation strategy may include one or more of the following degradation operations: The voice output module controls the output of interactive voice, which is used to confirm the driving intention and / or the task objectives of the sub-task to the user. Pause execution of the target subtask; Control the mobile device to stop moving.
[0142] When the confidence level of a subtask is low, the task planner can control the voice output module to output interactive voice. This interactive voice can be used to further confirm the user's driving intentions and / or the task objectives of the subtask, such as "Do you mean the open space next to that big tree on the right?" or "Do you mean you want to park the car in the first parking space on the left?" The user may reconfirm through gestures, voice, or other interactive methods to avoid misoperation and improve the accuracy of subtask execution.
[0143] It should be noted that information confirming the user's driving intentions and / or the task objectives of sub-tasks can also be output through the interface, and is not limited here.
[0144] In some embodiments, the degradation strategy may include multiple degradation operations, each corresponding to a different confidence interval. When the confidence level of the target subtask is lower than a preset confidence threshold, the task planner can determine the target confidence interval to which the confidence level of the target subtask belongs, and execute the degradation operation corresponding to the target confidence interval through the task planner.
[0145] Optionally, multiple degradation operations correspond to different confidence intervals. The smaller the maximum / minimum value of the confidence interval (i.e., the smaller the value contained in the confidence interval), the higher the security of the corresponding degradation operation. For example, the maximum value of the confidence interval corresponding to controlling the voice output module to output interactive voice is greater than the maximum value of the confidence interval corresponding to pausing the execution of the target subtask, and the maximum value of the confidence interval corresponding to pausing the execution of the target subtask is greater than the maximum value of the confidence interval corresponding to controlling the mobile device to stop moving. By executing different degradation operations in a tiered manner, appropriate degradation operations can be flexibly selected, ensuring the security of the mobile device while avoiding unnecessary impact on the user's operation of the mobile device, thus improving the user experience.
[0146] In the above implementation, when the confidence level of a subtask is too low, the task planner automatically executes a degradation strategy, thereby ensuring the security of the mobile device.
[0147] Step 608: Generate a driving action sequence based on the task execution instructions using the VLA model. The driving action sequence includes one or more driving actions.
[0148] Step 610: Based on the various driving actions included in the driving action sequence, control the movement of the mobile device sequentially.
[0149] It should be noted that the descriptions of steps 608 to 610 can be found in the relevant descriptions in the above embodiments, and will not be repeated here.
[0150] Optionally, during the execution of the target sub-task by VLA, the real-time status of the mobile device, such as real-time driving speed, driving direction, and location information, can be fed back to the task planner. The task planner can then monitor the execution of the target sub-task based on the real-time status fed back by VLA.
[0151] For example, Figure 7 This is a system architecture diagram of a motion control method in one embodiment. For example... Figure 7As shown, the system collects in-cabin visual data, out-of-cabin visual data, voice data, and equipment status data from the vehicle. The multimodal data is encoded to achieve semantic alignment before being input into the Vehicle Management Module (VLM). The VLM performs intent recognition and CoT task inference to obtain multiple sub-tasks. The VLM can then send task information for these sub-tasks to the task planner. The task planner can construct a long-term skill graph and / or a DAG (Directed Acyclic Graph) for each sub-task. Based on the long-term skill graph and the DAG, execution instructions for each sub-task are generated and sent to the Vehicle Automation Module (VLA). The long-term skill graph describes the functional units required to execute multiple sub-tasks, while the DAG defines the execution order and timing (e.g., preconditions) of each sub-task.
[0152] After receiving the task execution instruction, VLA performs trajectory planning and vehicle driving control, executes various sub-tasks, and feeds back the real-time status to the task rule generator. It can also feed back the execution results to VLM.
[0153] In this embodiment, by cooperating with the VLM, task planner and VLA, the user's vague or abstract intentions can be transformed into multiple sub-tasks that the mobile device can execute, and the mobile device can be accurately controlled to execute each sub-task, which improves the accuracy of controlling the mobile device. Moreover, each sub-task can be interpreted, monitored and rolled back, which improves the flexibility of controlling the mobile device and increases the success rate of completing driving intentions.
[0154] like Figure 8 As shown, in one embodiment, a motion control device 800 is provided, which can be applied to the above-mentioned mobile device. The motion control device 800 includes a data acquisition module 810, an intent recognition module 820, an anchoring module 830, a planning module 840, and a driving module 850.
[0155] The data acquisition module 810 is used for multi-source sensing data; the multi-source sensing data includes: voice data, environmental sensing data around the mobile device, and device status data.
[0156] The intent recognition module 820 is used to recognize the intent of the voice data to obtain the driving intent, which includes physical space preferences and action types.
[0157] Anchoring module 830 is used to anchor physical space preferences to the physical environment of environmental perception data to obtain anchoring results.
[0158] The planning module 840 is used to plan and generate an action planning sequence consisting of multiple sub-tasks based on anchoring results, environmental perception data, equipment status data, and action types.
[0159] The drive module 850 is used to drive the movement of a mobile device based on a sequence of motion planning.
[0160] In some embodiments, the planning module 840 is further configured to determine initial scene information based on environmental perception data and device status data, wherein the initial scene information describes the scene in which the mobile device is located before generating the action planning sequence; and to plan and generate an action planning sequence consisting of multiple sub-tasks based on the initial scene information, anchoring results and action type.
[0161] In some embodiments, the motion control device 800 further includes a determination module.
[0162] The judgment module is used to determine whether the constraints corresponding to the action planning sequence are met.
[0163] The drive module 850 is also used to drive the mobile device to move based on the action planning sequence if the constraints are met.
[0164] The constraints include one or more of the following: The current device capabilities of the mobile device meet the requirements for executing the action planning sequence. All sub-tasks comply with preset driving regulations or preset safety regulations; The total estimated execution time corresponding to the action planning sequence is less than or equal to the time threshold; The driving permission information obtained by the mobile device satisfies the permissions required to execute the action planning sequence.
[0165] In some embodiments, the intent recognition module 820 is further configured to input voice data into the VLM, and perform intent recognition on the voice data through the VLM to obtain the driving intent.
[0166] The anchoring module 830 is also used to input environmental perception data and device status data into the VLM, and through the VLM, anchor the physical space preference to the physical environment of the environmental perception data based on the environmental perception data to obtain the anchoring result.
[0167] In some embodiments, the planning module 840 is further configured to input real-time environmental perception data and real-time device status data into the VLM during the execution of multiple sub-tasks; and adjust the unexecuted sub-tasks by the VLM based on the real-time environmental perception data and real-time device status data.
[0168] In some embodiments, the drive module 850 includes an instruction input unit, an action generation unit, and a control unit.
[0169] The instruction input unit is used to input the task execution instructions corresponding to the target subtask to be executed into the VLA model.
[0170] The action generation unit is used to generate a sequence of driving actions based on the task execution instructions through the VLA model. The sequence of driving actions includes one or more driving actions.
[0171] The control unit is used to sequentially control the movement of the mobile device based on the various driving actions included in the driving action sequence.
[0172] In some embodiments, the motion control device 800 further includes a task determination module.
[0173] The task determination module is used to send task information of one or more subtasks to the task planner via VLM; the task planner then determines the target subtask to be executed based on the task information of one or more subtasks.
[0174] The instruction input unit is also used to send task execution instructions corresponding to the target subtask to the VLA model through the task planner.
[0175] In some embodiments, task information includes prerequisites for subtasks.
[0176] The instruction input unit is also used to send the task execution instruction corresponding to the target subtask to the VLA model when the task planner determines that the preconditions corresponding to the target subtask are met. The prerequisites include one or more of the following: The mobile device reaches the starting position corresponding to the target subtask; The target markers corresponding to the target sub-tasks were detected based on environmental perception data.
[0177] In some embodiments, the task information includes the confidence level and degradation strategy corresponding to the subtask. The motion control device 800 also includes a degradation module.
[0178] The degradation module is used to execute a degradation strategy through the task planner if the confidence level of the target subtask is lower than a preset confidence threshold.
[0179] In some embodiments, the degradation strategy includes one or more of the following degradation operations: The voice output module controls the output of interactive voice, which is used to confirm the driving intention and / or the task objectives of the sub-task to the user. Pause execution of the target subtask; Control the mobile device to stop moving.
[0180] In some embodiments, the degradation strategy includes multiple degradation operations. The degradation module is further configured to determine the target confidence interval to which the confidence level corresponding to the target subtask belongs, and execute the degradation operation corresponding to the target confidence interval through the task planner.
[0181] In some embodiments, the motion control device 800 further includes a feedback module.
[0182] The feedback module is used to obtain the execution result of the first subtask after it has been completed; the first subtask can be any one of multiple subtasks.
[0183] The planning module 840 is also used to re-plan and generate an action planning sequence based on the anchoring result, the current environmental perception data, the current device status data, and the action type if there is a deviation between the execution result of the first sub-task and the task objective of the first sub-task.
[0184] In some embodiments, device status data includes one or more of the following: driving speed, driving direction, location information, map information stored on the mobile device, and driving permission information obtained by the mobile device.
[0185] Figure 9 This is a structural block diagram of a mobile device in one embodiment. For example... Figure 9 As shown, the mobile device 900 may include one or more of the following components: a processor 910 and a memory 920 coupled to the processor 910, wherein the memory 920 may store one or more computer programs, which may be configured to implement the methods described in the above embodiments when executed by one or more processors 910.
[0186] The processor 910 may include one or more processing cores. The processor 910 connects to various parts within the mobile device 900 using various interfaces and lines, and performs various functions and processes data of the mobile device 900 by running or executing instructions, programs, code sets, or instruction sets stored in the memory 920, and by calling data stored in the memory 920. Optionally, the processor 910 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor 910 may integrate one or a combination of several of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the displayed content; and the modem handles wireless communication. It is understood that the modem may also not be integrated into the processor 910 and may be implemented separately using a communication chip.
[0187] The memory 920 may include random access memory (RAM) or read-only memory (ROM). The memory 920 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 920 may include a program storage area and a data storage area. The program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as touch functionality, sound playback functionality, image playback functionality, etc.), and instructions for implementing the various method embodiments described above. The data storage area may also store data created by the mobile device 900 during use.
[0188] Understandably, the mobile device 900 may include more or fewer structural elements than those shown in the block diagram above, such as radio frequency modules, power modules, physical buttons, Wi-Fi (Wireless Fidelity) modules, speakers, Bluetooth modules, sensors, etc., and may not be limited herein.
[0189] This application discloses an electronic device, including a memory and a processor. The memory stores a computer program, which, when executed by the processor, causes the electronic device to perform the methods described in the above embodiments.
[0190] This application discloses a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the methods described in the above embodiments.
[0191] This application discloses a computer program product, which includes a computer program, and when the computer program is executed by a processor in a mobile device, it enables the mobile device to implement the methods described in the above embodiments.
[0192] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, ROM, etc.
[0193] Any references to memory, storage, databases, or other media used herein may include non-volatile and / or volatile memory. Suitable non-volatile memory may include ROM, Programmable ROM (PROM), Erasable PROM (EPROM), Electrically Erasable PROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM), which is used as an external cache memory. By way of illustration and not limitation, RAM may take many forms, such as Static RAM (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchlink DRAM (SLDRAM), Rambus DRAM (RDRAM), and Direct Rambus DRAM (DRDRAM).
[0194] It should be understood that the phrase "one embodiment" or "an embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Those skilled in the art should also recognize that the embodiments described in the specification are optional embodiments, and the actions and modules involved are not necessarily essential to this application.
[0195] In the various embodiments of this application, it should be understood that the sequence number of each process does not necessarily imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.
[0196] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; they can be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.
[0197] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.
[0198] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0199] The foregoing has provided a detailed description of a motion control method, apparatus, and mobile device disclosed in the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and its core ideas. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.
Claims
1. A motion control method characterized by, Applied to mobile devices, the method includes: Acquire multi-source sensing data; wherein, the multi-source sensing data includes: voice data, environmental sensing data around the mobile device, and device status data; The voice data is subjected to intent recognition to obtain the driving intent, which includes physical space preferences and action types; The physical space preference is anchored to the physical environment of the environmental perception data to obtain the anchoring result; Based on the anchoring results, the environmental perception data, the device status data, and the action type, an action planning sequence consisting of multiple sub-tasks is planned and generated. The mobile device is driven to move based on the motion planning sequence.
2. The method of claim 1, wherein, Based on the anchoring result, the environmental perception data, the device status data, and the action type, an action planning sequence consisting of multiple sub-tasks is planned and generated, including: Based on the environmental perception data and the device status data, initial scene information is determined, which describes the scene in which the mobile device is located before generating the action planning sequence; Based on the initial scene information, the anchoring result, and the action type, an action planning sequence consisting of multiple sub-tasks is planned and generated.
3. The method according to claim 1, characterized in that, After the planning generates an action planning sequence consisting of multiple subtasks, the method further includes: Determine whether the constraints corresponding to the action planning sequence are met; If the constraints are met, then the step of driving the mobile device to move based on the action planning sequence is executed; The constraints include one or more of the following: The current device capabilities of the mobile device meet the requirements for executing the action planning sequence. All of the sub-tasks comply with preset driving regulations or preset safety regulations; The total estimated execution time corresponding to the action planning sequence is less than or equal to the time threshold. The driving permission information obtained by the mobile device satisfies the permission required to execute the action planning sequence.
4. The method according to any one of claims 1 to 3, characterized in that, The process of performing intent recognition on the voice data to obtain the driving intent includes: The speech data is input into the visual speech model (VLM), and the VLM performs intent recognition on the speech data to obtain the driving intent. The step of anchoring the physical space preference to the physical environment of the environmental perception data to obtain the anchoring result includes: The environmental perception data and device status data are input into the VLM. Based on the environmental perception data and device status data, the VLM anchors the physical space preference to the physical environment of the environmental perception data to obtain the anchoring result.
5. The method according to claim 4, characterized in that, The method further includes: During the execution of the multiple sub-tasks, real-time environmental perception data and real-time device status data are input into the VLM; The VLM adjusts unexecuted subtasks based on real-time environmental perception data and real-time device status data.
6. The method according to claim 4, characterized in that, The step of driving the mobile device to move based on the action planning sequence includes: Input the task execution instructions corresponding to the target subtask to be executed into the Visual Language Action (VLA) model; The VLA model generates a sequence of driving actions based on the task execution instructions, the sequence of driving actions including one or more driving actions; Based on the various driving actions included in the driving action sequence, the movement of the mobile device is controlled sequentially.
7. The method according to claim 6, characterized in that, Before inputting the task execution instruction corresponding to the target subtask to be executed into the Visual Language Action (VLA) model, the method further includes: The VLM sends task information for one or more subtasks to the task planner. The task planner determines the target subtask to be executed based on the task information of the one or more subtasks. The step of inputting the task execution instructions corresponding to the target subtask to be executed into the Visual Language Action (VLA) model includes: The task planner sends the task execution instructions corresponding to the target subtask to the VLA model.
8. The method according to claim 7, characterized in that, The task information includes the preconditions corresponding to the subtask; sending the task execution instruction corresponding to the target subtask to the VLA model through the task planner includes: If the task planner determines that the preconditions corresponding to the target subtask are met, it sends the task execution instruction corresponding to the target subtask to the VLA model. The preconditions include one or more of the following: The mobile device arrives at the starting position corresponding to the target subtask; The target markers corresponding to the target sub-task were detected based on environmental perception data.
9. The method according to claim 7, characterized in that, The task information includes the confidence level and degradation strategy corresponding to the sub-task; After determining the target subtask to be executed, the method further includes: If the confidence level of the target subtask is lower than the preset confidence threshold, the degradation strategy is executed through the task planner.
10. The method according to claim 9, characterized in that, The degradation strategy includes one or more of the following degradation operations: The voice output module is controlled to output interactive voice, which is used to confirm the driving intention and / or the task objective of the sub-task to the user. Pause the execution of the target subtask; Control the mobile device to stop moving.
11. The method according to claim 9 or 10, characterized in that, The downgrade strategy includes multiple downgrade operations, each of which corresponds to a different confidence interval. The execution of the degradation strategy through the task planner includes: Determine the target confidence interval to which the confidence level corresponding to the target subtask belongs, and execute the downgrade operation corresponding to the target confidence interval through the task planner.
12. The method according to any one of claims 1 to 3, characterized in that, The method further includes: If the first subtask is completed, obtain the execution result of the first subtask; the first subtask is any one of the plurality of subtasks. If the execution result of the first subtask deviates from the task objective of the first subtask, then the action planning sequence is re-planned and generated based on the anchoring result, the current environmental perception data, the current device status data, and the action type.
13. A motion control device, characterized in that, Applied to a mobile device, the device includes: The data acquisition module is used for multi-source sensing data; wherein, the multi-source sensing data includes: voice data, environmental sensing data around the mobile device, and device status data; An intent recognition module is used to perform intent recognition on the voice data to obtain driving intent, which includes physical space preferences and action types; An anchoring module is used to anchor the physical space preference to the physical environment of the environmental perception data to obtain the anchoring result; The planning module is used to plan and generate an action planning sequence consisting of multiple sub-tasks based on the anchoring result, the environmental perception data, the device status data, and the action type. A driving module is used to drive the mobile device to move based on the action planning sequence.
14. A mobile device, characterized in that, The device includes a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the mobile device performs the method as described in any one of claims 1 to 12.
