Data processing method and related apparatus therefor
By using language models for end-to-end data processing in autonomous driving and robot control systems, the problem of insufficient decision-making accuracy caused by decoupling of task modules in the prior art is solved, and the accuracy and flexibility of the control system are improved.
Patent Information
- Application Number
- PCT/CN2024/095786
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-06-01
- Filing Date
- 2024-05-28
- Publication Date
- 2024-12-05
AI Technical Summary
The decoupling design of each task module in the existing autonomous driving and robot control systems leads to poor decision-making accuracy, and the errors in the front-end module affect the decision-making results of downstream modules, and the control accuracy is insufficient.
Using an end-to-end control method based on language model, extract scene information from sensor data through the first language model, generate a second prompt text to query the information that needs to be known, and use the second language model to make decisions to realize sensor data and language model Interaction between them builds an end-to-end automatic control system.
It improves decision-making accuracy, uses the inductive reasoning ability of the language model to expand to the control field, realizes a unified representation of infinite perceptual needs, and enhances the flexibility and accuracy of the control system.
Smart Images

Figure CN2024095786_05122024_PF_FP_ABST
Abstract
Description
A data processing method and related device
[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office of China on June 1, 2023, with application number 202310645412.X and invention name “A data processing method and related devices thereof”, the entire contents of which are incorporated by reference into this application. Technical Field
[0002] The present application relates to the field of artificial intelligence, and in particular to a data processing method and related devices. Background Art
[0003] Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a branch of computer science that seeks to understand the essence of intelligence and develop new intelligent machines that can respond in a manner similar to human intelligence. AI also studies the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.
[0004] For robots or vehicles, when performing automatic control (for example, a robot automatically executes a task based on a user's task execution request, or a vehicle performs an autonomous driving task), three tasks are typically involved: perception, prediction, and decision-making. Taking autonomous driving as an example, the perception task involves receiving raw sensor data (such as images or lasers) as input and identifying dynamic targets (such as the size and position of pedestrians and vehicles) and static elements (such as lane lines and arrow signs on the ground) in the environment. The prediction task can receive the raw results of dynamic and static targets obtained by the perception task and predict the movement intentions of targets in the environment, mainly predicting the future movement trajectory and intentions of other vehicles. The decision-making task can receive the prediction results obtained by the prediction task, and based on the future driving intentions of other vehicles and the navigation task of the own vehicle, generate a decision conclusion, output a control signal, and control the vehicle for autonomous driving.
[0005] However, the existing architecture design decouples the functions of each task module from each other. They maintain a clear upstream and downstream relationship. Each module's output interface is predefined, allowing for serial connection. This approach offers the advantages of high interpretability and ease of modular development and maintenance. However, due to the decoupling of front-end and back-end modules, errors in the front-end module can also lead to erroneous decisions in downstream modules, resulting in poor control accuracy.
[0006] Summary of the Invention
[0007] This application provides a data processing method that can improve decision-making accuracy.
[0008] In a first aspect, the present application provides a data processing method, comprising: obtaining scene information through a first language model based on sensor data and a first prompt text; the sensor data is data collected from the surrounding environment of a target object, and the first prompt text indicates extracting scene information of the scene where the target object is located based on the sensor data; the target object is the vehicle itself or the robot itself; generating a second prompt text through a second language model based on the scene information and the current execution target of the target object, the second prompt text indicating a query for information that the target object needs to know when completing the execution target and is not included in the scene information; obtaining a reply text corresponding to the second prompt text through the first language model based on the sensor data and the second prompt text; and obtaining a control instruction for the target object through the second language model based on the reply text, the scene information, and the current execution target.
[0009] In an embodiment of the present application, the second language model obtains demand information (second prompt text) in the form of a query text based on different scenario information, and the first language model obtains a reply text based on the query text, and the reply text is passed to the second language model for decision-making. On the one hand, an end-to-end automatic control system is constructed through the interaction between the first language model and the second language model. Compared with a multi-stage decision-making system, the present application can improve decision-making accuracy. On the other hand, the architecture introduces a language model, which can expand the inductive reasoning ability of the large language model to the control field. The text expression of the language model is different from the traditional fixed format and can represent an infinite number of perceptual needs, thereby unifying the method of obtaining conclusions of various types of perceptual needs of the second language model.
[0010] In a possible implementation, the scene information includes information indicating a scene category and / or information about perceptual elements included in the scene.
[0011] In a possible implementation, the target object is a vehicle, and the method further includes:
[0012] Receive navigation instructions from the user;
[0013] A current execution target of the target object is obtained according to the current position of the vehicle and the navigation instruction.
[0014] In a possible implementation, the target object is a robot; and the method further includes:
[0015] Receive task instructions from users;
[0016] The current execution target of the target object is obtained according to the current posture of the robot, scene map information and the task instruction.
[0017] In a possible implementation, the information that needs to be known and is not included in the scene information includes:
[0018] Status information of a sensory element associated with the execution target; or
[0019] The method for achieving the execution goal.
[0020] In a possible implementation, the first language model and the second language model are chatGPT, GPT4, or chatGLM.
[0021] In a second aspect, the present application provides a data processing device, the device comprising:
[0022] A processing module is configured to obtain scene information through a first language model based on sensor data and a first prompt text; the sensor data is data collected about the surrounding environment of a target object, and the first prompt text indicates scene information of a scene in which the target object is located, extracted based on the sensor data; the target object is the vehicle itself or the robot itself; a second prompt text is generated through a second language model based on the scene information and the current execution target of the target object, the second prompt text indicating a query for information that the target object needs to know when completing the execution target and is not included in the scene information; a reply text corresponding to the second prompt text is obtained through the first language model based on the sensor data and the second prompt text; and a control instruction for the target object is obtained through the second language model based on the reply text, the scene information, and the current execution target.
[0023] In a possible implementation, the target object is a vehicle or a robot.
[0024] In a possible implementation, the scene information includes information indicating a scene category and / or information about perceptual elements included in the scene.
[0025] In a possible implementation, the target object is a vehicle, and the apparatus further includes:
[0026] The acquisition module is used to receive the user's navigation instructions;
[0027] A current execution target of the target object is obtained according to the current position of the vehicle and the navigation instruction.
[0028] In a possible implementation, the target object is a robot; and the apparatus further includes:
[0029] The acquisition module is used to receive the task instructions from the user;
[0030] The processing module is further configured to obtain a current execution target of the target object based on the current posture of the robot, scene map information, and the task instruction.
[0031] In a possible implementation, the information that needs to be known and is not included in the scene information includes:
[0032] Status information of a sensory element associated with the execution target; or
[0033] The method for achieving the execution goal.
[0034] In a possible implementation, the first language model and the second language model are chatGPT, GPT4, or chatGLM.
[0035] In a third aspect, the present application provides a data processing device, which includes a memory and a processor; the memory stores code, and the processor is configured to execute the code. When the code is executed, the server executes the method as described in the first aspect or any possible implementation of the first aspect.
[0036] In a fourth aspect, the present application provides a vehicle comprising the data processing device according to the third aspect.
[0037] In a fifth aspect, the present application provides a robot, which includes the data processing device as described in the third aspect.
[0038] In a sixth aspect, the present application provides a computer storage medium storing a computer program, which, when executed by a computer, enables the computer to implement the method of the first aspect or any possible implementation of the first aspect.
[0039] In a seventh aspect, the present application provides a computer program product, which stores instructions. When the instructions are executed by a computer, the computer implements the method of the first aspect or any possible implementation of the first aspect.
[0040] In an eighth aspect, the present application provides a chip system, comprising a processor configured to support a data processing device in implementing the functions described in the aforementioned aspects, such as transmitting or processing data or information described in the aforementioned methods. In one possible design, the chip system further comprises a memory configured to store program instructions and data necessary for executing or training the device. The chip system may consist solely of a chip or may include a chip and other discrete components. BRIEF DESCRIPTION OF THE DRAWINGS
[0041] Figure 1a is a structural diagram of the artificial intelligence main framework;
[0042] Figure 1b is a schematic diagram of the structure of an automatic driving device;
[0043] Figure 1c is a schematic diagram of the structure of a robot;
[0044] Figure 1d is a schematic diagram of the structure of a system;
[0045] FIG2 is a flowchart of a data processing method provided in an embodiment of the present application;
[0046] FIG3 is a flowchart of a data processing method provided in an embodiment of the present application;
[0047] FIG4 is a schematic diagram of the structure of a data processing device provided in an embodiment of the present application;
[0048] FIG5 is a schematic diagram of a structure of an execution device provided in an embodiment of the present application;
[0049] FIG6 is a schematic diagram of the structure of a server provided in an embodiment of the present application;
[0050] FIG7 is a schematic diagram of the structure of a chip provided in an embodiment of the present application. DETAILED DESCRIPTION
[0051] The following describes the embodiments of the present invention in conjunction with the accompanying drawings. The terms used in the embodiments of the present invention are only used to explain the specific embodiments of the present invention, and are not intended to limit the present invention.
[0052] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.
[0053] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, and this is merely a way of distinguishing the objects of the same properties when describing them in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.
[0054] First, let's describe the overall workflow of an AI system. See Figure 1a, which shows a schematic diagram of the main AI framework. This AI framework will be explained from two perspectives: the "intelligent information chain" (horizontal axis) and the "IT value chain" (vertical axis). The "intelligent information chain" reflects the entire process from data acquisition to processing. For example, this process could be the general progression of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, and intelligent execution and output. Throughout this process, data undergoes a condensed journey from "data-information-knowledge-wisdom." The "IT value chain," spanning the underlying infrastructure of human intelligence, information (provided and processed by technology), and the system's industrial ecosystem, reflects the value that AI brings to the information technology industry.
[0055] (1) Infrastructure
[0056] Infrastructure provides computing power for AI systems, enabling communication with the outside world and supporting this through a foundational platform. External communication occurs through sensors; computing power is provided by intelligent chips (CPUs, NPUs, GPUs, ASICs, FPGAs, and other hardware accelerators). The foundational platform includes a distributed computing framework and network-related platform guarantees and support, including cloud storage and computing, and interconnected networks. For example, sensors communicate with the outside world to acquire data, which is then fed into the intelligent chips within the distributed computing system provided by the foundational platform for computation.
[0057] (2) Data
[0058] Data above the infrastructure layer represents data sources for AI. This data includes graphics, images, voice, and text, as well as IoT data from traditional devices. This includes business data from existing systems and sensor data such as force, displacement, liquid level, temperature, and humidity.
[0059] (3) Data processing
[0060] Data processing generally includes data training, machine learning, deep learning, search, reasoning, decision-making, etc.
[0061] Among them, machine learning and deep learning can symbolize and formalize data for intelligent information modeling, extraction, preprocessing, and training.
[0062] Reasoning refers to the process of simulating human intelligent reasoning in computers or intelligent systems, using formalized information to perform machine thinking and solve problems based on reasoning control strategies. Typical functions are search and matching.
[0063] Decision-making refers to the process of making decisions after intelligent information is reasoned, and usually provides functions such as classification, sorting, and prediction.
[0064] (4) General ability
[0065] After the data has undergone the data processing mentioned above, some general capabilities can be further formed based on the results of the data processing, such as algorithms or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.
[0066] (5) Smart products and industry applications
[0067] Smart products and industry applications refer to the products and applications of artificial intelligence systems in various fields. They are the encapsulation of the overall artificial intelligence solution, which productizes intelligent information decision-making and realizes practical application. Its application areas mainly include: smart terminals, smart transportation, smart medical care, autonomous driving, smart cities, etc.
[0068] This application can be applied to the field of autonomous driving of vehicles or the field of control of robots.
[0069] The vehicle in the embodiment of the present application may refer to an internal combustion engine vehicle using an engine as a power source, a hybrid vehicle using an engine and an electric motor as a power source, an electric vehicle using an electric motor as a power source, and the like.
[0070] In an embodiment of the present application, the vehicle may include an automatic driving device 100 with an automatic driving function.
[0071] Referring to Figure 1b, Figure 1b is a functional block diagram of an autonomous driving device 100 with autonomous driving functionality provided by an embodiment of the present application. In one embodiment, the autonomous driving device 100 may include various subsystems, such as a travel system 102, a sensor system 104, a control system 106, one or more peripheral devices 108, as well as a power supply 110, a computer system 112, and a user interface 116. Optionally, the autonomous driving device 100 may include more or fewer subsystems, and each subsystem may include multiple components. In addition, each subsystem and component of the autonomous driving device 100 may be interconnected via wired or wireless connections.
[0072] Propulsion system 102 may include components that provide powered motion for autonomous driving device 100. In one embodiment, propulsion system 102 may include engine 118, energy source 119, transmission 120, and wheels / tires 121. Engine 118 may be an internal combustion engine, an electric motor, an air compression engine, or a combination of other types of engines, such as a hybrid engine consisting of a gasoline engine and an electric motor, or a hybrid engine consisting of an internal combustion engine and an air compression engine. Engine 118 converts energy source 119 into mechanical energy.
[0073] Examples of energy source 119 include gasoline, diesel, other petroleum-based fuels, propane, other compressed gas-based fuels, ethanol, solar panels, batteries, and other sources of electricity. Energy source 119 may also provide power to other systems of autonomous driving device 100.
[0074] The transmission 120 can transmit mechanical power from the engine 118 to the wheels 121. The transmission 120 may include a gearbox, a differential, and a drive shaft. In one embodiment, the transmission 120 may also include other components, such as a clutch. The drive shaft may include one or more shafts that can be coupled to one or more wheels 121.
[0075] The sensor system 104 may include several sensors that sense information about the environment surrounding the autonomous driving device 100. For example, the sensor system 104 may include a positioning system 122 (the positioning system may be a global positioning system (GPS) system, a BeiDou system, or other positioning systems), an inertial measurement unit (IMU) 124, a radar 126, a laser rangefinder 128, and a camera 130. The sensor system 104 may also include sensors that monitor the internal systems of the autonomous driving device 100 (e.g., an in-vehicle air quality monitor, a fuel gauge, an oil temperature gauge, etc.). Sensor data from one or more of these sensors may be used to detect objects and their corresponding characteristics (position, shape, direction, speed, etc.). This detection and recognition is a key function for the safe operation of the autonomous autonomous driving device 100.
[0076] Positioning system 122 may be used to estimate the geographic location of autopilot 100. IMU 124 is used to sense position and orientation changes of autopilot 100 based on inertial acceleration. In one embodiment, IMU 124 may be a combination of an accelerometer and a gyroscope.
[0077] Radar 126 may utilize radio signals to sense objects in the surrounding environment of autonomous driving device 100. In some embodiments, in addition to sensing objects, radar 126 may also be used to sense the speed and / or heading of the objects.
[0078] Radar 126 may include an electromagnetic wave transmitter and receiver. Based on the principle of radio wave transmission, radar 126 may be implemented as a pulse radar or a continuous wave radar. The continuous wave radar may be implemented as a frequency modulated continuous wave (FMCW) or frequency shift keying (FSK) depending on the signal waveform.
[0079] Radar 126 can detect objects using electromagnetic waves based on a time of flight (TOF) or phase-shift method, and can measure the position, distance, and relative speed of the detected object. To detect objects in front of, behind, or to the side of the vehicle, radar 126 can be positioned appropriately outside the vehicle. LiDAR 126 can detect objects using a TOF or phase-shift method using laser light, and can measure the position, distance, and relative speed of the detected object.
[0080] Alternatively, in order to detect objects located in front of, behind, or to the sides of the vehicle, the lidar 126 may be configured at a suitable location on the exterior of the vehicle.
[0081] The laser rangefinder 128 may utilize laser light to sense objects in the environment in which the autonomous driving device 100 is located. In some embodiments, the laser rangefinder 128 may include one or more laser sources, a laser scanner, and one or more detectors, among other system components.
[0082] The camera 130 may be used to capture multiple images of the surrounding environment of the autonomous driving device 100. The camera 130 may be a still camera or a video camera.
[0083] Optionally, to capture images of the vehicle's exterior, camera 130 may be located at an appropriate location on the vehicle's exterior. For example, to capture images from the front of the vehicle, camera 130 may be positioned within the vehicle's interior, close to the front windshield. Alternatively, camera 130 may be positioned around the front bumper or radiator grille. For example, to capture images from the rear of the vehicle, camera 130 may be positioned within the vehicle's interior, close to the rear window. Alternatively, camera 130 may be positioned around the rear bumper, trunk, or tailgate. For example, to capture images from the side of the vehicle, camera 130 may be positioned within the vehicle's interior, close to at least one of the side windows. Alternatively, camera 130 may be positioned around a side mirror, fender, or door.
[0084] In the embodiment of the present application, sensor data and the like may be acquired based on one or more sensors in the sensor system 104 .
[0085] Control system 106 controls the operation of autonomous driving device 100 and its components. Control system 106 may include various components, including a steering system 132 , a throttle 134 , a brake unit 136 , a sensor fusion algorithm 138 , a computer vision system 140 , a path control system 142 , and an obstacle avoidance system 144 .
[0086] The steering system 132 is operable to adjust the forward direction of the autonomous driving device 100. For example, in one embodiment, it may be a steering wheel system.
[0087] Throttle 134 is used to control the operating speed of engine 118 and, in turn, the speed of autopilot 100 .
[0088] Braking unit 136 is used to control the deceleration of autonomous driving device 100. Braking unit 136 can use friction to slow down wheels 121. In other embodiments, braking unit 136 can convert the kinetic energy of wheels 121 into electric current. Braking unit 136 can also take other forms to slow the rotation speed of wheels 121 and thus control the speed of autonomous driving device 100.
[0089] The computer vision system 140 can be operated to process and analyze images captured by the camera 130 to identify objects and / or features in the environment surrounding the autonomous driving device 100. The objects and / or features may include traffic signs, road boundaries, and obstacles. The computer vision system 140 can use object recognition algorithms, structure from motion (SFM) algorithms, video tracking, and other computer vision techniques. In some embodiments, the computer vision system 140 can be used to map the environment, track objects, estimate the speed of objects, and so on.
[0090] The route control system 142 is used to determine a driving route for the autonomous driving device 100. In some embodiments, the route control system 142 may combine data from the sensors 138, the positioning system 122, and one or more predetermined maps to determine a driving route for the autonomous driving device 100.
[0091] The obstacle avoidance system 144 is used to identify, assess, and avoid or otherwise negotiate potential obstacles in the environment of the autonomous driving device 100 .
[0092] Of course, in one example, the control system 106 may include additional or alternative components other than those shown and described, or may include fewer than some of the components shown.
[0093] The autonomous driving device 100 interacts with external sensors, other autonomous driving devices, other computer systems, or users via peripherals 108. Peripherals 108 may include a wireless communication system 146, an onboard computer 148, a microphone 150, and / or a speaker 152.
[0094] In some embodiments, peripheral devices 108 provide a means for the user of autonomous driving device 100 to interact with user interface 116. For example, onboard computer 148 can provide information to the user of autonomous driving device 100. User interface 116 can also operate onboard computer 148 to receive user input. Onboard computer 148 can be operated via a touch screen. In other cases, peripheral devices 108 can provide a means for autonomous driving device 100 to communicate with other devices located within the vehicle. For example, microphone 150 can receive audio (e.g., voice commands or other audio input) from the user of autonomous driving device 100. Similarly, speaker 152 can output audio to the user of autonomous driving device 100.
[0095] The wireless communication system 146 can communicate wirelessly with one or more devices directly or via a communication network. For example, the wireless communication system 146 can use 3G cellular communication, such as code division multiple access (CDMA), EVDO, global system for mobile communications (GSM) / general packet radio service (GPRS), or 4G cellular communication, such as long term evolution (LTE), or 5G cellular communication. The wireless communication system 146 can use WiFi to communicate with a wireless local area network (WLAN). In some embodiments, the wireless communication system 146 can use an infrared link, Bluetooth, or ZigBee to communicate directly with the device. Other wireless protocols, such as various autonomous driving device communication systems, for example, the wireless communication system 146 may include one or more dedicated short range communications (DSRC) devices, which may include public and / or private data communications between autonomous driving devices and / or roadside stations.
[0096] Power supply 110 can provide power to the various components of autonomous driving device 100. In one embodiment, power supply 110 can be a rechargeable lithium-ion or lead-acid battery. One or more battery packs of such batteries can be configured as a power source to provide power to the various components of autonomous driving device 100. In some embodiments, power supply 110 and energy source 119 can be implemented together, such as in some all-electric vehicles.
[0097] Some or all of the functionality of the autonomous driving device 100 is controlled by a computer system 112. The computer system 112 may include at least one processor 113 that executes instructions 115 stored in a non-transitory computer-readable medium, such as a memory 114. The computer system 112 may also be a plurality of computing devices that control individual components or subsystems of the autonomous driving device 100 in a distributed manner.
[0098] Processor 113 can be any conventional processor, such as commercially available central processing unit (CPU). Alternatively, the processor can be a dedicated device such as an application specific integrated circuit (ASIC) or other hardware-based processor. Although Figure 1b functionally illustrates a processor, memory, and other elements of the computer 110 in the same block, those skilled in the art will appreciate that the processor, computer, or memory can actually include multiple processors, computers, or memories that may or may not be stored in the same physical housing. For example, the memory can be a hard drive or other storage medium located in a housing different from the computer 110. Therefore, reference to a processor or computer will be understood to include reference to a collection of processors or computers or memories that may or may not operate in parallel. Different from using a single processor to perform the steps described herein, each of some components such as a steering assembly and a deceleration assembly can have its own processor that only performs calculations related to the functions of the component.
[0099] In various aspects described herein, the processor can be located remotely from the autonomous driving device and in wireless communication with the autonomous driving device. In other aspects, some of the processes described herein are performed on a processor disposed within the autonomous driving device while others are performed by a remote processor, including taking the necessary steps to execute a single maneuver.
[0100] In some embodiments, memory 114 may include instructions 115 (e.g., program logic) that are executable by processor 113 to perform various functions of autonomous driving device 100, including those described above. Memory 114 may also include additional instructions, including instructions for sending data to, receiving data from, interacting with, and / or controlling one or more of travel system 102, sensor system 104, control system 106, and peripherals 108.
[0101] In addition to instructions 115, memory 114 may also store data such as road maps, route information, the position, direction, speed, and other such autopilot data of the autopilot, and other information. This information may be used by the autopilot 100 and computer system 112 during operation of the autopilot 100 in autonomous, semi-autonomous, and / or manual modes.
[0102] The data processing method provided in the embodiment of the present application can be a software code stored in the memory 114. In addition, the models (e.g., the first language model and the second language model) in the embodiment of the present application can be stored in the memory 114. The processor 113 can retrieve the software code from the memory and execute the retrieved software code to implement the data processing method provided in the embodiment of the present application. After obtaining the control signal of the target vehicle, the control signal can be transmitted to the control system 106. The control system 106 can determine the driving strategy of the vehicle or directly perform driving control based on the control signal.
[0103] User interface 116 is used to provide information to or receive information from a user of autonomous driving device 100. Optionally, user interface 116 may include one or more input / output devices within the set of peripherals 108, such as wireless communication system 146, onboard computer 148, microphone 150, and speaker 152.
[0104] Computer system 112 may control functions of autonomous driving device 100 based on input received from various subsystems (e.g., travel system 102, sensor system 104, and control system 106) and from user interface 116. For example, computer system 112 may utilize input from control system 106 in order to control steering unit 132 to avoid obstacles detected by sensor system 104 and obstacle avoidance system 144. In some embodiments, computer system 112 may be operable to provide control over many aspects of autonomous driving device 100 and its subsystems.
[0105] Alternatively, one or more of the above components may be installed or associated separately from the autonomous driving device 100. For example, the memory 114 may be partially or completely separate from the autonomous driving device 100. The above components may be communicatively coupled together in a wired and / or wireless manner.
[0106] Optionally, the above components are only an example. In actual applications, the components in the above modules may be added or deleted according to actual needs. Figure 1b should not be understood as a limitation to the embodiments of the present application.
[0107] In addition, the embodiments of the present application can also be applied to the robot 13.
[0108] Next, the schematic structure of the robot 13 is introduced.
[0109] The embodiment of the present application can be applied to a robot 13 as shown in FIG1 c . As shown in FIG1 c , the robot 13 may include: a sensor module 110 , a driving device 120 , a control device 140 and a main control platform 130 .
[0110] Among them, the sensor module 110 may include one or more visual sensors 111 (such as cameras) (this embodiment of the present application may also be referred to as sensors), for example, it may include an ordinary optical camera, or it may be an infrared camera, a structured light sensor, or a time-of-flight (ToF) sensor. For example, the sensor module 110 may include an ordinary RGB camera or a red, yellow, yellow, and blue (RYYB) camera, and the camera device module may also include multiple cameras or sensors to form an RGB-D depth camera solution. For example, the RGB-D depth camera solution may include two RGB cameras to form a binocular solution, may include an RGB camera and a structured light sensor to form a structured light solution, or may include an RGB camera and a ToF sensor to form a ToF solution. This embodiment of the present application does not specifically limit this. In addition, the visual sensor 111 (such as a camera) may be a fixed-focus camera or a zoom camera, for example, with phase focus, laser focus and other capabilities.
[0111] It should be understood that the visual sensor 111 (e.g., a camera) can be carried on a motion unit, which is used to carry the visual sensor 111 (e.g., a camera) and drive the visual sensor 111 (e.g., a camera) to rotate. In one embodiment, the motion unit can drive the visual sensor 111 (e.g., a camera) to rotate with two degrees of freedom. If the direction indicated by the Z axis is directly in front of the camera, then the two degrees of freedom of rotation can include the rotation of the visual sensor 111 (e.g., a camera) with the x-axis as the rotation axis and the rotation of the visual sensor 111 (e.g., a camera) with the y-axis as the rotation axis. The motion unit drives the visual sensor 111 (such as a camera) to rotate by rotating a servo or a servo motor. For example, when the driving device is used to drive the visual sensor 111 (such as a camera) to rotate with two degrees of freedom, the motion unit can include two driving mechanisms driver 1 and driver 2, such as two servos or two servo motors, one of the servos 1 or servo motor 1 is used to control the rotation of the visual sensor 111 (such as a camera) with the x-axis as the rotation axis, and the other servo 2 or servo motor 2 is used to control the rotation of the visual sensor 111 (such as a camera) with the y-axis as the rotation axis. In other embodiments, the motion unit can drive the visual sensor 111 (e.g., a camera) to generate three-degree-of-freedom rotation, that is, add the rotation of the visual sensor 111 (e.g., a camera) with the z-axis as the rotation axis. Accordingly, the motion unit can also include three driving mechanisms driver 1, driver 2, and driver 3, such as three servos or three servo motors, one of which is a servo 1 or a servo motor 1, used to control the rotation of the visual sensor 111 (e.g., a camera) with the x-axis as the rotation axis, another servo 2 or a servo motor 2 is used to control the rotation of the visual sensor 111 (e.g., a camera) with the y-axis as the rotation axis, and another servo 3 or a servo motor 3 is used to control the rotation of the visual sensor 111 (e.g., a camera) with the z-axis as the rotation axis.
[0112] The sensor module 110 may further include a motion sensor 112 , which may be an odometer, an accelerometer, a speedometer, an inertial measurement unit, etc., for collecting mileage information of the robot 13 during driving, such as travel distance, trajectory, speed, and other information.
[0113] The force sensor 113 can be a sensor for detecting the force applied to the end of the robotic arm 142. As the force sensor 113, a pressure sensor that can detect force in a single-axis direction and a force sensor or torque sensor that can detect force components in multiple-axis directions can be used. Optionally, a six-axis force sensor can be used as the force sensor 113. The six-axis force sensor detects the magnitude of the force parallel to three detection axes that are orthogonal to each other in the inherent sensor coordinate system and the magnitude of the torque around the three detection axes. It should be noted that the force sensor 113 can be set at a position other than the position of the end of the robotic arm 142, for example, it can be set on one or more joints in the robotic arm 142.
[0114] The drive unit 120 may include components that provide powered movement for the robot 13. In one embodiment, the drive unit 120 may include an engine, an energy source, a transmission, and wheels / tires. The engine may be an internal combustion engine, an electric motor, an air compression engine, or a combination of other types of engines, such as a hybrid engine consisting of a gasoline engine and an electric motor, or a hybrid engine consisting of an internal combustion engine and an air compression engine. The engine converts the energy source into mechanical energy.
[0115] Examples of energy sources include gasoline, diesel, other petroleum-based fuels, propane, other compressed gas-based fuels, ethanol, solar panels, batteries, and other sources of electricity. The energy source can also provide energy for other systems of the robot 13.
[0116] A transmission can transmit mechanical power from the engine to the wheels. The transmission can include a differential and a drive shaft. In one embodiment, the transmission can also include other components, such as a clutch. The drive shaft can include one or more shafts that can be coupled to one or more wheels.
[0117] The main control platform 130 is the data processing and control center of the device. It establishes communication connections with the control device 140, the sensor module 110, and the drive device 120. For example, it can receive image data collected by the sensor module 110, process the image data, and send movement instructions to the drive device 120. The control device 140 may include a robotic arm 142. In some embodiments, the main control platform 130 may be an embedded computer platform, including but not limited to computer chips and software systems designed based on the X86 instruction set, the ARM instruction set, the RISC-V instruction set, or the MIPS instruction set. The main control platform 130 can control the posture of the robotic arm 142 through control instructions to perform tasks.
[0118] In one embodiment, the above-mentioned computer chip may include, for example, a processor 131 and a memory 132, wherein the processor 131 may include, for example: a central processing unit (CPU), a system on a chip (SoC), an application processor (AP), a microcontroller (microcontroller), a neural network processor (NPU) and / or a graphics processing unit (GPU), etc.; the memory 132 may include, for example, a non-volatile memory and a volatile memory, the non-volatile memory may be, for example, a flash memory (flash memory), including NAND flash, a solid-state drive, etc., and the volatile memory may be, for example, a synchronous dynamic random-access memory (SDRAM), etc.
[0119] In one embodiment, the software system may include an operating system and program instructions 133 running in the operating system. When the processor executes the program instructions, the apparatus shown in FIG1c executes each step of the data processing method provided in the embodiment of the present application.
[0120] In some embodiments, the memory 132 may contain program instructions 133 (e.g., program logic) that are executable by the processor 131 to perform various functions of the robot 13, including those described above. The memory 132 may also contain additional instructions, including instructions for sending data to, receiving data from, interacting with, and / or controlling one or more of the drive device 120, the sensor module 110, the control system, and peripheral devices.
[0121] In addition to the program instructions 133, the memory 132 may also store data such as road maps, route information, the position, direction, speed, and other such autopilot data of the autopilot, and other information. This information may be used by the robot 13 during operation in autonomous, semi-autonomous, and / or manual modes.
[0122] It should be understood that the data processing method in the embodiment of the present application involves algorithm processing related to artificial intelligence. Therefore, the architecture of the processor is not limited to the structure of the processor combined with the memory in Figure 1c, but can also be other hardware architectures (such as pure hardware or other software and hardware combination architectures).
[0123] The wireless communication system 150 can communicate wirelessly with one or more devices (e.g., servers) directly or via a communication network. For example, the wireless communication system 150 can use 3G cellular communications, such as code division multiple access (CDMA), EVDO, global system for mobile communications (GSM) / general packet radio service (GPRS), or 4G cellular communications, such as long term evolution (LTE), or 5G cellular communications. The wireless communication system 150 can use WiFi to communicate with a wireless local area network (WLAN). In some embodiments, the wireless communication system 150 can use an infrared link, Bluetooth, or ZigBee to communicate directly with the device. Other wireless protocols, such as various autonomous driving device communication systems, for example, the wireless communication system 150 may include one or more dedicated short range communications (DSRC) devices, which may include public and / or private data communications between autonomous driving devices and / or roadside stations.
[0124] Alternatively, one or more of the above components may be installed or associated separately from the robot 13. For example, the memory 132 may be partially or completely separate from the robot 13. The above components may be communicatively coupled together in a wired and / or wireless manner.
[0125] Optionally, the above components are only an example. In actual applications, the components in the above modules may be added or deleted according to actual needs. Figure 1c should not be understood as a limitation to the embodiments of the present application.
[0126] The steps related to the model reasoning process in the embodiment of the present application involve AI-related operations. The system architecture provided in the embodiment of the present application is introduced in detail below in conjunction with Figure 1d.
[0127] FIG1 d is a schematic diagram of the system architecture provided by an embodiment of the present application. As shown in FIG1 d , the system architecture 500 includes an execution device 510 , a training device 520 , a database 530 , a client device 540 , a data storage system 550 , and a data acquisition system 560 .
[0128] The execution device 510 includes a calculation module 511, an I / O interface 512, a pre-processing module 513, and a post-processing module 514. The calculation module 511 may include the target model / rule 501, and the pre-processing module 513 and the post-processing module 514 are optional.
[0129] The data collection device 560 is used to collect training samples. After collecting the training samples, the data collection device 560 stores these training samples in the database 530.
[0130] The training device 520 can be based on the training samples maintained in the database 530, and the neural network to be trained (for example, the first language model and the second language model in the embodiment of the present application) to obtain the target model / rule 501.
[0131] It should be understood that the training device 520 can perform a pre-training process on the neural network to be trained based on the training samples maintained in the database 530, or fine-tune the model based on the pre-training.
[0132] It should be noted that, in actual applications, the training samples maintained in the database 530 may not all be collected by the data acquisition device 560, but may also be received from other devices. It should also be noted that the training device 520 may not train the target model / rule 501 entirely based on the training samples maintained in the database 530, but may also obtain training samples from the cloud or other places for model training. The above description should not be used as a limitation on the embodiments of the present application.
[0133] The target model / rule 501 trained by the training device 520 can be applied to different systems or devices, such as the execution device 510 shown in FIG. 1 d , which can be a vehicle or a robot.
[0134] Specifically, the training device 520 may transfer the trained model to the execution device 510 .
[0135] Among them, the execution device 510 can be a vehicle or a robot. In one implementation, the training device 530 can perform the pre-training or fine-tuning process of the model, deploy the trained model to the execution device 510, and the execution device 510 can execute the trained model to implement the data processing method in the embodiment of the present application.
[0136] In FIG1d , the execution device 510 is configured with an input / output (I / O) interface 512 for data exchange with external devices. A user can input data (e.g., a navigation request or a task execution request in the embodiment of the present application) into the I / O interface 512 through a client device 540. Furthermore, the input data may also include sensor data.
[0137] Preprocessing module 513 and preprocessing module 514 are used to preprocess the input data received by I / O interface 512. It should be understood that preprocessing module 513 and preprocessing module 514 may be absent or only one preprocessing module may be present. If preprocessing module 513 and preprocessing module 514 are absent, computing module 511 may be used directly to process the input data.
[0138] When the execution device 510 preprocesses the input data, or when the computing module 511 of the execution device 510 performs calculations and other related processing, the execution device 510 can call the data, code, etc. in the data storage system 550 for corresponding processing, and can also store the data, instructions, etc. obtained from the corresponding processing in the data storage system 550.
[0139] Finally, the I / O interface 512 provides the processed results to the client device 540 and thus to the user.
[0140] In the scenario shown in FIG1d , the user can manually input data, and this "manual input data" can be operated through the interface provided by I / O interface 512. In another scenario, client device 540 can automatically send input data to I / O interface 512. If user authorization is required for client device 540 to automatically send input data, the user can set the corresponding permissions in client device 540. The user can view the results output by execution device 510 on client device 540, which can be presented in a specific form such as display, sound, or action. Client device 540 can also serve as a data acquisition terminal, collecting input data input into I / O interface 512 and output results output from I / O interface 512 as new sample data and storing them in database 530. Of course, collection can also be performed without client device 540, with I / O interface 512 directly storing the input data input into I / O interface 512 and output results output from I / O interface 512 as new sample data in database 530.
[0141] It is worth noting that FIG1d is merely a schematic diagram of a system architecture provided by an embodiment of the present application, and the positional relationships between the devices, components, modules, etc. shown in the figure do not constitute any limitation. For example, in FIG1d , data storage system 550 is an external memory relative to execution device 510. In other cases, data storage system 550 can also be placed in execution device 510. It should be understood that the execution device 510 can be deployed in client device 540.
[0142] From the inference side of the model:
[0143] In the embodiment of the present application, the computing module 511 of the above-mentioned execution device 510 can obtain the code stored in the data storage system 550 to implement the steps related to the model reasoning process in the embodiment of the present application.
[0144] In an embodiment of the present application, the computing module 511 of the execution device 510 may include a hardware circuit (such as an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a general-purpose processor, a digital signal processor (DSP), a microprocessor or a microcontroller, etc.), or a combination of these hardware circuits. For example, the training device 520 may be a hardware system with an instruction execution function, such as a CPU, DSP, etc., or a hardware system without an instruction execution function, such as an ASIC, FPGA, etc., or a combination of the above-mentioned hardware systems without an instruction execution function and hardware systems with an instruction execution function.
[0145] Specifically, the computing module 511 of the execution device 510 can be a hardware system with an execution instruction function, and the steps related to the model reasoning process provided in the embodiment of the present application can be software codes stored in the memory. The computing module 511 of the execution device 510 can obtain the software code from the memory and execute the obtained software code to implement the steps related to the model reasoning process provided in the embodiment of the present application.
[0146] It should be understood that the computing module 511 of the execution device 510 can be a combination of a hardware system that does not have the function of executing instructions and a hardware system that has the function of executing instructions. Some of the steps related to the model reasoning process provided in the embodiment of the present application can also be implemented by the hardware system that does not have the function of executing instructions in the computing module 511 of the execution device 510, which is not limited here.
[0147] From the training side of the model:
[0148] In an embodiment of the present application, the above-mentioned training device 520 can obtain the code stored in the memory (not shown in Figure 1d, which can be integrated into the training device 520 or deployed separately from the training device 520) to implement the steps related to model training in the embodiment of the present application.
[0149] In an embodiment of the present application, the training device 520 may include a hardware circuit (such as an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a general-purpose processor, a digital signal processor (DSP), a microprocessor or a microcontroller, etc.), or a combination of these hardware circuits. For example, the training device 520 may be a hardware system with an instruction execution function, such as a CPU, DSP, etc., or a hardware system without an instruction execution function, such as an ASIC, FPGA, etc., or a combination of the above-mentioned hardware systems without an instruction execution function and hardware systems with an instruction execution function.
[0150] It should be understood that the training device 520 can be a combination of a hardware system that does not have the function of executing instructions and a hardware system that has the function of executing instructions. Some of the steps related to model training provided in the embodiments of the present application can also be implemented by the hardware system in the training device 520 that does not have the function of executing instructions, which is not limited here.
[0151] The above is an introduction to the application architecture of the embodiment of the present application. Next, the intention prediction method provided by the embodiment of the present application is described in detail.
[0152] Since the embodiments of the present application involve neural networks, for ease of understanding, the relevant terms involved in the embodiments of the present application are first introduced below.
[0153] (1) Neural Network
[0154] A neural network may be composed of neural units. A neural unit may refer to an operation unit that takes xs (i.e., input data) and intercept = 1 as input. The output of the operation unit may be:
[0155] Where s = 1, 2, ... n, n is a natural number greater than 1, Ws is the weight of xs, and b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal of the neural unit into the output signal. The output signal of the activation function can be used as the input of the next convolutional layer, and the activation function can be a sigmoid function. A neural network is a network formed by connecting multiple single neural units mentioned above, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field. The local receptive field can be an area composed of several neural units.
[0156] (2) Convolutional neural network (CNN) is a deep neural network with a convolutional structure. Convolutional neural network contains a feature extractor consisting of a convolution layer and a subsampling layer, which can be regarded as a filter. The convolution layer refers to the neuron layer in the convolutional neural network that performs convolution processing on the input signal. In the convolution layer of the convolutional neural network, a neuron can only be connected to some neurons in the adjacent layer. A convolution layer usually contains several feature planes, and each feature plane can be composed of some rectangularly arranged neural units. The neural units in the same feature plane share weights, and the shared weights here are convolution kernels. Shared weights can be understood as the way of extracting features is independent of position. The convolution kernel can be formalized as a matrix of random size, and the convolution kernel can obtain reasonable weights through learning during the training process of the convolutional neural network. In addition, the direct benefit of shared weights is to reduce the connections between the layers of the convolutional neural network, while reducing the risk of overfitting.
[0157] (3) Deep Neural Networks
[0158] Deep Neural Network (DNN), also known as multi-layer neural network, can be understood as a neural network with many hidden layers. There is no special metric for "many" here. Based on the position of different layers in DNN, the neural network inside DNN can be divided into three categories: input layer, hidden layer, and output layer. Generally speaking, the first layer is the input layer, the last layer is the output layer, and the layers in between are all hidden layers. The layers are fully connected, that is, any neuron in the i-th layer must be connected to any neuron in the i+1-th layer. Although DNN looks complicated, the work of each layer is actually not complicated. Simply put, it is the following linear relationship expression: in, is the input vector, is the output vector, is the offset vector, W is the weight matrix (also called coefficient), and α() is the activation function. Each layer is just an input vector After such a simple operation, the output vector Since there are many DNN layers, the coefficient W and the offset vector The number of these parameters is also very large. The definition of these parameters in DNN is as follows: Take the coefficient W as an example: Assume that in a three-layer DNN, the linear coefficient from the 4th neuron in the second layer to the 2nd neuron in the third layer is defined as The superscript 3 represents the layer number of the coefficient W, while the subscripts correspond to the third layer index 2 of the output and the second layer index 4 of the input.
[0159] In summary, the coefficient from the kth neuron in the L-1th layer to the jth neuron in the Lth layer is defined as
[0160] It's important to note that the input layer has no W parameter. In deep neural networks, more hidden layers allow the network to better capture complex real-world situations. Theoretically, a model with more parameters has higher complexity and greater "capacity," meaning it can handle more complex learning tasks. Training a deep neural network is essentially the process of learning the weight matrix, with the ultimate goal of obtaining the weight matrices for all layers of a trained deep neural network (a weight matrix formed by the vectors W across many layers).
[0161] (4) Loss function
[0162] During the training of a deep neural network, because we want the output of the deep neural network to be as close as possible to the desired predicted value, we can compare the current network's predicted value with the desired target value and then update the weight vector of each layer of the neural network based on the difference between the two. (Of course, before the first update, there is usually an initialization process, which pre-configures the parameters for each layer in the deep neural network.) For example, if the network's predicted value is too high, the weight vector is adjusted to make it predict a lower value. This adjustment is continued until the deep neural network can predict the desired target value or a value very close to the desired target value. Therefore, it is necessary to predefine "how to compare the difference between the predicted value and the target value." This is the loss function (or objective function), which is an important equation used to measure the difference between the predicted value and the target value. For example, the loss function output value (loss) indicates a greater difference, so training a deep neural network becomes a process of minimizing this loss.
[0163] (5) Backpropagation algorithm
[0164] Convolutional neural networks can use the back propagation (BP) algorithm to correct the size of the parameters in the initial super-resolution model during training, reducing the reconstruction error loss of the super-resolution model. Specifically, the forward propagation of the input signal to the output generates an error loss. This error loss information is then backpropagated to update the parameters of the initial super-resolution model, thereby converging the error loss. The BP algorithm is a backward propagation movement dominated by the error loss, aiming to obtain the optimal super-resolution model parameters, such as the weight matrix.
[0165] (6) Natural language processing (NLP)
[0166] Natural language refers to human language, and natural language processing (NLP) is the processing of human language. Natural language processing is the process of systematically analyzing, understanding, and extracting information from text data in an intelligent and efficient manner. By using NLP and its components, we can manage very large amounts of text data, perform a large number of automated tasks, and solve a wide variety of problems, such as automatic summarization, machine translation (MT), named entity recognition (NER), relation extraction (RE), information extraction (IE), sentiment analysis, speech recognition, question answering, and topic segmentation.
[0167] (7) Pre-trained language model
[0168] A pretrained language model is a natural language sequence encoder that encodes each word in a natural language sequence into a vector representation for prediction. Its training consists of two phases. In the pre-training phase, the model is trained on a large amount of unsupervised text for language modeling tasks, thereby learning a word representation. In the fine-tuning phase, the model is initialized using the parameters learned in the pre-training phase and trained in a relatively short number of steps on downstream tasks such as text classification and sequence labeling. This allows the semantic information gained from pre-training to be successfully transferred to downstream tasks.
[0169] For robots or vehicles, when performing automatic control (for example, a robot automatically executes a task based on a user's task execution request, or a vehicle performs an autonomous driving task), three tasks are typically involved: perception, prediction, and decision-making. Taking autonomous driving as an example, the perception task involves receiving raw sensor data (such as images or lasers) as input and identifying dynamic targets (such as the size and position of pedestrians and vehicles) and static elements (such as lane lines and arrow signs on the ground) in the environment. The prediction task can receive the raw results of dynamic and static targets obtained by the perception task and predict the movement intentions of targets in the environment, mainly predicting the future movement trajectory and intentions of other vehicles. The decision-making task can receive the prediction results obtained by the prediction task, and based on the future driving intentions of other vehicles and the navigation task of the own vehicle, generate a decision conclusion, output a control signal, and control the vehicle for autonomous driving.
[0170] However, the existing architecture design decouples the functions of each task module from each other. They maintain a clear upstream and downstream relationship. Each module's output interface is predefined, allowing for serial connection. This approach offers the advantages of high interpretability and ease of modular development and maintenance. However, due to the decoupling of front-end and back-end modules, errors in the front-end module can also lead to erroneous decisions in downstream modules, resulting in poor control accuracy.
[0171] To address the aforementioned issues, the present application proposes an end-to-end control solution based on a language model. "End-to-end" means that the system can directly output control signals based on input sensor data. Information transmission between system modules is no longer a fixed, agreed-upon format. Instead, it allows for full information sharing and dissemination.
[0172] Referring to FIG. 2 , FIG. 2 is a flow chart of a data processing method provided in an embodiment of the present application. As shown in FIG. 2 , the data processing method provided in an embodiment of the present application includes:
[0173] 201. Obtain scene information through a first language model based on sensor data and a first prompt text; the sensor data is data collected from the surrounding environment of a target object, and the first prompt text indicates that scene information of the scene where the target object is located is extracted based on the sensor data, and the target object is the vehicle itself or the robot itself.
[0174] In a possible implementation, the execution subject of the embodiment corresponding to FIG2 may be the target object itself, or a server, or some steps may be executed by the target object itself and some steps may be executed by the server, which is not limited here.
[0175] In one possible implementation, the target object is a robot; the target object can receive task instructions input by the user, for example, the task instructions can instruct the robot to perform a task, such as "help me get some drinks from the kitchen" or "help me get the water bottle on the table".
[0176] In a possible implementation, the target object is a vehicle, and the target object can receive navigation instructions from the user. For example, the navigation instruction can indicate the destination of the vehicle's automatic driving, such as "please navigate to school."
[0177] After receiving a task instruction (such as the navigation instructions and robot task instructions described above), the objectives required to execute each stage of the task indicated by the task instruction can be obtained. The target object needs to sequentially achieve the objectives of each stage to complete the task corresponding to the task instruction. During the process of the target object executing the task corresponding to the task instruction, it is necessary to determine the current objective to be achieved and then determine the control strategy that can achieve the current objective.
[0178] Specifically, when the target object is performing a task, it can determine the current task that needs to be executed (that is, the current execution target of the target object in the embodiment of the present application) based on the instructions input by the user (such as the task instructions or navigation instructions introduced above) and its own state. As time goes by, the target object continues to execute the execution targets that need to be executed at each time until the task indicated by the instructions input by the user is achieved.
[0179] For example, the robot may obtain the current execution target of the target object according to the current posture of the robot, scene map information, and the task instruction.
[0180] Optionally, the robot can obtain the execution goals (including the current execution goals) required to achieve each stage of the task indicated by the task instruction through the second language model based on the robot's current posture, scene map information and the task instruction.
[0181] In a possible implementation, the vehicle may obtain the current execution target of the target object according to the current position of the vehicle and the navigation instruction.
[0182] Optionally, the vehicle can obtain the execution targets (including the current execution targets) required to implement each stage of the task indicated by the task instruction through the second language model based on the current position of the vehicle and the navigation instruction (which may also include map information).
[0183] Exemplarily, the vehicle can generate the execution targets required in each stage (such as a coarse-grained driving planning path) based on the current position of the vehicle and the navigation instructions. For example, it may include: first driving out of the basement, and then safely driving to the community according to the navigation route after reaching the ground, entering the gate after arriving at the community gate, and parking in the parking space after identifying an empty parking space.
[0184] In one possible implementation, the target object can determine the current target to be executed based on its current state. To execute the target, it is necessary to obtain the current scene information and combine the scene information (as well as information such as the recovery text described later) to determine the control information, thereby enabling the target object to achieve the target in the current scene.
[0185] Next, we will introduce how to obtain scene information:
[0186] In a possible implementation, the target object is a vehicle, and the sensor data is data collected from the surrounding environment of the target object. For example, the sensor data may be data collected by an image sensor, data collected by a radar, etc.
[0187] In a possible implementation, the target object is a robot, and the sensor data is data collected from the surrounding environment of the target object. For example, the sensor data may be data collected by an image sensor, data collected by a radar, etc.
[0188] In one possible implementation, the first language model (which may also be referred to as a language-perception model in this embodiment of the present application) may be a large language model such as chatGPT, GPT4, or chatGLM. This embodiment of the present application does not limit the type of language model.
[0189] In one possible implementation, the first language model can receive input of high-frequency sensor data and input of a first prompt text, wherein the first prompt text can indicate the extraction of scene information of the scene where the target object is located based on the sensor data. For example, taking the target object as a vehicle as an example, the first prompt text can be "Please perceive the type of environment around the vehicle and the perceptual elements in the environment." The first prompt text can guide the first language model to output a perception result (that is, the environmental information in the embodiment of the present application). Exemplarily, the environmental information can include the category of the current scene (such as basement, open ground, tunnel, intersection, etc.), and the perceptual elements of the current scene, such as the position of vehicles and pedestrians, lane line detection results, etc.
[0190] Among them, the perception element can be an object that affects the target object's execution of the task.
[0191] 202. Generate a second prompt text using a second language model based on the scenario information and the current execution target of the target object, where the second prompt text indicates a query for information that the target object needs to know when completing the execution target and that is not included in the scenario information.
[0192] The scenario information and the current execution target of the target object may be input into the second language model in the form of text (eg, prompt).
[0193] In one possible implementation, after obtaining the scenario information, a second prompt text can be generated through a second language model based on the scenario information and the current execution target of the target object. The second prompt text indicates a query for information that the target object needs to know when completing the execution target and is not included in the scenario information.
[0194] In a possible implementation, if the scenario information obtained by the first language model already contains all the information required for the current execution goal of the target object, the second language model can directly obtain the control instructions for achieving the current execution goal.
[0195] However, due to complex scenarios or unexpected situations, the scenario information obtained by the first language model is often difficult to include all the information required for the target object's current execution goal. In this case, the second language model can output an active query text (that is, the second prompt text in the embodiment of the present application). The second prompt text indicates a query for information that the target object needs to know when completing the execution goal and is not included in the scenario information.
[0196] In one possible implementation, the information that needs to be known and is not included in the scenario information includes: state information of a perception element associated with the execution target. A perception element here refers to an object associated with the execution target. For example, if the scenario category in the current scenario information is an intersection and the current execution target is to go straight through the intersection, the perception element may be the state of a traffic light, or how long it will take for a red traffic light to turn green.
[0197] For example, the current execution goal is to go straight through the intersection, the perception element may be a traffic light, and the second prompt text may be: Please identify the status of the traffic light, is it possible to go straight?.
[0198] For example, the current execution target is entering a residential gate, the sensing element may be a parking gate pole, and the second prompt text may be: Is the parking gate pole raised? Can the vehicle pass?
[0199] For example, the current execution target is parking, the sensing element may be the vacant state of a parking space, and the second prompt text may be: Is there a vacant parking space?
[0200] In a possible implementation, the information that needs to be known and is not included in the scenario information includes: a method for implementing the execution target.
[0201] For example, the current execution goal is to drive out of the basement. In the basement scenario, since there is no GPS positioning information and no map information available, the second prompt text may be "How to drive out of the basement?"
[0202] 203. Obtain a reply text corresponding to the second prompt text using the first language model according to the sensor data and the second prompt text.
[0203] In one possible implementation, the first language model can perform secondary reasoning based on the sensor data and the second prompt text, and output high-level environmental information that meets the requirements of the question indicated in the second prompt text (that is, the response text corresponding to the second prompt text) to support the decision of the second language model.
[0204] The following uses a vehicle as an example to describe how the first and second language models collaborate to generate decision instructions in different driving scenarios.
[0205] 1) Underground garage scenario: There is no GPS positioning information in the underground garage, and no map information is available. Therefore, the second language model can generate a second prompt text: "How to drive out of the underground garage?" and input the second prompt text into the first language model. The first language model identifies the underground garage exit guide sign in the environment based on the sensor data and gives a reply text: Exit guide sign detected, please turn right out of the underground garage. After receiving the reply text, the second language model can generate a driving instruction (also referred to as a control instruction in the embodiment of the present application): Turn right out of the underground garage.
[0206] 2) Intersection scenario: The second language model generates a second prompt based on the scenario and the current execution goal: "Please identify the traffic light status. Can I go straight?" This second prompt is fed to the first language model, which recognizes the sensor data and responds with a textual response: "The traffic light is currently red. Passable in 5 seconds." Upon receiving this response, the second language model generates a driving instruction: "The traffic light is currently red. Stop and wait for 5 seconds."
[0207] 3) Gate Entry Scenario: When a vehicle arrives at the gate, the second language model generates a second prompt: Is the parking gate pole raised? Can the vehicle pass? This second prompt is fed into the first language model. The first language model identifies the gate pole status based on sensor data and outputs a response: The gate pole is raised, allowing passage. The second language model then generates a driving instruction based on this response: Go straight and enter the gate.
[0208] 4) Parking scenario: After the vehicle arrives at a residential parking lot, the second language model generates a second prompt: "Are there any available parking spaces?" and inputs this second prompt into the first language model. The first language model identifies parking spaces in the environment based on sensor data and responds: "Two available spaces found to the right, available for parking." After receiving this response, the second language model generates a driving decision instruction: "Park into the available space to the right."
[0209] 204. Obtain a control instruction for the target object through the second language model according to the reply text, the scenario information, and the current execution target.
[0210] In one possible implementation, after obtaining the reply text, the first language model can input the reply text into the second language model. The second language model can obtain the control instructions of the target object through the second language model based on the reply text, the scenario information and the current execution target.
[0211] Among them, the control instructions can be coarse-grained control targets (such as stopping and waiting for 5 seconds and then going straight, turning left, turning right, etc.), or they can be fine-grained hardware control signals, such as the control signal for the rotation of the steering wheel, the control signal for the rotation of the robotic arm joint, etc.
[0212] In an embodiment of the present application, the second language model obtains demand information (second prompt text) in the form of a query text based on different scenario information, and the first language model obtains a reply text based on the query text, and the reply text is passed to the second language model for decision-making. On the one hand, an end-to-end automatic control system is constructed through the interaction between the first language model and the second language model. Compared with a multi-stage decision-making system, the present application can improve decision-making accuracy. On the other hand, the architecture introduces a language model, which can expand the inductive reasoning ability of the large language model to the control field. The text expression of the language model is different from the traditional fixed format and can represent an infinite number of perceptual needs, thereby unifying the method of obtaining conclusions of various types of perceptual needs of the second language model.
[0213] As shown in Figure 3, Figure 3 is a schematic diagram of a process in a typical scenario of autonomous driving, in which sensor data and a first query text can be input into the first language model in the perception module, the first language model can obtain scene information and input it into the second language model of the decision control model, the second language model can obtain a second query text based on the coarse-grained planning goal and scene information, and input the second query text into the first language model, the first language model can obtain a reply text corresponding to the second query text, and the second language model (or a control instruction generation module separate from the second language model) can obtain a control instruction based on the reply text and scene information and other information (it should be understood that the control instruction generation model in Figure 3 may belong to the second language model or not).
[0214] Next, we will take autonomous driving and robot control as examples to introduce two implementation examples of the present application:
[0215] In an autonomous driving scenario, a user's navigation instruction might be: "Navigate home from the company basement." The second language model is fed the navigation instruction: "Navigate home from the company basement." Based on the positioning information and navigation instruction, the second language model generates a coarse-grained driving plan: first exit the basement, then safely drive to the residential complex following the navigation route upon reaching the surface. Upon reaching the residential complex, the vehicle enters the gate, identifies an empty parking space, and parks in it. The first language model receives high-frequency basic perception text (i.e., the first prompt text) as "vehicle surroundings" input and outputs the current driving scene (e.g., basement, open ground, tunnel, intersection, etc.) and the corresponding perception elements of the scene, such as the location of vehicles and pedestrians, and lane detection results. Based on the received current driving scene and corresponding perception elements, the second language model determines whether there are any perception elements to be identified in the current scene. If so, it generates a perception element active query text (i.e., the second prompt text) and sends it to the first language model. For example, the second language model receives the current driving scenario as an "intersection" and the coarse-grained planned path as "go straight." The second language model generates the following perceptual element active query text: "Please identify the traffic light status. Can I go straight?" The first language model receives the perceptual element active query text (also known as the second prompt text), identifies the environmental perceptual elements, and performs secondary reasoning based on the currently identified environmental perceptual results. It then outputs high-level environmental information (also known as the response text) that meets the question requirements: "The current traffic light is red. Passage is allowed in 5 seconds." After receiving the perceptual reasoning results, the second language model outputs the driving decision instruction: "The current traffic light is red. Stop and wait 5 seconds." Based on the coarse-grained planned path and other previously identified perceptual elements, it then outputs the following driving decision instruction: "The current traffic light at the intersection is red. Stop and wait 5 seconds."
[0216] In a robot control scenario, a user's task instruction might be: "Get me some drinks from the kitchen." The task instruction text, "Get me some drinks from the kitchen," is input to the second language model. Based on the input text, the current scene map, and its own physical quantities, the second language model generates a coarse-grained navigation planning path. The first language model receives high-frequency basic perception text (i.e., the first prompt text) as "robot surrounding environment" input, along with an environmental feature map and basic sensor perception data. It outputs the current driving scene it perceives and identifies, along with the corresponding perception elements. Based on the received current driving scene and corresponding perception elements, the second language model determines whether there are any perception elements to be identified in the current scene. If so, it generates a perception element active query text and sends it to the first language model. The first language model receives the perception element active query text, identifies the environmental perception elements (i.e., the second prompt text), and sends it to the first language model. The first language model receives the perception element active query text (i.e., the second prompt text), identifies the environmental perception elements, and performs secondary reasoning based on the currently identified environmental perception results, outputting high-level environmental information (i.e., the response text) that meets the requirements of the question. After receiving the perception reasoning results, the second language model outputs driving decision instructions based on the coarse-grained planned path and other previously identified perception elements until the task of navigating to the kitchen is completed.
[0217] The above describes the data processing method provided in the embodiment of the present application from the perspective of the method. Next, the data processing device provided in the embodiment of the present application is described from the perspective of the device.
[0218] 4 , which is a schematic diagram of the structure of a data processing device provided in an embodiment of the present application, as shown in FIG4 , the device 400 includes:
[0219] Processing module 401 is used to obtain scene information through a first language model based on sensor data and a first prompt text; the sensor data is data collected from the surrounding environment of a target object, and the first prompt text indicates scene information of the scene where the target object is located extracted based on the sensor data; the target object is the vehicle itself or the robot itself; based on the scene information and the current execution target of the target object, a second prompt text is generated through a second language model, the second prompt text indicating a query for information that the target object needs to know when completing the execution target and is not included in the scene information; based on the sensor data and the second prompt text, a reply text corresponding to the second prompt text is obtained through the first language model; based on the reply text, the scene information, and the current execution target, a control instruction for the target object is obtained through the second language model.
[0220] For a detailed description of the processing module 401 , reference may be made to the description of steps 201 to 204 in the above embodiment, which will not be repeated here.
[0221] In a possible implementation, the scene information includes information indicating a scene category and / or information about perceptual elements included in the scene.
[0222] In a possible implementation, the target object is a vehicle, and the apparatus further includes:
[0223] Acquisition module 402, for receiving navigation instructions from the user;
[0224] A current execution target of the target object is obtained according to the current position of the vehicle and the navigation instruction.
[0225] In a possible implementation, the target object is a robot; and the apparatus further includes:
[0226] Acquisition module 402, for receiving a task instruction from a user;
[0227] The processing module 401 is further configured to obtain the current execution target of the target object according to the current posture of the robot, scene map information, and the task instruction.
[0228] In a possible implementation, the information that needs to be known and is not included in the scene information includes:
[0229] Status information of a sensory element associated with the execution target; or
[0230] The method for achieving the execution goal.
[0231] In a possible implementation, the first language model and the second language model are chatGPT, GPT4, or chatGLM.
[0232] It should be noted that the information interaction, execution process, etc. between the modules / units of the above-mentioned device are based on the same concept as the method embodiment of the present application, and the technical effects they bring are the same as those of the method embodiment of the present application. For specific contents, please refer to the description in the method embodiment shown above in the embodiment of the present application, and no further details will be given here.
[0233] Next, an execution device provided in an embodiment of the present application is introduced. Please refer to Figure 5. Figure 5 is a structural schematic diagram of the execution device provided in an embodiment of the present application. The execution device 500 can be specifically manifested as a vehicle control device or a robot, etc., which is not limited here. Specifically, the execution device 500 includes: a receiver 501, a transmitter 502, a processor 503 and a memory 504 (wherein the number of processors 503 in the execution device 500 can be one or more, and Figure 5 takes one processor as an example), wherein the processor 503 may include an application processor 5031 and a communication processor 5032. In some embodiments of the present application, the receiver 501, the transmitter 502, the processor 503 and the memory 504 may be connected via a bus or other means.
[0234] The memory 504 may include a read-only memory and a random access memory, and provides instructions and data to the processor 503. A portion of the memory 504 may also include non-volatile random access memory (NVRAM). The memory 504 stores processor and operation instructions, executable modules, or data structures, or subsets or extensions thereof. The operation instructions may include various operation instructions for implementing various operations.
[0235] Processor 503 controls the operation of the execution device. In specific applications, the various components of the execution device are coupled together via a bus system. In addition to a data bus, the bus system may also include a power bus, a control bus, and a status signal bus. However, for clarity, all of these buses are referred to as a bus system in the figure.
[0236] The methods disclosed in the above embodiments of the present application can be applied to or implemented by the processor 503. The processor 503 can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by hardware integrated logic circuits in the processor 503 or by software instructions. The above processor 503 can be a general-purpose processor, a digital signal processor (DSP), a microprocessor, or a microcontroller, and can further include an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The processor 503 can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of the present application can be directly implemented as being executed by a hardware decoding processor, or can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium well-known in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in memory 504, and processor 503 reads the information in memory 504 and performs the steps of the above method in conjunction with its hardware.
[0237] Receiver 501 can be used to receive input digital or character information and generate signal input related to executing relevant settings and function control of the device. Transmitter 502 can be used to output digital or character information through the first interface; transmitter 502 can also be used to send instructions to the disk pack through the first interface to modify the data in the disk pack.
[0238] In an embodiment of the present application, in one case, the processor 503 is used to execute the data processing method described in the embodiment corresponding to Figure 2.
[0239] The present application also provides a server. Please refer to Figure 6, which is a schematic diagram of the structure of a server provided in an embodiment of the present application. The data processing device described in the embodiment corresponding to Figure 4 can be deployed on the server 600 to implement the functions of the data processing device in the embodiment corresponding to Figure 4. Specifically, the server 600 is implemented by one or more servers. The server 600 may have relatively large differences due to different configurations or performance. It may include one or more central processing units (CPUs) 619 (for example, one or more processors) and memory 632, and one or more storage media 630 (for example, one or more mass storage devices) for storing application programs 642 or data 644. Among them, the memory 632 and storage medium 630 can be temporary storage or permanent storage. The program stored in the storage medium 630 may include one or more modules (not shown in the figure), each module may include a series of instruction operations in the server. Furthermore, the central processing unit 619 can be configured to communicate with the storage medium 630 to execute a series of instruction operations in the storage medium 630 on the server 600.
[0240] The server 600 may also include one or more power supplies 626, one or more wired or wireless network interfaces 650, one or more input and output interfaces 658; or one or more operating systems 641, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.
[0241] In the embodiment of the present application, the central processing unit 619 is used to execute the data processing method provided in the embodiment corresponding to Figure 2.
[0242] An embodiment of the present application also provides a computer program product, which, when running on a computer, enables the computer to execute the steps executed by the aforementioned execution device, or enables the computer to execute the steps executed by the aforementioned training device.
[0243] A computer-readable storage medium is also provided in an embodiment of the present application, which stores a program for signal processing. When the computer-readable storage medium is run on a computer, it enables the computer to execute the steps executed by the aforementioned execution device, or enables the computer to execute the steps executed by the aforementioned training device.
[0244] The execution device, training device or terminal device provided in the embodiments of the present application can specifically be a chip, and the chip includes: a processing unit and a communication unit. The processing unit can be, for example, a processor, and the communication unit can be, for example, an input / output interface, a pin or a circuit. The processing unit can execute the computer execution instructions stored in the storage unit to enable the chip in the execution device to execute the data processing method described in the above embodiment, or to enable the chip in the training device to execute the data processing method described in the above embodiment. Optionally, the storage unit is a storage unit in the chip, such as a register, a cache, etc. The storage unit can also be a storage unit located outside the chip in the wireless access device, such as a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM), etc.
[0245] Specifically, see Figure 7, which is a schematic diagram of the structure of a chip provided in an embodiment of the present application. This chip can be represented as a neural network processor NPU 700. NPU 700 is mounted on the host CPU (host CPU) as a coprocessor and is assigned tasks by the host CPU. The core of the NPU is arithmetic circuit 703, which is controlled by controller 704 to extract matrix data from memory and perform multiplication operations.
[0246] In some implementations, the arithmetic circuit 703 includes multiple processing units (PEs). In some implementations, the arithmetic circuit 703 is a two-dimensional systolic array. The arithmetic circuit 703 can also be a one-dimensional systolic array or other electronic circuitry capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 703 is a general-purpose matrix processor.
[0247] For example, assume there are input matrix A, weight matrix B, and output matrix C. The computation circuit retrieves the corresponding data of matrix B from weight memory 702 and caches it on each PE in the computation circuit. The computation circuit then retrieves the data of matrix A from input memory 701 and performs a matrix operation on matrix B. The partial or final matrix result is stored in accumulator 708.
[0248] Unified memory 706 is used to store input and output data. Weight data is directly transferred to weight memory 702 through the Direct Memory Access Controller (DMAC) 705. Input data is also transferred to unified memory 706 through the DMAC.
[0249] BIU stands for Bus Interface Unit, i.e., bus interface unit 710 , which is used for interaction between the AXI bus, DMAC, and instruction fetch buffer (IFB) 709 .
[0250] The bus interface unit 710 (BIU) is used for the instruction fetch memory 709 to obtain instructions from the external memory, and is also used for the storage unit access controller 705 to obtain the original data of the input matrix A or the weight matrix B from the external memory.
[0251] DMAC is mainly used to transfer input data in the external memory DDR to the unified memory 706 or transfer weight data to the weight memory 702 or transfer input data to the input memory 701.
[0252] The vector calculation unit 707 includes multiple operation processing units. When necessary, it further processes the output of the operation circuit, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. It is mainly used for non-convolutional / fully connected layer network calculations in neural networks, such as batch normalization, pixel-level summation, and upsampling of feature planes.
[0253] In some implementations, the vector calculation unit 707 can store the processed output vector in the unified memory 706. For example, the vector calculation unit 707 can apply a linear function or a nonlinear function to the output of the operation circuit 703, such as linear interpolation of the feature plane extracted by the convolution layer, or accumulate a vector of values to generate an activation value. In some implementations, the vector calculation unit 707 generates a normalized value, a pixel-level summed value, or both. In some implementations, the processed output vector can be used as an activation input to the operation circuit 703, for example, for use in subsequent layers in a neural network.
[0254] An instruction fetch buffer 709 connected to the controller 704 is used to store instructions used by the controller 704;
[0255] Unified memory 706, input memory 701, weight memory 702 and instruction fetch memory 709 are all on-chip memories. External memories are private to the NPU hardware architecture.
[0256] The processor mentioned in any of the above places can be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits for controlling the execution of the above program.
[0257] It should also be noted that the device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed across multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment. In addition, in the drawings of the device embodiments provided in this application, the connection relationship between the modules indicates that there is a communication connection between them, which can be specifically implemented as one or more communication buses or signal lines.
[0258] Through the description of the above embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course can also be implemented by special hardware including application-specific integrated circuits, special CPUs, special memories, special components, etc. In general, all functions performed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structures used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits, etc. However, for the present application, software program implementation is a better implementation method in most cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer's floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk or optical disk, etc., and includes a number of instructions to enable a computer device (which can be a personal computer, training equipment, or network equipment, etc.) to execute the method of each embodiment of the present application.
[0259] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.
[0260] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website, a computer, a training device or a data center by wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) mode to another website, a computer, a training device or a data center. The computer-readable storage medium can be any available medium that a computer can store or a data storage device such as a training device, a data center that includes one or more available media integrations. The available medium can be a magnetic medium, (such as a floppy disk, a hard disk, a magnetic tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid-state drive (SSD)).
Claims
1. A data processing method, characterized in that: The method comprises: According to the sensor data and the first prompt text, scene information is obtained through the first language model; the sensor data is data collected about the surrounding environment of the target object, and the first prompt text indicates to extract scene information of the scene where the target object is located according to the sensor data; the target object is the vehicle itself or the robot itself; Generate a second prompt text through a second language model according to the scenario information and the current execution target of the target object, wherein the second prompt text indicates a query for information that the target object needs to know when completing the execution target and that is not included in the scenario information; Obtaining a reply text corresponding to the second prompt text through the first language model according to the sensor data and the second prompt text; According to the reply text, the scenario information and the current execution target, a control instruction of the target object is obtained through the second language model.
2. The method according to claim 1, characterized in that: The scene information includes information indicating a scene category and / or information of perceptual elements included in the scene.
3. The method according to claim 1 or 2, characterized in that: The target object is a vehicle, and the method further includes: Receive navigation instructions from the user; A current execution target of the target object is obtained according to the current position of the vehicle and the navigation instruction.
4. The method according to claim 1 or 2, characterized in that: The target object is a robot; the method further comprises: Receive task instructions from users; The current execution target of the target object is obtained according to the current posture of the robot, the scene map information and the task instruction.
5. The method according to any one of claims 1 to 4, characterized in that: The information that needs to be known and is not included in the scene information includes: state information of a sensory element associated with the execution target; or The implementation method of the execution goal.
6. A data processing device, characterized in that: The device comprises: A processing module is used to obtain scene information through a first language model according to sensor data and a first prompt text; the sensor data is data collected from the surrounding environment of a target object, and the first prompt text indicates that scene information of a scene where the target object is located is extracted according to the sensor data; the target object is the vehicle itself or the robot itself; according to the scene information and the current execution target of the target object, a second prompt text is generated through a second language model, and the second prompt text indicates a query of information that the target object needs to know when completing the execution target and is not included in the scene information; according to the sensor data and the second prompt text, a reply text corresponding to the second prompt text is obtained through the first language model; according to the reply text, the scene information and the current execution target, a control instruction of the target object is obtained through the second language model.
7. The device according to claim 6, characterized in that The scene information includes information indicating a scene category and / or information of perceptual elements included in the scene.
8. The device according to claim 6 or 7, characterized in that The target object is a vehicle, and the device further comprises: An acquisition module, used to receive navigation instructions from users; A current execution target of the target object is obtained according to the current position of the vehicle and the navigation instruction.
9. The device according to any one of claims 6 to 8, characterized in that: The target object is a robot; the device also includes: An acquisition module, used to receive task instructions from users; The processing module is further used to obtain the current execution target of the target object according to the current posture of the robot, scene map information and the task instruction.
10. The device according to any one of claims 6 to 9, characterized in that: The information that needs to be known and is not included in the scene information includes: state information of a sensory element associated with the execution target; or The implementation method of the execution goal.
11. A data processing device, characterized in that: The server comprises a memory and a processor; the memory stores codes, the processor is configured to execute the codes, and when the codes are executed, the server executes the method according to any one of claims 1 to 5.
12. A computer storage medium, characterized in that: The computer storage medium stores a computer program, which, when executed by a computer, enables the computer to implement the method according to any one of claims 1 to 5.
13. A computer program product, characterized in that The computer program product stores instructions, which, when executed by a computer, enable the computer to implement the method according to any one of claims 1 to 5.
14. A chip, comprising a processor, characterized in that: The processor is used to support a data processing device to implement the method according to any one of claims 1 to 5.
Citation Information
Patent Citations
A data processing method and related apparatus thereof
CN116701586B
Voice recognition method and device and electronic equipment
CN111816165A
Man-machine interaction training method and device based on reinforcement learning strategy
CN114417086A
Method, device and equipment for generating chat reply based on dynamic Prompt and medium
CN115599896A
Data processing method and related device thereof
CN116701586A
Cited By
Vehicle control method, server and computer readable storage medium
CN120148502A
Environment data processing method and electronic equipment
CN120256877A
Parameter estimation system and method
WO2026143473A1
