A data processing method and related apparatus thereof

Abstract

A data processing method applied to the field of artificial intelligence, the method comprising: obtaining scene information according to sensor data and a first prompt text through a first language model; generating a second prompt text according to the scene information and a current execution target of a target object (a vehicle itself or a robot itself) through a second language model, the second prompt text indicating a query for information that needs to be known when completing the execution target of the target object and is not included in the scene information; obtaining a reply text corresponding to the second prompt text according to the sensor data and the second prompt text through the first language model; and obtaining a control instruction of the target object according to the reply text, the scene information and the current execution target through the second language model. The application constructs an end-to-end automatic control system through the interaction between the first language model and the second language model, and can improve the decision accuracy compared with a multi-stage decision system.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and in particular to a data processing method and related apparatus. Background Technology

[0002] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0003] For robots or vehicles, automatic control typically involves three tasks: perception, prediction, and decision-making. Taking autonomous driving as an example, the perception task involves receiving raw sensor data (images or lasers, etc.) as input and identifying dynamic targets (such as pedestrians and vehicles, their size and position) and static elements (such as lane lines and arrows on the ground). The prediction task receives the raw results of the perception task regarding dynamic and static targets and infers the movement intentions of targets in the environment, primarily predicting the future trajectory and intentions of other vehicles. The decision-making task receives the prediction results from the prediction task, and based on the future driving intentions of other vehicles and the vehicle's navigation task, generates a decision conclusion, outputs control signals, and controls the vehicle to perform autonomous driving.

[0004] However, in the existing architecture, the functions of each task module are decoupled, exhibiting a clear upstream and downstream relationship. This is achieved by pre-defining the output interfaces of each module for interconnection. While this approach offers advantages such as strong interpretability and ease of modular development and maintenance, the decoupling of front-end and back-end modules means that if a front-end module malfunctions, the downstream module's decision-making will also be flawed, resulting in poor control precision. Summary of the Invention

[0005] This application provides a data processing method that can improve decision-making accuracy.

[0006] In a first aspect, this application provides a data processing method, the method comprising: obtaining scene information based on sensor data and a first prompt text using a first language model; wherein the sensor data is data collected from the surrounding environment of a target object, and the first prompt text indicates the extraction of scene information of the scene where the target object is located based on the sensor data; the target object is a vehicle itself or a robot itself; generating a second prompt text based on the scene information and the current execution target of the target object using a second language model, the second prompt text indicating the query for information that the target object needs to know to complete the execution target and which is not included in the scene information; obtaining a response text corresponding to the second prompt text based on the sensor data and the second prompt text using the first language model; and obtaining control instructions for the target object based on the response text, the scene information, and the current execution target using the second language model.

[0007] In this embodiment, the second language model obtains the query text form of the requirement information (second prompt text) based on different scenario information. The first language model obtains the response text based on the query text, and the response text is passed to the second language model for decision-making. On the one hand, an end-to-end automatic control system is constructed through the interaction between the first language model and the second language model. Compared with multi-stage decision-making systems, this application can improve decision-making accuracy. On the other hand, the introduction of language models into this architecture can extend the inductive reasoning ability of the large language model to the control domain. The expression mode of the language model's text is different from the traditional fixed format and can represent an infinite number of perceptual needs, thereby unifying the acquisition method of various types of perceptual need conclusions of the second language model.

[0008] In one possible implementation, the scene information includes information indicating the scene category and / or information about the perceptual elements included in the scene.

[0009] In one possible implementation, the target object is a vehicle, and the method further includes:

[0010] Receive navigation commands from the user;

[0011] Based on the vehicle's current location and the navigation instructions, the target object's current execution objective is obtained.

[0012] In one possible implementation, the target object is a robot; the method further includes:

[0013] Receive task instructions from the user;

[0014] Based on the robot's current posture, scene map information, and the task instructions, the current execution target of the target object is obtained.

[0015] In one possible implementation, the information that needs to be known but is not included in the scenario information includes:

[0016] The state information of the sensing element associated with the execution target; or,

[0017] The method for achieving the execution objective.

[0018] In one possible implementation, the first language model and the second language model are chatGPT, GPT4, or chatGLM.

[0019] Secondly, this application provides a data processing apparatus, the apparatus comprising:

[0020] The processing module is configured to obtain scene information based on sensor data and a first prompt text using a first language model; the sensor data is data collected from the surrounding environment of the target object, and the first prompt text indicates the extraction of scene information of the scene where the target object is located based on the sensor data; the target object is the vehicle itself or the robot itself; based on the scene information and the current execution goal of the target object, a second prompt text is generated using a second language model, the second prompt text indicating the query for information that the target object needs to know to complete the execution goal but is not included in the scene information; based on the sensor data and the second prompt text, a response text corresponding to the second prompt text is obtained using the first language model; based on the response text, the scene information, and the current execution goal, control instructions for the target object are obtained using the second language model.

[0021] In one possible implementation, the target object is a vehicle or a robot.

[0022] In one possible implementation, the scene information includes information indicating the scene category and / or information about the perceptual elements included in the scene.

[0023] In one possible implementation, the target object is a vehicle, and the device further includes:

[0024] The acquisition module is used to receive navigation commands from the user;

[0025] Based on the vehicle's current location and the navigation instructions, the target object's current execution objective is obtained.

[0026] In one possible implementation, the target object is a robot; the device further includes:

[0027] The acquisition module is used to receive task instructions from the user;

[0028] The processing module is further configured to obtain the current execution target of the target object based on the robot's current posture, scene map information, and the task instructions.

[0029] In one possible implementation, the information that needs to be known but is not included in the scenario information includes:

[0030] The state information of the sensing element associated with the execution target; or,

[0031] The method for achieving the execution objective.

[0032] In one possible implementation, the first language model and the second language model are chatGPT, GPT4, or chatGLM.

[0033] Thirdly, this application provides a data processing apparatus, the server including a memory and a processor; the memory stores code, and the processor is configured to execute the code, and when the code is executed, the server performs the method as described in the first aspect or any possible implementation thereof.

[0034] Fourthly, this application provides a vehicle that includes the data processing apparatus as described in the third aspect.

[0035] Fifthly, this application provides a robot, the vehicle of which includes the data processing device as described in the third aspect.

[0036] In a sixth aspect, this application provides a computer storage medium storing a computer program that, when executed by a computer, causes the computer to perform the method as described in the first aspect or any possible implementation thereof.

[0037] In a seventh aspect, this application provides a computer program product that stores instructions that, when executed by a computer, cause the computer to perform the method as described in the first aspect or any possible implementation thereof.

[0038] Eighthly, this application provides a chip system including a processor for supporting a data processing device in implementing the functions involved in the foregoing aspects, such as transmitting or processing data involved in the foregoing methods; or, information. In one possible design, the chip system further includes a memory for storing program instructions and data necessary for executing or training the device. The chip system may be composed of chips or may include chips and other discrete devices. Attached Figure Description

[0039] Figure 1aA structural diagram illustrating the main framework of artificial intelligence;

[0040] Figure 1b This is a schematic diagram of the structure of an autonomous driving device;

[0041] Figure 1c This is a schematic diagram of the structure of a robot;

[0042] Figure 1d This is a schematic diagram of the system structure;

[0043] Figure 2 A flowchart illustrating a data processing method provided in an embodiment of this application;

[0044] Figure 3 A flowchart illustrating a data processing method provided in an embodiment of this application;

[0045] Figure 4 A schematic diagram of the structure of a data processing apparatus provided in an embodiment of this application;

[0046] Figure 5 A schematic diagram of the structure of the execution device provided in the embodiments of this application;

[0047] Figure 6 A schematic diagram of the structure of a server provided in an embodiment of this application;

[0048] Figure 7 This is a schematic diagram of a chip structure provided in an embodiment of this application. Detailed Implementation

[0049] The embodiments of the present invention will now be described with reference to the accompanying drawings. The terminology used in the embodiments section is for illustrative purposes only and is not intended to limit the scope of the invention.

[0050] The embodiments of this application will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.

[0051] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0052] First, the overall workflow of the artificial intelligence system is described; please refer to [link / reference]. Figure 1a , Figure 1a The diagram illustrates a structural framework for artificial intelligence (AI). The framework is further elaborated below along two dimensions: the "Intelligent Information Chain" (horizontal axis) and the "IT Value Chain" (vertical axis). The "Intelligent Information Chain" reflects a series of processes from data acquisition to processing. For example, this process could be a general sequence of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, and intelligent execution and output. In this process, data undergoes a condensation process of "data—information—knowledge—wisdom." The "IT Value Chain" reflects the value that AI brings to the information technology industry, from the underlying infrastructure of human intelligence and information (provided and processed through technological means) to the industrial ecosystem of the system.

[0053] (1) Infrastructure

[0054] Infrastructure provides computing power to support artificial intelligence systems, enabling communication with the external world and providing support through a basic platform. This communication occurs through sensors; computing power is provided by intelligent chips (hardware acceleration chips such as CPUs, NPUs, GPUs, ASICs, and FPGAs); and the basic platform includes distributed computing frameworks and related platform guarantees and support, which may include cloud storage and computing, interconnected networks, etc. For example, sensors communicate with the outside world to acquire data, and this data is provided to intelligent chips in the distributed computing system provided by the basic platform for computation.

[0055] (2) Data

[0056] The data at the next layer of infrastructure is used to represent the data sources in the field of artificial intelligence. The data involves graphics, images, voice, text, and IoT data from traditional devices, including business data from existing systems and sensor data such as force, displacement, liquid level, temperature, and humidity.

[0057] (3) Data processing

[0058] Data processing typically includes methods such as data training, machine learning, deep learning, search, reasoning, and decision-making.

[0059] Among them, machine learning and deep learning can perform intelligent information modeling, extraction, preprocessing, and training on data, including symbolization and formalization.

[0060] Reasoning refers to the process in which, in a computer or intelligent system, the machine thinks and solves problems by simulating human intelligent reasoning, based on reasoning control strategies and using formalized information. Typical functions include search and matching.

[0061] Decision-making refers to the process of making decisions based on intelligent information after reasoning, and it typically provides functions such as classification, sorting, and prediction.

[0062] (4) General ability

[0063] After the data processing mentioned above, the results of the data processing can be used to form some general capabilities, such as algorithms or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0064] (5) Smart Products and Industry Applications

[0065] Intelligent products and industry applications refer to products and applications of artificial intelligence systems in various fields. They are the encapsulation of overall artificial intelligence solutions, productizing intelligent information decision-making and realizing practical applications. Their application areas mainly include: intelligent terminals, intelligent transportation, intelligent healthcare, autonomous driving, smart cities, etc.

[0066] This application can be applied to the field of autonomous driving of vehicles or the field of robot control.

[0067] The vehicle in this application embodiment may refer to an internal combustion engine vehicle that uses an engine as a power source, a hybrid vehicle that uses both an engine and an electric motor as power sources, an electric vehicle that uses an electric motor as a power source, and so on.

[0068] In this embodiment of the application, the vehicle may include an autonomous driving device 100 with autonomous driving function.

[0069] Reference Figure 1b , Figure 1bThis is a functional block diagram of an autonomous driving device 100 with autonomous driving capabilities provided in an embodiment of this application. In one embodiment, the autonomous driving device 100 may include various subsystems, such as a driving system 102, a sensor system 104, a control system 106, one or more peripheral devices 108, a power supply 110, a computer system 112, and a user interface 116. Optionally, the autonomous driving device 100 may include more or fewer subsystems, and each subsystem may include multiple components. Furthermore, each subsystem and component of the autonomous driving device 100 may be interconnected via wired or wireless means.

[0070] The mobility system 102 may include components that provide powered motion for the autonomous driving device 100. In one embodiment, the mobility system 102 may include an engine 118, an energy source 119, a transmission 120, and wheels / tires 121. The engine 118 may be an internal combustion engine, an electric motor, an air-compressed engine, or other types of engine combinations, such as a hybrid engine consisting of a gasoline engine and an electric motor, or a hybrid engine consisting of an internal combustion engine and an air-compressed engine. The engine 118 converts the energy source 119 into mechanical energy.

[0071] Examples of energy sources 119 include gasoline, diesel, other petroleum-based fuels, propane, other compressed gas-based fuels, ethanol, solar panels, batteries, and other sources of electricity. Energy source 119 may also provide power to other systems of the autonomous driving device 100.

[0072] The transmission 120 can transmit mechanical power from the engine 118 to the wheels 121. The transmission 120 may include a gearbox, a differential, and a drive shaft. In one embodiment, the transmission 120 may also include other components, such as a clutch. The drive shaft may include one or more axles that can be coupled to one or more wheels 121.

[0073] Sensor system 104 may include several sensors that sense information about the environment surrounding autonomous driving device 100. For example, sensor system 104 may include a positioning system 122 (which may be a Global Positioning System (GPS) system, a BeiDou system, or another positioning system), an inertial measurement unit (IMU) 124, radar 126, a laser rangefinder 128, and a camera 130. Sensor system 104 may also include sensors monitoring the internal systems of autonomous driving device 100 (e.g., in-vehicle air quality monitor, fuel gauge, oil temperature gauge, etc.). Sensor data from one or more of these sensors can be used to detect objects and their corresponding characteristics (position, shape, orientation, speed, etc.). This detection and identification is a key function for the safe operation of autonomous driving device 100.

[0074] The positioning system 122 can be used to estimate the geographic location of the autonomous driving device 100. The IMU 124 is used to sense changes in the position and orientation of the autonomous driving device 100 based on inertial acceleration. In one embodiment, the IMU 124 can be a combination of an accelerometer and a gyroscope.

[0075] Radar 126 can use radio signals to sense objects in the surrounding environment of the autonomous driving device 100. In some embodiments, in addition to sensing objects, radar 126 can also be used to sense the speed and / or direction of travel of objects.

[0076] Radar 126 may include an electromagnetic wave transmitting unit and a receiving unit. In terms of electromagnetic wave transmission principle, radar 126 can be implemented as either pulse radar or continuous wave radar. In the continuous wave radar mode, radar 126 can be implemented as either frequency-modulated continuous wave (FMCW) or frequency shift keying (FSK) depending on the signal waveform.

[0077] Radar 126 uses electromagnetic waves as a medium to detect objects based on time-of-flight (TOF) or phase-shift methods, and detects the position of the detected object, the distance to the detected object, and the relative velocity. To detect objects located in front of, behind, or to the side of the vehicle, radar 126 can be configured at an appropriate location on the exterior of the vehicle. LiDAR 126 uses laser light as a medium to detect objects based on TOF or phase-shift methods, and detects the position of the detected object, the distance to the detected object, and the relative velocity.

[0078] Alternatively, in order to detect objects located in front of, behind or to the side of the vehicle, the lidar 126 may be configured at an appropriate location on the exterior of the vehicle.

[0079] The laser rangefinder 128 can use lasers to sense objects in the environment where the autonomous driving device 100 is located. In some embodiments, the laser rangefinder 128 may include one or more laser sources, a laser scanner, and one or more detectors, as well as other system components.

[0080] Camera 130 can be used to capture multiple images of the surrounding environment of the autonomous driving device 100. Camera 130 can be a still camera or a video camera.

[0081] Optionally, to acquire images of the vehicle's exterior, the camera 130 can be located at an appropriate position outside the vehicle. For example, to acquire images of the front of the vehicle, the camera 130 can be positioned inside the vehicle's interior, close to the windshield. Alternatively, the camera 130 can be positioned around the front bumper or radiator grille. For example, to acquire images of the rear of the vehicle, the camera 130 can be positioned inside the vehicle's interior, close to the rear window. Alternatively, the camera 130 can be positioned around the rear bumper, trunk, or tailgate. For example, to acquire images of the sides of the vehicle, the camera 130 can be positioned inside the vehicle's interior, close to at least one of the side windows. Alternatively, the camera 130 can be positioned around a side mirror, fender, or door.

[0082] In this embodiment of the application, sensor data can be acquired based on one or more sensors in the sensor system 104, etc.

[0083] The control system 106 controls the operation of the autonomous driving device 100 and its components. The control system 106 may include various elements, including a steering system 132, a throttle 134, a braking unit 136, a sensor fusion algorithm 138, a computer vision system 140, a route control system 142, and an obstacle avoidance system 144.

[0084] The steering system 132 is operable to adjust the forward direction of the automatic driving device 100. For example, in one embodiment, it can be a steering wheel system.

[0085] Throttle 134 is used to control the operating speed of engine 118 and thus the speed of automatic driving device 100.

[0086] Braking unit 136 is used to control the deceleration of the automatic driving device 100. Braking unit 136 can use friction to slow down the wheel 121. In other embodiments, braking unit 136 can convert the kinetic energy of the wheel 121 into electrical current. Braking unit 136 may also take other forms to slow down the rotational speed of the wheel 121 to control the speed of the automatic driving device 100.

[0087] The computer vision system 140 is operable to process and analyze images captured by the camera 130 to identify objects and / or features in the surrounding environment of the autonomous driving device 100. These objects and / or features may include traffic signals, road boundaries, and obstacles. The computer vision system 140 may use object recognition algorithms, structure from motion (SFM) algorithms, video tracking, and other computer vision techniques. In some embodiments, the computer vision system 140 may be used to map the environment, track objects, estimate object velocities, and so on.

[0088] The route control system 142 is used to determine the driving route of the autonomous driving device 100. In some embodiments, the route control system 142 may combine data from the sensor 138, the positioning system 122, and one or more predetermined maps to determine the driving route for the autonomous driving device 100.

[0089] The obstacle avoidance system 144 is used to identify, assess and avoid or otherwise traverse potential obstacles in the environment of the autonomous driving device 100.

[0090] Of course, in one instance, the control system 106 may include additional or alternative components besides those shown and described. Alternatively, some of the components shown above may be reduced.

[0091] The autonomous driving device 100 interacts with external sensors, other autonomous driving devices, other computer systems, or users via peripheral devices 108. Peripheral devices 108 may include a wireless communication system 146, an onboard computer 148, a microphone 150, and / or a speaker 152.

[0092] In some embodiments, peripheral device 108 provides a means for the user of autonomous driving device 100 to interact with user interface 116. For example, on-board computer 148 may provide information to the user of autonomous driving device 100. User interface 116 may also operate on-board computer 148 to receive user input. On-board computer 148 may be operated via touchscreen. In other cases, peripheral device 108 may provide a means for autonomous driving device 100 to communicate with other devices located within the vehicle. For example, microphone 150 may receive audio (e.g., voice commands or other audio input) from the user of autonomous driving device 100. Similarly, speaker 152 may output audio to the user of autonomous driving device 100.

[0093] The wireless communication system 146 can communicate wirelessly with one or more devices directly or via a communication network. For example, the wireless communication system 146 can use 3G cellular communication, such as code division multiple access (CDMA), EVDO, Global System for Mobile Communications (GSM) / General Packet Radio Service (GPRS), or 4G cellular communication, such as long term evolution (LTE), or 5G cellular communication. The wireless communication system 146 can communicate using WiFi and a wireless local area network (WLAN). In some embodiments, the wireless communication system 146 can communicate directly with devices using an infrared link, Bluetooth, or ZigBee. Other wireless protocols, such as various autonomous driving device communication systems, may also be used. For example, the wireless communication system 146 may include one or more dedicated short-range communications (DSRC) devices, which may include public and / or private data communications between autonomous driving devices and / or roadside stations.

[0094] Power source 110 can provide power to various components of autonomous driving device 100. In one embodiment, power source 110 can be a rechargeable lithium-ion or lead-acid battery. One or more such battery packs can be configured to provide power to various components of autonomous driving device 100. In some embodiments, power source 110 and energy source 119 can be implemented together, as is the case in some fully electric vehicles.

[0095] Some or all of the functions of the autonomous driving device 100 are controlled by a computer system 112. The computer system 112 may include at least one processor 113, which executes instructions 115 stored in a non-transitory computer-readable medium such as memory 114. The computer system 112 may also be multiple computing devices that control individual components or subsystems of the autonomous driving device 100 in a distributed manner.

[0096] Processor 113 can be any conventional processor, such as a commercially available central processing unit (CPU). Alternatively, the processor can be a special-purpose device such as an application-specific integrated circuit (ASIC) or other hardware-based processor. Although Figure 1b The processor, memory, and other components of computer 110 within the same block are functionally illustrated; however, those skilled in the art will understand that the processor, computer, or memory may actually include multiple processors, computers, or memories that may or may not be stored in the same physical housing. For example, memory may be a hard disk drive or other storage media located in a housing different from that of computer 110. Therefore, references to processors or computers will be understood to include references to a collection of processors or computers or memories that may or may not operate in parallel. Unlike using a single processor to perform the steps described herein, some components, such as steering and deceleration components, may each have their own processor that performs calculations only relevant to the component's specific function.

[0097] In the various aspects described herein, the processor may be located remotely from the autonomous driving device and may communicate wirelessly with the autonomous driving device. In other aspects, some of the processes described herein are executed on a processor located within the autonomous driving device, while others are executed by a remote processor, including taking the necessary steps to perform a single operation.

[0098] In some embodiments, memory 114 may contain instructions 115 (e.g., program logic) that can be executed by processor 113 to perform various functions of the autonomous driving device 100, including those described above. Memory 114 may also contain additional instructions, including instructions to send data to, receive data from, interact with, and / or control one or more of the mobility system 102, sensor system 104, control system 106, and peripheral devices 108.

[0099] In addition to instruction 115, memory 114 may also store data such as road maps, route information, the location, direction, speed, and other data of the autonomous driving device, as well as other information. This information can be used by the autonomous driving device 100 and computer system 112 during operation of the autonomous, semi-autonomous, and / or manual modes.

[0100] The data processing method provided in this application embodiment can be software code stored in memory 114. Furthermore, the models in this application embodiment (e.g., a first language model, a second language model) can be stored in memory 114. The processor 113 can retrieve the software code from the memory and execute the retrieved software code to implement the data processing method provided in this application embodiment. After obtaining the control signal of the target vehicle, the control signal can be transmitted to the control system 106. The control system 106 can determine the vehicle's driving strategy based on the control signal or directly perform driving control.

[0101] User interface 116 is used to provide information to or receive information from the user of autonomous driving device 100. Optionally, user interface 116 may include one or more input / output devices within a set of peripheral devices 108, such as wireless communication system 146, vehicle computer 148, microphone 150, and speaker 152.

[0102] Computer system 112 can control the functions of autonomous driving device 100 based on input received from various subsystems (e.g., driving system 102, sensor system 104, and control system 106) and from user interface 116. For example, computer system 112 can utilize input from control system 106 to control steering unit 132 to avoid obstacles detected by sensor system 104 and obstacle avoidance system 144. In some embodiments, computer system 112 is operable to provide control over many aspects of autonomous driving device 100 and its subsystems.

[0103] Alternatively, one or more of these components may be installed separately from or associated with the autonomous driving device 100. For example, the memory 114 may exist partially or completely separately from the autonomous driving device 100. The components may be communicatively coupled together in a wired and / or wireless manner.

[0104] Optionally, the components described above are merely examples. In actual applications, components in each of the above modules may be added or removed as needed. Figure 1b This should not be construed as a limitation on the embodiments of this application.

[0105] Furthermore, the embodiments of this application can also be applied to robot 13.

[0106] Next, we will introduce the architecture diagram of robot 13.

[0107] The embodiments of this application can be applied to, for example... Figure 1c In robot 13 shown. As Figure 1c As shown, the robot 13 may include: a sensor module 110, a drive device 120, a control device 140, and a main control platform 130.

[0108] The sensor module 110 may include one or more vision sensors 111 (e.g., cameras) (which may also be referred to as sensors in this embodiment). For example, it may include a common optical camera, an infrared camera, a structured light sensor, or a time-of-flight (ToF) sensor. For instance, the sensor module 110 may include a common RGB camera or a red-yellow-yellow-blue (RYYB) camera. The camera module may also include multiple cameras or sensors forming an RGB-D depth camera solution. For instance, the RGB-D depth camera solution may include two RGB cameras forming a binocular solution, one RGB camera and one structured light sensor forming a structured light solution, or one RGB camera and one ToF sensor forming a ToF solution. This embodiment does not specifically limit the specific implementation. Furthermore, the vision sensor 111 (e.g., a camera) may be a fixed-focus camera or a zoom camera, and may possess capabilities such as phase detection autofocus or laser autofocus.

[0109] It should be understood that the visual sensor 111 (e.g., a camera) can be mounted on a motion unit, which carries the visual sensor 111 (e.g., the camera) and drives the visual sensor 111 (e.g., the camera) to rotate. In one embodiment, the motion unit can drive the visual sensor 111 (e.g., the camera) to rotate with two degrees of freedom. If the direction pointed to by the Z-axis is directly in front of the camera, then the two degrees of freedom rotation can include the rotation of the visual sensor 111 (e.g., the camera) about the x-axis and the rotation of the visual sensor 111 (e.g., the camera) about the y-axis. The motion unit drives the vision sensor 111 (e.g., a camera) to rotate, which can be achieved by the rotation of a servo motor or a servo motor. For example, when the drive device is used to drive the vision sensor 111 (e.g., a camera) to generate two degrees of freedom rotation, the motion unit can include two drive mechanisms, driver 1 and driver 2, such as two servo motors or two servo motors. One servo motor 1 or servo motor 1 is used to control the rotation of the vision sensor 111 (e.g., a camera) about the x-axis, and the other servo motor 2 or servo motor 2 is used to control the rotation of the vision sensor 111 (e.g., a camera) about the y-axis. In other embodiments, the motion unit can drive the vision sensor 111 (e.g., a camera) to generate three degrees of freedom rotation, that is, increase the rotation of the vision sensor 111 (e.g., the camera) about the z-axis. Accordingly, the motion unit may also include three drive mechanisms driver 1, driver 2 and driver 3, such as three servo motors or three servo motors. One servo motor 1 or servo motor 1 is used to control the rotation of the vision sensor 111 (e.g., the camera) about the x-axis, another servo motor 2 or servo motor 2 is used to control the rotation of the vision sensor 111 (e.g., the camera) about the y-axis, and another servo motor 3 or servo motor 3 is used to control the rotation of the vision sensor 111 (e.g., the camera) about the z-axis.

[0110] The sensor module 110 may also include a motion sensor 112, which may be an odometer, accelerometer, speedometer, inertial measurement unit, etc., used to collect mileage information of the robot 13 during its travel, such as travel distance, trajectory, speed and other information.

[0111] Force sensor 113 can be a sensor used to detect the force applied to the end effector of robotic arm 142. Force sensor 113 can be a pressure sensor capable of detecting force in a single-axis direction, or a force sensor or torque sensor capable of detecting force components in multiple axial directions. Optionally, a six-axis force sensor can be used as force sensor 113. A six-axis force sensor detects the magnitude of the force parallel to three mutually orthogonal detection axes in an inherent sensor coordinate system and the magnitude of the torque about the three detection axes. It should be noted that force sensor 113 can be located at a position other than the end effector of robotic arm 142; for example, it can be located on one or more joints of robotic arm 142.

[0112] The drive unit 120 may include components that provide powered motion for the robot 13. In one embodiment, the drive unit 120 may include an engine, an energy source, a transmission, and wheels / tires. The engine may be an internal combustion engine, an electric motor, an air compressor engine, or other combinations of engines, such as a hybrid engine consisting of a gasoline engine and an electric motor, or a hybrid engine consisting of an internal combustion engine and an air compressor engine. The engine converts the energy source into mechanical energy.

[0113] Examples of energy sources include gasoline, diesel, other petroleum-based fuels, propane, other compressed gas-based fuels, ethanol, solar panels, batteries, and other sources of electricity. These energy sources can also power other systems within the robot 13.

[0114] A transmission system can transmit mechanical power from an engine to the wheels. The transmission system may include a differential and a drive shaft. In one embodiment, the transmission system may also include other components, such as a clutch. The drive shaft may include one or more axles that can be coupled to one or more wheels.

[0115] The main control platform 130 is the data processing and control center of the device. The main control platform 130 establishes communication connections with the control device 140, sensor module 110, and drive device 120. For example, it can receive image data collected by the sensor module 110, process the image data, and send movement commands to the drive device 120. The control device 140 may include a robotic arm 142. In some embodiments, the main control platform 130 may be an embedded computer platform, including but not limited to computer chips and software systems designed based on x86, ARM, RISC-V, or MIPS instruction sets. The main control platform 130 can perform tasks by controlling the posture of the robotic arm 142 through control commands.

[0116] In one embodiment, the aforementioned computer chip may include, for example, a processor 131 and a memory 132. The processor 131 may include, for example, a central processing unit (CPU), a system-on-a-chip (SoC), an application processor (AP), a microcontroller, a neural-network processing unit (NPU), and / or a graphics processing unit (GPU). The memory 132 may include, for example, non-volatile memory and volatile memory. Non-volatile memory may include flash memory, such as NAND flash, solid-state drives, etc., and volatile memory may include synchronous dynamic random-access memory (SDRAM), etc.

[0117] In one embodiment, the software system may include an operating system and program instructions 133 running on the operating system. When the processor executes the program instructions, it causes... Figure 1c The apparatus shown performs each step of the data processing method provided in the embodiments of this application.

[0118] In some embodiments, memory 132 may contain program instructions 133 (e.g., program logic) that can be executed by processor 131 to perform various functions of robot 13, including those described above. Memory 132 may also contain additional instructions, including instructions for sending data to, receiving data from, interacting with, and / or controlling one or more of the drive unit 120, sensor module 110, control system, and peripheral devices.

[0119] In addition to program instructions 133, memory 132 may also store data such as road maps, route information, the position, direction, speed, and other data of the autonomous driving device, as well as other information. This information can be used by robot 13 during operation in autonomous, semi-autonomous, and / or manual modes.

[0120] It should be understood that the data processing methods in the embodiments of this application involve artificial intelligence-related algorithm processing; therefore, the architecture of the processor is not limited to... Figure 1c The processor combined with memory can also be other hardware architectures (such as pure hardware or other hardware and software combined architectures).

[0121] The wireless communication system 150 can communicate wirelessly with one or more devices (e.g., servers) directly or via a communication network. For example, the wireless communication system 150 can use 3G cellular communication, such as code division multiple access (CDMA), EVDO, Global System for Mobile Communications (GSM) / General Packet Radio Service (GPRS), or 4G cellular communication, such as long term evolution (LTE), or 5G cellular communication. The wireless communication system 150 can communicate using WiFi and a wireless local area network (WLAN). In some embodiments, the wireless communication system 150 can communicate directly with devices using an infrared link, Bluetooth, or ZigBee. Other wireless protocols, such as various autonomous driving device communication systems, may also be used. For example, the wireless communication system 150 may include one or more dedicated short range communications (DSRC) devices, which may include public and / or private data communications between autonomous driving devices and / or roadside stations.

[0122] Alternatively, one or more of these components may be installed separately from or associated with robot 13. For example, memory 132 may exist partially or completely separately from robot 13. The components may be communicatively coupled together in a wired and / or wireless manner.

[0123] Optionally, the components described above are merely examples. In actual applications, components in each of the above modules may be added or removed as needed. Figure 1c This should not be construed as a limitation on the embodiments of this application.

[0124] The steps related to the model inference process in the embodiments of this application involve AI-related calculations, which will be discussed below. Figure 1d The system architecture provided in the embodiments of this application will be described in detail.

[0125] Figure 1d This is a schematic diagram of the system architecture provided for an embodiment of this application. Figure 1d As shown, the system architecture 500 includes an execution device 510, a training device 520, a database 530, a client device 540, a data storage system 550, and a data acquisition system 560.

[0126] The execution device 510 includes a calculation module 511, an I / O interface 512, a preprocessing module 513, and a preprocessing module 514. The calculation module 511 may include a target model / rule 501, while the preprocessing modules 513 and 514 are optional.

[0127] The data acquisition device 560 is used to collect training samples. After collecting the training samples, the data acquisition device 560 stores these training samples in the database 530.

[0128] The training device 520 can maintain training samples in the database 530 to obtain the target model / rule 501 from the neural network to be trained (e.g., the first language model and the second language model in the embodiments of this application).

[0129] It should be understood that the training device 520 can perform a pre-training process on the neural network to be trained based on the training samples maintained in the database 530, or fine-tune the model based on the pre-training.

[0130] It should be noted that in practical applications, the training samples maintained in database 530 may not all come from the data acquisition device 560; they may also be received from other devices. Furthermore, it should be noted that training device 520 may not necessarily train the target model / rule 501 entirely based on the training samples maintained in database 530; it may also obtain training samples from the cloud or other sources for model training. The above description should not be construed as limiting the embodiments of this application.

[0131] The target model / rule 501 trained using training device 520 can be applied to different systems or devices, such as... Figure 1d The execution device 510 shown can be a vehicle or a robot, etc.

[0132] Specifically, the training device 520 can transfer the trained model to the execution device 510.

[0133] The execution device 510 can be a vehicle or a robot. In one implementation, the training device 530 can perform the pre-training or fine-tuning process of the model, deploy the trained model to the execution device 510, and the execution device 510 can execute the trained model to implement the data processing method in the embodiments of this application.

[0134] exist Figure 1dIn this embodiment, the execution device 510 is configured with an input / output (I / O) interface 512 for data interaction with external devices. Users can input data (such as navigation requests or task execution requests in this embodiment) into the I / O interface 512 through the client device 540. In addition, the input data may also include sensor data.

[0135] Preprocessing modules 513 and 514 are used to preprocess the input data received from the I / O interface 512. It should be understood that preprocessing modules 513 and 514 may be absent, or only one preprocessing module may be used. When preprocessing modules 513 and 514 are absent, the calculation module 511 can be used directly to process the input data.

[0136] During the preprocessing of input data by the execution device 510, or during the calculation module 511 of the execution device 510 performing calculations and other related processes, the execution device 510 can call data, code, etc. in the data storage system 550 for corresponding processing, or store the data, instructions, etc. obtained from the corresponding processing into the data storage system 550.

[0137] Finally, the I / O interface 512 provides the processing result to the client device 540, thereby providing it to the user.

[0138] exist Figure 1d In the illustrated scenario, the user can manually provide input data, which can be done through the interface provided by I / O interface 512. Alternatively, the client device 540 can automatically send input data to I / O interface 512. If user authorization is required for the client device 540 to automatically send input data, the user can set the corresponding permissions in the client device 540. The user can view the output results of the execution device 510 on the client device 540, which can be presented in various forms such as display, sound, or animation. The client device 540 can also act as a data acquisition terminal, collecting the input data and output results of the input I / O interface 512 as shown in the figure, and storing them as new sample data in database 530. Alternatively, data can be collected directly from the I / O interface 512 without going through the client device 540, using the input data and output results of the input I / O interface 512 as shown in the figure, and storing them as new sample data in database 530.

[0139] It is worth noting that, Figure 1d This is merely a schematic diagram of a system architecture provided in an embodiment of this application. The positional relationships between the devices, components, modules, etc., shown in the diagram do not constitute any limitation. For example, in Figure 1dIn this context, the data storage system 550 is an external storage device relative to the execution device 510. However, in other cases, the data storage system 550 may also be placed within the execution device 510. It should be understood that the aforementioned execution device 510 may be deployed within the client device 540.

[0140] From the inference side of the model:

[0141] In this embodiment, the computing module 511 of the execution device 520 can obtain the code stored in the data storage system 550 to implement the steps related to the model reasoning process in this embodiment.

[0142] In this embodiment of the application, the computing module 511 of the execution device 520 may include hardware circuits (such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), general-purpose processors, digital signal processors (DSPs), microprocessors or microcontrollers, etc.) or combinations of these hardware circuits. For example, the training device 520 may be a hardware system with instruction execution capabilities, such as a CPU or DSP, or a hardware system without instruction execution capabilities, such as an ASIC or FPGA, or a combination of the aforementioned hardware systems without instruction execution capabilities and hardware systems with instruction execution capabilities.

[0143] Specifically, the computing module 511 of the execution device 520 can be a hardware system with the function of executing instructions. The steps related to the model inference process provided in this application embodiment can be software code stored in the memory. The computing module 511 of the execution device 520 can obtain the software code from the memory and execute the obtained software code to implement the steps related to the model inference process provided in this application embodiment.

[0144] It should be understood that the computing module 511 of the execution device 520 can be a combination of a hardware system without the function of executing instructions and a hardware system with the function of executing instructions. Some steps related to the model reasoning process provided in the embodiments of this application can also be implemented by the hardware system in the computing module 511 of the execution device 520 without the function of executing instructions, which is not limited here.

[0145] From the training side of the model:

[0146] In this embodiment of the application, the training device 520 can access the memory ( Figure 1d(Not shown in the diagram, but can be integrated into the training device 520 or deployed separately from the training device 520) The code stored in the diagram can be used to implement the steps related to model training in the embodiments of this application.

[0147] In this embodiment of the application, the training device 520 may include hardware circuits (such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), general-purpose processors, digital signal processors (DSPs), microprocessors or microcontrollers, etc.) or combinations of these hardware circuits. For example, the training device 520 may be a hardware system with instruction execution capabilities, such as a CPU or DSP, or a hardware system without instruction execution capabilities, such as an ASIC or FPGA, or a combination of the aforementioned hardware systems without instruction execution capabilities and hardware systems with instruction execution capabilities.

[0148] It should be understood that the training device 520 can be a combination of a hardware system without the function of executing instructions and a hardware system with the function of executing instructions. Some steps related to the training of the neutralization model provided in the embodiments of this application can also be implemented by the hardware system in the training device 520 without the function of executing instructions, which is not limited here.

[0149] The application architecture of the embodiments of this application has been described above. The intent prediction method provided by the embodiments of this application will be described in detail below.

[0150] Since the embodiments of this application involve neural networks, the relevant terms involved in the embodiments of this application will be introduced below for ease of understanding.

[0151] (1) Neural Network

[0152] Neural networks can be composed of neural units, which can refer to units represented by x. s The arithmetic unit that takes input data and an intercept of 1 as input can output:

[0153]

[0154] Where s = 1, 2, ..., n, n is a natural number greater than 1, W s For x sThe weights are denoted by b, where b is the bias of the neural unit. f is the activation function of the neural unit, used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into an output signal. The output signal of this activation function can be used as the input of the next convolutional layer; the activation function can be the sigmoid function. A neural network is a network formed by connecting multiple of the above-mentioned individual neural units together; that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field, which can be a region composed of several neural units.

[0155] (2) A convolutional neural network (CNN) is a deep neural network with a convolutional structure. A CNN contains a feature extractor consisting of convolutional layers and subsampling layers, which can be viewed as a filter. A convolutional layer refers to the layer of neurons in a CNN that performs convolutional processing on the input signal. In a convolutional layer of a CNN, a neuron can be connected to only some of the neurons in its neighboring layers. A convolutional layer typically contains several feature planes, each composed of rectangularly arranged neural units. Neural units on the same feature plane share weights, which are the convolutional kernel. Shared weights can be understood as the way features are extracted being independent of their location. The convolutional kernel can be formalized as a matrix of random size, and during the training process of the CNN, the kernel can learn reasonable weights. Furthermore, the direct benefit of shared weights is reducing the connections between layers in the CNN, while also reducing the risk of overfitting.

[0156] (3) Deep Neural Networks

[0157] Deep Neural Networks (DNNs), also known as multilayer neural networks, can be understood as neural networks with many hidden layers, though there's no specific metric for "many." DNNs can be categorized into three layers based on their position: input layers, hidden layers, and output layers. Generally, the first layer is the input layer, the last layer is the output layer, and the layers in between are hidden layers. All layers are fully connected, meaning that any neuron in the i-th layer is connected to any neuron in the (i+1)-th layer. Although DNNs appear complex, the operation of each layer is actually quite simple, resembling a linear relationship as follows: in, It is the input vector. It is the output vector. α is the offset vector, W is the weight matrix (also called coefficients), and α() is the activation function. Each layer is simply an adjustment of the input vector. The output vector is obtained through such a simple operation. Because DNNs have many layers, the coefficients W and the offset vector... The number of these parameters is therefore quite large. The definitions of these parameters in a DNN are as follows: Taking the coefficient W as an example: Assuming a three-layer DNN, the linear coefficient from the 4th neuron in the second layer to the 2nd neuron in the third layer is defined as... The superscript 3 represents the layer number where coefficient W is located, while the subscript corresponds to the third layer index 2 of the output and the second layer index 4 of the input.

[0158] In summary: the coefficients from the k-th neuron in layer L-1 to the j-th neuron in layer L are defined as follows:

[0159] It's important to note that the input layer does not have a W parameter. In deep neural networks, more hidden layers allow the network to better represent complex real-world situations. Theoretically, the more parameters a model has, the higher its complexity and "capacity," meaning it can perform more complex learning tasks. Training a deep neural network is essentially the process of learning the weight matrix, with the ultimate goal of obtaining the weight matrix of all layers in the trained deep neural network (a weight matrix formed by the vectors W from many layers).

[0160] (4) Loss Function

[0161] In training a deep neural network, to ensure the output closely approximates the desired predicted value, we compare the network's prediction with the target value. Based on the difference, we update the weight vector of each layer (usually pre-configuring parameters before the initial update). For example, if the prediction is too high, the weight vector is adjusted to predict a lower value. This adjustment continues until the deep neural network predicts the target value or a value very close to it. Therefore, we need to predefine "how to compare the difference between the predicted and target values," which is the loss function or objective function. These are important equations used to measure the difference between the predicted and target values. Taking the loss function as an example, a higher output value (loss) indicates a greater difference, and training the deep neural network becomes a process of minimizing this loss.

[0162] (5) Backpropagation algorithm

[0163] Convolutional neural networks can employ backpropagation (BP) to correct the parameters in the initial super-resolution model during training, thereby reducing the reconstruction error loss. Specifically, forward propagation of the input signal to the output generates an error loss; this error loss information is then propagated back to update the parameters in the initial super-resolution model, leading to convergence of the error loss. The backpropagation algorithm is an error-loss-driven backpropagation process aimed at obtaining the optimal parameters of the super-resolution model, such as the weight matrix.

[0164] (6) Natural Language Processing (NLP)

[0165] Natural language is human language, and Natural Language Processing (NLP) is the processing of human language. NLP is a systematic process of analyzing, understanding, and extracting information from text data in an intelligent and efficient manner. By using NLP and its components, we can manage very large amounts of text data, perform numerous automated tasks, and solve a wide variety of problems, such as automatic summarization, machine translation (MT), named entity recognition (NER), relation extraction (RE), information extraction (IE), sentiment analysis, speech recognition, question answering systems, and topic segmentation, among others.

[0166] (7) Pre-trained language model

[0167] A pre-trained language model is a natural language sequence encoder that encodes each word in a natural language sequence into a vector representation for prediction tasks. Its training consists of two phases. In the pre-training phase, the model is trained on a large-scale unsupervised text environment to learn word representations. In the fine-tuning phase, the model is initialized using the parameters learned in the pre-training phase and then trained on downstream tasks such as text classification and sequence labeling with fewer steps, successfully transferring the semantic information obtained in pre-training to downstream tasks.

[0168] For robots or vehicles, automatic control typically involves three tasks: perception, prediction, and decision-making. Taking autonomous driving as an example, the perception task involves receiving raw sensor data (images or lasers, etc.) as input and identifying dynamic targets (such as pedestrians and vehicles, their size and position) and static elements (such as lane lines and arrows on the ground). The prediction task receives the raw results of the perception task regarding dynamic and static targets and infers the movement intentions of targets in the environment, primarily predicting the future trajectory and intentions of other vehicles. The decision-making task receives the prediction results from the prediction task, and based on the future driving intentions of other vehicles and the vehicle's navigation task, generates a decision conclusion, outputs control signals, and controls the vehicle to perform autonomous driving.

[0169] However, in the existing architecture, the functions of each task module are decoupled, exhibiting a clear upstream and downstream relationship. This is achieved by pre-defining the output interfaces of each module for interconnection. While this approach offers advantages such as strong interpretability and ease of modular development and maintenance, the decoupling of front-end and back-end modules means that if a front-end module malfunctions, the downstream module's decision-making will also be flawed, resulting in poor control precision.

[0170] To address the aforementioned problems, this application proposes an end-to-end control scheme based on a language model. "End-to-end" means that the system can directly output control signals based on input sensor data. Information transmission between system modules is no longer based on pre-defined, fixed formats, but rather on full information sharing and dissemination.

[0171] Reference Figure 2 , Figure 2 This is a flowchart illustrating a data processing method provided in an embodiment of this application, such as... Figure 2 As shown, the data processing method provided in this application embodiment includes:

[0172] 201. Based on sensor data and a first prompt text, scene information is obtained through a first language model; the sensor data is data collected from the surrounding environment of the target object, and the first prompt text indicates that scene information of the scene where the target object is located is extracted based on the sensor data, and the target object is the vehicle itself or the robot itself.

[0173] In one possible implementation, Figure 2 The execution entity in the corresponding embodiment can be the target object itself, or a server, or some steps can be executed by the target object itself and some steps can be executed by the server; this is not limited here.

[0174] In one possible implementation, the target object is a robot; the target object can receive task instructions input by the user, such as task instructions that tell the robot to perform tasks, such as "get me some drinks from the kitchen" or "get me the water bottle on the table".

[0175] In one possible implementation, the target object is a vehicle that can receive navigation instructions from the user. For example, the navigation instructions can indicate the destination for the vehicle's autonomous driving, such as "Please navigate to the school".

[0176] Upon receiving a task instruction (such as the navigation instruction and the robot's task instruction described above), the execution objectives required for each stage of the task indicated by the task instruction can be obtained. The target object needs to sequentially achieve the objectives of each stage to accomplish the task corresponding to the task instruction. During the execution of the task corresponding to the task instruction by the target object, it is necessary to determine what the current objective needs to be achieved, and then determine the control strategy that can achieve the current objective.

[0177] Specifically, when the target object is performing a task, it can determine the task to be performed at the moment based on the user's input instructions (such as the task instructions or navigation instructions described above) and its own state (that is, the current execution target of the target object in this application embodiment). As time goes by, the target object continuously executes the execution targets that need to be performed at each time until the task indicated by the user's input instructions is achieved.

[0178] For example, the robot can obtain the current execution target of the target object based on the robot's current posture, scene map information, and the task instructions.

[0179] Optionally, the robot can obtain the execution targets (including the current execution target) required to achieve each stage of the task indicated by the task command through a second language model, based on the robot's current posture, scene map information, and the task command.

[0180] In one possible implementation, the vehicle can determine the current execution target of the target object based on the vehicle's current location and the navigation instructions.

[0181] Optionally, the vehicle can obtain the execution targets (including the current execution target) required for each stage of the task indicated by the task instruction through a second language model, based on the vehicle's current location and the navigation instructions (which may also include map information).

[0182] For example, a vehicle can generate the execution objectives (e.g., coarse-grained driving planning path) required for each stage based on the vehicle's current location and navigation instructions. These objectives may include: first driving out of the underground parking garage, then safely driving to the community according to the navigation route after reaching the ground level, entering the gate after reaching the community entrance, and parking in the parking space after identifying an empty parking space.

[0183] In one possible implementation, the target object can determine the goal to be executed based on its current state. To execute the goal, it needs to obtain the current scene information and combine it with the scene information (as well as information such as the recovery text to be discussed later) to determine the control information, thereby enabling the target object to achieve the goal within the current scene.

[0184] Next, we will explain how to obtain scene information:

[0185] In one possible implementation, the target object is a vehicle, and the sensor data is data collected from the surrounding environment of the target object. For example, the sensor data can be data collected by an image sensor, data collected by radar, etc.

[0186] In one possible implementation, the target object is a robot, and the sensor data is data collected from the surrounding environment of the target object. For example, the sensor data can be data collected by an image sensor, data collected by radar, etc.

[0187] In one possible implementation, the first language model (which may also be referred to as a language-perception model in this application embodiment) can be a large language model such as chatGPT, GPT4 or chatGLM. This application embodiment does not limit the type of language model.

[0188] In one possible implementation, the first language model can receive high-frequency sensor data input and a first prompt text input. The first prompt text can instruct the extraction of scene information about the target object's location based on the sensor data. For example, taking a vehicle as the target object, the first prompt text could be requesting the perception of the environment type surrounding the vehicle and the perception elements within that environment. The first prompt text can guide the first language model to output perception results (i.e., the environmental information in this embodiment). For example, the environmental information can include the category of the current scene (e.g., underground parking, open ground, tunnel, intersection, etc.) and the perception elements of the current scene, such as the positions of vehicles and pedestrians, lane detection results, etc.

[0189] Among them, the perceptual elements can be objects that affect the target object's task execution.

[0190] 202. Based on the scenario information and the current execution target of the target object, generate a second prompt text through a second language model. The second prompt text indicates the query for information that the target object needs to know to complete the execution target and that is not included in the scenario information.

[0191] The scene information and the current execution target of the target object can be input into the second language model in the form of text (e.g., prompt).

[0192] In one possible implementation, after obtaining the scene information, a second prompt text can be generated using a second language model based on the scene information and the current execution goal of the target object. The second prompt text indicates the query for information that the target object needs to know to complete the execution goal and that is not included in the scene information.

[0193] In one possible implementation, if the scenario information obtained by the first language model already contains all the information required for the current execution goal of the target object, the second language model can directly obtain the control instructions to achieve the current execution goal.

[0194] However, due to complex scenarios or unexpected situations, the scenario information obtained by the first language model often cannot contain all the information required by the target object for its current execution goal. In this case, the second language model can output active query text (that is, the second prompt text in the embodiments of this application). The second prompt text indicates the query for information that the target object needs to know when completing the execution goal and that is not included in the scenario information.

[0195] In one possible implementation, the information that needs to be known but is not included in the scene information includes: the state information of the sensing elements associated with the execution target. Here, the sensing elements are objects associated with the execution target. For example, if the scene category in the current scene information is an intersection, and the current execution target is to go straight through the intersection, then the sensing elements could be the state of the traffic lights, or how long it will take for a traffic light that is red to turn green, etc.

[0196] For example, if the current execution goal is to proceed straight through the intersection, the sensing element could be a traffic light, and the second prompt text could be: "Please identify the status of the traffic light; is it permissible to proceed straight?"

[0197] For example, if the current execution goal is to enter the community gate, the sensing element could be the parking barrier gate, and the second prompt text could be: "Is the parking barrier gate raised? Can the vehicle pass?"

[0198] For example, if the current execution goal is parking, the perception element can be the vacancy status of the parking space, and the second prompt text can be: Are there any vacant parking spaces?

[0199] In one possible implementation, the information that needs to be known but is not included in the scenario information includes: the implementation method of the execution target.

[0200] For example, the current execution goal is to drive out of the underground parking garage. In the underground parking garage scenario, since there is no GPS positioning information and no map information is available, the second prompt text could be "How do I drive out of the underground parking garage?".

[0201] 203. Based on the sensor data and the second prompt text, obtain the response text corresponding to the second prompt text through the first language model.

[0202] In one possible implementation, the first language model can perform secondary reasoning based on sensor data and the second prompt text, and output higher-level environmental information (i.e., the response text corresponding to the second prompt text) that meets the requirements of the question indicated in the second prompt text, in order to support the decision of the second language model.

[0203] The following example, using a vehicle as the target object, describes how the first language model and the second language model collaborate to generate decision instructions in different driving scenarios.

[0204] 1) Underground parking garage scenario: The underground parking garage lacks GPS positioning information and map information. Therefore, the second language model can generate a second prompt text: "How to drive out of the underground parking garage?", and input this second prompt text into the first language model. The first language model identifies the underground parking garage exit guidance sign in the environment based on sensor data and provides a response text: "Exit guidance sign detected, please turn right to exit the underground parking garage." After receiving the response text, the second language model can generate a driving instruction (which can also be referred to as a control instruction in this embodiment): "Turn right to exit the underground parking garage."

[0205] 2) Intersection Scenario: The second language model can generate a second prompt text based on the scenario information and the current execution objective: Please identify the traffic light status; is it permissible to proceed straight? This second prompt text is input into the first language model. After identifying the traffic light status based on sensor data, the first language model provides a response text: The current traffic light is red; you may proceed in 5 seconds. Upon receiving the response text, the second language model generates a driving instruction: The current intersection light is red; stop and wait for 5 seconds.

[0206] 3) Gate Entry Scenario: After the vehicle arrives at the entrance of the community, the second language model generates a second prompt text: "Is the parking gate arm raised? Can the vehicle pass?" This second prompt text is input into the first language model. After recognizing the status of the gate arm ahead based on sensor data, the first language model outputs a reply text: "Gate arm raised, you can pass." Then, the second language model generates a driving instruction based on the reply text: "Go straight into the gate."

[0207] 4) Parking Scenario: After the vehicle arrives at the community parking lot, the second language model generates a second prompt text: "Are there any empty parking spaces?", and inputs this second prompt text into the first language model. The first language model identifies parking spaces in the environment based on sensor data and provides a response text: "Two empty parking spaces found to the right front; you can park here." After receiving the response text, the second language model can generate a driving decision instruction: "Park in the empty parking space to the right front."

[0208] 204. Based on the response text, the scenario information, and the current execution target, the control instructions for the target object are obtained through the second language model.

[0209] In one possible implementation, after receiving the response text, the first language model can input the response text into the second language model. The second language model can then obtain the control instructions for the target object based on the response text, the scenario information, and the current execution target.

[0210] The control commands can be coarse-grained control objectives (such as stopping and waiting for 5 seconds before proceeding straight, turning left, or turning right) or fine-grained hardware control signals, such as control signals for steering wheel rotation or control signals for robotic arm joint rotation.

[0211] In this embodiment, the second language model obtains the query text form of the requirement information (second prompt text) based on different scenario information. The first language model obtains the response text based on the query text, and the response text is passed to the second language model for decision-making. On the one hand, an end-to-end automatic control system is constructed through the interaction between the first language model and the second language model. Compared with multi-stage decision-making systems, this application can improve decision-making accuracy. On the other hand, the introduction of language models into this architecture can extend the inductive reasoning ability of the large language model to the control domain. The expression mode of the language model's text is different from the traditional fixed format and can represent an infinite number of perceptual needs, thereby unifying the acquisition method of various types of perceptual need conclusions of the second language model.

[0212] like Figure 3 As shown, Figure 3 This is a schematic diagram of a typical autonomous driving scenario. Sensor data and a first query text are input into the first language model of the perception module. The first language model obtains scene information and inputs it into the second language model of the decision control model. The second language model obtains the second query text based on the coarse-grained planning objective and scene information, and inputs the second query text into the first language model. The first language model obtains the corresponding response text for the second query text. The second language model (or a control command generation module separate from the second language model) can generate control commands based on the response text and scene information, etc. (It should be understood that...) Figure 3 The control instruction generation model in the model may or may not belong to the second language model.

[0213] The following sections will use autonomous driving and robot control as examples to illustrate two implementation diagrams of the embodiments of this application:

[0214] In an autonomous driving scenario, a user's navigation command could be: "Start from the company's underground parking garage and navigate home." The second language model is input with the navigation command text: "Start from the company's underground parking garage and navigate home." Based on the location information and navigation command information, the second language model generates a coarse-grained driving planning path: first, exit the underground parking garage; then, upon reaching the ground level, safely drive along the navigation route to the residential area; upon reaching the entrance, enter the gate; identify an available parking space, and park. The first language model receives high-frequency basic perception text (i.e., the first prompt text) as input for the "vehicle's surrounding environment," and outputs the perceived current driving scene (e.g., underground parking garage, open ground, tunnel, intersection, etc.) and the corresponding perception elements, such as the positions of vehicles and pedestrians, lane detection results, etc. Based on the received current driving scene and corresponding perception elements, the second language model determines whether there are any perception elements to be identified in the current scene. If so, it generates a perception element active query text (i.e., the second prompt text) and sends it to the first language model. For example, the second language model receives the current driving scenario as an "intersection" and the coarse-grained planned path as "go straight." The second language model generates the active query text for the perception elements as "Please identify the traffic light status; is it permissible to go straight?". The first language model receives the active query text (i.e., the second prompt text), identifies the environmental perception elements, and performs secondary reasoning based on the currently identified environmental perception results, outputting higher-order environmental information that meets the question requirements (i.e., the response text): "The current traffic light is red; you can proceed in 5 seconds." After receiving the perception reasoning results, the second language model, based on the coarse-grained planned path and other previously identified perception elements, outputs the driving decision instruction: "The current intersection light is red; stop and wait for 5 seconds."

[0215] In a robot control scenario, the user's task instruction could be: "Get me some drinks from the kitchen." The second language model receives this task instruction text: "Get me some drinks from the kitchen." Based on the input text, the current scene map, and its own physical quantities, the second language model generates a coarse-grained navigation planning path. The first language model receives high-frequency basic perception text (i.e., the first prompt text) as input for the "robot's surrounding environment," and receives environmental feature maps and basic sensor perception data, outputting the perceived current driving scene and the corresponding perception elements. The second language model, based on the received current driving scene and corresponding perception elements, determines whether there are any perception elements to be identified in the current scene. If so, it generates an active query text for the perception element and sends it to the first language model. The first language model receives the active query text for the perception element, identifies the environmental perception elements (i.e., the second prompt text), and sends it back to the first language model. The first language model receives the active query text for the perception element (i.e., the second prompt text), identifies the environmental perception elements, and based on the currently identified environmental perception results, performs secondary reasoning to output higher-order environmental information (i.e., the response text) that meets the problem requirements. After receiving the perceptual reasoning results, the second language model outputs driving decision instructions based on the coarse-grained path planning and other previously identified perceptual elements, until the task of navigating to the kitchen is completed.

[0216] The data processing method provided in the embodiments of this application has been described above from the perspective of methodology. Next, the data processing apparatus provided in the embodiments of this application will be described from the perspective of apparatus.

[0217] Reference Figure 4 , Figure 4 This is a schematic diagram of the structure of a data processing apparatus provided in an embodiment of this application, such as... Figure 4 As shown, the device 400 includes:

[0218] The processing module 401 is configured to obtain scene information based on sensor data and a first prompt text using a first language model; the sensor data is data collected from the surrounding environment of the target object, and the first prompt text indicates the extraction of scene information of the scene where the target object is located based on the sensor data; the target object is the vehicle itself or the robot itself; based on the scene information and the current execution goal of the target object, a second prompt text is generated using a second language model, the second prompt text indicating the query for information that the target object needs to know to complete the execution goal but is not included in the scene information; based on the sensor data and the second prompt text, a response text corresponding to the second prompt text is obtained using the first language model; based on the response text, the scene information, and the current execution goal, control instructions for the target object are obtained using the second language model.

[0219] For a detailed description of the processing module 401, please refer to the description of steps 201 to 204 in the above embodiments, which will not be repeated here.

[0220] In one possible implementation, the scene information includes information indicating the scene category and / or information about the perceptual elements included in the scene.

[0221] In one possible implementation, the target object is a vehicle, and the device further includes:

[0222] Module 402 is used to receive navigation instructions from the user;

[0223] Based on the vehicle's current location and the navigation instructions, the target object's current execution objective is obtained.

[0224] In one possible implementation, the target object is a robot; the device further includes:

[0225] Module 402 is used to receive user task instructions;

[0226] The processing module 401 is further configured to obtain the current execution target of the target object based on the robot's current posture, scene map information, and the task instructions.

[0227] In one possible implementation, the information that needs to be known but is not included in the scenario information includes:

[0228] The state information of the sensing element associated with the execution target; or,

[0229] The method for achieving the execution objective.

[0230] In one possible implementation, the first language model and the second language model are chatGPT, GPT4, or chatGLM.

[0231] It should be noted that the information interaction and execution process between the modules / units of the above-mentioned device are based on the same concept as the method embodiment of this application, and the resulting technical effects are the same as those of the method embodiment of this application. For details, please refer to the description in the method embodiment shown above in the embodiment of this application, and it will not be repeated here.

[0232] The following describes an execution device provided in an embodiment of this application. Please refer to [link / reference]. Figure 5 , Figure 5This is a schematic diagram of an execution device provided in an embodiment of this application. The execution device 500 can specifically be a vehicle control device or a robot, etc., and is not limited thereto. Specifically, the execution device 500 includes: a receiver 501, a transmitter 502, a processor 503, and a memory 504 (wherein the execution device 500 may have one or more processors 503). Figure 5 (Taking a processor as an example), processor 503 may include application processor 5031 and communication processor 5032. In some embodiments of this application, receiver 501, transmitter 502, processor 503 and memory 504 may be connected via a bus or other means.

[0233] Memory 504 may include read-only memory and random access memory, and provides instructions and data to processor 503. A portion of memory 504 may also include non-volatile random access memory (NVRAM). Memory 504 stores processor and operation instructions, executable modules, or data structures, or subsets thereof, or extended sets thereof, wherein the operation instructions may include various operation instructions for implementing various operations.

[0234] Processor 503 controls the operation of the execution device. In specific applications, the various components of the execution device are coupled together through a bus system, which may include not only the data bus, but also power buses, control buses, and status signal buses. However, for clarity, all buses are referred to as the bus system in the diagram.

[0235] The methods disclosed in the embodiments of this application can be applied to processor 503, or implemented by processor 503. Processor 503 can be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit in the hardware of processor 503 or by instructions in the form of software. The processor 503 can be a general-purpose processor, a digital signal processor (DSP), a microprocessor or a microcontroller, and may further include an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The processor 503 can implement or execute the methods, steps and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or can be executed by a combination of hardware and software modules in the decoding processor. The software module can reside in a mature storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory 504, and processor 503 reads the information from memory 504 and, in conjunction with its hardware, completes the steps of the above method.

[0236] Receiver 501 can be used to receive input digital or character information, and to generate signal inputs related to the settings and function control of the execution device. Transmitter 502 can be used to output digital or character information through the first interface; transmitter 502 can also be used to send instructions to the disk group through the first interface to modify the data in the disk group.

[0237] In one embodiment of this application, the processor 503 is configured to execute... Figure 2 The data processing method described in the corresponding embodiment.

[0238] This application also provides a server; please refer to [link / reference]. Figure 6 , Figure 6 This is a schematic diagram of a server structure provided in an embodiment of this application. Server 600 can be deployed with... Figure 4 The data processing apparatus described in the corresponding embodiment is used to implement Figure 4In accordance with the function of the data processing device in the corresponding embodiment, specifically, server 600 is implemented by one or more servers. Server 600 can vary significantly due to different configurations or performance, and may include one or more central processing units (CPUs) 619 (e.g., one or more processors) and memory 632, and one or more storage media 630 (e.g., one or more mass storage devices) for storing application programs 642 or data 644. The memory 632 and storage media 630 can be temporary or persistent storage. The program stored in storage media 630 may include one or more modules (not shown in the figure), each module may include a series of instruction operations on the server. Furthermore, the CPU 619 may be configured to communicate with storage media 630 and execute the series of instruction operations in storage media 630 on server 600.

[0239] Server 600 may also include one or more power supplies 626, one or more wired or wireless network interfaces 650, one or more input / output interfaces 658; or one or more operating systems 641, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0240] In this embodiment, the central processing unit 619 is used to execute... Figure 2 The data processing method provided in the corresponding embodiment.

[0241] This application also provides a computer program product that, when run on a computer, causes the computer to perform steps as performed by the aforementioned execution device, or causes the computer to perform steps as performed by the aforementioned training device.

[0242] This application also provides a computer-readable storage medium storing a program for signal processing, which, when run on a computer, causes the computer to perform steps as performed by the aforementioned execution device, or causes the computer to perform steps as performed by the aforementioned training device.

[0243] The execution device, training device, or terminal device provided in this application embodiment can specifically be a chip. The chip includes a processing unit and a communication unit. The processing unit can be, for example, a processor, and the communication unit can be, for example, an input / output interface, pins, or circuits. The processing unit can execute computer execution instructions stored in the storage unit to cause the chip within the execution device to execute the data processing method described in the above embodiments, or to cause the chip within the training device to execute the data processing method described in the above embodiments. Optionally, the storage unit can be a storage unit within the chip, such as a register or cache. Alternatively, the storage unit can be a storage unit located outside the chip within the wireless access device, such as a read-only memory (ROM) or other types of static storage devices capable of storing static information and instructions, such as random access memory (RAM).

[0244] For details, please refer to Figure 7 , Figure 7 This is a schematic diagram of a chip provided in an embodiment of this application. The chip can be represented as a neural network processor (NPU) 700. The NPU 700 is mounted as a coprocessor on the host CPU, and tasks are assigned by the host CPU. The core of the NPU is the arithmetic circuit 703, which is controlled by a controller 704 to retrieve matrix data from the memory and perform multiplication operations.

[0245] In some implementations, the arithmetic circuit 703 internally includes multiple processing engines (PEs). In some implementations, the arithmetic circuit 703 is a two-dimensional pulsating array. The arithmetic circuit 703 can also be a one-dimensional pulsating array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 703 is a general-purpose matrix processor.

[0246] For example, suppose we have an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit retrieves the corresponding data of matrix B from the weight memory 702 and caches it in each PE of the arithmetic circuit. The arithmetic circuit retrieves the data of matrix A from the input memory 701 and performs matrix operations with matrix B. The partial result or the final result of the obtained matrix is ​​stored in the accumulator 708.

[0247] Unified memory 706 is used to store input and output data. Weight data is directly transferred to weight memory 702 via Direct Memory Access Controller (DMAC) 705. Input data is also transferred to unified memory 706 via DMAC.

[0248] BIU stands for Bus Interface Unit 710, which is used for interaction between the AXI bus and the DMAC and the Instruction Fetch Buffer (IFB) 709.

[0249] The Bus Interface Unit (BIU) 710 is used by the instruction fetch memory 709 to fetch instructions from external memory, and also by the memory access controller 705 to fetch the original data of the input matrix A or the weight matrix B from external memory.

[0250] The DMAC is mainly used to move input data from external memory DDR to unified memory 706, or to weight data to weight memory 702, or to input data to input memory 701.

[0251] The vector computation unit 707 includes multiple arithmetic processing units that further process the output of the computation circuit when necessary, such as vector multiplication, vector addition, exponential operations, logarithmic operations, size comparisons, etc. It is mainly used for computation in non-convolutional / fully connected layers of neural networks, such as batch normalization, pixel-level summation, and upsampling of feature planes.

[0252] In some implementations, the vector computation unit 707 can store the processed output vector in the unified memory 706. For example, the vector computation unit 707 can apply a linear function, or a nonlinear function, to the output of the computation circuit 703, such as performing linear interpolation on the feature planes extracted by the convolutional layer, or, for example, accumulating a vector of values ​​to generate activation values. In some implementations, the vector computation unit 707 generates normalized values, pixel-level summed values, or both. In some implementations, the processed output vector can be used as an activation input to the computation circuit 703, for example, for use in subsequent layers of the neural network.

[0253] The instruction fetch buffer 709 connected to the controller 704 is used to store the instructions used by the controller 704;

[0254] Unified memory 706, input memory 701, weighted memory 702, and instruction fetch memory 709 are all on-chip memories. External memory is proprietary to this NPU hardware architecture.

[0255] The processor mentioned above can be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits used to control the execution of the above program.

[0256] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the accompanying drawings of the device embodiments provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0257] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods of the various embodiments of this application.

[0258] In the above embodiments, the implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, in the form of a computer program product.

[0259] This computer program product includes one or more computer instructions. When these computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

Claims

1. A data processing method, characterized in that, The method includes: Based on sensor data and a first prompt text, scene information is obtained through a first language model; the sensor data is data collected from the surrounding environment of the target object, and the first prompt text indicates that scene information of the scene where the target object is located is extracted based on the sensor data; the target object is the vehicle itself or the robot itself. Based on the scenario information and the current execution goal of the target object, a second prompt text is generated through a second language model. The second prompt text indicates the query for information that the target object needs to know to complete the execution goal and that is not included in the scenario information. Based on the sensor data and the second prompt text, the response text corresponding to the second prompt text is obtained through the first language model; Based on the response text, the scenario information, and the current execution target, the control instructions for the target object are obtained through the second language model.

2. The method according to claim 1, characterized in that, The scene information includes information indicating the scene category and / or information about the perceptual elements included in the scene.

3. The method according to claim 1 or 2, characterized in that, The target object is a vehicle, and the method further includes: Receive navigation commands from the user; Based on the vehicle's current location and the navigation instructions, the target object's current execution objective is obtained.

4. The method according to claim 1 or 2, characterized in that, The target object is a robot; the method further includes: Receive task instructions from the user; Based on the robot's current posture, scene map information, and the task instructions, the current execution target of the target object is obtained.

5. The method according to any one of claims 1 to 4, characterized in that, The information that needs to be known but is not included in the scenario information includes: The state information of the sensing element associated with the execution target; or, The method for achieving the execution objective.

6. A data processing apparatus, characterized in that, The device includes: The processing module is used to obtain scene information based on sensor data and a first prompt text through a first language model; the sensor data is data collected from the surrounding environment of the target object, and the first prompt text indicates the extraction of scene information of the scene where the target object is located based on the sensor data; the target object is the vehicle itself or the robot itself. The processing module is further configured to generate a second prompt text based on the scene information and the current execution target of the target object through a second language model. The second prompt text indicates the query for information that the target object needs to know when completing the execution target and that is not included in the scene information. The processing module is further configured to obtain the response text corresponding to the second prompt text based on the sensor data and the second prompt text, using the first language model; The processing module is further configured to obtain control instructions for the target object through the second language model based on the reply text, the scene information, and the current execution target.

7. The apparatus according to claim 6, characterized in that, The scene information includes information indicating the scene category and / or information about the perceptual elements included in the scene.

8. The apparatus according to claim 6 or 7, characterized in that, The target object is a vehicle, and the device further includes: The acquisition module is used to receive navigation commands from the user; Based on the vehicle's current location and the navigation instructions, the target object's current execution objective is obtained.

9. The apparatus according to any one of claims 6 to 8, characterized in that, The target object is a robot; the device further includes: The acquisition module is used to receive task instructions from the user; The processing module is further configured to obtain the current execution target of the target object based on the robot's current posture, scene map information, and the task instructions.

10. The apparatus according to any one of claims 6 to 9, characterized in that, The information that needs to be known but is not included in the scenario information includes: The state information of the sensing element associated with the execution target; or, The method for achieving the execution objective.

11. A data processing apparatus, characterized in that, It includes a memory and a processor; the memory stores code, and the processor is configured to execute the code, wherein when the code is executed, the processor performs the method as described in any one of claims 1 to 5.

12. A computer storage medium, characterized in that, The computer storage medium stores a computer program that, when executed by the computer, causes the computer to perform the method described in any one of claims 1 to 5.

13. A computer program product, characterized in that, The computer program product stores instructions that, when executed by a computer, cause the computer to perform the method described in any one of claims 1 to 5.

14. A chip, comprising a processor, characterized in that, The processor is used to support the data processing device in implementing the method as described in any one of claims 1 to 5.

Images