Vehicle driving control method, device, equipment and medium

Abstract

The embodiment of the invention relates to a vehicle driving control method and device, equipment and a medium, and the method is applied to a vehicle, and comprises the steps: obtaining the perception data of a vehicle sensor, processing the perception data through a preset visual language model VLM based on a first prompt word of a thinking chain, obtaining an initial scene description text, obtaining the external data of a vehicle road cloud, and obtaining the external data of the vehicle road cloud; processing the external data through a VLM based on a second cue word of a thinking chain to obtain a candidate scene description text, and merging the initial scene description text and the candidate scene description text according to a preset text cue splicing format to obtain an enhanced scene description text, and analyzing the enhanced scene description text through a VLM based on a third prompt word of the thinking chain, obtaining an analysis result, and sending the analysis result to a vehicle controller, so that the vehicle controller performs vehicle driving control based on the analysis result. Therefore, more accurate scene understanding can be provided, driving planning and control are optimized, and safety and efficiency are improved.

Description

Technical Field

[0001] The present disclosure relates to the field of vehicle control technology, and in particular to a vehicle driving control method, device, equipment and medium. Background Art

[0002] At present, autonomous driving technology mainly relies on the vehicle's own perception system, such as radar, camera, etc., as well as the vehicle's internal computing unit for data processing and decision-making.

[0003] In the related technology, additional information support is provided through communication between the vehicle and the external environment. However, the existing methods of integrating external information with the vehicle system have problems such as low efficiency, inflexible information processing, and difficulty in adapting to complex scenarios. Summary of the invention

[0004] In order to solve the above technical problems or at least partially solve the above technical problems, the present disclosure provides a vehicle driving control method, device, equipment and medium.

[0005] The embodiment of the present disclosure provides a vehicle driving control method, which is applied to a vehicle, including:

[0006] Acquire the perception data of the vehicle sensor, and process the perception data based on the first prompt word of the thought chain through a preset visual language model VLM (Vision Language Model) to obtain an initial scene description text;

[0007] Acquire external data of the vehicle-road cloud, and process the external data through the visual language model VLM based on the second prompt word of the thought chain to obtain a candidate scene description text;

[0008] Merging the initial scene description text and the candidate scene description text according to a preset text prompt splicing format to obtain an enhanced scene description text;

[0009] The enhanced scene description text is analyzed by the visual language model VLM based on the third prompt word of the thought chain, and the analysis result is sent to the vehicle controller, so that the vehicle controller performs vehicle driving control based on the analysis result.

[0010] Optionally, two-way communication is established between the vehicle and the cloud server, and the acquisition of external data of the vehicle-road-cloud includes:

[0011] Receiving a plurality of data information or a first external scene description text obtained based on a plurality of data sources and sent by the cloud server; wherein the cloud server processes the plurality of data information based on a preset prompt word template through a preset large language model LLM (Large Language Model) to obtain the first external scene description text; and / or,

[0012] Receive roadside data sent by roadside equipment.

[0013] Optionally, the processing of the external data by the visual language model VLM based on the second prompt word of the thought chain to obtain a candidate scene description text includes:

[0014] Convert the multiple data information and the roadside data through the visual language model VLM to obtain a first external description text;

[0015] The first external description text is set as an external sentence based on the second prompt word to obtain the candidate scene description text.

[0016] Optionally, the processing of the external data based on the second prompt word of the thought chain by the visual language model VLM to obtain a candidate scene description text includes:

[0017] The roadside data is converted by the visual language model VLM to obtain a second external description text, and the second external description text is set as an external sentence based on the second prompt word to obtain the second external scene description text;

[0018] The first external scene description text and the second external scene description text are combined as the candidate scene description text.

[0019] Optionally, the method further includes:

[0020] A request instruction is sent to the cloud server, so that the cloud server obtains a subscription identifier based on the request instruction, and determines the multiple data sources based on the subscription identifier.

[0021] Optionally, the method further includes:

[0022] Sending the driving information of the vehicle to the cloud server, so that the cloud server can monitor and obtain the multiple data information in real time based on the driving information; and / or,

[0023] Send an information acquisition request to the cloud server, so that the cloud server acquires the multiple data information based on the information acquisition request.

[0024] Optionally, the analyzing the enhanced scene description text based on the third prompt word of the thought chain by the visual language model VLM to obtain an analysis result includes:

[0025] Performing static attribute analysis on the enhanced scene description text to obtain a static analysis result;

[0026] Performing motion state analysis on the enhanced scene description text to obtain a dynamic analysis result;

[0027] The static analysis result and the dynamic analysis result are set based on the third prompt word to obtain an analysis description text, and the analysis description text is input into the visual language model VLM to obtain the analysis result.

[0028] The present disclosure also provides a vehicle driving control device, including:

[0029] A first acquisition module, used to acquire perception data of vehicle sensors;

[0030] A first processing module is used to process the perception data based on the first prompt word of the thought chain through a preset visual language model VLM to obtain an initial scene description text;

[0031] The second acquisition module is used to acquire external data of the vehicle-road cloud;

[0032] A second processing module is used to process the external data based on the second prompt word of the thought chain through the visual language model VLM to obtain a candidate scene description text;

[0033] A merging module, used to merge the initial scene description text and the candidate scene description text according to a preset text prompt splicing format to obtain an enhanced scene description text;

[0034] An analysis module, configured to analyze the enhanced scene description text based on the third prompt word of the thought chain through the visual language model VLM to obtain an analysis result;

[0035] The first sending module is used to send the analysis result to the vehicle controller so that the vehicle controller performs vehicle driving control based on the analysis result.

[0036] An embodiment of the present disclosure also provides an electronic device, which includes: a processor; a memory for storing executable instructions of the processor; the processor is used to read the executable instructions from the memory and execute the instructions to implement the vehicle driving control method provided by the embodiment of the present disclosure.

[0037] The embodiment of the present disclosure also provides a computer-readable storage medium, which stores a computer program, and the computer program is used to execute the vehicle driving control method provided by the embodiment of the present disclosure.

[0038] The embodiment of the present disclosure also provides a computer program product, including a computer program, wherein the computer program implements the vehicle driving control method described in the above aspect when executed by a processor.

[0039] The technical solution provided by the embodiment of the present disclosure has the following advantages over the prior art: the vehicle driving control solution provided by the embodiment of the present disclosure includes: obtaining the perception data of the vehicle sensor, processing the perception data based on the first prompt word of the thinking chain through the preset visual language model VLM to obtain the initial scene description text; obtaining the external data of the vehicle-road cloud, processing the external data based on the second prompt word of the thinking chain through the visual language model VLM to obtain the candidate scene description text; merging the initial scene description text and the candidate scene description text according to the preset text prompt splicing format to obtain the enhanced scene description text; analyzing the enhanced scene description text based on the third prompt word of the thinking chain through the visual language model VLM, obtaining the analysis result and sending it to the vehicle controller, so that the vehicle controller performs vehicle driving control based on the analysis result. It solves the technical problems of low efficiency, inflexible information processing, and difficulty in adapting to complex scenes when integrating vehicle-road cloud information with the driving system in the existing way, and can provide more accurate scene understanding, thereby optimizing the planning and control of autonomous driving, improving safety and efficiency; in addition, it can also adapt to complex traffic environments and dynamic changes, such as traffic accidents, road construction, etc., making driving control more flexible and reliable. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0041] Figure 1 A schematic flow chart of a vehicle driving control method provided by an embodiment of the present disclosure;

[0042] Figure 2 A flow chart of another vehicle driving control method provided by an embodiment of the present disclosure;

[0043] Figure 3 An example diagram of vehicle driving control provided by an embodiment of the present disclosure;

[0044] Figure 4A schematic diagram of the structure of a vehicle driving control device provided by an embodiment of the present disclosure;

[0045] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0046] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0047] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0048] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0049] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0050] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0051] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0052] At present, autonomous driving technology mainly relies on the vehicle's own perception system, such as radar, camera, etc., as well as the vehicle's internal computing unit for data processing and decision-making; vehicle-road-cloud technology can be used as an auxiliary means to provide additional information support through the communication between the vehicle and the external environment. Usually, the roadside or cloud server can provide a lot of valuable information, which needs to be integrated with the vehicle's single-vehicle intelligence.

[0053] In existing vehicle driving control methods, information from the roadside or cloud server is often used as an input to the rule set in planning and control; however, the above-mentioned method of integrating vehicle-road-cloud information with the driving system has problems such as low efficiency, inflexible information processing, and difficulty in adapting to complex scenarios; especially when processing unstructured information, such as traffic incidents, road construction and other dynamic information, existing methods often cannot effectively use this information to optimize driving decisions.

[0054] In response to the above problems, the present disclosure implements a vehicle driving control method, which obtains perception data from a vehicle sensor, processes the perception data through a preset visual language model VLM based on a first prompt word of a thinking chain, and obtains an initial scene description text; obtains external data of a vehicle-road-cloud, processes the external data through the visual language model VLM based on a second prompt word of the thinking chain, and obtains a candidate scene description text; merges the initial scene description text and the candidate scene description text according to a preset text prompt splicing format to obtain an enhanced scene description text; analyzes the enhanced scene description text through the visual language model VLM based on a third prompt word of the thinking chain, and obtains an analysis result and sends it to a vehicle controller, so that the vehicle controller performs vehicle driving control based on the analysis result. It solves the technical problems of low efficiency, inflexible information processing, and difficulty in adapting to complex scenarios when integrating vehicle-road-cloud information with driving systems in existing methods. It can convert unstructured vehicle-road-cloud information into a structured description language, improve the availability of information, and effectively integrate external information into the end-to-end autonomous driving decision-making process. It can provide more accurate scene understanding, thereby optimizing autonomous driving planning and control, and improving safety and efficiency. In addition, it can also adapt to complex traffic environments and dynamic changes, such as traffic accidents, road construction, etc., making driving control more flexible and reliable.

[0055] Figure 1 The present invention provides a flow chart of a vehicle driving control method provided by an embodiment of the present invention. The method can be executed by a vehicle driving control device, wherein the device can be implemented by software and / or hardware and can generally be integrated into an electronic device. Figure 1 As shown, the method includes:

[0056] Step 101: Acquire the perception data of the vehicle sensor, and process the perception data based on the first prompt word of the thought chain through a preset visual language model VLM to obtain an initial scene description text.

[0057] In the disclosed embodiment, after the vehicle starts the automatic driving mode or assisted driving mode or other vehicle control instructions, the perception data of the vehicle sensor can be obtained, and the perception data can be processed by the pre-trained visual language model VLM based on the first prompt word of the thinking chain to obtain the initial scene description text.

[0058] Among them, vehicle sensors include cameras, lidar and other devices, and perception data include videos, pictures, point clouds and other data, that is, the perception data can be multimodal data; among them, VLM is a multimodal model that can learn from data such as images and texts at the same time. It is a generative model that can accept multimodal data inputs such as images and texts and generate text outputs.

[0059] In the disclosed embodiment, the VLM is obtained in advance through training, and there will be question and answer data with similar prompt words during the training process. Through a large amount of data, the VLM can learn to process multimodal data and generate scene description text; wherein, the scene description text refers to a detailed description of the vehicle's surroundings, which is used to assist decision-making and planning.

[0060] In the disclosed embodiment, the thinking chain has a corresponding prompt word for each processing step, and the performance of the large model on complex reasoning tasks is enhanced by requiring the model to explicitly output the intermediate step-by-step reasoning steps before outputting the final answer; that is, by allowing the large model to gradually participate in the process of decomposing a complex problem into step-by-step sub-problems and solving them one by one, the performance of the large model can be significantly improved.

[0061] In the disclosed embodiment, the perception data (point cloud, image, etc.) of the vehicle sensor (such as camera, radar) is used as the input of the visual language model VLM on the vehicle side, and the VLM is allowed to complete the initial scene description text through the system sentence in the first prompt word.

[0062] For example, the system sentence in the first prompt word is to describe the current driving situation. After the perception data is input into the VLM, the initial scene description text is output including time: xx, road type: community entrance, lane line: white dotted line, key target: pedestrian on the right is at [(x1, y1)(x2, y3)].

[0063] Step 102: Acquire external data of the vehicle-road cloud, and process the external data through the visual language model VLM based on the second prompt word of the thought chain to obtain a candidate scene description text.

[0064] In the embodiments of the present disclosure, vehicle-road-cloud refers to the technology of vehicle-to-external communication including vehicle-to-vehicle, vehicle-to-infrastructure, vehicle-to-pedestrian, etc. Therefore, there are many ways to obtain external data of vehicle-road-cloud. As an example, a first external scene description text sent by a cloud server is obtained. That is, the cloud server pre-processes a variety of data information through a preset large language model LLM based on a preset prompt word template to obtain the first external scene description text and sends it to the vehicle. That is, the vehicle does not need to process it, and can directly use the first external scene description text as a candidate scene description text for merging with the initial scene description text, thereby further improving the processing efficiency on the vehicle side.

[0065] As another example, a variety of data information obtained based on multiple data sources is received from a cloud server, roadside data sent by a roadside device is received, the various data information and the roadside data are converted through a visual language model VLM to obtain a first external description text, and the first external description text is set as an external sentence based on a second prompt word to obtain a candidate scene description text.

[0066] As another example, a first external scene description text sent by a cloud server is obtained, roadside data is converted through a visual language model VLM to obtain a second external description text, and the second external description text is set as an external sentence based on a second prompt word to obtain a second external scene description text, and the first external scene description text and the second external scene description text are merged as a candidate scene description text.

[0067] Step 103: merge the initial scene description text and the candidate scene description text according to a preset text prompt splicing format to obtain an enhanced scene description text.

[0068] Among them, a text prompt splicing format is pre-set, such as directly merging according to character strings, etc. Specifically, the visual language model VLM on the vehicle side is used to semantically understand the external information obtained from the roadside equipment and the cloud server to generate an external description text. For example, the traffic monitoring data is converted into "There is construction 500 meters ahead, and congestion is expected", and the data of the smart parking lot is converted into "There are many vacant parking spaces in area A, and the nearest parking space to the car is numbered 301"; the external description text is placed in a preset prompt word template through an external sentence. The external sentence can be understood as a special prompt sentence. During the VLM training stage, the sentence has been used to enable the VLM to recognize and strengthen the reasoning logic through the external sentence. The initial scene description text obtained in the previous step is merged to obtain an enhanced scene description text.

[0069] For example, the system statement is to describe the current driving situation. After the perception data is input into the VLM, the initial scene description text is output including time: xx, road type: community entrance, lane line: white dashed line, key target: pedestrian on the right is at [(x1, y1)(x2, y2)]; the external statement is parking space: the community allows external vehicles to park, and the area with the most vacant parking spaces is P1.

[0070] Step 104: Analyze the enhanced scene description text based on the third prompt word of the thought chain through the visual language model VLM, obtain the analysis result and send it to the vehicle controller, so that the vehicle controller performs vehicle driving control based on the analysis result.

[0071] In the disclosed embodiment, the enhanced scene description text is analyzed based on the third prompt word of the thought chain through the visual language model VLM, and there are many ways to obtain the analysis result. As an example, the enhanced scene description text is analyzed for key objects and environment to obtain the analysis result; as another example, the enhanced scene description text is analyzed for static attributes to obtain static analysis results, and the enhanced scene description text is analyzed for motion states to obtain dynamic analysis results. The static analysis result and the dynamic analysis result are set based on the third prompt word to obtain the analysis description text, and the analysis description text is input into the visual language model VLM to obtain the analysis result.

[0072] Specifically, after obtaining the enhanced scene description text, a more comprehensive scene analysis of the current driving scene is performed through the visual language model VLM, including analysis of key objects and analysis of the potential driving impact of key objects on the vehicle. The analysis results are input into the planning and control stage of the vehicle's end-to-end autonomous driving system.

[0073] More specifically, the VLM model trained by the thinking chain is analyzed through the preset third prompt word to obtain the enhanced scene description text. That is, after obtaining the description of the key objects in the driving scene, the VLM analyzes the key object characteristics and the potential driving impact of the key objects on the vehicle, such as analyzing static attributes and motion states; among them, static attributes describe the inherent attributes of the object, such as numbers or slogans in road signs, cargo that exceeds the length of the truck body, etc. These attributes help the vehicle to determine some potential driving risks; motion states include position, direction, and action, which describe the motion state of an object over a period of time, similar to the traditional trajectory prediction module; thus, the VLM will adaptively output the possible aspects of an object in the text.

[0074] For example, the system statement is the key goal: Truck A is in front of the right [x1, x2...xn], and the current environmental enhancement information is xxx. According to the key goal and environmental information, the key objects and their impact on the ego vehicle are analyzed and described. The analysis result is that the low-speed truck A blocks the current lane, and the ego vehicle needs to slow down and maintain a safe distance.

[0075] Furthermore, the analysis result is sent to the vehicle controller so that the vehicle controller performs vehicle driving control based on the analysis result.

[0076] Specifically, VLM is based on multimodal input data (video data and text data, etc.) to predict the next vehicle path and low-level control signals such as speed and steering angle; for example, the analysis result is that the self-state and historical trajectory are […]], please determine the meta-operations, decisions and plan future path points; thus, the control instructions can be determined as [accelerate, move slightly to the left, and go straight at a constant speed]; decision: accelerate and move slightly to the left to overtake the vehicle in front, and then go straight at a constant speed; path points: [(x1, y1), (x2, y2),..., (xn, yn)].

[0077] The vehicle driving control scheme provided by the embodiment of the present disclosure obtains the perception data of the vehicle sensor, processes the perception data based on the first prompt word of the thinking chain through the preset visual language model VLM, and obtains the initial scene description text; obtains the external data of the vehicle-road cloud, processes the external data based on the second prompt word of the thinking chain through the visual language model VLM, and obtains the candidate scene description text; merges the initial scene description text and the candidate scene description text according to the preset text prompt splicing format to obtain the enhanced scene description text; analyzes the enhanced scene description text based on the third prompt word of the thinking chain through the visual language model VLM, obtains the analysis result and sends it to the vehicle controller, so that the vehicle controller performs vehicle driving control based on the analysis result. Through the data fusion on the vehicle side, the surrounding environment can be more comprehensively perceived, the recognition ability of the automatic driving system for dynamic and static obstacles is improved, the localized application of VLM enables the rapid understanding of complex scene descriptions, reduces decision delays, improves the real-time and accuracy of automatic driving, and the combination of global and local information provides more comprehensive data support for the automatic driving system, so that it can adapt to more changing driving scenes, and has better scalability, so as to add new information sources or sensors in the future.

[0078] Figure 2 This is a flow chart of another vehicle driving control method provided by an embodiment of the present disclosure. This embodiment further optimizes the above vehicle driving control method based on the above embodiment. Figure 2 As shown, the method includes:

[0079] Step 201: Acquire the perception data of the vehicle sensor, and process the perception data based on the first prompt word of the thought chain through a preset visual language model VLM to obtain an initial scene description text.

[0080] It should be noted that step 201 is the same as step 101 , and for detailed description, please refer to the description of step 101 in the above embodiment.

[0081] Step 202: Receive various data information obtained based on various data sources and sent by the cloud server and roadside data sent by the roadside equipment.

[0082] Step 203: convert the various data information and roadside data through the visual language model VLM to obtain an external description text, and set the first external description text as an external sentence based on the second prompt word to obtain a candidate scene description text.

[0083] In an embodiment of the present disclosure, the vehicle may send a request instruction to the cloud server, so that the cloud server obtains a subscription identifier based on the request instruction and determines multiple data sources based on the subscription identifier.

[0084] In the disclosed embodiment, the vehicle may send the vehicle's driving information to a cloud server so that the cloud server can perform real-time monitoring based on the driving information to obtain a variety of data information; and / or send an information acquisition request to the cloud server so that the cloud server can acquire a variety of data information based on the information acquisition request.

[0085] Specifically, the cloud server, as a remote information acquisition center, is responsible for subscribing external data required for driving from a variety of data sources (such as traffic monitoring systems, weather services, municipal announcements, smart parking lots, etc.), which can be sent directly to the vehicle or converted into structured description text through the large language model LLM of the cloud server and the corresponding prompt word template; the above information is transmitted to the vehicle through the vehicle-cloud communication mechanism. Among them, LLM refers to a complex algorithm model for understanding and generating natural language.

[0086] Specifically, there will be sensing devices and computing devices on the roadside to conduct short-range real-time information exchange through the vehicle-road communication mechanism; the vehicle side, as the client, will establish a stable communication mechanism with the cloud server side, and through two-way communication, a push-pull combination (the cloud server actively pushes or the vehicle client actively pulls) is used to obtain and update structured description texts.

[0087] In other words, the vehicle can actively report driving information (such as speed, location, and destination), and can also actively obtain additional information required for vehicle-side regulation and control, such as the road conditions of the highway ahead and the countdown of traffic lights in the real-time traffic system; the cloud server, as an information agent, can actively push information to the vehicle, such as sudden traffic accidents in the current area; it can also respond to the vehicle's request for information; the cloud can also form an overall plan based on the information of the autonomous driving networked vehicle and send it to the vehicle.

[0088] Therefore, through information registration and subscription, the enhanced information source can be flexibly configured and conveniently accessed to strengthen the end-to-end model of the vehicle side; the two-way communication mechanism between the vehicle side and the cloud server side allows the vehicle side to actively request information and the cloud side to actively push information, which improves the real-time and accuracy of the information; and the VLM is deployed on the vehicle side to directly understand the semantics of the environmental description around the vehicle without sending the data to a remote server, reducing communication delays and data transmission costs.

[0089] Step 204: merge the initial scene description text and the candidate scene description text according to a preset text prompt splicing format to obtain an enhanced scene description text.

[0090] Among them, a text prompt splicing format is preset, such as directly merging according to character strings, etc. Specifically, the initial scene description text and the candidate scene description text are merged according to the preset text prompt splicing format to obtain an enhanced scene description text.

[0091] Therefore, it is proposed to convert the information of roadside equipment or cloud servers into a structured description language and integrate it with the scene description on the vehicle side. This is different from the usual practice of directly inputting data in a limited format as planning and control rules. By integrating the local perception data on the vehicle side and the global scene description of the cloud server, an enhanced scene description text is formed. This combination is superior to the limitations of the existing technology that usually considers local information independently, and improves the decision-making ability of the autonomous driving system in complex environments.

[0092] Step 205: Perform static attribute analysis on the enhanced scene description text to obtain a static analysis result, and perform motion state analysis on the enhanced scene description text to obtain a dynamic analysis result.

[0093] Step 206: Set the static analysis result and the dynamic analysis result based on the third prompt word to obtain an analysis description text, and input the analysis description text into the visual language model VLM to obtain the analysis result and send it to the vehicle controller, so that the vehicle controller performs vehicle driving control based on the analysis result.

[0094] In the disclosed embodiment, after obtaining the enhanced scene description text, a more comprehensive scene analysis of the current driving scene is performed through the VLM, including analysis of key objects and analysis of the potential driving impact of key objects on the vehicle, and the analysis results are input into the planning and control stage of the vehicle's end-to-end autonomous driving system.

[0095] The disclosed embodiments use vehicle-side computing combined with a vehicle-road-cloud system to enhance the scene understanding capability of an autonomous driving vehicle. For example, Figure 3 An example diagram of vehicle driving control shown in FIG. 1 includes: a cloud server (cloud server) can access information sources such as weather services and municipal announcements to obtain a variety of data information, and process the various data information through the LLM of the cloud server to generate a scene description and send it to the vehicle client (vehicle); the vehicle-road includes roadside traffic monitoring equipment, smart parking lots, etc. to obtain roadside data and send it to the vehicle client; the vehicle client can generate a scene description (such as Figure 3 shown <system>Describe the current driving situation, <vlm>Time: xx, Road type: Community entrance, Lane line: White dashed line, Key target: Pedestrian on the right side is in [(x1, y1), (x2, y2)]), and the additional information provided by the cloud server and roadside equipment is integrated with the generated scene description to obtain the enhanced scene description (such as Figure 3 shown <system>Describe the current driving situation, <vlm>Time: xx, Road type: Community entrance, Lane line: White dashed line, Key target: Pedestrian on the right side at [(x1, y1), (x2, y2)]), <extra>Parking space: The community allows external vehicles to park, and the area with the most vacant parking spaces is P1); then, the enhanced scene description is analyzed to obtain the analysis results for planning control (such as Figure 3 shown <system>Self-state and historical trajectory, time: xx, road type: community entrance, lane line: white dashed line, key target: pedestrian on the right side at [(x1, y1), (x2, y2)]), parking space: the community allows external vehicles to park, the area with the most vacant parking spaces is P1, please confirm the driving trajectory and control operation, <vlm>Planning: Drive into P1 area and park, control operation: [slow down, keep straight, turn right at the end of the road...]).

[0096] Therefore, a progressive thinking chain of scene description, scene analysis and hierarchical planning is preset, and different prompt words correspond to the three modules of perception, planning and control in the traditional autonomous driving process. With the help of the powerful generalization and cognitive ability of the large visual language model, better performance than traditional modules is achieved.

[0097] As an example of an application scenario, after the autonomous driving enters parking lots in shopping malls, residential areas, etc., it enhances the scene description by receiving the parking space information in the parking lot, so that the system can make decisions to assist the vehicle in automatically planning to enter the target vacant parking space; as an example of another application scenario, in the time-limited bus lanes on the expressway, the self-driving system can identify and respond to the rules of the time-limited bus lanes through the enhanced scene description, intelligently plan the driving route, avoid restricted time periods, and make efficient use of non-restricted time periods.

[0098] As a result, unstructured vehicle-road-cloud information can be converted into a structured description language, which improves the availability of information and can effectively integrate external information into the end-to-end autonomous driving decision-making process; through integration with the end-to-end autonomous driving system, this solution can provide more accurate scene understanding and enhance the ability to understand complex traffic scenarios, thereby optimizing autonomous driving planning and control and improving safety and efficiency; the solution can also adapt to complex traffic environments and dynamic changes, such as traffic accidents, road construction, etc., making the autonomous driving system more flexible and reliable; the solution provides a paradigm that can easily allow more smart devices or information sources that are helpful for autonomous driving to be easily accessed to strengthen the end-to-end model on the vehicle side.

[0099] It should be noted that in addition to its application in the field of autonomous driving, it can also be used for urban traffic management and optimization for intelligent transportation systems, providing real-time traffic information and driving assistance for Internet of Vehicles services, and remote monitoring of vehicles for fleet management and vehicle maintenance; it can also be used in areas such as drone navigation and intelligent robot mission planning. In addition, the description language generation part can be replaced or improved with different large speech models to adapt to different application scenarios and improve the accuracy of information conversion.

[0100] Figure 4 This is a schematic diagram of the structure of a vehicle driving control device provided by an embodiment of the present disclosure. The device can be implemented by software and / or hardware and can generally be integrated into an electronic device. Figure 4 As shown, the device comprises:

[0101] A first acquisition module 401 is used to acquire perception data of a vehicle sensor;

[0102] A first processing module 402 is used to process the perception data based on the first prompt word of the thought chain through a preset visual language model VLM to obtain an initial scene description text;

[0103] The second acquisition module 403 is used to acquire external data of the vehicle-road cloud;

[0104] A second processing module 404 is used to process the external data based on the second prompt word of the thought chain through the visual language model VLM to obtain a candidate scene description text;

[0105] A merging module 405 is used to merge the initial scene description text and the candidate scene description text according to a preset text prompt splicing format to obtain an enhanced scene description text;

[0106] An analysis module 406 is used to analyze the enhanced scene description text based on the third prompt word of the thought chain through the visual language model VLM to obtain an analysis result;

[0107] The first sending module 407 is used to send the analysis result to the vehicle controller, so that the vehicle controller performs vehicle driving control based on the analysis result.

[0108] Optionally, two-way communication is established between the vehicle and the cloud server, and the second acquisition module 403 is specifically used to:

[0109] Receiving a plurality of data information or a first external scene description text obtained based on a plurality of data sources and sent by the cloud server; wherein the cloud server processes the plurality of data information based on a preset prompt word template through a preset large language model LLM (Large Language Model) to obtain the first external scene description text; and / or,

[0110] Receive roadside data sent by roadside equipment.

[0111] Optionally, the second processing module 404 is specifically configured to:

[0112] Convert the multiple data information and the roadside data through the visual language model VLM to obtain a first external description text;

[0113] The first external description text is set as an external sentence based on the second prompt word to obtain the candidate scene description text.

[0114] Optionally, the second processing module 404 is specifically configured to:

[0115] The roadside data is converted by the visual language model VLM to obtain a second external description text, and the second external description text is set as an external sentence based on the second prompt word to obtain the second external scene description text;

[0116] The first external scene description text and the second external scene description text are combined as the candidate scene description text.

[0117] Optionally, the device further includes:

[0118] The second sending module is used to send a request instruction to the cloud server, so that the cloud server obtains a subscription identifier based on the request instruction, and determines the multiple data sources based on the subscription identifier.

[0119] Optionally, the device further includes:

[0120] a third sending module, configured to send the driving information of the vehicle to the cloud server, so that the cloud server can perform real-time monitoring based on the driving information to obtain the multiple data information; and / or,

[0121] The fourth sending module is used to send an information acquisition request to the cloud server, so that the cloud server acquires the multiple data information based on the information acquisition request.

[0122] Optionally, the analysis module 406 is specifically configured to:

[0123] Performing static attribute analysis on the enhanced scene description text to obtain a static analysis result;

[0124] Performing motion state analysis on the enhanced scene description text to obtain a dynamic analysis result;

[0125] The static analysis result and the dynamic analysis result are set based on the third prompt word to obtain an analysis description text, and the analysis description text is input into the visual language model VLM to obtain the analysis result.

[0126] The vehicle driving control device provided in the embodiments of the present disclosure can execute the vehicle driving control method provided in any embodiment of the present disclosure, and has functional modules and beneficial effects corresponding to the execution method.

[0127] The embodiments of the present disclosure also provide a computer program product, including a computer program / instruction, which, when executed by a processor, implements the vehicle driving control method provided by any embodiment of the present disclosure.

[0128] Figure 5 This is a schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure. Figure 5 , which shows a schematic diagram of the structure of an electronic device 500 suitable for implementing the embodiment of the present disclosure. The electronic device 500 in the embodiment of the present disclosure may include but is not limited to mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle terminals (such as vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 5 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0129] like Figure 5 As shown, the electronic device 500 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage device 508 into a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the electronic device 500 are also stored. The processing device 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0130] Typically, the following devices may be connected to the I / O interface 505: an input device 506 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 507 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 508 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 509. The communication device 509 may allow the electronic device 500 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 5 The electronic device 500 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0131] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 509, or installed from the storage device 508, or installed from the ROM 502. When the computer program is executed by the processing device 501, the above-mentioned functions defined in the vehicle driving control method of the embodiment of the present disclosure are executed.

[0132] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0133] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (Hyper Text Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0134] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0135] The above-mentioned computer-readable medium carries one or more programs. When the above-mentioned one or more programs are executed by the electronic device, the electronic device: obtains the perception data of the vehicle sensor, processes the perception data based on the first prompt word of the thinking chain through the preset visual language model VLM, and obtains the initial scene description text; obtains the external data of the vehicle-road-cloud, processes the external data based on the second prompt word of the thinking chain through the visual language model VLM, and obtains the candidate scene description text; merges the initial scene description text and the candidate scene description text according to the preset text prompt splicing format to obtain the enhanced scene description text; analyzes the enhanced scene description text based on the third prompt word of the thinking chain through the visual language model VLM, obtains the analysis result and sends it to the vehicle controller, so that the vehicle controller performs vehicle driving control based on the analysis result.

[0136] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including, but not limited to, object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0137] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0138] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the name of a unit does not, in some cases, limit the unit itself.

[0139] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0140] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0141] According to one or more embodiments of the present disclosure, the present disclosure provides an electronic device, including:

[0142] processor;

[0143] a memory for storing instructions executable by the processor;

[0144] The processor is used to read the executable instructions from the memory and execute the instructions to implement any vehicle driving control method provided in the present disclosure.

[0145] According to one or more embodiments of the present disclosure, the present disclosure provides a computer-readable storage medium, wherein the storage medium stores a computer program, and the computer program is used to execute any vehicle driving control method as provided by the present disclosure.

[0146] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0147] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0148] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.< / vlm> < / system> < / extra> < / vlm> < / system> < / vlm> < / system>

Claims

1. A vehicle driving control method, characterized in that: Applied to a vehicle, the method comprises: Acquire the perception data of the vehicle sensor, and process the perception data based on the first prompt word of the thought chain through a preset visual language model VLM to obtain an initial scene description text; Acquire external data of the vehicle-road cloud, and process the external data through the visual language model VLM based on the second prompt word of the thought chain to obtain a candidate scene description text; Merging the initial scene description text and the candidate scene description text according to a preset text prompt splicing format to obtain an enhanced scene description text; The enhanced scene description text is analyzed by the visual language model VLM based on the third prompt word of the thought chain, and the analysis result is sent to the vehicle controller, so that the vehicle controller performs vehicle driving control based on the analysis result.

2. The vehicle driving control method according to claim 1, characterized in that: The vehicle and the cloud server establish two-way communication, and the acquisition of external data of the vehicle-road-cloud includes: Receiving a plurality of data information or a first external scene description text obtained based on a plurality of data sources and sent by the cloud server; wherein the cloud server processes the plurality of data information based on a preset prompt word template through a preset large language model LLM to obtain the first external scene description text; and / or, Receive roadside data sent by roadside equipment.

3. The vehicle driving control method according to claim 2, characterized in that: The processing of the external data based on the second prompt word of the thought chain by the visual language model VLM to obtain a candidate scene description text includes: Convert the multiple data information and the roadside data through the visual language model VLM to obtain a first external description text; The first external description text is set as an external sentence based on the second prompt word to obtain the candidate scene description text.

4. The vehicle driving control method according to claim 2, characterized in that: The processing of the external data based on the second prompt word of the thought chain by the visual language model VLM to obtain a candidate scene description text includes: The roadside data is converted by the visual language model VLM to obtain a second external description text, and the second external description text is set as an external sentence based on the second prompt word to obtain the second external scene description text; The first external scene description text and the second external scene description text are combined as the candidate scene description text.

5. The vehicle driving control method according to claim 2, characterized in that: The method further comprises: A request instruction is sent to the cloud server, so that the cloud server obtains a subscription identifier based on the request instruction, and determines the multiple data sources based on the subscription identifier.

6. The vehicle driving control method according to claim 2, characterized in that: The method further comprises: Sending the driving information of the vehicle to the cloud server, so that the cloud server can monitor and obtain the multiple data information in real time based on the driving information; and / or, Send an information acquisition request to the cloud server, so that the cloud server acquires the multiple data information based on the information acquisition request.

7. The vehicle driving control method according to claim 1, characterized in that: The step of analyzing the enhanced scene description text based on the third prompt word of the thought chain by the visual language model VLM to obtain an analysis result includes: Performing static attribute analysis on the enhanced scene description text to obtain a static analysis result; Performing motion state analysis on the enhanced scene description text to obtain a dynamic analysis result; The static analysis result and the dynamic analysis result are set based on the third prompt word to obtain an analysis description text, and the analysis description text is input into the visual language model VLM to obtain the analysis result.

8. A vehicle driving control device, characterized in that: Applied to vehicles, including: A first acquisition module, used to acquire perception data of vehicle sensors; A first processing module is used to process the perception data based on the first prompt word of the thought chain through a preset visual language model VLM to obtain an initial scene description text; The second acquisition module is used to acquire external data of the vehicle-road cloud; A second processing module is used to process the external data based on the second prompt word of the thought chain through the visual language model VLM to obtain a candidate scene description text; A merging module, used to merge the initial scene description text and the candidate scene description text according to a preset text prompt splicing format to obtain an enhanced scene description text; An analysis module, configured to analyze the enhanced scene description text based on the third prompt word of the thought chain through the visual language model VLM to obtain an analysis result; The first sending module is used to send the analysis result to the vehicle controller so that the vehicle controller performs vehicle driving control based on the analysis result.

9. An electronic device, characterized in that: The electronic device comprises: processor; a memory for storing instructions executable by the processor; The processor is used to read the executable instructions from the memory and execute the instructions to implement the vehicle driving control method described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and the computer program is used to execute the vehicle driving control method described in claims 1-7 above.

Images