Driving reminding method and device, vehicle, storage medium and program product

Abstract

This disclosure relates to a driving reminder method, device, vehicle, storage medium, and program product. It can acquire vehicle driving environment data; based on the driving environment data, determine the first driving behavior of other vehicles within a preset area surrounding the vehicle; and based on the first driving behavior, control the vehicle to output target reminder information, which is used to remind the vehicle user to drive safely. In this way, the driving behavior of other vehicles around the vehicle can be determined based on the vehicle's driving environment data, and then safe driving reminders can be provided based on the identification results of other vehicles' driving behavior. The output reminder information is more consistent with actual road conditions, effectively improving driving safety and reducing the probability of traffic accidents.

Description

Technical Field

[0001] This disclosure relates to the field of smart cockpit technology, and in particular to a driving reminder method, device, vehicle, storage medium, and program product. Background Technology

[0002] Competitive driving behavior (commonly known as road rage) typically refers to anger arising from traffic congestion or misunderstanding of other drivers' actions. In this state, drivers may engage in unsafe behaviors such as reckless lane changes, aggressive overtaking, and running yellow lights; in severe cases, they may even attack other vehicles or drivers. The number of traffic accidents caused by road rage is steadily increasing each year, seriously impacting public transportation safety. Therefore, it is necessary to identify and intervene in driving behaviors to reduce traffic accidents. Summary of the Invention

[0003] To overcome the problems existing in related technologies, this disclosure provides a driving reminder method, device, vehicle, storage medium, and program product.

[0004] According to a first aspect of the present disclosure, a driving reminder method is provided, comprising: Acquire vehicle driving environment data; Based on the driving environment data, determine the first driving behavior of other vehicles within the surrounding preset area; Based on the first driving behavior, the vehicle is controlled to output target prompt information, which is used to remind the user of the vehicle to drive safely.

[0005] Optionally, determining the first driving behavior of other vehicles within a preset area surrounding the vehicle based on the driving environment data includes: Based on the driving environment data, determine the road condition description information corresponding to the preset area; The first driving behavior is determined based on the road condition description information.

[0006] Optionally, the driving environment data includes environmental images of the vehicle's surroundings in at least one direction; The step of determining the road condition description information corresponding to the preset area based on the driving environment data includes: Obtain the preset task description instruction corresponding to the driving environment data; After inputting the preset task description instruction and the environmental image of at least one direction around the vehicle into the visual language model, the road condition description information output by the model is obtained. The preset task description instruction is a natural language description of the recognition task currently being performed by the visual language model.

[0007] Optionally, the visual language model includes a visual encoder, a text encoder, and a large language model; The step of inputting the preset task description instruction and the environmental image of at least one direction around the vehicle into the visual language model to obtain the road condition description information output by the model includes: The environmental image in at least one direction is input into the visual encoder to obtain the first vector code of the image; The preset task description instruction is input into the text encoder to obtain the second vector code of the text; After inputting the first vector encoding and the second vector encoding into the large language model, the traffic condition description information is output.

[0008] Optionally, the driving environment data also includes image recognition results after performing image recognition on the environmental images in the at least one direction; The step of determining the road condition description information corresponding to the preset area based on the driving environment data includes: After inputting the preset task description instruction, the environmental image of at least one direction, and the image recognition result into the visual language model, the road condition description information output by the model is obtained.

[0009] Optionally, the first driving behavior includes the current driving behavior and / or predicted future driving behavior of the other vehicles.

[0010] Optionally, controlling the vehicle to output target prompt information based on the first driving behavior includes: If it is determined that a first preset driving behavior exists in the first driving behavior, the cause of the first preset driving behavior is identified according to the road condition description information, wherein the first preset driving behavior includes the driving behavior of other vehicles that affect the driving safety of the vehicle. The target prompt information is output based on the cause of the first preset driving behavior.

[0011] Optionally, identifying the cause of the first preset driving behavior based on the road condition description information includes: The road surface data of the roads within the preset area are determined based on the road condition description information; The cause of the first preset driving behavior is identified based on the road surface data.

[0012] Optionally, the method further includes: If the road surface data determines that the road within the preset area meets the preset risk warning conditions, the vehicle is controlled to issue a risk warning.

[0013] Optionally, the method further includes: Obtain the vehicle's status data; The step of determining the first driving behavior of other vehicles within a preset area surrounding the vehicle based on the driving environment data includes: Based on the driving environment data and the vehicle status data, the first driving behavior and the second driving behavior of the vehicle are determined by the visual language model, wherein the vehicle status data includes the vehicle's driving status data and / or the user status data of the user inside the vehicle.

[0014] Optionally, controlling the vehicle to output target prompt information based on the first driving behavior includes: Based on the first driving behavior and the second driving behavior, the vehicle is controlled to output the target prompt information.

[0015] According to a second aspect of the present disclosure, a driving reminder device is provided, the device comprising: The acquisition module is configured to acquire vehicle driving environment data; The determination module is configured to determine the first driving behavior of other vehicles within a preset area surrounding the vehicle based on the driving environment data. The control module is configured to control the vehicle to output target prompt information based on the first driving behavior, the target prompt information being used to remind the user of the vehicle to drive safely.

[0016] According to a third aspect of the present disclosure, a vehicle is provided, comprising: processor; Memory used to store processor-executable instructions; The processor is configured to perform the steps of the method described in the first aspect of this disclosure.

[0017] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps of the method described in the first aspect of the present disclosure.

[0018] According to a fifth aspect of the present disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps of the method described in the first aspect of the present disclosure.

[0019] The technical solutions provided by the embodiments of this disclosure can include the following beneficial effects: based on the vehicle's driving environment data, the driving behavior of other vehicles around the vehicle is determined. This allows for the issuance of safe driving reminders based on the identification results of the driving behavior of other vehicles around the vehicle, effectively influencing the driver's decision-making. Compared to driving behavior identification based on the vehicle's own state data, the identification results can reflect the actual road conditions of the vehicle's driving environment. Therefore, issuing safe driving reminders based on the first driving behavior can effectively improve driving safety and reduce the probability of traffic accidents.

[0020] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0021] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0022] Figure 1 This is a flowchart illustrating a driving reminder method according to an exemplary embodiment.

[0023] Figure 2 It is based on Figure 1 The illustrated embodiment shows a flowchart of a driving reminder method.

[0024] Figure 3 It is based on Figure 2 The illustrated embodiment shows a flowchart of a driving reminder method.

[0025] Figure 4 This is a schematic diagram illustrating the architecture of a visual language model according to an exemplary embodiment.

[0026] Figure 5 It is based on Figure 1 The illustrated embodiment shows a flowchart of a driving reminder method.

[0027] Figure 6 It is based on Figure 5 The illustrated embodiment shows a flowchart of a driving reminder method.

[0028] Figure 7 It is based on Figure 1 The illustrated embodiment shows a flowchart of a driving reminder method.

[0029] Figure 8 This is a block diagram illustrating a driving reminder device according to an exemplary embodiment.

[0030] Figure 9 This is a block diagram illustrating a vehicle according to an exemplary embodiment. Detailed Implementation

[0031] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0032] It should be noted that all actions involving the acquisition of signals, information, or data in this disclosure are carried out in compliance with the relevant data protection laws and policies of the country where the location is situated, and with authorization from the owner of the relevant device.

[0033] This disclosure is primarily applied to scenarios involving the identification or prediction of vehicle driving behavior. In such scenarios, the vehicle can be controlled to provide safe driving reminders or to soothe the driver's emotions based on the identified or predicted driving behavior.

[0034] For example, this driving behavior can include competitive driving. Competitive driving behavior mainly includes actions a driver might take when angry, such as reckless lane changes, aggressive overtaking, running yellow lights, using foul language, and attacking other vehicles. Road rage often arises suddenly and without warning, sometimes even instinctively. Even people who are normally very emotionally stable can experience road rage while driving. Because road rage-related traffic accidents are rising year after year, and with the increase in the number of motor vehicles and improvements in vehicle performance, the negative social impact of road rage will continue to expand. Therefore, it is necessary to identify and intervene in competitive driving behavior in a timely manner to reduce the occurrence of traffic accidents caused by road rage.

[0035] In related technologies, rule-based or small-scale model (usually a small model deployed on the vehicle) methods can be used to identify and address road rage in drivers of their own vehicles. Examples include rule-based recognition of sudden acceleration (or deceleration), lane-crossing, and vehicle-mounted small-scale model recognition of driver facial expressions and body posture. Typically, an in-vehicle camera captures the driver's facial expressions, sensors collect vehicle driving status data, and then rules or onboard small models are used to determine the driver's behavior. If the behavior is determined to be road rage, the driver can be alerted to safety. However, this presents several problems: First, it only identifies the driver's own behavior and cannot assess the safety of other vehicles or the overall driving environment. Second, using rules or small models to judge driving behavior based on driver expressions and vehicle driving status has low accuracy and cannot provide effective explanations for the identified behavior. Furthermore, rule-based or onboard small-scale model-based driver behavior identification can only identify and intervene after competitive driving behavior has occurred, and cannot effectively prevent dangerous driving behavior from happening.

[0036] To address the aforementioned problems, this disclosure provides a driving reminder method, device, vehicle, storage medium, and program product. The specific embodiments of this disclosure will be described in detail below with reference to the accompanying drawings.

[0037] Figure 1 This is a flowchart illustrating a driving reminder method according to an exemplary embodiment, which can be applied to a vehicle. Figure 1 As shown, the method includes the following steps.

[0038] In step S11, the vehicle's driving environment data is acquired.

[0039] The driving environment data refers to data that characterizes the environment surrounding the vehicle. For example, the driving environment data may include environmental images in at least one direction around the vehicle. By performing image recognition on the environmental images, information such as pedestrians, other vehicles, obstacles, and road surface data (such as lane lines, road surface smoothness, road markings, etc.) around the vehicle can be determined.

[0040] In this step, the vehicle can acquire environmental images from at least one direction around the vehicle using surround-view cameras.

[0041] In step S12, based on the driving environment data, the first driving behavior of other vehicles within a preset area surrounding the vehicle is determined.

[0042] Since this driving environment data represents the environmental data surrounding the vehicle, including driving information of other vehicles around the vehicle (such as whether they forcibly change lanes, overtake, and speed information), this step can determine the primary driving behavior of other vehicles around the vehicle based on the driving environment data. For example, this primary driving behavior can include competitive driving behaviors such as overtaking, changing lanes, rapid acceleration, rapid deceleration, and cutting off other vehicles. Furthermore, the preset area can be understood as the area surrounding the vehicle centered on the vehicle itself. For example, the preset area could be a rectangular area around the vehicle within a first distance range in front of and behind the vehicle, and within a second distance range to the left and right of the vehicle.

[0043] It should be noted that "determining" in this step can include both identification and prediction; that is, it can identify the current driving behavior of other vehicles, and it can also predict the possible future driving behavior of other vehicles. In other words, the first driving behavior includes the current driving behavior of other vehicles around the vehicle and / or the predicted future driving behavior.

[0044] In step S13, based on the first driving behavior, the vehicle is controlled to output target prompt information, which is used to remind the user of the vehicle to drive safely.

[0045] The target prompt information may include one or more of the following: the identification result of the first driving behavior (such as whether it is a dangerous driving behavior), the explanation of the cause of the first driving behavior, and the information to soothe the emotions of the user in the vehicle.

[0046] Based on the identification of potential dangerous vehicles around the vehicle by the first driving behavior, and based on the target prompt information, the driver of the vehicle is informed, which helps the driver to predict and avoid dangerous factors in a timely manner.

[0047] In addition, when controlling the vehicle to output the target prompt information in this step, it can be output in the form of voice, or in the form of text (such as text prompts on the vehicle's central control screen), or in a combination of voice and text. This disclosure does not limit the specific form of the target prompt information.

[0048] Using the above method, the driving behavior of other vehicles around the vehicle is determined based on the vehicle's driving environment data. This allows for the identification of safe driving behaviors based on the recognition results of these other vehicles, effectively influencing the driver's decision-making. Compared to identifying driving behavior based on the vehicle's own status data, the identification results reflect the actual road conditions of the driving environment. Therefore, providing safe driving reminders based on primary driving behaviors can effectively improve driving safety, reduce the probability of traffic accidents, and also enhance the driving and riding experience for drivers and passengers, as well as the intelligence level of the smart cockpit.

[0049] Figure 2 It is based on Figure 1 The illustrated embodiment shows a flowchart of a driving reminder method, as follows: Figure 2 As shown, step S12 includes the following sub-steps: In step S121, road condition description information corresponding to the preset area is determined based on driving environment data.

[0050] The driving environment data may include environmental images in at least one direction around the vehicle. These environmental images may include, for example, a front view image of the vehicle, a surround view image of the vehicle (generally composed of environmental images in the four directions of the vehicle, collected at the same time, and a rear view image of the vehicle).

[0051] The road condition description information refers to the natural language description of the road conditions in a preset area identified based on the driving environment data. This road condition description information can include natural language descriptions of various types of road conditions, such as road type (e.g., highway, ring road, urban road), road environment (e.g., daytime, nighttime, weather, visibility), special road conditions (e.g., ramps, lane occupancy, traffic control), driving behavior of other vehicles (e.g., lane changing, lane crossing, overtaking, cutting off other vehicles), and predictions of competitive driving behavior.

[0052] During this step, the road condition description information can be determined using the Vision Language Model (VLM) based on the driving environment data.

[0053] The visual language model refers to a multimodal large model that can learn from both images and text. Typical visual language models include MiniGPT-4, LLaVA, Qwen-VL, etc. These models can combine pre-trained visual encoders with large language models, enabling large language models to handle many tasks involving image input.

[0054] The core of the visual language model used in this disclosure lies in its powerful multimodal learning capability. Traditional computer vision models mainly focus on tasks such as image recognition and classification, while natural language processing models focus on text generation and understanding. In this disclosure, VLM integrates the advantages of these two models to achieve cross-modal understanding and generation of image and text data. That is, the input of the visual language model used in this disclosure can be image data and text data, and the output can be text data.

[0055] It should be noted that the entity executing step S12 can be either the vehicle or the server corresponding to the vehicle.

[0056] One possible implementation is to determine the initial driving behavior locally on the vehicle side. That is, the vehicle can determine the initial driving behavior based on driving environment data, using a visual language model pre-deployed in the vehicle.

[0057] For example, a vehicle can collect driving environment data collected during historical driving as input samples for model training. The actual driving behaviors of other vehicles around the vehicle corresponding to the driving environment data collected at different times can be used as training labels for the model. After pre-training the preset model, the visual language model is obtained. The model can then be deployed on the vehicle's infotainment system so that the vehicle can determine the first driving behavior locally based on the real-time collected driving environment data through the visual language model.

[0058] Considering that visual language models are usually large in scale, there are certain limitations to their deployment on the local vehicle system. Therefore, other types of deep learning models can be pre-deployed on the vehicle system. In this way, the first driving behavior can be determined locally in the vehicle based on the driving environment data through these other types of deep learning models.

[0059] In another possible implementation, considering that visual language models are usually large in scale and suitable for deployment in the cloud (i.e., servers), this disclosure can also send the driving environment data collected by the vehicle to the server at a preset frequency. In this way, the server can determine the first driving behavior based on the received driving environment data through the visual language model and send it to the vehicle, so that the vehicle can output guidance on driving based on the first driving behavior and improve driving safety.

[0060] Figure 3 It is based on Figure 2 The illustrated embodiment presents a flowchart of a driving alert method. In this disclosure, during the process of determining road condition description information corresponding to a preset area based on driving environment data using a vehicle or server, methods such as... Figure 3 The steps shown are followed. (As shown) Figure 3 As shown, step S121 includes the following sub-steps: In step S1211, a preset task description instruction corresponding to the driving environment data is obtained.

[0061] The preset task description instruction can be a model input instruction corresponding to the visual language model, determined based on the driving environment data. Furthermore, the preset task description instruction is a natural language description of the recognition task currently being performed by the visual language model; that is, the preset task description instruction can be a natural language description of the recognition and / or prediction task to be performed by the visual language model, which is then used as input data for the model along with the driving environment data. For example, the preset task description instruction could be: "Based on the current surrounding environment, analyze the behavioral motivations of surrounding vehicles and assess the driving risks of other vehicles."

[0062] In step S1212, the preset task description instruction and environmental images of at least one direction around the vehicle are input into the visual language model to obtain the road condition description information output by the model.

[0063] For example, Figure 4 This is a schematic diagram illustrating the architecture of a visual language model according to an exemplary embodiment, such as... Figure 4 As shown, the visual language model includes a visual encoder, a text encoder, and a large language model. During this step, an environmental image from at least one direction can be input into the visual encoder to obtain a first vector code for the image; the preset task description instruction can be input into the text encoder to obtain a second vector code for the text; and the first and second vector codes can be input into the large language model to output the road condition description information.

[0064] In yet another possible embodiment of this disclosure, the driving environment data may further include image recognition results obtained by performing image recognition on the environmental images in at least one direction. For example, the vehicle may obtain the image recognition results by performing image recognition on the environmental images in at least one direction based on a vehicle-side perception model.

[0065] The vehicle-side perception model can be a visual model deployed on the vehicle. This visual model can perform target detection on environmental images collected by the vehicle from each direction. The image recognition result can include the target detection result, which can include the location and category information of each target object in the environmental image. The target object can include, for example, other vehicles, pedestrians, obstacles, and other objects around the vehicle.

[0066] Thus, in the process of determining the road condition description information corresponding to the preset area based on driving environment data, this disclosure can input the preset task description instruction, the environmental image of at least one direction, and the image recognition result into the visual language model to obtain the road condition description information output by the model.

[0067] For example, the preset task description instruction and the image recognition result can be input together into a program such as... Figure 4The text encoder in the visual language model shown obtains the second vector encoding of the text, and inputs the environmental image in at least one direction as shown. Figure 4 The visual encoder shown obtains the first vector code of the image, and then inputs the first vector code and the second vector code into the large language model to output the road condition description information.

[0068] In this way, the image recognition result, along with the driving environment data, is input into the visual language model, which helps the visual language model learn from the input data from more dimensions, improves the model's understanding ability, and thus improves the accuracy of the model's output results.

[0069] In step S122, the first driving behavior is determined based on the road condition description information.

[0070] In this step, the vehicle or server can perform rule matching based on the road condition description information to determine the target natural language description of other vehicles' driving behavior contained in the road condition description information, and take the driving behavior described in the target natural language description as the first driving behavior.

[0071] As mentioned above, the first driving behavior includes the current driving behavior of other vehicles around the vehicle and / or the predicted future driving behavior.

[0072] For example, suppose a server uses a visual language model to predict the future driving behavior of other vehicles around a vehicle based on its driving environment data. The vehicle can collect this driving environment data in real time and upload it to the server at a preset frequency (e.g., once per second). The server can then use this data to predict the driving behavior of other vehicles. Assuming the vehicle uploads data to the server every second, the server receives multiple consecutive frames of image data collected within that second. This data includes temporal information. The visual language model can then learn global temporal information based on each uploaded frame and previous uploads, enabling it to predict the behavior of other vehicles and provide early warnings to the driver. Furthermore, it can use strategies such as ambient music and lighting to soothe the driver's emotions, avoiding the lag problem of intervention only after competitive driving behavior has already occurred.

[0073] Figure 5 It is based on Figure 1 The illustrated embodiment shows a flowchart of a driving reminder method, as follows: Figure 5 As shown, step S13 includes the following sub-steps: In step S131, if it is determined that there is a first preset driving behavior in the first driving behavior, the cause of the first preset driving behavior is identified according to the road condition description information, wherein the first preset driving behavior includes the driving behavior of other vehicles that affect the driving safety of the vehicle.

[0074] In step S132, the target prompt information is output according to the cause of the first preset driving behavior.

[0075] During the execution of step S131, the road surface data of the roads within the preset area can be determined based on the road condition description information. That is, the road condition description information can also include description information of the road surface data of the roads within the preset area. In this way, the cause of the first preset driving behavior can be identified based on the road surface data. Among them, the road surface data can include whether the roads within the preset area are special road conditions (such as ramps, lane occupancy, controlled road sections, etc.).

[0076] One objective of this disclosure is to output target prompt information based on the identified first driving behavior, thereby providing safe driving reminders to vehicle users. In one possible application scenario, when other vehicles around the driver engage in competitive driving behaviors such as sudden lane changes or overtaking, the driver may over-attribute these behaviors to intentional or malicious actions rather than external objective factors. For example, a lane change by another vehicle to avoid an invisible obstacle might be perceived as malicious cutting off the driver, leading to road rage and potentially unsafe driving. Therefore, when the identified or predicted first driving behaviors of other vehicles include those affecting the driver's safety (i.e., the first pre-defined driving behavior), it is necessary to identify the cause of this first pre-defined driving behavior and output the cause to the user (i.e., to evaluate the first pre-defined driving behavior in a more objective way). This can minimize misunderstandings by the driver (i.e., prevent the driver from thinking in a subjectively malicious direction) and improve driving safety.

[0077] In the process of identifying the cause of the first preset driving behavior based on the road surface data, it is possible to determine whether the road within the preset area around the vehicle is a special road condition (such as ramps, lane occupancy, controlled road sections, etc.) based on the road surface data. Thus, if it is determined that the first preset driving behavior exists in the first driving behavior, and if it is determined based on the road surface data that the road within the preset area around the vehicle is a special road condition, it can be determined that the cause of the first preset driving behavior may be due to the existence of the special road condition leading to the first preset driving behavior of other vehicles.

[0078] For example, suppose that based on the road surface data it is determined that there is an obstacle blocking the road in a preset area around the vehicle. Other vehicles may suddenly change lanes to avoid the obstacle blocking the road. In this case, the target prompt message output by the vehicle control system for the first preset driving behavior may be: "The vehicle ahead changed lanes suddenly because there is an obstacle in its lane. Please drive safely." The above example is only for illustration and this disclosure does not limit it.

[0079] In this way, the target prompt information can explain the first preset driving behavior of other vehicles to the user of the vehicle. The user of the vehicle can understand the reason for the first preset driving behavior of other vehicles in a timely manner based on the target prompt information output by the vehicle. This makes it easier to understand the behavior of other vehicles, avoids road rage caused by misunderstandings, and improves driving safety.

[0080] Figure 6 It is based on Figure 5 The illustrated embodiment shows a flowchart of a driving reminder method, as follows: Figure 6 As shown, the method also includes the following steps: In step S14, if the road within the preset area is determined to meet the preset risk warning conditions based on the road surface data, the vehicle is controlled to issue a risk warning.

[0081] The preset risk warning conditions refer to pre-defined road conditions that may affect the driver's safety. For example, if road data indicates that there is a ramp ahead of the vehicle, the driver should be wary of other vehicles making sudden lane changes. Similarly, if road data indicates that there is a controlled road section ahead of the vehicle, the driver should be reminded to avoid the controlled road section and be wary of other vehicles changing lanes. Therefore, the preset risk warning conditions can include at least one road condition within the preset area, such as a ramp, a road encroachment, or a controlled road section.

[0082] For example, if the road data indicates that there is an on-ramp ahead of the vehicle, the risk warning message that can be controlled to be output by the vehicle could be: "There is an on-ramp ahead. Beware of other vehicles changing lanes suddenly. Please keep your attention or change lanes to the left in advance." This example is merely illustrative and is not intended to limit the scope of this disclosure.

[0083] Based on the above method, when it is determined that other vehicles' initial driving behaviors contain a first preset driving behavior that affects the driving safety of the vehicle, the vehicle can be controlled to output target prompt information. This target prompt information can explain the cause of the other vehicle's first preset driving behavior. Furthermore, if the road surface data determines that the road within a preset area meets preset risk warning conditions, the vehicle can also be controlled to issue a risk warning.

[0084] Figure 7 It is based on Figure 1 The illustrated embodiment shows a flowchart of a driving reminder method, as follows: Figure 7 As shown, the method also includes the following steps: In step S15, the vehicle's self-state data is acquired.

[0085] The vehicle status data may include vehicle driving status data and / or user status data of users inside the vehicle (such as the driver). The driving status data may include, for example, vehicle speed, acceleration, accelerator / brake pedal opening, steering wheel angle, etc. The user status data may include user facial expression data and user posture data. For example, the user status data may be represented based on user images, that is, the user status data includes user images.

[0086] In this step, the vehicle can collect driving status data through sensors and collect facial expression data and posture data of the user through cameras in the cockpit.

[0087] Thus, during the execution of step S12, the first driving behavior and the second driving behavior of the vehicle can be determined through a visual language model based on driving environment data and vehicle status data.

[0088] The second driving behavior refers to the driving behavior of the vehicle itself, which may also include competitive driving behaviors such as overtaking, changing lanes, rapid acceleration, rapid deceleration, and cutting off other vehicles.

[0089] Similar to the method described above for determining the first driving action based on driving environment data using a visual language model, there are two ways to simultaneously determine both the first and second driving actions. One approach is to determine them locally on the vehicle side, where the vehicle can determine both actions based on driving environment data and its own vehicle status data using a pre-deployed visual language model. The other approach is to send the driving environment data and vehicle status data to a server, allowing the server to determine the first and second driving actions using a visual language model.

[0090] In this step, the vehicle or server can determine a preset task description instruction based on its own vehicle status data and the driving environment data; the user image and an environmental image in at least one direction are input as follows: Figure 4 The visual encoder shown obtains the third vector code of the image; the driving state data and the preset task description instruction are input into the text encoder to obtain the fourth vector code of the text; the third vector code and the fourth vector code are input into the large language model to output the second driving behavior.

[0091] The preset task description instruction can be a model input instruction corresponding to the visual language model, determined based on the vehicle's state data. This preset task description instruction can be a natural language description of the recognition and / or prediction task to be performed by the visual language model, used together with the driving state data as input data for the model. For example, the preset task description instruction could be: "Based on the current surrounding environment and vehicle state data, please analyze the behavioral motivations of surrounding vehicles and your own vehicle, and assess the driving risks of other vehicles and your own vehicle."

[0092] It should be noted that the second driving behavior may also include the vehicle's current driving behavior and / or the predicted future driving behavior of the vehicle.

[0093] For example, suppose a server predicts a vehicle's future driving behavior using a visual language model based on the vehicle's status data. The vehicle can collect this status data in real time and upload it to the server at a preset frequency (e.g., once per second). The server can then use this data to predict the vehicle's future driving behavior. Assuming the vehicle uploads data to the server every second, the server receives multiple consecutive frames of data collected within that second. This data includes temporal information. The visual language model can then learn global temporal information based on each uploaded frame and previous data uploads, enabling prediction of the vehicle's behavior and providing early warnings for driver safety. Furthermore, it can use strategies such as ambient music and lighting to soothe the driver's emotions, avoiding the lag problem of intervention only after competitive driving behavior has already occurred.

[0094] It should also be noted that after using the driving environment data and the vehicle's status data as input to the visual language model, the model can simultaneously output the first driving behavior of other vehicles within a preset area around the vehicle, as well as the second driving behavior of the vehicle itself. These first and second driving behaviors constitute the overall judgment of the vehicle's driving behavior. Providing safe driving reminders based on the recognition results of this overall driving behavior can effectively improve driving safety and reduce the probability of traffic accidents.

[0095] For example, a vehicle can send environmental images, driving status data, and user status data from at least one direction to a server; then, it receives a first driving behavior and a second driving behavior output by the server through a visual language model based on the environmental images, driving status data, and user status data from at least one direction. For instance, the vehicle can upload collected front view images, surround view images, rear view images, vehicle speed, acceleration, accelerator / brake pedal opening, steering wheel angle, driver's facial emotions, attention, and posture data to the server. After the server inputs the received data into a pre-trained visual language model, the visual language model can output road condition description information corresponding to a preset area. This road condition description information may include descriptions of the current driving behavior (such as lane changing, crossing lines, cutting off other vehicles, etc.) of the vehicle and other vehicles around it, identified by the visual language model based on the input data; it may also include descriptions of the driving tendencies of the vehicle and other vehicles around it, predicted by the visual language model based on the input data. These driving tendencies may include the probability that the vehicle and / or other vehicles around it will engage in competitive driving behavior at a preset future time (the next time).

[0096] In addition, the road condition description information may also include descriptions of the road environment and road surface data of the road where the vehicle is currently located. The road environment may include the type of road (such as highway, ring road, urban road, etc.) and information such as the brightness (such as day or night), visibility and weather of the environment in which the road is located. The road surface data may include whether it is a special road condition (such as ramp, lane occupancy, controlled section, etc.). The above examples are only illustrative and this disclosure does not limit it.

[0097] After determining the first driving behavior and the second driving behavior, such as Figure 7 As shown, by executing step S13, the vehicle can be controlled to output target prompt information based on the first driving behavior and the second driving behavior.

[0098] The target prompt information may include one or more of the following: the identification results of the first driving behavior and the second driving behavior (such as whether it is a dangerous driving behavior), the explanation of the cause of the first driving behavior and the second driving behavior, and the information to soothe the emotions of the users in the vehicle.

[0099] In the process of controlling the vehicle to output target prompt information based on a first driving behavior and a second driving behavior, this disclosure can, if it is determined that a first preset driving behavior exists in the first driving behavior, identify the cause of the first preset driving behavior, and then output target prompt information according to the cause of the first preset driving behavior. The specific implementation method for identifying the cause of the first preset driving behavior can be referred to the description above, and will not be repeated here. If it is determined that the second driving behavior belongs to the second preset driving behavior, a first prompting strategy corresponding to the second driving behavior is determined, and target prompt information is output according to the first prompting strategy. The second preset driving behavior includes vehicle driving behaviors that affect the driving safety of the vehicle.

[0100] For example, the second preset driving behavior may include driving behaviors that affect vehicle driving safety, such as sudden lane changes, rapid acceleration, and sudden deceleration. It is understandable that, if the second preset driving behavior is determined to exist within the vehicle's second driving behavior, it is necessary to promptly output target prompt information to avoid further dangerous traffic accidents.

[0101] In one possible implementation, a pre-defined correspondence between competitive driving behaviors of the vehicle and prompting strategies can be established. This prompting strategy may include the text content of the prompt message, the prompting method (such as voice or text), and an instruction message indicating whether to execute a preset emotional calming strategy. Thus, based on the current second driving behavior and the correspondence, a first prompting strategy corresponding to the current second driving behavior can be determined, and then the target prompt message can be output according to the first prompting strategy.

[0102] For example, assuming the second driving behavior is determined to be rapid acceleration, the first prompt strategy corresponding to the rapid acceleration can be determined as "Prompt text content: The current driving mode is not very safe. Don't rush, drive slowly, and listen to some music to relax; Prompt method: voice output; play music A". At this time, the vehicle can be controlled to play the voice prompt message "The current driving mode is not very safe. Don't rush, drive slowly, and listen to some music to relax" in the vehicle cabin, and then play music A. This is just an example, and this disclosure does not limit it.

[0103] Furthermore, as mentioned above, the road condition description information may also include descriptions of the road environment where the vehicle is currently located. This road environment may include, for example, road type (such as highway, urban road, ring road, etc.) and weather and brightness information. Thus, in this disclosure, a second prompting strategy corresponding to the road environment can be determined; and the vehicle can be controlled to output target prompting information according to the second prompting strategy.

[0104] Similarly, the correspondence between road environment and prompting strategy can be preset. Then, based on the current road environment, the corresponding second prompting strategy is determined, and the vehicle is controlled to output the target prompting information according to the second prompting strategy. The second prompting strategy can also include information such as the text content and prompting method of the prompting information.

[0105] For example, assuming that the vehicle is currently in a nighttime environment and is driving on a highway based on the road environment, a voice prompt message can be output in the vehicle's cabin: "Currently driving on a highway at night, please enter a service area to rest in time to avoid fatigue driving." This is just an example, and this disclosure does not limit it.

[0106] Using the above method, the driving behavior of both the vehicle and other surrounding vehicles can be determined simultaneously based on the vehicle's driving environment data and its own vehicle status data. This allows for the provision of safe driving reminders based on the overall driving behavior recognition results, effectively influencing the driver's decision-making. Compared to driving behavior recognition based solely on vehicle status data, the recognition results are more comprehensive, thus effectively improving driving safety and reducing the probability of traffic accidents.

[0107] Figure 8 This is a block diagram illustrating a driving reminder device according to an exemplary embodiment, such as... Figure 8 As shown, the device includes: The acquisition module 801 is configured to acquire vehicle driving environment data; The determination module 802 is configured to determine the first driving behavior of other vehicles within a preset area surrounding the vehicle based on the driving environment data. The control module 803 is configured to control the vehicle to output target prompt information based on the first driving behavior, the target prompt information being used to remind the user of the vehicle to drive safely.

[0108] Optionally, the determining module 802 is configured to determine road condition description information corresponding to the preset area based on the driving environment data; and determine the first driving behavior based on the road condition description information.

[0109] Optionally, the driving environment data includes environmental images of the vehicle's surroundings in at least one direction; The determining module 802 is configured to acquire a preset task description instruction corresponding to the driving environment data; input the preset task description instruction and the environmental image of at least one direction around the vehicle into the visual language model to obtain the road condition description information output by the model, wherein the preset task description instruction is a natural language description of the recognition task currently being performed by the visual language model.

[0110] Optionally, the visual language model includes a visual encoder, a text encoder, and a large language model; the determining module 802 is configured to input the environmental image of at least one direction into the visual encoder to obtain a first vector code of the image; input the preset task description instruction into the text encoder to obtain a second vector code of the text; and input the first vector code and the second vector code into the large language model to output the road condition description information.

[0111] Optionally, the driving environment data also includes image recognition results after image recognition of the environmental images in the at least one direction; the determining module 802 is configured to input the preset task description instruction, the environmental images in the at least one direction and the image recognition results into the visual language model to obtain the road condition description information output by the model.

[0112] Optionally, the first driving behavior includes the current driving behavior and / or predicted future driving behavior of the other vehicles.

[0113] Optionally, the control module 803 is configured to, when determining that a first preset driving behavior exists in the first driving behavior, identify the cause of the first preset driving behavior based on the road condition description information, wherein the first preset driving behavior includes the driving behavior of other vehicles that affect the driving safety of the vehicle; and output the target prompt information based on the cause of the first preset driving behavior.

[0114] Optionally, the control module 803 is configured to determine the road surface data of the road within the preset area based on the road condition description information; and to identify the cause of the first preset driving behavior based on the road surface data.

[0115] Optionally, the control module 803 is further configured to control the vehicle to provide a risk warning when the road surface data determines that the road within the preset area meets the preset risk warning conditions.

[0116] Optionally, the acquisition module 801 is further configured to acquire the vehicle's self-state data; The determining module 802 is further configured to determine the first driving behavior and the second driving behavior of the vehicle based on the driving environment data and the vehicle status data through the visual language model, wherein the vehicle status data includes the driving status data of the vehicle and / or the user status data of the user inside the vehicle.

[0117] Optionally, the control module 803 is configured to control the vehicle to output the target prompt information based on the first driving behavior and the second driving behavior.

[0118] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0119] This disclosure also provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the steps of the driving reminder method provided in this disclosure.

[0120] Figure 9 This is a block diagram illustrating a vehicle according to an exemplary embodiment. For example, vehicle 900 can be a hybrid vehicle, a non-hybrid vehicle, an electric vehicle, a fuel cell vehicle, or other types of vehicle. Vehicle 900 can be an autonomous vehicle, a semi-autonomous vehicle, or a non-autonomous vehicle.

[0121] Reference Figure 9 The vehicle 900 may include various subsystems, such as an infotainment system 910, a perception system 920, a decision control system 930, a drive system 940, and a computing platform 950. The vehicle 900 may also include more or fewer subsystems, and each subsystem may include multiple components. Furthermore, each subsystem and component of the vehicle 900 can be interconnected via wired or wireless means.

[0122] In some embodiments, the infotainment system 910 may include a communication system, an entertainment system, and a navigation system, etc.

[0123] The perception system 920 may include several sensors for sensing information about the environment surrounding the vehicle 900. For example, the perception system 920 may include a global positioning system (which may be GPS, BeiDou, or other positioning systems), an inertial measurement unit (IMU), lidar, millimeter-wave radar, ultrasonic radar, and a camera device.

[0124] The decision control system 930 may include a computing system, a vehicle controller, a steering system, a throttle, and a braking system.

[0125] The drive system 940 may include components that provide powered motion to the vehicle 900. In one embodiment, the drive system 940 may include an engine, an energy source, a transmission system, and wheels. The engine may be one or a combination of internal combustion engines, electric motors, and compressed air engines. The engine is capable of converting energy provided by the energy source into mechanical energy.

[0126] Some or all of the functions of the vehicle 900 are controlled by a computing platform 950. The computing platform 950 may include at least one processor 951 and a first memory 952, the processor 951 being able to execute instructions 953 stored in the first memory 952.

[0127] The processor 951 can be any conventional processor, such as a commercially available CPU. The processor may also include, for example, a Graphics Processing Unit (GPU), a Field Programmable Gate Array (FPGA), a System on Chip (SOC), an Application Specific Integrated Circuit (ASIC), or a combination thereof.

[0128] The first memory 952 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0129] In addition to instruction 953, the first memory 952 can also store data, such as road maps, route information, vehicle position, direction, speed, and other data. The data stored in the first memory 952 can be used by the computing platform 950.

[0130] In this embodiment of the disclosure, the processor 951 may execute instructions 953 to complete all or part of the steps of the driving reminder method described above.

[0131] In another exemplary embodiment, a computer program product is also provided, the computer program product comprising a computer program executable by a programmable device, the computer program having a code portion for performing the driving reminder method described above when executed by the programmable device.

[0132] Those skilled in the art will also understand that the various illustrative logical blocks and steps listed in the embodiments of this application can be implemented by electronic hardware, computer software, or a combination of both. Whether such functionality is implemented through hardware or software depends on the specific application and the overall system design requirements. Those skilled in the art can implement the described functionality using various methods for each specific application, but such implementation should not be construed as exceeding the scope of protection of the embodiments of this application.

[0133] Furthermore, the term “exemplary” is used herein to mean serving as an example, instance, or illustration. Any aspect or design described herein as “exemplary” is not necessarily to be construed as advantageous compared to other aspects or designs. Rather, the use of the term “exemplary” is intended to present the concept in a concrete manner. As used herein, the term “or” is intended to mean an inclusive “or” rather than an exclusive “or.” That is, unless otherwise specified or clear from the context, “X applies A or B” is intended to mean any of the natural inclusive arrangements. That is, “X applies A or B” satisfies any of the foregoing instances if X applies A; X applies B; or both X applies A and B. Additionally, unless otherwise specified or clear from the context to refer to the singular form, the articles “a” and “an” as used in this application and the appended claims are generally understood to mean “one or more.”

[0134] Similarly, although this disclosure has been shown and described with respect to one or more implementations, equivalent variations and modifications will occur to those skilled in the art upon reading and understanding this specification and the accompanying drawings. This disclosure includes all such modifications and variations and is limited only by the scope of the claims. In particular, with respect to the various functions performed by the components described above (e.g., elements, resources, etc.), unless otherwise indicated, the terminology used to describe such components is intended to correspond to any component (functionally equivalent) that performs the specific function of the described component, even if structurally not equivalent to the disclosed structure. Furthermore, although specific features of this disclosure may have been disclosed with respect to only one of several implementations, such features may be combined with one or more other features of other implementations, as may be desired and advantageous to any given or particular application. Moreover, with regard to the terms “comprising,” “owning,” “having,” “having,” or variations thereof as used in the detailed description or claims, such terms are intended to be inclusive in a manner similar to the term “including.”

[0135] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the appended claims.

[0136] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

[0137] In the above detailed description, reference has been made to the accompanying drawings, which illustrate specific aspects of this disclosure by way of illustration. In this regard, terms indicating direction or positional relationship, such as “center,” “longitudinal,” “lateral,” “length,” “width,” “thickness,” “upper,” “lower,” “front,” “rear,” “left,” “right,” “vertical,” “horizontal,” “top,” “bottom,” “inner,” “outer,” “clockwise,” “counterclockwise,” “axial,” “radial,” and “circumferential,” are used with reference to the orientation of the described figures. Since components of the described device can be positioned in multiple different orientations, directional terms are used for illustrative purposes and not for limitation. It should be understood that other aspects can be utilized and structural or logical changes can be made without departing from the concept of this disclosure. Therefore, the following detailed description should not be considered limiting.

[0138] It should be understood that, unless otherwise specifically indicated, features of various embodiments of this disclosure described herein can be combined with each other. As used herein, the term “and / or” includes any one of the relevant listed items and any combination of any two or more; similarly, “at least one of…” includes any one of the relevant listed items and any combination of any two or more.

[0139] It should be understood that, unless otherwise expressly specified and limited, the terms "joining," "attaching," "installing," "connecting," "linking," "fixing," etc., used in the embodiments of this disclosure should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection, an electrical connection, or a connection that allows communication between them; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components, unless otherwise expressly limited. Those skilled in the art can understand the specific meaning of the above terms herein based on the specific circumstances.

[0140] Furthermore, the term "above" as used herein with respect to components, elements, or material layers formed or located "above" a surface may be used to indicate that the component, element, or material layer is "indirectly" positioned (e.g., placed, formed, deposited, etc.) on the surface such that one or more additional components, elements, or layers are arranged between the surface and the component, element, or material layer. However, the term "above" as used with respect to components, elements, or material layers formed or located "above" a surface may also optionally have a specific meaning: that the component, element, or material layer is "directly" positioned (e.g., placed, formed, deposited, etc.) on the surface, for example, in direct contact with the surface.

[0141] Although terms such as “first,” “second,” and “third” may be used herein to describe various components, parts, regions, layers, or sections, these components, parts, regions, layers, or sections are not limited to these terms. Rather, these terms are used only to distinguish one component, part, region, layer, or section from another. Therefore, without departing from the teachings of the examples described herein, the first component, part, region, layer, or section mentioned in the examples may also be referred to as the second component, part, region, layer, or section. Furthermore, the terms “first” and “second” are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as “first” or “second” may explicitly or implicitly include at least one of that feature. In the description herein, “a plurality” means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0142] It should be understood that spatial relative terms, such as “above,” “upper,” “below,” and “lower,” are used herein to describe the relationship between one element and another shown in the figures. In addition to the orientation depicted in the figures, these spatial relative terms are also intended to encompass different orientations of the device in use or operation. For example, if the device in the figures is flipped, an element described as “above” or “upper” relative to another element would be “below” or “lower” relative to that other element. Thus, depending on the spatial orientation of the device, the term “above” encompasses both above and below orientations. Devices may have other orientations (e.g., rotated 90 degrees or in other orientations), and the spatial relative terms used herein should be interpreted accordingly.

Claims

1. A driving reminder method, characterized in that, The method includes: Acquire vehicle driving environment data; Based on the driving environment data, determine the first driving behavior of other vehicles within a preset area surrounding the vehicle; Based on the first driving behavior, the vehicle is controlled to output target prompt information, which is used to remind the user of the vehicle to drive safely.

2. The method according to claim 1, characterized in that, The step of determining the first driving behavior of other vehicles within a preset area surrounding the vehicle based on the driving environment data includes: Based on the driving environment data, determine the road condition description information corresponding to the preset area; The first driving behavior is determined based on the road condition description information.

3. The method according to claim 2, characterized in that, The driving environment data includes environmental images of the vehicle's surroundings in at least one direction. The step of determining the road condition description information corresponding to the preset area based on the driving environment data includes: Obtain the preset task description instruction corresponding to the driving environment data; After inputting the preset task description instruction and the environmental image of at least one direction around the vehicle into the visual language model, the road condition description information output by the model is obtained. The preset task description instruction is a natural language description of the recognition task currently being performed by the visual language model.

4. The method according to claim 3, characterized in that, The visual language model includes a visual encoder, a text encoder, and a large language model; The step of inputting the preset task description instruction and the environmental image of at least one direction around the vehicle into the visual language model to obtain the road condition description information output by the model includes: The environmental image in at least one direction is input into the visual encoder to obtain the first vector code of the image; The preset task description instruction is input into the text encoder to obtain the second vector code of the text; After inputting the first vector encoding and the second vector encoding into the large language model, the traffic condition description information is output.

5. The method according to claim 3, characterized in that, The driving environment data also includes the image recognition result after performing image recognition on the environmental image in the at least one direction; The step of determining the road condition description information corresponding to the preset area based on the driving environment data includes: After inputting the preset task description instruction, the environmental image of at least one direction, and the image recognition result into the visual language model, the road condition description information output by the model is obtained.

6. The method according to any one of claims 1-5, characterized in that, The first driving behavior includes the current driving behavior and / or predicted future driving behavior of the other vehicles.

7. The method according to claim 2, characterized in that, The step of controlling the vehicle to output target prompt information based on the first driving behavior includes: If it is determined that a first preset driving behavior exists in the first driving behavior, the cause of the first preset driving behavior is identified according to the road condition description information, wherein the first preset driving behavior includes the driving behavior of other vehicles that affect the driving safety of the vehicle. The target prompt information is output based on the cause of the first preset driving behavior.

8. The method according to claim 7, characterized in that, Identifying the cause of the first preset driving behavior based on the road condition description information includes: The road surface data of the roads within the preset area are determined based on the road condition description information; The cause of the first preset driving behavior is identified based on the road surface data.

9. The method according to claim 8, characterized in that, The method further includes: If the road surface data determines that the road within the preset area meets the preset risk warning conditions, the vehicle is controlled to issue a risk warning.

10. The method according to claim 3, characterized in that, The method further includes: Obtain the vehicle's status data; The step of determining the first driving behavior of other vehicles within a preset area surrounding the vehicle based on the driving environment data includes: Based on the driving environment data and the vehicle status data, the first driving behavior and the second driving behavior of the vehicle are determined by the visual language model, wherein the vehicle status data includes the vehicle's driving status data and / or the user status data of the user inside the vehicle.

11. The method according to claim 10, characterized in that, The step of controlling the vehicle to output target prompt information based on the first driving behavior includes: Based on the first driving behavior and the second driving behavior, the vehicle is controlled to output the target prompt information.

12. A driving reminder device, characterized in that, The device includes: The acquisition module is configured to acquire vehicle driving environment data; The determination module is configured to determine the first driving behavior of other vehicles within a preset area surrounding the vehicle based on the driving environment data. The control module is configured to control the vehicle to output target prompt information based on the first driving behavior, the target prompt information being used to remind the user of the vehicle to drive safely.

13. A vehicle, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to perform the steps of the method according to any one of claims 1-11.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program performs the steps of the method described in any one of claims 1-11.

15. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the steps of the method according to any one of claims 1-11.

Images