Method and device for generating an outside environment picture and vehicle

Abstract

The application discloses a method and device for generating an out-of-vehicle environment picture and a vehicle, and relates to the technical field of vehicles. The method comprises the following steps: determining a three-dimensional semantic map corresponding to an out-of-vehicle environment of a towing vehicle based on point cloud data and image data corresponding to the out-of-vehicle environment; determining a virtual model corresponding to a towing device included in the towing vehicle based on current posture data and towing parameters of the towing device; displaying a virtual field of view picture corresponding to the towing vehicle based on the three-dimensional semantic map and the virtual model; and the virtual field of view picture comprises out-of-vehicle environment information of the towing vehicle. In this way, the field of view coverage and accuracy of the generated out-of-vehicle environment picture can be improved.

Description

Technical Field

[0001] This application relates to the field of vehicle technology, and in particular to a method, apparatus and vehicle for generating images of the external environment of a vehicle. Background Technology

[0002] With the development of the new energy intelligent connected vehicle industry, trailer transportation has become increasingly widely used in logistics, public transportation, and other fields due to its advantages such as improved freight efficiency and optimized resource integration. During trailer operation, the completeness of the driver's perception of the surrounding environment directly determines driving safety; therefore, the industry has developed various vision assistance technologies. However, these technologies are prone to blind spots, resulting in low coverage and accuracy of the generated external environment image.

[0003] Therefore, improving the field of view coverage and accuracy of the generated images of the vehicle's external environment has become an urgent problem to be solved. Summary of the Invention

[0004] This application provides a method, apparatus, and vehicle for generating images of the external environment of a vehicle, which can improve the field of view coverage and accuracy of the generated images of the external environment of the vehicle.

[0005] In a first aspect, embodiments of this application provide a method for generating an external environment image, the method comprising:

[0006] Based on the point cloud data and image data corresponding to the external environment of the trailer vehicle, a three-dimensional semantic map corresponding to the external environment is determined.

[0007] Based on the current attitude data and towing parameters of the trailer included in the trailer vehicle, determine the virtual model corresponding to the trailer.

[0008] Based on a 3D semantic map and a virtual model, a virtual view of the trailer vehicle is generated and displayed; the virtual view includes information about the trailer vehicle's external environment.

[0009] In this embodiment, the 3D semantic map corresponding to the external environment is determined based on the multi-dimensional environmental information corresponding to the external environment of the trailer vehicle. This multi-dimensional environmental information includes a wide range of information and field of view information surrounding the trailer vehicle. Therefore, the 3D semantic map includes a large amount of information and a wide range of field of view environmental information around the trailer vehicle. Since the virtual model corresponding to the towing device is determined based on the current posture data and towing parameters of the towing device, the virtual model can accurately synchronize the posture changes of the towing device. Thus, based on the 3D semantic map and the virtual model, an accurate virtual field of view with depth perception and high field of view coverage can be determined, thereby improving the field of view coverage and accuracy of the generated external environment image. Consequently, after displaying the virtual field of view image in the trailer vehicle's main vehicle, the driver's field of view coverage and the accuracy of the external environment can be improved, which in turn helps to improve the driving safety of the trailer vehicle.

[0010] In one optional implementation of the first aspect, a virtual view of the trailer vehicle is generated and displayed based on a three-dimensional semantic map and a virtual model, including: overlaying the virtual model onto the three-dimensional semantic map to obtain an initial virtual view; optimizing the initial virtual view based on the driving state data of the trailer vehicle to obtain and display the virtual view of the trailer vehicle.

[0011] By adopting this implementation method, since the driving status data corresponding to the trailer vehicle can reflect the driver's current driving situation, by using the driving status data to optimize the initial virtual view screen corresponding to the trailer vehicle, the optimized virtual view screen can be more adapted to the driver's driving state. This can help to display a virtual view screen that is adapted to the driver's driving state, thereby improving the driving safety of the trailer vehicle.

[0012] In one optional implementation of the first aspect, optimizing the initial virtual view based on the driving status data corresponding to the trailer vehicle to obtain the virtual view corresponding to the trailer vehicle includes: determining the driving risk assessment result of the trailer vehicle within a preset time period; and optimizing the initial virtual view based on the driving status data corresponding to the trailer vehicle to obtain the virtual view corresponding to the trailer vehicle when the driving risk assessment result indicates that there is a driving risk for the trailer vehicle within the preset time period.

[0013] Using this implementation method, if the driving risk assessment result of the towed vehicle within a preset time period indicates that there is a driving risk within the preset time period, the initial virtual view screen can be optimized in advance based on the driving status data of the towed vehicle. In this way, a virtual view screen that is adapted to the driving status can be shown to the driver in advance, thereby minimizing the possibility of the driver misjudging the risk due to the mismatch between the virtual view screen and the driving status, thus improving driving safety.

[0014] In one optional implementation of the first aspect, determining the driving risk assessment result of the towed vehicle within a preset time period includes: using a trajectory prediction model, based on a three-dimensional semantic map, a virtual model, and driving state data corresponding to the towed vehicle, determining a first predicted driving trajectory of the virtual model within a preset time period, and a second predicted driving trajectory of obstacles; obstacles include obstacles within a preset distance range of the towed vehicle; and determining the driving risk assessment result of the towed vehicle within a preset time period based on the first predicted driving trajectory and the second predicted driving trajectory.

[0015] Using this implementation method, the driving risk assessment results of the towed vehicle within a preset time period can be quickly determined.

[0016] In one optional implementation of the first aspect, a virtual view of the trailer vehicle is generated and displayed based on a three-dimensional semantic map and a virtual model, including: determining the virtual view of the trailer vehicle based on the three-dimensional semantic map and the virtual model; determining the driving scenario of the trailer vehicle; determining the target display view of the virtual view of the trailer vehicle from multiple preset display views based on the driving status data and driving scenario of the trailer vehicle; and displaying the virtual view of the trailer vehicle based on the target display view.

[0017] By adopting this implementation method, the target display angle corresponding to the virtual field of view is determined from multiple preset display angles based on driving status data and driving scenario, and the virtual field of view is displayed based on the target display angle. In this way, the virtual field of view can be actively adapted to the driver's line of sight, rather than being passively viewed. The driver does not need to frequently switch the display angle of the virtual field of view, thereby improving the driving safety of the towed vehicle.

[0018] In one optional implementation of the first aspect, a virtual view of the trailer vehicle is generated and displayed based on a three-dimensional semantic map and a virtual model, including: obtaining scene adaptation parameters corresponding to the trailer vehicle, wherein the scene adaptation parameters include at least one of geographical area adaptation parameters, weather adaptation parameters, and trailer device adaptation parameters; and generating and displaying the virtual view of the trailer vehicle based on the scene adaptation parameters, the three-dimensional semantic map, and the virtual model.

[0019] By adopting this implementation method, a virtual field of view corresponding to the trailer vehicle is generated based on scene adaptation parameters, 3D semantic map and virtual model. This not only determines an accurate virtual field of view with depth and high field of view coverage, but also makes the determined virtual field of view more suitable for the current application scenario.

[0020] In one optional implementation of the first aspect, determining a three-dimensional semantic map corresponding to the external environment of the trailer vehicle based on point cloud data and image data corresponding to the external environment of the trailer vehicle includes: acquiring point cloud data and image data corresponding to the external environment of the trailer vehicle; performing time synchronization processing on the point cloud data and image data to obtain target point cloud data and target image data; and generating a three-dimensional semantic map corresponding to the external environment of the trailer vehicle based on the target point cloud data and target image data.

[0021] By adopting this implementation method, a three-dimensional semantic map containing a large amount of environmental information and a wide field of view around the trailer vehicle can be accurately determined based on point cloud data and image data corresponding to the external environment of the vehicle, thereby providing a data foundation for subsequent display of external environment information.

[0022] In one optional implementation of the first aspect, the towing parameters include factory parameters and load information; determining the virtual model corresponding to the towing device based on the current attitude data and towing parameters of the towing device included in the towing vehicle includes: performing noise reduction processing on the current attitude data, factory parameters and load information of the towing device included in the towing vehicle to obtain target current attitude data, target factory parameters and target load information; and generating the virtual model corresponding to the towing device based on the target current attitude data, target factory parameters and target load information.

[0023] Using this implementation method, a virtual model corresponding to the towing device can be generated simply and accurately, thereby providing a data foundation for the subsequent display of external environment information.

[0024] In an optional implementation of the first aspect, the method further includes: upon detecting a control operation for switching the display perspective of the virtual view screen, switching the display perspective of the virtual view screen to obtain a virtual view screen with the switched perspective; and displaying the virtual view screen with the switched perspective.

[0025] This implementation method enables accurate presentation of the virtual view and convenient interaction with the driver, thereby improving the driving safety of trailer vehicles.

[0026] Secondly, embodiments of this application provide an apparatus for generating images of the external environment of a vehicle, the apparatus comprising:

[0027] The determination module is used to determine the three-dimensional semantic map corresponding to the external environment of the trailer based on the point cloud data and image data corresponding to the external environment of the trailer.

[0028] The determination module is also used to determine the virtual model corresponding to the trailer based on the current attitude data and towing parameters of the trailer included in the trailer vehicle;

[0029] The generation and display module is used to generate and display the virtual view of the trailer based on the 3D semantic map and virtual model; the virtual view includes the external environment information of the trailer.

[0030] Thirdly, embodiments of this application provide a vehicle, including a memory and a vehicle controller. The memory stores a computer program, and the vehicle controller executes the computer program to implement the steps of the method provided in the first aspect above.

[0031] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a vehicle controller, implements the steps of the method provided in the first aspect above.

[0032] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a vehicle controller, implements the steps of the method provided in the first aspect above.

[0033] Regarding the beneficial effects of any of the technical solutions in the second to fifth aspects mentioned above, refer to the beneficial effects of the corresponding technical solutions in the first aspect; repeated examples will not be listed here. Attached Figure Description

[0034] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0035] Figure 1 This is an optional flowchart illustrating a method for generating an external vehicle environment image provided in an embodiment of this application;

[0036] Figure 2 This is a schematic diagram of an optional structure of a trajectory prediction model provided in an embodiment of this application;

[0037] Figure 3 This is another optional flowchart illustrating a method for generating an external environment image provided in an embodiment of this application;

[0038] Figure 4 This is a schematic diagram of an optional architecture for a system for displaying external environmental information of a towed vehicle, provided in an embodiment of this application.

[0039] Figure 5 This is an optional structural schematic diagram of a device for generating images of the vehicle's external environment provided in an embodiment of this application;

[0040] Figure 6 This is an optional structural schematic diagram of a vehicle provided in an embodiment of this application. Detailed Implementation

[0041] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0042] With the development of the new energy intelligent connected vehicle industry, trailer transportation has become increasingly widely used in logistics, public transportation, and other fields due to its advantages such as improved freight efficiency and optimized resource integration. During trailer operation, the driver's complete perception of the surrounding environment directly determines driving safety; therefore, the industry has developed various vision assistance technologies.

[0043] Currently, relevant towing vision assistance technologies are mainly divided into three categories: First, the combination of traditional optical rearview mirrors and blind spot mirrors, which expands the field of vision through mechanical structures; second, electronic rearview mirror systems, which use high-definition cameras to collect images of the vehicle's surroundings and display them on the driver's cockpit screen after processing; and third, dedicated towing assistance imaging systems, which adopt a "smart hardware + smartphone" model, allowing the driver to obtain 1080P high-definition rear images by scanning a QR code with their mobile phone. In addition, some Advanced Driving Assistance Systems (ADAS) integrate data from cameras, ultrasonic sensors, and other sources to achieve basic blind spot warnings and reversing trajectory assistance, further improving the convenience of towing. However, all of the above methods have fixed blind spots, especially in non-lane cruise scenarios such as towing vehicles turning, changing lanes, and reversing into parking spaces. The area connecting the towing device and the lead vehicle, the sides and rear of the towed vehicle, and the area under the rear of the vehicle cannot be effectively covered, resulting in lower driving safety.

[0044] Based on this, embodiments of this application provide a method, apparatus, and vehicle for generating an external environment image, wherein the method includes: determining a three-dimensional semantic map corresponding to the external environment based on point cloud data and image data corresponding to the external environment of the trailer vehicle; determining a virtual model corresponding to the trailer device based on the current posture data and towing parameters of the trailer device included in the trailer vehicle; generating and displaying a virtual view image corresponding to the trailer vehicle based on the three-dimensional semantic map and the virtual model; the virtual view image includes external environment information of the trailer vehicle. This method utilizes a multi-dimensional semantic map of the external environment, which is determined based on the multi-dimensional environmental information surrounding the trailer vehicle. This multi-dimensional environmental information includes a wide range of information and field of view around the trailer vehicle. Furthermore, the virtual model of the towing device is determined based on its current posture data and towing parameters. Therefore, the virtual model can accurately synchronize with the posture changes of the towing device. Based on the three-dimensional semantic map and the virtual model, an accurate virtual field of view with depth perception and high coverage can be determined, thus improving the coverage and accuracy of the generated external environment image. Consequently, when the virtual field of view is displayed in the trailer vehicle's driver compartment, the driver's field of view coverage and the accuracy of the external environment are improved, ultimately enhancing the driving safety of the trailer vehicle.

[0045] The method for generating images of the vehicle's external environment provided in the embodiments of this application will be described below.

[0046] Please see Figure 1 , Figure 1 This is an optional flowchart illustrating a method for generating an external environment image of a vehicle, provided in an embodiment of this application. This method can be executed by the vehicle's overall controller. Figure 1 As shown, the method for generating the vehicle's external environment image may include, but is not limited to, the following steps:

[0047] S101. Based on the point cloud data and image data corresponding to the external environment of the trailer vehicle, determine the three-dimensional semantic map corresponding to the external environment.

[0048] The trailer vehicle may include a tractor unit and a trailer.

[0049] The point cloud data can be collected using millimeter-wave radar and / or lidar.

[0050] Among them, a 3D semantic map refers to an environmental model that adds semantic information to a 3D spatial map. A 3D semantic map transforms a traditional map containing location and shape into a structured environmental model that includes object categories and attributes, and is understandable by machines.

[0051] In an optional implementation, prior to step S101, the vehicle controller may further acquire point cloud data and image data corresponding to the external environment of the trailer vehicle, collected by a data acquisition device deployed in the trailer vehicle. The data acquisition device may include, but is not limited to, cameras, millimeter-wave radar, and / or lidar.

[0052] S102. Based on the current attitude data and towing parameters of the trailer included in the trailer vehicle, determine the virtual model corresponding to the trailer.

[0053] The current attitude data of the trailer hitch may include, but is not limited to, the yaw angle and pitch angle of the trailer hitch. In some embodiments, the current attitude data may be collected by attitude sensors deployed at the connection point between the trailer and the trailer hitch. Optionally, the attitude sensors may include, but are not limited to, gyroscopes, angular displacement sensors, etc.

[0054] The towing parameters may include the factory parameters of the towing device and load information.

[0055] In one optional implementation, the vehicle controller determines the virtual model corresponding to the towing device based on the current attitude data and towing parameters of the towing device included in the trailer vehicle. This can be achieved by dynamically generating a 1:1 scale virtual model corresponding to the towing device using a parametric modeling algorithm based on the current attitude data and towing parameters of the towing device included in the trailer vehicle.

[0056] In one alternative implementation, the vehicle controller can also update the virtual model in real time based on the real-time attitude data of the trailer.

[0057] It should be noted that this application does not limit the execution order of steps S101 and S102. For example, the vehicle controller may execute step S101 first and then step S102; or it may execute step S102 first and then step S101; or it may execute steps S101 and S102 simultaneously, etc.

[0058] S103. Based on the three-dimensional semantic map and virtual model, generate and display the virtual view of the trailer vehicle; the virtual view includes the external environment information of the trailer vehicle.

[0059] In one optional implementation, the vehicle controller displays a virtual view of the trailer vehicle based on a 3D semantic map and a virtual model. This can be achieved by: generating a virtual view of the trailer vehicle based on the 3D semantic map and the virtual model; and then displaying the virtual view.

[0060] In some embodiments, the vehicle controller displays the virtual view screen, which may be done by displaying the virtual view screen in the main vehicle of the trailer. Optionally, displaying the virtual view screen in the main vehicle of the trailer may be done in at least one of the instrument display screen, central control screen, or passenger-side screen in the cabin of the main vehicle of the trailer; it may also be displayed on the head-up display (HUD) of the main vehicle of the trailer, etc., without limitation.

[0061] In this embodiment, the 3D semantic map corresponding to the external environment is determined based on the multi-dimensional environmental information corresponding to the external environment of the trailer vehicle. This multi-dimensional environmental information includes a wide range of information and field of view information surrounding the trailer vehicle. Therefore, the 3D semantic map includes a large amount of information and a wide range of field of view environmental information around the trailer vehicle. Since the virtual model corresponding to the towing device is determined based on the current posture data and towing parameters of the towing device, the virtual model can accurately synchronize the posture changes of the towing device. Thus, based on the 3D semantic map and the virtual model, an accurate virtual field of view with depth perception and high field of view coverage can be determined, thereby improving the field of view coverage and accuracy of the generated external environment image. Consequently, after displaying the virtual field of view image in the trailer vehicle's main vehicle, the driver's field of view coverage and the accuracy of the external environment can be improved, which in turn helps to improve the driving safety of the trailer vehicle.

[0062] In one alternative implementation, Figure 1 In step S103 of the method for generating the external environment image of the vehicle shown, the way the vehicle controller generates and displays the virtual field of view image corresponding to the trailer vehicle based on the three-dimensional semantic map and the virtual model can be: superimposing the virtual model onto the three-dimensional semantic map to obtain the initial virtual field of view image; optimizing the initial virtual field of view image based on the driving state data corresponding to the trailer vehicle to obtain the virtual field of view image corresponding to the trailer vehicle and displaying the virtual field of view image.

[0063] In some embodiments, the vehicle controller overlays the virtual model onto a 3D semantic map to obtain an initial virtual view. This can be achieved by using a ray tracing rendering algorithm to overlay the virtual model onto the 3D semantic map, thus ensuring that the lighting and perspective relationships between the virtual model and the real environment are consistent, thereby providing a realistic sense of depth.

[0064] The driving status data may include, but is not limited to, the driver's line of sight, and driving operation data reflecting driving intentions. Optionally, the driving operation data may include, but is not limited to, accelerator pedal opening, brake pedal opening, steering wheel angle, and steering speed.

[0065] For example, assuming the driving status data includes the driver's line of sight being focused on the right-side rearview mirror, the vehicle controller can perform vision clarity enhancement processing on the initial virtual view of the right-side towing area to obtain an optimized virtual view.

[0066] In some embodiments, the vehicle controller optimizes the initial virtual view based on the driving state data corresponding to the towed vehicle to obtain a virtual view corresponding to the towed vehicle. This can be achieved by: determining the driving risk assessment result of the towed vehicle within a preset time period; and, if the driving risk assessment result indicates that there is a driving risk for the towed vehicle within the preset time period, optimizing the initial virtual view based on the driving state data corresponding to the towed vehicle to obtain a virtual view corresponding to the towed vehicle. In this way, when a driving risk is predicted, a virtual view adapted to the driving state can be displayed to the driver in advance, thereby minimizing the possibility of misjudging the risk due to a mismatch between the virtual view and the driving state, thus improving driving safety.

[0067] The preset duration refers to the preset time after the current time. Optionally, the preset duration can be determined based on expert experience, multiple trials, or is custom-defined, etc., without limitation here. For example, the preset duration is 4-8 seconds.

[0068] In this embodiment, the vehicle controller determines the driving risk assessment result of the towed vehicle within a preset time period by: using a trajectory prediction model, based on a 3D semantic map, a virtual model, and the driving state data corresponding to the towed vehicle, determining a first predicted driving trajectory of the virtual model within the preset time period, and a second predicted driving trajectory of obstacles; obstacles include obstacles within a preset distance range of the towed vehicle; based on the first and second predicted driving trajectories, the driving risk assessment result of the towed vehicle within the preset time period is determined. This allows for the rapid determination of the driving risk assessment result of the towed vehicle within the preset time period.

[0069] Optionally, the trajectory prediction model can be a lightweight Visual Language Action (VLA) model. The core of a VLA model is translating visual and linguistic understanding into concrete physical actions. It receives visual perception results and linguistic instructions, and then outputs a sequence of actions to be executed.

[0070] Optionally, the trajectory prediction model can be trained in the following ways:

[0071] Step 1: The server acquires multiple traffic scene videos, sensor data, and driving status data, and trains a large trajectory prediction model based on these data to obtain a pre-trained trajectory prediction model. This pre-trained trajectory prediction model is equipped with causal reasoning and environmental understanding capabilities.

[0072] Among them, the pre-trained trajectory prediction model can also be called the teacher model, which has a large number of model parameters (such as 32B).

[0073] Optionally, the server can be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides cloud computing services, etc. There are no restrictions here.

[0074] Step 2: The server inputs the sample 3D semantic map, the sample virtual model corresponding to the sample towing device, and the sample driving state data corresponding to the sample towing device into the pre-trained trajectory prediction model to obtain the first predicted driving trajectory of the first sample and the second predicted driving trajectory of the first sample.

[0075] Step 3: The server sends the sample 3D semantic map, the sample virtual model corresponding to the sample towing device, the sample driving state data corresponding to the sample towing device, and the first predicted driving trajectory and the second predicted driving trajectory of the first sample to the vehicle controller. Correspondingly, the vehicle controller receives the sample 3D semantic map, the sample virtual model corresponding to the sample towing device, the sample driving state data corresponding to the sample towing device, and the first predicted driving trajectory and the second predicted driving trajectory of the first sample.

[0076] Step 4: The vehicle controller inputs the sample 3D semantic map, the sample virtual model corresponding to the sample towing device, and the sample driving state data corresponding to the sample towing device into the initial trajectory prediction model to obtain the first predicted driving trajectory of the second sample and the second predicted driving trajectory of the second sample.

[0077] Step 5: Train the initial trajectory prediction model by minimizing the directions of the first predicted driving trajectory of the second sample and the actual first predicted driving trajectory, the directions of the second predicted driving trajectory of the second sample and the actual second predicted driving trajectory, the directions of minimizing the differences between the first predicted driving trajectory of the first sample and the first predicted driving trajectory of the second sample, and the directions of minimizing the differences between the second predicted driving trajectory of the first sample and the second predicted driving trajectory of the second sample, to obtain the trajectory prediction model.

[0078] The actual first predicted driving trajectory is the actual driving trajectory of the sample towing device within a preset time period; the actual second predicted driving trajectory is the actual driving trajectory of obstacles within a preset distance range of the towing vehicle within a preset time period.

[0079] In other words, during the training of the trajectory prediction model, the vehicle controller not only needs the trajectory prediction model to be accurate, but also needs the trajectory prediction model's output to be as close as possible to the output of the pre-trained trajectory prediction model.

[0080] The trajectory prediction model can also be called a student model. The trajectory prediction model has fewer parameters than the pre-trained trajectory prediction model. For example, the trajectory prediction model has 3.2B parameters.

[0081] In some embodiments, the structure of the trajectory prediction model can be found in [reference needed]. Figure 2 As shown, Figure 2 This is a schematic diagram of an optional structure of a trajectory prediction model provided in an embodiment of this application. For example... Figure 2 As shown, the trajectory prediction model includes an input feature encoding layer, a temporal context modeling layer, a trajectory prediction decoding layer, and an output constraint and post-processing layer. The input data for the input feature encoding layer includes a 3D semantic map, a virtual model, vehicle-specific driving state data, and historical trajectories.

[0082] The input feature encoding layer is used to extract features from the 3D semantic map, virtual model, vehicle-related driving status data, and historical trajectory, and to perform feature fusion processing on the extracted features (3D semantic map features, virtual model features, driving status data features, and historical trajectory features) to obtain multi-source fusion encoded features.

[0083] The temporal context modeling layer includes a lightweight Transformer encoder and a scene attention module. The input data of the temporal context modeling layer is multi-source fusion encoded features. After the multi-source fusion encoded features are processed by the lightweight Transformer encoder and the scene attention module, temporal context features can be obtained.

[0084] The trajectory prediction decoding layer includes a multi-head trajectory decoder. The input data of the trajectory prediction decoding layer is temporal context features, and the output data includes the predicted driving trajectory of the virtual model output by the multi-head trajectory decoder, the predicted driving trajectory of the obstacle, as well as the trajectory confidence and risk probability.

[0085] The output constraint and post-processing layer includes a trajectory smoothing filter module, a dynamics constraint module, and a collision risk assessment module. The input data for this layer consists of the predicted driving trajectory of the virtual model, the predicted driving trajectory of the obstacles, trajectory confidence, and risk probability. Specifically, the trajectory smoothing module smooths and filters the predicted driving trajectories of the virtual model and the obstacles, resulting in processed predicted trajectories for both. The dynamics constraint module constrains these trajectories, resulting in constrained predicted trajectories for both. The collision risk assessment module, based on the constrained predicted trajectories of the virtual model and the obstacles, determines the driving risk assessment result for the trailer vehicle corresponding to the virtual model within a preset time period.

[0086] In some embodiments, the vehicle controller may also upload vehicle driving data and operating status data to the server, enabling the server to retrain the pre-trained trajectory prediction model. Driving data may include, but is not limited to, multi-dimensional environmental information, vehicle attitude data, and environmental semantic information during vehicle operation; operating status data may include, but is not limited to, the computational load of the trajectory prediction model, the health status of each data acquisition device in the vehicle, and the feedback of the trajectory prediction model's inference results.

[0087] In some implementations, the vehicle controller can also download an upgrade package of a pre-trained trajectory prediction model from the server to further optimize the local trajectory prediction model, thereby improving the accuracy of trajectory prediction.

[0088] Optionally, the vehicle controller calls the trajectory prediction model to determine the first and second predicted driving trajectories of the virtual model within a preset time period based on the 3D semantic map, the virtual model, and the driving state data corresponding to the trailer vehicle. This can be achieved by: performing spatial alignment processing on the 3D semantic map and the virtual model to obtain the processed 3D semantic map and the processed virtual model; and inputting the processed 3D semantic map and the processed virtual model into the trajectory prediction model to obtain the first and second predicted driving trajectories of the virtual model within a preset time period.

[0089] Optionally, the vehicle controller determines the driving risk assessment result of the towed vehicle within a preset time period based on the first predicted driving trajectory and the second predicted driving trajectory. This can be done as follows: if it is determined that the first predicted driving trajectory and the second predicted driving trajectory overlap, the driving risk assessment result of the towed vehicle within the preset time period is determined to be that there is a driving risk; if it is determined that the first predicted driving trajectory and the second predicted driving trajectory do not overlap, the driving risk assessment result of the towed vehicle within the preset time period is determined to be that there is no driving risk.

[0090] The vehicle controller can determine whether the first predicted driving trajectory and the second predicted driving trajectory overlap by performing an intersection operation on the first predicted driving trajectory and the second predicted driving trajectory in three-dimensional space.

[0091] Optionally, the vehicle controller can also determine the probability of collision between the towed vehicle and the obstacle, the time urgency, and the severity of the consequences if the driving risk assessment result of the towed vehicle within a preset time period is determined to be a driving risk; and determine the risk level corresponding to the driving risk based on the collision probability, time urgency, and severity of the consequences.

[0092] For example, suppose the vehicle controller determines that the driving risk assessment result of the towed vehicle in the next 5 seconds is that there is a driving risk on the left rear. In this case, the vehicle controller can automatically adjust the presentation angle of the virtual view of the left rear based on the driving status data, and enhance the clarity of the virtual view of the left rear.

[0093] By adopting this implementation method, since the driving status data corresponding to the trailer vehicle can reflect the driver's current driving situation, by using the driving status data to optimize the initial virtual view screen corresponding to the trailer vehicle, the optimized virtual view screen can be more adapted to the driver's driving state. This can help to display a virtual view screen that is adapted to the driver's driving state, thereby improving the driving safety of the trailer vehicle.

[0094] In one alternative implementation, Figure 1 In step S103 of the method for generating the external environment image of the vehicle shown, the vehicle controller generates and displays the virtual field of view image corresponding to the trailer vehicle based on the three-dimensional semantic map and the virtual model, including: determining the virtual field of view image corresponding to the trailer vehicle based on the three-dimensional semantic map and the virtual model; determining the driving scenario corresponding to the trailer vehicle; determining the target display perspective corresponding to the virtual field of view image from multiple preset display perspectives based on the driving status data and driving scenario corresponding to the trailer vehicle; and displaying the virtual field of view image based on the target display perspective.

[0095] In some embodiments, the vehicle controller determines the virtual view of the towed vehicle based on a 3D semantic map and a virtual model. This can be achieved by using a ray tracing rendering algorithm to overlay the virtual model onto the 3D semantic map, thus obtaining the virtual view of the towed vehicle. The virtual model in the virtual view is rendered with semi-transparent highlighting to ensure a clear distinction between the virtual model and the real environment.

[0096] In some embodiments, the driving scenarios corresponding to the trailer vehicle may include, but are not limited to, high-speed driving scenarios, reversing driving scenarios, and cornering driving scenarios.

[0097] In some embodiments, the multiple preset display perspectives may include, but are not limited to, at least two of the following: a panoramic bird's-eye view, a driver's side towing perspective, a passenger's side towing perspective, a rear towing perspective, and a close-up perspective of the connection between the towing vehicle and the towing device.

[0098] For example, assuming the driving status data includes the driver's line of sight being on the driver's side and the driving scenario is high-speed driving, the vehicle controller can determine that the target display view corresponding to the virtual field of view is the driver's side towing view. Assuming the driving status data includes the driver decelerating and the driving scenario is reversing, the vehicle controller can determine that the target display view corresponding to the virtual field of view is a panoramic bird's-eye view.

[0099] In some embodiments, the vehicle controller can also adaptively generate the target display perspective corresponding to the virtual field of view based on the driving status data and driving scenario of the trailer vehicle.

[0100] In some embodiments, the vehicle controller can also adaptively switch the target display perspective corresponding to the virtual view screen. The perspective switching delay can be controlled within a preset time range, such as 5ms-6ms. This minimizes the possibility of display stuttering or distortion in the virtual view screen.

[0101] By adopting this implementation method, the target display angle corresponding to the virtual field of view is determined from multiple preset display angles based on driving status data and driving scenario, and the virtual field of view is displayed based on the target display angle. In this way, the virtual field of view can be actively adapted to the driver's line of sight, rather than being passively viewed. The driver does not need to frequently switch the display angle of the virtual field of view, thereby improving the driving safety of the towed vehicle.

[0102] In one alternative implementation, Figure 1In step S103 of the method for generating the external environment image of the vehicle shown, the vehicle controller generates and displays the virtual field of view image corresponding to the trailer vehicle based on the three-dimensional semantic map and the virtual model. This can be achieved by: obtaining the scene adaptation parameters corresponding to the trailer vehicle, including at least one of the geographical area adaptation parameters, weather adaptation parameters, and trailer device adaptation parameters; and generating and displaying the virtual field of view image corresponding to the trailer vehicle based on the scene adaptation parameters, the three-dimensional semantic map, and the virtual model.

[0103] Among them, the geographical area adaptation parameters may include, but are not limited to, the adaptation parameters corresponding to narrow urban roads, rural roads without markings, or the adaptation parameters corresponding to special scenarios such as road construction; the weather adaptation parameters may include, but are not limited to, the adaptation parameters corresponding to specific environments such as heavy rain, heavy fog, and heavy snow; the trailer adaptation parameters may include the adaptation parameters corresponding to trailers of different models, sizes, and load information.

[0104] In some embodiments, the scene adaptation parameters may include geographic area adaptation parameters, weather adaptation parameters, and towing device adaptation parameters. The vehicle controller obtains the scene adaptation parameters corresponding to the towing vehicle by: obtaining the geographic area adaptation parameters corresponding to the towing vehicle based on a first correspondence, wherein the first correspondence includes correspondences between multiple geographic areas and multiple parameters; obtaining the weather adaptation parameters corresponding to the towing vehicle based on a second correspondence, wherein the second correspondence includes correspondences between multiple weather conditions and multiple parameters; and obtaining the towing device adaptation parameters corresponding to the towing vehicle based on a third correspondence, wherein the third correspondence includes correspondences between multiple towing devices and multiple parameters.

[0105] Optionally, the first correspondence can be a table stored locally on the vehicle controller (denoted as the first correspondence table), or it can be a table stored in a database that is accessible to the vehicle controller (denoted as the first correspondence table). The first correspondence table includes correspondences between various geographical regions and multiple parameters.

[0106] Optionally, the second correspondence can be a table stored locally on the vehicle controller (denoted as the second correspondence table), or it can be a table stored in a database that is accessible to the vehicle controller (denoted as the second correspondence table). The second correspondence table includes correspondences between various weather conditions and multiple parameters.

[0107] Optionally, the third correspondence can be a table stored locally on the vehicle controller (denoted as the third correspondence table), or it can be a table stored in a database that is accessible to the vehicle controller (denoted as the third correspondence table). The third correspondence table includes the correspondence between various towing devices and multiple parameters.

[0108] In some embodiments, the vehicle controller generates a virtual view of the trailer vehicle based on scene adaptation parameters, a 3D semantic map, and a virtual model. This can be achieved by: overlaying the virtual model onto the 3D semantic map to obtain an initial virtual view of the trailer vehicle; updating the initial virtual view of the trailer vehicle based on the scene adaptation parameters to obtain an updated virtual view of the trailer vehicle; and using the updated virtual view of the trailer vehicle as the virtual view of the trailer vehicle.

[0109] In other embodiments, the vehicle controller generates a virtual view of the towed vehicle based on scene adaptation parameters, a 3D semantic map, and a virtual model. This can be achieved by inputting the scene adaptation parameters, the 3D semantic map, and the virtual model into a virtual view generator to obtain the virtual view of the towed vehicle. In this embodiment, the scene adaptation parameters are used to constrain the virtual view before it is generated.

[0110] By adopting this implementation method, a virtual field of view corresponding to the trailer vehicle is generated based on scene adaptation parameters, 3D semantic map and virtual model. This not only determines an accurate virtual field of view with depth and high field of view coverage, but also makes the determined virtual field of view more suitable for the current application scenario.

[0111] In one alternative implementation, Figure 1 In step S101 of the method for generating the vehicle exterior environment image shown, the vehicle controller determines the three-dimensional semantic map corresponding to the vehicle exterior environment based on the point cloud data and image data corresponding to the vehicle exterior environment of the trailer vehicle. This can be achieved by: acquiring the point cloud data and image data corresponding to the vehicle exterior environment of the trailer vehicle; performing time synchronization processing on the point cloud data and image data to obtain target point cloud data and target image data; and generating the three-dimensional semantic map corresponding to the vehicle exterior environment based on the target point cloud data and target image data.

[0112] In some embodiments, the vehicle controller can also acquire multi-dimensional environmental information corresponding to the external environment of the trailer vehicle, collected by data acquisition devices deployed in the trailer. These data acquisition devices may include, but are not limited to, cameras, and at least two of millimeter-wave radar and / or lidar. This fusion of multiple data acquisition devices can compensate for the imaging deficiencies of a single camera in adverse weather conditions, improving the accuracy of external environment detection.

[0113] The trailer can be equipped with multiple cameras, including front-view, surround-view, side-view, and rear-view cameras. Each camera also integrates adaptive cleaning and heating functions, which improves image clarity in extreme environments (such as heavy rain, fog, and blizzards), thereby enhancing adaptability to extreme conditions and improving the accuracy of image data collected under such conditions.

[0114] Millimeter-wave radar can be used to accurately obtain the distance and relative speed between the trailer and obstacles around the trailer, and it is not affected by weather or climate.

[0115] Among them, lidar can be used to accurately obtain the three-dimensional spatial coordinates and distance information of obstacles around the trailer vehicle.

[0116] In some implementations, the vehicle controller performs time synchronization processing on point cloud data and image data to obtain target point cloud data and target image data. This can be achieved by: performing time synchronization processing on point cloud data and image data to obtain time-synchronized point cloud data and time-synchronized image data; and then performing denoising and format conversion processing on the time-synchronized point cloud data and time-synchronized image data to obtain target point cloud data and target image data.

[0117] In some embodiments, the vehicle controller generates a 3D semantic map corresponding to the external environment of the vehicle based on target point cloud data and target image data. This can be achieved by: inputting the target point cloud data and target image into a feature extractor to obtain point cloud features corresponding to the target point cloud data and image features corresponding to the target image data; constructing a geometric environment based on the point cloud features using a point cloud matching algorithm; and estimating the relative motion poses of each object within a preset distance range of the towed vehicle based on the image features using feature matching; and performing semantic filling on the geometric environment based on the relative motion poses of each object to obtain a 3D semantic map corresponding to the external environment of the vehicle.

[0118] Optionally, the vehicle controller can also incrementally update the 3D semantic map based on the multi-dimensional environmental information corresponding to the real-time external environment collected during the towing vehicle's movement. In scenarios where there are changes, such as road construction or temporary roadblocks, the vehicle controller can compare time-series data to remove invalid map elements in real time and update the latest road topology to ensure the consistency between the 3D semantic map and the real environment.

[0119] By adopting this implementation method, a three-dimensional semantic map containing a large amount of environmental information and a wide field of view around the trailer vehicle can be accurately determined based on point cloud data and image data corresponding to the external environment of the vehicle, thereby providing a data foundation for subsequent display of external environment information.

[0120] In one alternative implementation, Figure 1 In step S102 of the method for generating the vehicle exterior environment image shown, the towing parameters include factory parameters and load information. The vehicle controller determines the virtual model corresponding to the towing device based on the current attitude data and towing parameters of the towing device included in the towing vehicle. This can be done by: performing noise reduction processing on the current attitude data, factory parameters, and load information of the towing device included in the towing vehicle to obtain the target current attitude data, target factory parameters, and target load information; and generating the virtual model corresponding to the towing device based on the target current attitude data, target factory parameters, and target load information.

[0121] The current attitude data may include the yaw angle and pitch angle of the trailer; the factory parameters may include the model and size of the trailer.

[0122] In some embodiments, factory parameters can be read via a Controller Area Network (CAN) bus, and may include, but are not limited to, the model and size of the trailer hitch. Load information can be collected via weight sensors deployed at the connection point between the trailer and the trailer hitch.

[0123] In some embodiments, the vehicle controller generates a virtual model corresponding to the trailer device based on the target's current attitude data, target factory parameters, and target load information. This can be achieved by dynamically generating a 1:1 scale virtual model corresponding to the trailer device using a parametric modeling algorithm based on the current attitude data, factory parameters, and load information of the trailer device included in the trailer vehicle.

[0124] Using this implementation method, a virtual model of the trailer can be generated simply and accurately based on the attitude data, factory parameters, and load information of the trailer, thus providing a data foundation for the subsequent display of external environment information.

[0125] In one alternative implementation, Figure 1 In the method for generating the external environment image shown, the vehicle controller can also switch the display perspective of the virtual field of view when it detects a control operation for switching the display perspective of the virtual field of view, and obtain the virtual field of view after switching the perspective; and display the virtual field of view after switching the perspective.

[0126] In some embodiments, the control operation may be a gesture operation, a voice operation, or a button operation, etc., and there is no limitation here.

[0127] For example, gesture operation could be switching the display view by waving your hand within a preset distance range of the displayed virtual view. Voice operation could be the driver saying "Show right-side towed view." Button operation could be the driver pressing a control on the steering wheel to switch the virtual view.

[0128] In some embodiments, the vehicle controller may also output alarm information and / or enhance the warning signs in the virtual view screen when it detects that the driver's hands have left the steering wheel or the driver's line of sight has deviated from the direction of travel; and control the vehicle to decelerate if no response to the alarm information or warning signs is detected within a preset time.

[0129] Optionally, the alarm information can be voice information (such as a voice prompt asking you to drive safely) or light information (such as flashing lights in the vehicle), etc., without limitation.

[0130] This implementation method enables accurate presentation of the virtual view and convenient interaction with the driver, thereby improving the driving safety of trailer vehicles.

[0131] The following is combined Figure 3 This paper provides an overall description of the method for generating images of the vehicle's external environment provided in the embodiments of this application. Please refer to [link to relevant documentation]. Figure 3 , Figure 3 This is another optional flowchart illustrating a method for generating an external vehicle environment image provided in an embodiment of this application. For example... Figure 3 As shown, the method for generating the vehicle's external environment image may include, but is not limited to, the following steps:

[0132] S301. Obtain the current attitude data, factory parameters, and load information of the trailer included in the trailer vehicle, and generate a virtual model corresponding to the trailer based on the current attitude data, factory parameters, and load information.

[0133] In some embodiments, the vehicle controller generates a virtual model corresponding to the towing device based on the front attitude data, factory parameters, and load information. This can be achieved by: denoising the current attitude data, factory parameters, and load information to obtain the target current attitude data, target factory parameters, and target load information; and generating a 1:1 scale virtual model based on the target current attitude data, target factory parameters, and target load information using a parametric modeling algorithm.

[0134] S302. Acquire omnidirectional point cloud data and omnidirectional image data corresponding to the external environment of the trailer vehicle; and generate a three-dimensional semantic map corresponding to the external environment based on the omnidirectional point cloud data and omnidirectional image data.

[0135] In some embodiments, the vehicle controller generates a 3D semantic map of the external environment based on omnidirectional point cloud data and omnidirectional image data. This can be achieved by employing a point cloud-image fusion algorithm to generate the 3D semantic map of the external environment based on omnidirectional point cloud data and omnidirectional image data. This allows for accurate identification of elements such as lane lines, curbs, obstacles (e.g., vehicles, pedestrians, cyclists), and traffic signs, thus enabling environmental reconstruction in scenarios without high-precision maps.

[0136] In some embodiments, the vehicle controller can also update the three-dimensional semantic map in real time.

[0137] S303. Overlay the virtual model onto the 3D semantic map to obtain the initial virtual view.

[0138] In some embodiments, the vehicle controller may employ a ray tracing rendering algorithm to overlay the virtual model onto a three-dimensional semantic map to obtain the initial virtual view of the towed vehicle.

[0139] S304. Based on the three-dimensional semantic map, the virtual model, and the driving status data corresponding to the trailer vehicle, determine the first predicted driving trajectory of the virtual model within a preset time period, and the second predicted driving trajectory of the obstacles; the obstacles include obstacles within a preset distance range of the trailer vehicle.

[0140] In some embodiments, the vehicle controller determines the first predicted driving trajectory of the virtual model and the second predicted driving trajectory of the obstacle within a preset time period based on the three-dimensional semantic map, the virtual model and the driving state data corresponding to the trailer vehicle. This can be achieved by inputting the three-dimensional semantic map, the virtual model and the driving state data corresponding to the trailer vehicle into the aforementioned trajectory prediction model to obtain the first predicted driving trajectory of the virtual model and the second predicted driving trajectory of the obstacle within a preset time period.

[0141] S305. Based on the first predicted driving trajectory and the second predicted driving trajectory, determine the driving risk assessment result of the trailer vehicle for a preset time period.

[0142] In some embodiments, the vehicle controller determines the driving risk assessment result of the towed vehicle within a preset time period based on the first predicted driving trajectory and the second predicted driving trajectory. This can be as follows: if it is determined that the first predicted driving trajectory and the second predicted driving trajectory overlap, the driving risk assessment result of the towed vehicle within the preset time period is determined to be that there is a driving risk; if it is determined that the first predicted driving trajectory and the second predicted driving trajectory do not overlap, the driving risk assessment result of the towed vehicle within the preset time period is determined to be that there is no driving risk.

[0143] S306. If the driving risk assessment result indicates that the towed vehicle has a driving risk within a preset time period, the initial virtual field of view is optimized based on the driving status data of the towed vehicle to obtain the target virtual field of view.

[0144] For example, suppose the vehicle controller determines that the driving risk assessment result of the towed vehicle in the next 5 seconds is that there is a driving risk on the left rear. In this case, the vehicle controller can automatically adjust the presentation angle of the virtual view of the left rear based on the driving status data, and enhance the clarity of the virtual view of the left rear.

[0145] S307. Based on the driving status data and driving scenario of the trailer vehicle, determine the target display perspective corresponding to the target virtual field of view from multiple preset display perspectives.

[0146] In some embodiments, the relevant description of step S307 can be found in the preceding description, and will not be repeated here.

[0147] S308. Based on the target display perspective, display the virtual view of the target.

[0148] In some embodiments, the vehicle controller may also switch the display view of the target virtual view when it detects a control operation for switching the display view of the target virtual view, thereby obtaining a virtual view after the view is switched; and display the virtual view after the view is switched.

[0149] In this embodiment, since the three-dimensional semantic map corresponding to the external environment of the vehicle is determined based on the omnidirectional point cloud data and omnidirectional image data corresponding to the external environment of the trailer vehicle, the three-dimensional semantic map includes environmental information of the omnidirectional field of view around the trailer vehicle. Since the virtual model corresponding to the towing device is determined based on the current posture data and towing parameters of the towing device, the virtual model can accurately synchronize the posture changes of the towing device. Thus, based on the three-dimensional semantic map and the virtual model, an accurate virtual field of view with depth perception and high field of view coverage can be determined, thereby improving the field of view coverage and accuracy of the generated external environment image. Therefore, after the virtual field of view image is displayed in the main vehicle of the trailer vehicle, the field of view coverage seen by the driver (especially the field of view coverage of the connection area between the main vehicle and the towing device, and the area behind the towing device) and the accuracy of the external environment can be improved, which can help improve the driving safety of the trailer vehicle.

[0150] In one alternative implementation, Figure 1 and Figure 3 In the method for generating images of the vehicle's external environment shown, the vehicle controller can also establish a communication connection with other vehicles or terminals within a preset distance range of the towed vehicle to obtain long-distance traffic information and expand the field of vision perception range.

[0151] In one optional implementation, the vehicle controller can also send multi-dimensional environmental data corresponding to the external environment of the towed vehicle, the current attitude data of the towing device, and towing parameters to the server. Correspondingly, the server receives the multi-dimensional environmental data, the current attitude data of the towing device, and the towing parameters; the server stores the multi-dimensional environmental data, the current attitude data of the towing device, and the towing parameters. This allows fleet managers or users to remotely check the stability of the towing device connection and whether there is overload or abnormal swaying, even when they are not in the vehicle.

[0152] In this embodiment, the server can also determine whether there is an abnormal connection between the towing vehicle and the towing device (such as a loose link mechanism or excessive fluctuation in current attitude data) by comparing the current attitude data of the towing device with at least one of the towing parameters. If an abnormal connection is determined, an alarm message and driving information for adjusting the driving state of the towing vehicle are sent to the vehicle controller. The vehicle controller receives the alarm message and the driving information accordingly and controls the movement of the towing vehicle based on the driving information. This helps to minimize the occurrence of towing accidents caused by connection failures.

[0153] The towing parameters may include factory parameters and load information.

[0154] For example, assuming the initial load of the trailer is 25kg, and the load of the trailer currently obtained by the server (denoted as the current load) is 20kg, the server can determine that there is an anomaly at the connection between the main vehicle and the trailer by comparing the initial load and the current load.

[0155] In this implementation, the server can also send scene adaptation parameters to the vehicle controller based on the monitored multi-dimensional environmental data, so that the vehicle controller can generate a virtual view of the trailer vehicle based on the scene adaptation parameters, the three-dimensional semantic map and the virtual model.

[0156] In an optional implementation, this application also provides a schematic diagram of the architecture of a system for displaying external environmental information of a towed vehicle. Please refer to... Figure 4 , Figure 4 This is a schematic diagram of an optional architecture for a system displaying external environment information of a towed vehicle, provided in an embodiment of this application. For example... Figure 4 As shown, the system includes a multi-source perception input layer, a VLA multimodal unified coding layer, a VLA context reasoning and decision module, a vision optimization and automotive-grade constraint output module, and an in-vehicle system execution layer.

[0157] The multi-source perception input layer is used to input data collected by visual sensors, 3D semantic maps, textual semantic prior data, current attitude data and towing parameters of the trailer hitch, vehicle driving data, and driving status data. Visual sensors include cameras and LiDAR, and the data collected includes image data from cameras and point cloud data from LiDAR. The 3D semantic map includes lanes, obstacles, and drivable areas. Textual semantic prior data includes traffic rules and scene knowledge. The current attitude data of the trailer hitch includes yaw angle and pitch angle, and the towing parameters include the dimensions of the trailer hitch. Vehicle driving data includes vehicle speed, steering angle, and acceleration. Driving status data includes the driver's line of sight and driving intentions.

[0158] The VLA multimodal unified coding layer includes a visual feature encoder, a semantic feature encoder, a vehicle driving feature encoder, a driving state feature encoder, and a multimodal fusion and temporal alignment module. Specifically, the visual feature encoder extracts features from data acquired by visual sensors; the semantic feature encoder extracts features from 3D semantic maps and textual semantic prior data; the vehicle driving feature encoder extracts features from the current attitude data and towing parameters of the trailer, as well as vehicle driving data; the driving state feature encoder extracts features from driving state data; and the multimodal fusion and temporal alignment module performs feature fusion and temporal alignment processing on the extracted multimodal features to obtain a unified feature vector.

[0159] The VLA contextual reasoning and decision-making module includes a lightweight Transformer encoder, a driving intention reasoning submodule, a trajectory prediction submodel, a risk prediction submodule, and a contextual feature fusion submodule. Specifically, the lightweight Transformer encoder is used to understand the global scene semantics based on a unified feature vector and an onboard 3.2B student VLA model obtained through knowledge distillation; the driving intention reasoning submodel is used to determine the driving intention based on the global scene semantics; the trajectory prediction submodel is used to determine the predicted trajectory of the towing device and obstacles based on the global scene semantics; the risk prediction submodule is used to assess the collision risk within a preset time period; and the contextual feature fusion submodule is used to infer the virtual view of the towing vehicle based on the driving intention, the predicted trajectory of the towing device and obstacles, and the collision risk assessment results.

[0160] The vision optimization and automotive-grade constraint output module includes a vision angle decision submodule, a display enhancement instruction submodule, an automotive-grade real-time constraint submodule, and an anomaly fallback protection submodule. Specifically, the vision angle decision submodule determines the optimal viewing angle and / or display area for the virtual vision image; the display enhancement instruction submodule highlights and / or displays warning labels for risk areas; the automotive-grade real-time constraint submodule constrains the frame rate and latency of the virtual vision image, such as constraining the frame rate to be greater than or equal to 40Hz and the latency to be less than or equal to 10ms; and the anomaly fallback protection submodule adjusts the virtual vision image to a conservative vision mode in the event of anomalies.

[0161] The in-vehicle system execution layer includes a virtual view generation module, a multimodal interaction module, and a vehicle controller. The virtual view generation module fuses and renders the virtual and real view images to obtain the displayed view. The multimodal interaction module enables the driver to interact with the displayed view via the central control screen or head-up display. The vehicle controller provides safety warnings based on the displayed view.

[0162] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0163] Based on the same inventive concept, this application also provides an apparatus for generating an external environment image to implement the above-described method for generating an external environment image. The solution provided by this apparatus is similar to the solution described in the above-described method. Therefore, the specific limitations of one or more embodiments of the apparatus for generating an external environment image provided below can be found in the limitations of the method for generating an external environment image described above, and will not be repeated here.

[0164] Please see Figure 5 , Figure 5 This is an optional structural schematic diagram of a device for generating images of the vehicle's external environment provided in an embodiment of this application. For example... Figure 5 As shown, the device for generating the image of the vehicle's external environment may include, but is not limited to:

[0165] The determination module 501 is used to determine the three-dimensional semantic map corresponding to the external environment of the trailer based on the point cloud data and image data corresponding to the external environment of the trailer.

[0166] The determination module 501 is also used to determine the virtual model corresponding to the towing device based on the current attitude data and towing parameters of the towing device included in the trailer vehicle.

[0167] The generation and display module 502 is used to generate and display the virtual view of the trailer based on the three-dimensional semantic map and the virtual model; the virtual view includes the external environment information of the trailer.

[0168] In some embodiments, when the generation and display module 502 generates and displays a virtual view of the trailer vehicle based on a three-dimensional semantic map and a virtual model, it is specifically used to: overlay the virtual model onto the three-dimensional semantic map to obtain an initial virtual view; optimize the initial virtual view based on the driving state data of the trailer vehicle to obtain a virtual view of the trailer vehicle; and display the virtual view.

[0169] In some embodiments, when the generation and display module 502 optimizes the initial virtual view based on the driving state data corresponding to the trailer vehicle to obtain the virtual view corresponding to the trailer vehicle, the determination module 501 is also used to determine the driving risk assessment result of the trailer vehicle within a preset time period; the generation and display module 502 is used to optimize the initial virtual view based on the driving state data corresponding to the trailer vehicle to obtain the virtual view corresponding to the trailer vehicle when the driving risk assessment result indicates that there is a driving risk in the trailer vehicle within the preset time period.

[0170] In some embodiments, when determining the driving risk assessment result of the towed vehicle within a preset time period, the determining module 501 is specifically used to: determine the first predicted driving trajectory of the virtual model and the second predicted driving trajectory of obstacles within a preset time period based on the trajectory prediction model, the three-dimensional semantic map, the virtual model and the driving state data corresponding to the towed vehicle; the obstacles include obstacles within a preset distance range of the towed vehicle; and determine the driving risk assessment result of the towed vehicle within a preset time period based on the first predicted driving trajectory and the second predicted driving trajectory.

[0171] In some embodiments, when the generation and display module 502 generates and displays the virtual view of the trailer vehicle based on the 3D semantic map and the virtual model, the determination module 501 is further used to determine the virtual view of the trailer vehicle based on the 3D semantic map and the virtual model; determine the driving scenario corresponding to the trailer vehicle; and determine the target display view corresponding to the virtual view from multiple preset display views based on the driving status data and driving scenario corresponding to the trailer vehicle. The generation and display module 502 is used to display the virtual view based on the target display view.

[0172] In some embodiments, when the generation and display module 502 generates and displays a virtual view of the trailer vehicle based on a 3D semantic map and a virtual model, it is specifically used to: obtain scene adaptation parameters corresponding to the trailer vehicle, the scene adaptation parameters including at least one of geographical area adaptation parameters, weather adaptation parameters, and trailer device adaptation parameters; generate a virtual view of the trailer vehicle based on the scene adaptation parameters, the 3D semantic map, and the virtual model; and display the virtual view.

[0173] In some embodiments, when determining a three-dimensional semantic map corresponding to the external environment based on point cloud data and image data corresponding to the external environment of the trailer vehicle, the determining module 501 is specifically used to: acquire point cloud data and image data corresponding to the external environment of the trailer vehicle; perform time synchronization processing on the point cloud data and image data to obtain target point cloud data and target image data; and generate a three-dimensional semantic map corresponding to the external environment based on the target point cloud data and target image data.

[0174] In some embodiments, the towing parameters include factory parameters and load information; when determining the virtual model corresponding to the towing device based on the current attitude data and towing parameters of the towing device included in the towing vehicle, the determining module 501 is specifically used to: perform noise reduction processing on the current attitude data, factory parameters and load information of the towing device included in the towing vehicle to obtain target current attitude data, target factory parameters and target load information; and generate the virtual model corresponding to the towing device based on the target current attitude data, target factory parameters and target load information.

[0175] In some embodiments, the device may further include a processing module, which is configured to switch the display view of the virtual field of view when a control operation for switching the display view of the virtual field of view is detected, thereby obtaining a virtual field of view after the view is switched; and a generation and display module 502 is configured to display the virtual field of view after the view is switched.

[0176] It is understood that the specific implementation of each module in the vehicle exterior environment image generation device provided in this application embodiment and the beneficial effects that can be achieved can be referred to the description of the aforementioned vehicle exterior environment image generation method embodiment, and will not be repeated here.

[0177] Each module in the aforementioned device for generating images of the vehicle's external environment can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of the in-vehicle terminal device in hardware form, or stored in the memory of the external environment information display device of the towed vehicle in software form, so that the processor can call and execute the corresponding operations of each module.

[0178] In one exemplary embodiment, a vehicle is provided whose internal structure diagram can be as follows: Figure 6 As shown, the vehicle includes a processor, memory, input / output interface, communication interface, display unit, and input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interface. The vehicle's processor provides computing and control capabilities. The vehicle's memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The vehicle's input / output interface is used for exchanging information between the processor and external devices. The vehicle's communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements a method for generating a picture of the vehicle's external environment. The vehicle's display unit is used to form a visually visible image and can be a display screen, projection device, or virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the vehicle can be a touch layer covering the display screen, or it can be a button, trackball, or touchpad installed in the vehicle.

[0179] Those skilled in the art will understand that Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the vehicle to which the present application is applied. A specific vehicle may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0180] In one optional implementation, this application also provides a system for generating images of the vehicle's external environment, which may include, but is not limited to, a multi-source perception module, a processing module, a virtual field of view generation module, a multimodal interaction module, and a vehicle network communication module.

[0181] The multi-source perception module can be used to collect comprehensive data on the trailer vehicle, the trailer assembly, and the external environment of the trailer vehicle. The multi-source perception module may include an environmental perception unit, a trailer assembly perception unit, and a driving status data perception unit.

[0182] The environmental perception unit can be composed of 8 high-definition cameras (including front-view, surround-view, side-view, and rear-view cameras with a resolution of ≥4K), 4 millimeter-wave radars, and 2 lidars, covering the vehicle's all-around area without blind spots; the cameras integrate adaptive cleaning and heating functions to improve adaptability to extreme environments; the lidars are used to accurately obtain the 3D spatial coordinates and distance information of surrounding obstacles.

[0183] The towing device sensing unit can be composed of attitude sensors (including gyroscopes, angular displacement sensors, etc.) and weight sensors installed at the connection between the main vehicle and the towing device. The attitude sensors are used to collect the attitude data of the towing device in real time (including yaw angle, pitch angle, connection fastness status, etc.), and the weight sensors are used to collect the load information of the towing device. It supports reading the factory parameters such as the model and size of the towing device via the CAN bus.

[0184] The driving status data perception unit consists of a camera and millimeter-wave radar in the cockpit, used to monitor the driver's line of sight, head posture, and hand operation status in real time, and to determine the driver's attention distribution and operating intentions.

[0185] The processing module can be used to determine the 3D semantic map and generate the virtual model corresponding to the towed device based on the lightweight VLA model. The processing module may include a data preprocessing unit, a 3D semantic map generation unit, and a virtual model generation unit.

[0186] The data preprocessing unit is used to perform synchronous calibration, noise reduction, and format conversion on the data from each sensor; it uses timestamp alignment technology to eliminate the delay difference between multiple sensor data and ensure the consistency of data timing.

[0187] The 3D semantic map generation unit is used to generate a 3D semantic map of the vehicle's external environment based on the collected full-dimensional data of the external environment through a point cloud-image fusion algorithm. It can accurately identify elements such as lane lines, curbs, obstacles (vehicles, pedestrians, cyclists), and traffic signs. It supports real-time reconstruction and updating of the environment in scenarios without high-precision maps.

[0188] The virtual model generation unit is used to generate a 1:1 scale virtual model of the trailer based on the current attitude data, factory parameters, and load information of the trailer using a parametric modeling algorithm.

[0189] In some embodiments, the processing module further includes a risk prediction and vision optimization unit. This unit, based on a 3D semantic map and a virtual model, uses VLA's causal reasoning capabilities to predict driving risks (such as obstacle collisions, changes in blind spots, etc.) within a preset timeframe. If a driving risk exists, it combines driving status data to determine the optimal field of view and viewing angle. For example, if driving status data indicates that the driver's gaze is focused on the right-side rearview mirror, the risk prediction and vision optimization unit can automatically enhance the clarity and detail of the right-side towing area.

[0190] The virtual view generation module is used to achieve real-time fusion of virtual models and 3D semantic maps to generate multi-view virtual view images. The virtual view generation module may include a fusion rendering unit and a multi-view generation unit.

[0191] The fusion rendering unit is used to accurately overlay virtual models onto a 3D semantic map using ray tracing rendering algorithms, ensuring that the virtual model and the real environment have consistent lighting and perspective relationships, providing a realistic sense of depth; it supports automatic switching of rendering modes according to driving scenarios, such as enhancing night vision rendering in night scenes and enhancing contrast rendering in rainy scenes.

[0192] The multi-view generation unit is used to generate at least two basic views: panoramic bird's-eye view, driver's side towed view, passenger side towed view, rear towed view, and close-up view of the connecting parts; it supports adaptive generation of custom views based on the driver's operating intentions and driving scenarios; the view switching delay is ≤5ms, with no screen stuttering or distortion.

[0193] The multimodal interaction module is used to accurately present the virtual field of view and facilitate convenient interaction with the driver. The multimodal interaction module may include a display unit and an interaction control unit.

[0194] The display unit is used to display a virtual view using an integrated flexible display screen in the cockpit (including but not limited to the driver's instrument display screen, the central control screen, and the passenger screen) and / or a head-up display system; the virtual view can automatically switch the display area according to the driver's line of sight, for example, when driving at high speed, the view of the trailer ahead is given priority on the AR-HUD, and when reversing, the panoramic bird's-eye view is presented on the central control screen.

[0195] Interactive control unit: Supports three interaction methods: voice control (such as "show right-side towed view"), gesture control (such as waving to switch the view) and steering wheel button control; It has a driver off-road detection function. When it detects that the driver's hands are off the steering wheel or the driver's line of sight is off the road, it will issue an audible and visual alarm and gradually increase the visibility warning signs. If there is no response, it can link the vehicle to reduce the driving speed.

[0196] The vehicle-to-everything (V2X) communication module can be used to enable data interaction between the trailer vehicle and the cloud platform, such as uploading data and system operating status to the cloud platform, and / or downloading VLA model upgrade packages and scene adaptation parameters from the cloud platform; it can also be used to enable communication with other vehicles or infrastructure within a preset distance range to obtain long-distance traffic information and expand the field of vision perception range; it can also be used to report vehicle driving status information and real-time attitude data of the trailer device to the cloud platform so that the cloud platform can remotely detect the status of the trailer device, etc.

[0197] In one exemplary embodiment, this application provides a vehicle, including a memory and a vehicle controller, wherein the memory stores a computer program; when the vehicle controller executes the computer program, it implements the steps in the above-described methods for generating images of the vehicle's external environment.

[0198] In one exemplary embodiment, this application provides a computer-readable storage medium having a computer program stored thereon. When executed by a vehicle controller, the computer program implements the steps in the methods for generating the above-described external environment images.

[0199] In one exemplary embodiment, this application provides a computer program product, including a computer program. When executed by a vehicle controller, the computer program implements the steps in the methods for generating the various external environment images described above.

[0200] It should be noted that the data involved in this application (including but not limited to data used for analysis, data stored, data displayed, etc.) are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0201] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0202] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0203] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for generating an image of the external environment of a vehicle, characterized in that, The method includes: Based on the point cloud data and image data corresponding to the external environment of the trailer vehicle, a three-dimensional semantic map corresponding to the external environment is determined. Based on the current attitude data and towing parameters of the towing device included in the towing vehicle, a virtual model corresponding to the towing device is determined; Based on the three-dimensional semantic map and the virtual model, an initial virtual view of the trailer vehicle is generated. Using a trajectory prediction model, based on the 3D semantic map, the virtual model, and the driving status data corresponding to the trailer vehicle, a first predicted driving trajectory of the virtual model and a second predicted driving trajectory of obstacles are determined within a preset time period; the obstacles include obstacles within a preset distance range of the trailer vehicle. Based on the first predicted driving trajectory and the second predicted driving trajectory, the driving risk assessment result of the trailer vehicle within the preset time period is determined; If the driving risk assessment result indicates that the trailer vehicle has a driving risk within the preset time period, based on the driving status data and driving scenario of the trailer vehicle, a target display view corresponding to the virtual field of view is determined from multiple preset display views; the initial virtual field of view is optimized based on the target display view to obtain the virtual field of view corresponding to the trailer vehicle and the virtual field of view is displayed.

2. The method according to claim 1, characterized in that, The step of determining the driving risk assessment result of the towed vehicle within the preset time period based on the first predicted driving trajectory and the second predicted driving trajectory includes: If it is determined that the first predicted driving trajectory and the second predicted driving trajectory overlap, the driving risk assessment result of the trailer vehicle within a preset time period is determined to be that there is a driving risk.

3. The method according to claim 1, characterized in that, Based on the virtual model and the 3D semantic map, an initial virtual view of the trailer vehicle is generated, including: Obtain the scene adaptation parameters corresponding to the trailer vehicle, wherein the scene adaptation parameters include at least one of the following: geographical area adaptation parameters, weather adaptation parameters, and trailer device adaptation parameters; Based on the scene adaptation parameters, the 3D semantic map, and the virtual model, an initial virtual view of the trailer vehicle is generated.

4. The method according to any one of claims 1 to 3, characterized in that, Based on point cloud data and image data corresponding to the external environment of the trailer vehicle, a three-dimensional semantic map corresponding to the external environment is determined, including: Acquire point cloud data and image data corresponding to the external environment of the trailer vehicle; The point cloud data and the image data are time-synchronized to obtain target point cloud data and target image data. Based on the target point cloud data and the target image data, a three-dimensional semantic map corresponding to the vehicle's external environment is generated.

5. The method according to claim 4, characterized in that, The step of generating a 3D semantic map corresponding to the vehicle's external environment based on the target point cloud data and the target image data includes: The target point cloud data and the target image are input into a feature extractor to obtain the point cloud features corresponding to the target point cloud data and the image features corresponding to the target image data. Based on the point cloud features, a geometric environment is constructed, and based on the image features, the relative motion poses of each object within a preset distance range of the trailer vehicle are determined. Based on the relative motion poses of the objects, semantic filling is performed on the geometric environment to obtain a three-dimensional semantic map corresponding to the vehicle exterior environment.

6. The method according to claim 1 or 2, characterized in that, The towing parameters include factory parameters and load information; based on the current attitude data and towing parameters of the towing device included in the towing vehicle, a virtual model corresponding to the towing device is determined, including: The current attitude data of the towing device included in the trailer vehicle, the factory parameters, and the load information are denoised to obtain the target current attitude data, the target factory parameters, and the target load information. Based on the target's current attitude data, the target's factory parameters, and the target's load information, a virtual model corresponding to the towing device is generated.

7. A device for generating images of the external environment of a vehicle, characterized in that, The device includes: The determination module is used to determine a three-dimensional semantic map corresponding to the external environment of the trailer based on point cloud data and image data corresponding to the external environment of the trailer. The determining module is further configured to determine the virtual model corresponding to the towing device based on the current attitude data and towing parameters of the towing device included in the towing vehicle. The generation and display module is used to generate an initial virtual view of the trailer vehicle based on the three-dimensional semantic map and the virtual model. The determining module is used to determine, through a trajectory prediction model, a first predicted driving trajectory of the virtual model and a second predicted driving trajectory of obstacles within a preset time period, based on the three-dimensional semantic map, the virtual model, and the driving state data corresponding to the trailer vehicle; the obstacles include obstacles within a preset distance range of the trailer vehicle; and based on the first predicted driving trajectory and the second predicted driving trajectory, determine the driving risk assessment result of the trailer vehicle within the preset time period. The generation and display module is further configured to, when the driving risk assessment result indicates that the trailer vehicle has a driving risk within the preset time period, determine the target display view corresponding to the virtual field of view from multiple preset display views based on the driving status data and driving scenario corresponding to the trailer vehicle; optimize the initial virtual field of view based on the target display view to obtain the virtual field of view corresponding to the trailer vehicle and display the virtual field of view.

8. A vehicle, comprising a memory and a vehicle controller, characterized in that, The memory stores a computer program; when the vehicle controller executes the computer program, it implements the steps of the method described in any one of claims 1 to 6.

Images