Method and system for adaptive viewpoint recommendation in vehicle operations

Abstract

A computer-implemented method and system are disclosed for recommending an optimal viewpoint in a vehicle environment based on real-time sensor data and predicted object trajectories. The method comprises obtaining images from cameras and receiving vehicle sensor signals, generating a three-dimensional (3D) scene, and constructing a bird's eye view (BEV) image. Objects are detected in the BEV image, and a multi-view trajectory model predicts their future positions. The system updates vehicle sensor signals based on these predictions and recommends at least one viewpoint using a deep learning-based viewpoint recommendation model. The recommended viewpoint is rendered from the 3D scene and may be used to assist in vehicle operation by issuing alerts, initiating control actions, or modifying the human-machine interface. The invention enables proactive, context-aware visualization and control, improving safety, situational awareness, and driver support in dynamic driving scenarios.

Description

[0001] 202405595

[0002] 1

[0003] Method and System for Adaptive Viewpoint Recommendation in Vehicle Operations

[0004] FIELD OF THE INVENTION

[0005] The present invention relates generally to the field of driver assistance systems in vehicles. More specifically, it pertains to a method and system for recommending optimal viewpoints based on real-time and predicted positions of surrounding objects, using machine learning models to enhance driver awareness and vehicle safety in dynamic traffic environments.

[0006] BACKGROUND OF THE INVENTION

[0007] In modern driving scenarios — such as parking, navigating narrow streets, or maneuvering in tight urban spaces — drivers often face limited visibility and blind spots that can lead to accidents or near misses. To address these challenges, current driver assistance systems (DAS) have incorporated technologies such as surround-view cameras, ultrasonic sensors, and proximity detectors. These systems aim to enhance situational awareness by providing visual feedback from multiple angles around the vehicle.

[0008] Existing systems for viewpoint selection can be broadly categorized into two types: manual and automatic. Manual systems require the driver to interact with a touchscreen or HMI (Human-Machine Interface) to select a desired camera view. While this offers flexibility, it introduces cognitive load and distraction, especially in time-sensitive situations. Automatic systems, on the other hand, attempt to switch viewpoints based on real-time sensor inputs such as steering angle, gear position, or proximity readings. These systems reduce driver interaction but are often reactive and limited in scope.

[0009] A significant limitation of current automated systems is their reliance on real-time data alone. Most systems use ultrasonic sensors with a limited range of 2-3 meters and do not incorporate predictive modeling. As a result, they are unable to anticipate the future positions of dynamic traffic participants such as pedestrians, cyclists, or 202405595

[0010] 2 other vehicles. This reactive behavior can lead to delayed or suboptimal viewpoint recommendations, especially in fast-changing environments.

[0011] Furthermore, while some systems utilize Bird’s Eye View (BEV) images to provide a top-down perspective, these images are typically static and limited in spatial coverage. They are not integrated into the decision-making process for viewpoint selection and do not support temporal reasoning or trajectory forecasting. Consequently, these systems fail to provide context-aware recommendations that account for the future behavior of surrounding objects.

[0012] Another drawback is the lack of integration between visual data and sensor signals. Current systems treat camera images and sensor readings as separate inputs, without fusing them into a unified representation. This limits the system’s ability to make informed decisions based on a holistic understanding of the environment. In summary, existing driver assistance systems are constrained by their short-range sensing, lack of predictive capabilities, and limited integration of multimodal data. They do not offer proactive viewpoint recommendations that adapt to dynamic traffic conditions or anticipate future events.

[0013] SUMMARY OF THE INVENTION

[0014] The present invention addresses these limitations by providing a method and system for adaptive viewpoint recommendation in vehicle operations. The invention utilizes a machine learning model to generate a three-dimensional (3D) scene of the vehicle’s surroundings from camera images. From this 3D scene, a Bird’s Eye View (BEV) image is generated and used to detect surrounding objects. A multi-view trajectory model predicts the future positions of these objects by rendering multiple views and extracting features. The predicted trajectories are then used to update vehicle sensor signals values, which are fused with the BEV image to recommend an optimal viewpoint using a viewpoint recommendation model. The recommended view is rendered and displayed to the driver or used by the vehicle system to assist in navigation, hazard avoidance, or automated control. This approach enables proactive, context-aware viewpoint selection that enhances safety, reduces driver distraction, and improves overall situational awareness. The disclosed solution is 202405595

[0015] 3 illustrated by the subject-matter of the appended independent claims. The appended dependent claims address optional embodiments of the present disclosure.

[0016] According to a first aspect of the invention, there is provided a computer-implemented method for recommending a viewpoint for vehicle control assistance, the method comprising: obtaining a plurality of images captured by one or more cameras mounted on the vehicle, and one or more vehicle sensor signals; generating a three-dimensional, 3D, scene of surroundings of the vehicle based on the plurality of images; generating, from the 3D scene, a bird’s eye view, BEV, image; detecting one or more objects surrounding the vehicle based on the BEV image; predicting, by a multi-view trajectory model, one or more future trajectories for each of the one or more objects by rendering multiple views of each object from the 3D scene and extracting features from each view; updating one or more values of the vehicle sensor signals based on the one or more predicted trajectories; and recommending, by a viewpoint recommendation model, at least one viewpoint based on the updated vehicle sensor signals values and the BEV image.

[0017] The disclosed method achieves a synergistic technical effect by integrating perception and prediction modules into a unified pipeline that enhances situational awareness and decision-making in vehicle environments. The step of obtaining a plurality of images from the cameras, multi-modal input foundation that captures both environmental context and vehicle state (e.g., four fisheye cameras capturing front, rear, and side views). Generating a 3D scene (e.g. using a machine learning model, such as a neural radiance field), enables spatially accurate and occlusion-aware reconstruction of the surroundings (e.g., reconstructing a 5-meter radius scene showing a cyclist partially occluded by a parked van). The transformation of this 3D scene into a bird’s eye view (BEV) image provides a compact and interpretable spatial layout that facilitates efficient object detection (e.g., projecting the 3D scene into a top-down map showing a pedestrian at 1.2 meters behind the vehicle). Detecting objects in the BEV image allows the system to identify dynamic agents and potential hazards (e.g., detecting a pedestrian with 0.91 confidence and a cyclist with 0.88 confidence using a YOLO-based detector). 202405595

[0018] 4

[0019] The use of a multi-view trajectory model to predict future object trajectories introduces a forward-looking capability, enabling the system to anticipate rather than merely react to environmental changes (e.g., rendering five views of the cyclist and predicting a curved path as they approach the vehicle from the left). Updating vehicle sensor signals values based on these predictions effectively fuses real-time and predictive data, enhancing the temporal resolution of the vehicle’s perception system (e.g., modifying the rear ultrasonic sensor value from 1.5 meters to 0.5 meters based on the pedestrian’s predicted position in 2.5 seconds).

[0020] Finally, the viewpoint recommendation model leverages both the updated sensor data and the BEV image to select the most contextually appropriate viewpoint (e.g., recommending a rear-left diagonal view to simultaneously visualize the pedestrian and cyclist). Collectively, these steps contribute to a technical improvement in automated driving assistance by enabling proactive, context-aware viewpoint selection that enhances safety, reduces driver workload, and supports real-time decision-making in complex driving scenarios.

[0021] In embodiments, the method may further comprise rendering a recommended view image corresponding to the at least one viewpoint using the 3D scene. The rendering may be performed using virtual camera parameters associated with the selected viewpoint and may involve querying a volumetric scene representation such as a neural radiance field (NeRF).

[0022] This rendering step enables the generation of photorealistic, occlusion-aware images tailored to the current driving context (e.g., rendering a rear-left diagonal view showing a pedestrian and cyclist in predicted positions). The technical effect is to provide the driver or vehicle system with a high-fidelity visualization that enhances spatial understanding and supports informed decision-making.

[0023] In embodiments, the method may further comprise using the recommended view image to assist in vehicle operation by at least one of: issuing a driver alert, initiating an automated braking or steering maneuver, or modifying a display interface to emphasize a potential hazard. 202405595

[0024] 5

[0025] This step translates visual information into actionable outcomes (e.g., overlaying a red bounding box on a pedestrian predicted to cross behind the vehicle and triggering an audible alert). This enhances safety by enabling timely interventions and reducing the cognitive load on the driver.

[0026] In embodiments, the one or more vehicle sensor signals may comprise at least one of: a steering angle signal, a gear position signal, a proximity sensor signal, a turn indicator signal, a wheel direction signal, an acceleration pedal signal, or an ultrasonic sensor signal.

[0027] These signals provide real-time insight into the vehicle’s operational state (e.g., detecting that the vehicle is reversing with the left turn signal activated and a steering angle of -15°). This enables context-aware viewpoint selection and hazard anticipation based on the vehicle’s intended motion.

[0028] In embodiments, generating the BEV image may comprise applying inverse perspective mapping (IPM) to transform camera images into a top-down view, or using a deep learning-based BEV generation model trained on annotated top-view datasets to infer spatial layouts from the 3D scene. This transformation facilitates spatial reasoning and object detection (e.g., projecting a 3D scene into a BEV image showing a pedestrian at 1 .2 meters behind the vehicle). The technical effect is to produce a compact, interpretable representation that supports efficient downstream processing.

[0029] In embodiments, detecting the one or more objects may comprise applying a convolutional neural network (CNN)-based object detection model. This enables accurate identification of dynamic and static elements in the environment (e.g., detecting a cyclist with 0.88 confidence in the BEV image using a YOLO model). This ensures reliable object localization, which is critical for trajectory prediction and viewpoint reasoning. 202405595

[0030] 6

[0031] In embodiments, predicting the one or more future trajectories may comprise rendering a plurality of views of each object from the 3D scene using virtual camera parameters centered on the object; extracting spatial-aware features from each view using a shared convolutional neural network; and applying a collaborative correlation filter to track object movement across the views.

[0032] This multi-view approach enhances robustness in motion prediction (e.g., tracking a cyclist’s curved path across five rendered views). This improves trajectory accuracy, especially in the presence of occlusions or erratic motion. In embodiments, the method may further comprise triggering a trajectory prediction network when a tracking confidence score for one or more views falls below a predefined threshold. This fallback mechanism ensures continuity in prediction (e.g., using an LSTM network to estimate a pedestrian’s path after temporary occlusion). The technical effect is to maintain system responsiveness and reliability under uncertain conditions.

[0033] In embodiments, the viewpoint recommendation model may comprise a deep learning model selected from the group consisting of a transformer network, a diffusion model, a 3D convolutional neural network, or a long short-term memory (LSTM) network. These architectures enable context-aware viewpoint selection (e.g., a transformer model attending to both a pedestrian and cyclist to recommend a rear-left diagonal view). This contributes to improved adaptability and accuracy in dynamic driving scenarios.

[0034] In embodiments, the at least one viewpoint may be selected from a predefined set of viewpoint classes, each associated with a camera configuration. This structured approach simplifies deployment and ensures consistency (e.g., selecting from classes like “rear view,” “rear-left diagonal,” or “top-down front”). This enables reduced computational complexity and improved interpretability of the system’s outputs.

[0035] In embodiments, rendering the recommended view image may comprise using the machine learning model and camera parameters associated with the at least one 202405595

[0036] 7 viewpoint. This enables accurate and photorealistic rendering (e.g., using NeRF to generate a rear-left diagonal view with occlusion handling). The technical effect is to provide high-quality visual feedback that enhances driver awareness and system transparency.

[0037] In embodiments, the method may further comprise determining whether any objects are detected in the BEV image; and activating the multi-view trajectory model only if at least one object is detected, and otherwise deactivating the multi-view trajectory model. This conditional activation conserves computational resources (e.g., bypassing trajectory prediction when no objects are present). The technical effect is improved system efficiency and responsiveness.

[0038] In embodiments, updating the one or more values of the vehicle sensor signals may comprises: projecting the predicted future trajectories of the detected objects into a coordinate frame of the vehicle sensors, thereby obtaining projected future trajectories; computing a projected future distance from the vehicle to the detected objects based on the projected future trajectories; and modifying at least one value of the vehicle sensor signals to reflect the projected future distance to the object, wherein the at least one value optionally comprises proximity or ultrasonic sensor signals values. This fusion of predictive and real-time data enhances temporal awareness (e.g., updating the rear proximity sensor to 0.5 meters based on a pedestrian’s predicted position). This enables proactive hazard anticipation and improved safety.

[0039] In embodiments, the BEV image may comprise a temporal sequence of BEV images generated from current and previous timestamps, and the viewpoint recommendation model may be trained to process temporal sequences to improve prediction accuracy. This temporal modeling captures motion trends (e.g., using a sequence of BEV images to anticipate a cyclist’s approach). This enables enhanced robustness and accuracy in dynamic environments. 202405595

[0040] 8

[0041] In embodiments, the machine learning model may comprise a three-dimensional neural radiance field (NeRF) model configured to learn a volumetric representation of the scene from the plurality of images. This enables high-fidelity, view-consistent scene reconstruction (e.g., modeling a 3D environment with occluded pedestrians and dynamic lighting). This also enables improved rendering quality and spatial reasoning for viewpoint selection.

[0042] In embodiments, the method may further comprise using the one or more vehicle sensor signals — such as steering angle, wheel direction, IMU data, or GPS velocity — to refine camera pose estimation or spatial alignment of the plurality of images during the generation of the three-dimensional (3D) scene. This means that, in addition to relying on visual data from the cameras, the system may incorporate real-time sensor inputs to more accurately determine the position and orientation of each camera relative to the vehicle and the environment. This refinement may be applied before or during the 3D reconstruction process, allowing the machine learning model (e.g., a NeRF or depth estimation network) to correct for motion-induced distortions, calibration drift, or misalignment between frames.

[0043] The inclusion of vehicle sensor signals in the 3D scene generation process results in a more geometrically consistent and temporally stable reconstruction of the vehicle’s surroundings. For example, if the vehicle is turning while capturing images, steering angle and IMU data may help correct for rotational offsets, ensuring that the reconstructed scene accurately reflects the real-world layout. This improved spatial alignment enhances the accuracy of downstream modules such as object detection, trajectory prediction, and viewpoint recommendation. It also reduces the likelihood of visual artifacts or misinterpretations caused by camera misalignment, thereby increasing the robustness of the system in dynamic or low-visibility conditions. Overall, this integration supports more reliable perception and safer vehicle operation.

[0044] According to another aspect of the invention, there is provided a system for recommending a viewpoint for vehicle control assistance, the system comprising: one or more cameras configured to capture a plurality of images of surroundings of 202405595

[0045] 9 a vehicle; one or more vehicle sensors configured to generate vehicle sensor signals; a module configured to generate a 3D scene based on the plurality of images; a BEV generation model configured to generate a BEV image from the 3D scene; an object detection model configured to detect one or more objects in the BEV image a multi-view trajectory model configured to predict future trajectories of the one or more objects; a trajectory aggregation module configured to update one or more values of the vehicle sensor signals based on the predicted trajectories; a viewpoint recommendation model configured to recommend at least one viewpoint based on the updated vehicle sensor signals values and the BEV image;

[0046] In embodiments, the multi-view trajectory model may comprise: a rendering engine configured to generate object-centric views from the 3D scene; a feature extraction module configured to extract features from each view; a correlation filter module configured to track object motion across views; an a fallback trajectory prediction network configured to estimate trajectories when tracking confidence is below a threshold.

[0047] In embodiments, the system may further comprise: a rendering module configured to generate a recommended view image based on the at least one viewpoint; and a control module configured to use the recommended view image to assist in vehicle operation by at least one of: issuing a driver alert, initiating an automated braking or steering maneuver, or modifying a display interface to emphasize a potential hazard.

[0048] According to another aspect of the present disclosure, there is provided a computer program comprising instructions which, when the computer program is executed by a computer, cause the computer to carry out any one of the methods disclosed herein.

[0049] According to a further aspect of the disclosure, there is provided a computer-readable data carrier having stored thereon the computer program disclosed herein. 202405595

[0050] 10

[0051] According to a further aspect of the disclosure, there is provided an apparatus comprising: one or more interfaces for communication; a memory; and a data processing circuit configured to carry out any one of the methods disclosed herein.

[0052] Examples will now be further described with reference to the figures in which:

[0053] Figure 1 illustrates an environment for a viewpoint recommendation for vehicle control assistance according to an embodiment of the present disclosure;

[0054] Figure 2 illustrates a flow diagram for activating or de-activating a multi-view trajectory according to an embodiment of the present disclosure;

[0055] Figure 3 illustrates a flow diagram of a multi-view trajectory, in accordance with some embodiments of the present disclosure;

[0056] Figure 4 illustrates a flow diagram for recommending a viewpoint, in accordance with some embodiments of the present disclosure;

[0057] Figure 5 illustrates a block diagram for a viewpoint recommendation system, in accordance with some embodiments of the present disclosure;

[0058] Figure 6 illustrates a method for a viewpoint recommendation for vehicle control assistance, in accordance with some embodiments of the present disclosure; and Figure 7 illustrates a block diagram of an embodiment of an apparatus to execute the methods described herein.

[0059] Detailed Description

[0060] The present invention provides a method and system for adaptive viewpoint recommendation in vehicle operations. The invention may be implemented in a driver assistance system configured to enhance situational awareness by intelligently selecting and rendering optimal viewpoints based on both real-time sensor data and predicted future positions of surrounding objects. The system may be particularly beneficial in scenarios such as parking, navigating narrow streets, or maneuvering in dynamic traffic environments, where visibility and timely decision-making are critical. 202405595

[0061] 11

[0062] In one embodiment, the system may comprise one or more surround-view cameras mounted on the vehicle, configured to capture a plurality of images from different angles around the vehicle. These images may be synchronized with data from one or more vehicle sensors, which may include ultrasonic sensors, proximity sensors, steering angle sensors, gear position sensors, and turn indicators. The synchronized image and sensor data may be received by a processing unit for further analysis.

[0063] The processing unit may employ a machine learning model to generate a three-dimensional (3D) scene of the vehicle’s surroundings based on the captured images. In some embodiments, the machine learning model may comprise a three-dimensional neural radiance field (NeRF) model, which may be configured to learn a volumetric representation of the scene. The 3D scene may encode spatial and appearance information, allowing for the rendering of arbitrary viewpoints with occlusion awareness.

[0064] From the generated 3D scene, a bird’s eye view (BEV) image may be constructed. The BEV image may be generated using inverse perspective mapping or a deep learning-based projection model. This image may provide a top-down spatial layout of the environment, which may be particularly suitable for object detection and trajectory analysis.

[0065] An object detection model may be applied to the BEV image to identify surrounding traffic participants, such as pedestrians, cyclists, and other vehicles. The detection model may be implemented using a convolutional neural network (CNN), such as YOLO, Faster R-CNN, or SSD. YOLO stands for You Only Look Once, Faster R-CNN stands for Faster Region-based Convolutional Neural Network, and SSD stands for Single Shot MultiBox Detector. The model may output bounding boxes and class labels for each detected object, which may then be used for trajectory prediction.

[0066] For each detected object, a multi-view trajectory model may be activated to predict future positions. The model may render multiple views of each object from the 3D 202405595

[0067] 12 scene using virtual camera parameters centered on the object. Features may be extracted from each view using a shared CNN, and a collaborative correlation filter may be applied to track object motion across the views. If the tracking confidence falls below a predefined threshold, a fallback trajectory prediction network may be triggered to estimate the object’s future trajectory.

[0068] The predicted trajectories may be used to update the vehicle’s one or more sensor signals values. For example, the proximity or ultrasonic sensor values may be modified to reflect the projected future positions of the detected objects. This fusion of predicted data with real-time sensor signals may enable the system to anticipate potential hazards and respond proactively.

[0069] The updated sensor data or signals values, along with the BEV image, may be input to a viewpoint recommendation model. This model may be implemented using a deep learning architecture such as a transformer, a 3D convolutional neural network (3D CNN), or a long short-term memory (LSTM) network. The model may output a viewpoint class label corresponding to the optimal camera angle for the current and predicted scene context.

[0070] Based on the selected viewpoint class, the system may render a recommended view image using the 3D scene and the associated camera parameters. The rendered image may be displayed to the driver via a human-machine interface (HMI) or used by the vehicle’s control system. The system may use the recommended view image to assist in vehicle operation by issuing a driver alert, initiating an automated braking or steering maneuver, or modifying the display interface to emphasize a potential hazard.

[0071] In some embodiments, the system may include logic to activate or deactivate the multi-view trajectory model based on object detection results. If no objects are detected in the BEV image, the trajectory model may remain inactive to conserve computational resources. If objects are present, the model may be activated to predict their future positions. 202405595

[0072] 13

[0073] Additionally, the BEV image may be generated as a temporal sequence using current and previous timestamps. This temporal information may be used by the viewpoint recommendation model to learn motion patterns and improve prediction accuracy in dynamic environments.

[0074] The invention provides several technical advantages over existing systems. It may enable proactive viewpoint selection based on predicted object motion, reduce driver distraction through automated and context-aware view switching, and improve safety by facilitating early hazard detection and response. The system may also support efficient computation through selective module activation and may be adaptable to both manual and autonomous driving modes.

[0075] Figure 1 illustrates an exemplary environment 100 in which the viewpoint recommendation system may be implemented within a vehicle. The environment 100 may include a viewpoint recommendation and visualization system 107 installed in a vehicle operated by a user 111. The system 107 may be operably connected to one or more (e.g. surround-view) cameras 103, one or more vehicle sensors 105, and a user interface 109.

[0076] The one or more (e.g. surround-view) cameras 103 may be mounted at strategic positions around the vehicle, such as the front grille, rear bumper, and side mirrors, to provide a 360-degree visual coverage of the vehicle’s surroundings. These cameras may capture high-resolution images from multiple angles, which may be used to construct a comprehensive visual representation of the environment. The images may be synchronized and timestamped to ensure consistency with sensor data.

[0077] The one or more vehicle sensors 105 may include, but are not limited to, ultrasonic sensors, proximity sensors, steering angle sensors, gear position sensors, wheel direction sensors, turn indicator sensors, and acceleration pedal sensors. These sensors may provide real-time data regarding the vehicle’s motion state, orientation, and proximity to surrounding objects. For example, the ultrasonic sensors may 202405595

[0078] 14 detect the distance to nearby obstacles, while the steering angle sensor may indicate the direction in which the vehicle is turning.

[0079] The viewpoint recommendation and visualization system 107 may be configured to receive the images from the cameras 103 and the sensor signals from the sensors 105. Based on this input, the system may generate a three-dimensional (3D) scene of the vehicle’s surroundings using a machine learning model. In some embodiments, the model may comprise a neural radiance field (NeRF) that enables volumetric scene reconstruction from multi-view images.

[0080] Once the 3D scene is generated, the system may derive a bird’s eye view (BEV) image, which may provide a top-down perspective of the environment. This BEV image may be used to detect surrounding objects and predict their future trajectories. The predicted trajectories may then be used to update the sensor data or signals values, enabling the system to anticipate potential hazards or interactions.

[0081] The system 107 may further include a viewpoint recommendation model that analyzes the updated sensor data and BEV image to determine the most contextually appropriate viewpoint. This viewpoint may be rendered into a recommended view image using the 3D scene and displayed on the user interface 109. The user interface 109 may be a dashboard screen, heads-up display, or any other suitable HMI component that allows the driver 111 to view the recommended perspective.

[0082] In some embodiments, the recommended view image may be used not only for display but also to assist in vehicle operation. For example, the system may issue a visual or auditory alert if a pedestrian is predicted to cross behind the vehicle, or it may initiate an automated braking maneuver if a collision is imminent. This integration of visual data, predictive modeling, and control logic may significantly enhance driver awareness and vehicle safety. 202405595

[0083] 15

[0084] Figure 2 illustrates a flow diagram representing the logic for activating or deactivating the multi-view trajectory prediction module in accordance with one embodiment of the present invention. This mechanism may be implemented to optimize computational efficiency by selectively enabling trajectory prediction only when relevant objects are detected in the vehicle’s surroundings.

[0085] The process may begin with the acquisition of a plurality of images from one or more regular or surround-view cameras 203 mounted on the vehicle. These cameras may be positioned to provide overlapping fields of view, enabling full 360-degree coverage of the vehicle’s environment. The captured images may be synchronized and processed to construct a coherent visual dataset representing the current scene.

[0086] The images may then be input to a machine learning model, such as a neural radiance field (NeRF), to generate a three-dimensional (3D) scene 205. This 3D scene may encode spatial and appearance information in a volumetric format, allowing for the rendering of arbitrary viewpoints and the extraction of object-centric views. The 3D scene may serve as the foundational representation for subsequent processing steps.

[0087] From the 3D scene 205, a bird’s eye view (BEV) image 207 may be generated. The BEV image may provide a top-down perspective of the environment, offering a spatially intuitive layout of nearby objects and obstacles. This image may be constructed using inverse perspective mapping or a learned projection model, and may cover a configurable area around the vehicle, such as a 5-meter radius.

[0088] The BEV image 207 may then be analyzed using an object detection model 209. This model may be implemented using a convolutional neural network (CNN) architecture, such as YOLOv8 or Faster R-CNN, trained to detect common traffic participants including pedestrians, cyclists, and vehicles. The model may output bounding boxes and class labels for each detected object, along with confidence scores. 202405595

[0089] 16

[0090] A decision point may follow the object detection step. If one or more objects are detected in the BEV image 207 with confidence above a predefined threshold, the system may activate the multi-view trajectory prediction module 211. This module may then proceed to render multiple views of each detected object from the 3D scene, extract features, and predict future trajectories. These predictions may be used in downstream processes such as sensor data (also called sensor signals values in this disclosure) updating and viewpoint recommendation.

[0091] Conversely, if no objects are detected in the BEV image 207, or if the confidence scores fall below the threshold, the system may deactivate the multi-view trajectory module 213. In this state, the system may rely solely on real-time sensor data without engaging in predictive modeling, thereby conserving computational resources and reducing latency.

[0092] For example, consider a scenario in which the vehicle is reversing into a parking space. If a pedestrian is detected walking behind the vehicle at a distance of 2.5 meters, the object detection model may trigger the activation of the trajectory module. The system may then predict that the pedestrian will cross behind the vehicle within 2.5 seconds, prompting the system to recommend a rear-left diagonal view and issue a driver alert. In contrast, if the vehicle is stationary in an empty parking lot with no detected objects, the trajectory module may remain inactive, and the system may maintain a default rear view.

[0093] This activation logic may ensure that the system remains responsive and efficient, focusing computational resources on dynamic and safety-critical situations while minimizing unnecessary processing in static or low-risk environments.

[0094] Figure 3 illustrates a flow diagram representing the process of multi-view trajectory prediction for objects detected in the vehicle’s surroundings, in accordance with one embodiment of the present invention. This process may be initiated when one or more objects are identified in the bird’s eye view (BEV) image, as described in the previous figure. 202405595

[0095] 17

[0096] The process may begin with the generation of a three-dimensional (3D) scene 303 of the environment surrounding the vehicle. This 3D scene may be constructed using a machine learning model, such as a neural radiance field (NeRF), based on a plurality of images captured by regular or surround-view cameras. The 3D scene may encode spatial and appearance information in a volumetric format, enabling the rendering of arbitrary viewpoints with high fidelity and occlusion awareness.

[0097] Once the 3D scene 303 is available, the system may render multiple views of each detected object. These views may be generated by adjusting virtual camera parameters to focus on the object from different angles and distances. For example, if a pedestrian is detected walking behind the vehicle, the system may render five views from the rear, rear-left, rear-right, top-down, and diagonal perspectives. These views may capture different aspects of the object’s motion and appearance.

[0098] Each of the rendered views may then be processed by a feature extractor 305. The feature extractor may comprise a shared convolutional neural network (CNN) that operates across all views to extract spatial-aware features. These features may include object contours, motion cues, and contextual information from the surrounding scene. The use of shared weights across views may ensure consistency and reduce computational overhead.

[0099] The extracted features may be passed to a collaborative correlation filter 307, which may be configured to track the object’s movement across the different views. The correlation filter may compute similarity scores between features in consecutive frames or views, enabling the system to infer the object’s trajectory over time. The filter may be updated online to adapt to changes in object appearance or motion.

[0100] In cases where the tracking confidence is low — such as when the object is partially occluded or moving erratically — a trajectory prediction network (TPN) 309 may be triggered. The TPN may be implemented using a recurrent neural network (RNN), long short-term memory (LSTM), or transformer architecture. It may take as input the features from the available views and output a predicted trajectory for the object over a future time horizon, such as the next 2-3 seconds. 202405595

[0101] 18

[0102] Alternatively, in some embodiments, trajectory estimation 311 may be performed directly on the BEV image using simpler techniques such as Kalman filtering or CNN-based regression. While this approach may be less accurate than multi-view prediction, it may serve as a fallback in low-complexity scenarios or when computational resources are constrained.

[0103] For example, consider a scenario in which a cyclist is detected approaching the vehicle from the left at a speed of 3.5 meters per second. The system may render multiple views of the cyclist from the 3D scene, extract features, and track the cyclist’s motion using the correlation filter. If the cyclist briefly disappears behind a parked car, the TPN may be activated to predict the cyclist’s re-emergence and future path. This predicted trajectory may then be used to update the vehicle’s sensor data (signals values) and inform the viewpoint recommendation process.

[0104] The multi-view trajectory prediction process described in Figure 3 may enable the system to anticipate the future positions of dynamic objects with high accuracy and robustness. By leveraging multiple perspectives and advanced feature modeling, the system may provide a reliable foundation for proactive viewpoint selection and hazard mitigation.

[0105] Figure 4 illustrates a flow diagram representing the process of viewpoint recommendation based on updated sensor data and a bird’s eye view (BEV) image, in accordance with one embodiment of the present invention. This process may be executed after the system has detected surrounding objects and predicted their future trajectories, as described in the previous figures.

[0106] The process may begin with the generation of a BEV image 403 from a previously constructed three-dimensional (3D) scene. The 3D scene may be generated using a machine learning model, such as a neural radiance field (NeRF), based on a plurality of images captured by regular or surround-view cameras. The BEV image 403 may provide a top-down perspective of the environment surrounding the vehicle and may be rendered using inverse perspective mapping or a learned projection 202405595

[0107] 19 model. The BEV image may serve as a spatially intuitive representation for downstream processing.

[0108] In parallel, vehicle sensor signal information 405 may be obtained from one or more sensors mounted on the vehicle. These sensors may include, but are not limited to, ultrasonic sensors, proximity sensors, steering angle sensors, gear position sensors, wheel direction sensors, turn indicator sensors, and acceleration pedal sensors. The sensor data may provide real-time information about the vehicle’s motion state, orientation, and proximity to surrounding objects.

[0109] The system may then update the vehicle sensor data 405 based on the predicted future trajectories of the detected objects. For example, if a pedestrian is predicted to cross behind the vehicle within the next 2.5 seconds, the rear proximity sensor value may be updated to reflect the pedestrian’s projected position at that future time. This update may be performed by projecting the predicted trajectory into the coordinate frame of the vehicle sensors and modifying the corresponding signals values accordingly. In some embodiments, the BEV image 403 may remain unchanged, while in others, the future positions of objects may be overlaid onto the BEV image to create a temporally enriched representation.

[0110] The updated sensor data and the BEV image 403 may then be provided as input to a neural network 407 configured to recommend an optimal viewpoint. The neural network 407 may be implemented using a deep learning architecture such as a transformer, a 3D convolutional neural network (3D CNN), or a long short-term memory (LSTM) network. The model may be trained on a dataset comprising BEV images, sensor signals, and labeled viewpoint classes, enabling it to learn complex spatial and temporal patterns.

[0111] The output of the neural network 407 may be a viewpoint class number 409, which may correspond to a predefined camera configuration or rendering angle. For example, the model may output a class label corresponding to a rear-left diagonal view if a cyclist is predicted to approach from the left while the vehicle is reversing. 202405595

[0112] 20

[0113] The viewpoint class may be selected to maximize visibility of relevant objects and minimize blind spots, thereby enhancing driver awareness and safety.

[0114] The selected viewpoint class 409 may then be used to render a recommended view image from the 3D scene using the associated camera parameters. This image may be displayed to the driver via a human-machine interface (HMI) or used by the vehicle’s control system to assist in navigation or hazard avoidance.

[0115] For instance, consider a scenario in which the vehicle is preparing to reverse out of a driveway while a cyclist is approaching from the left at 3.5 meters per second. The system may detect the cyclist in the BEV image, predict their trajectory, and update the left-side proximity sensor accordingly. The neural network may then recommend a rear-left diagonal view, which may be rendered and displayed to the driver. If the cyclist’s predicted path intersects with the vehicle’s projected path, the system may also issue a visual alert or initiate a braking maneuver.

[0116] This viewpoint recommendation process may enable the system to proactively adapt to dynamic traffic conditions, providing the driver with the most relevant visual information for safe and informed decision-making.

[0117] Figure 5 illustrates a block diagram of a viewpoint recommendation system 500 for vehicle control assistance, in accordance with one or more embodiments of the present invention. The system 500 may be implemented within a vehicle to provide real-time, context-aware viewpoint recommendations based on both current sensor data and predicted future positions of surrounding objects.

[0118] The system 500 may include a plurality of functional modules, each configured to perform a specific task in the viewpoint recommendation pipeline. These modules may be implemented in software, hardware, or a combination thereof, and may be operably coupled via one or more communication buses or data interfaces.

[0119] In one embodiment, the system 500 may include a module that generates a three-dimensional (3D) scene of the vehicle’s surroundings based on the acquired 202405595

[0120] 21 plurality of images. A machine learning model 503 inside the module may be configured to generate the 3D scene of the vehicle’s surroundings. The model 503 may receive as input a plurality of images captured by (e.g. surround-view) cameras mounted on the vehicle. These images may be processed to construct a volumetric representation of the environment. In some embodiments, the machine learning model 503 may comprise a neural radiance field (NeRF), which may encode the scene as a continuous function of spatial coordinates and viewing direction, enabling photorealistic rendering from arbitrary viewpoints.

[0121] The output of the 3D scene generation module 503 may be provided to a bird’s eye view (BEV) generation model 505. The BEV generation model 505 may transform the 3D scene into a top-down, orthographic projection that provides a spatially intuitive layout of the environment. This BEV image may serve as a basis for object detection and trajectory prediction.

[0122] An object detection model 507 may receive the BEV image and identify one or more objects in the vehicle’s vicinity. The model 507 may be implemented using a convolutional neural network trained to detect common traffic participants such as pedestrians, cyclists, and vehicles. The output of the object detection model may include bounding boxes, class labels, and confidence scores for each detected object.

[0123] If one or more objects are detected, a multi-view trajectory model 509 may be activated. This model may render multiple views of each object from the 3D scene using virtual camera parameters. A feature extraction module may process each view to extract spatial-aware features, which may then be passed to a collaborative correlation filter for tracking. If tracking confidence is insufficient, a fallback trajectory prediction network may be triggered to estimate the object’s future path.

[0124] The predicted trajectories may be aggregated by an aggregate trajectory model 511 , which may also receive real-time vehicle sensor data. The sensor data may include signals from ultrasonic sensors, steering angle sensors, gear position sensors, and other sources. The aggregate trajectory model 511 may update the 202405595

[0125] 22 vehicle sensor signals values based on the predicted future positions of the detected objects. For example, if a pedestrian is predicted to cross behind the vehicle, the rear proximity sensor value may be adjusted to reflect the anticipated distance at a future time.

[0126] The updated sensor data and the BEV image may be provided to a viewpoint recommendation model 513. This model may be implemented using a deep learning architecture such as a transformer, a 3D convolutional neural network, or a long short-term memory network. The model 513 may analyze the spatial and temporal context to determine the most appropriate viewpoint for the current driving scenario. The output may be a viewpoint class label corresponding to a predefined camera configuration.

[0127] The selected viewpoint may be used to render a recommended view image from the 3D scene using the associated camera parameters. This image may be displayed to the driver via a human-machine interface (HMI) or used by the vehicle’s control system to assist in navigation, hazard avoidance, or automated driving maneuvers.

[0128] For example, if a cyclist is detected approaching from the left while the vehicle is reversing, the system may predict the cyclist’s trajectory, update the left-side sensor data, and recommend a rear-left diagonal view. This view may be rendered and displayed to the driver, along with a visual alert if the cyclist’s path intersects with the vehicle’s projected path.

[0129] The modular architecture of the system 500 may allow for efficient processing, scalability, and integration with existing vehicle systems. Each module may operate independently or in coordination with others, enabling the system to adapt to a wide range of driving scenarios and environmental conditions.

[0130] The method 600 disclosed herein may be executed by a driver assistance system installed in a vehicle and may involve a sequence of steps that transform raw sensor and image data into a context-aware visual recommendation used to assist vehicle operation. 202405595

[0131] 23

[0132] In embodiments, the method may begin by receiving at least one or more vehicle sensor signals and a plurality of images captured by regular or surround-view cameras mounted on the vehicle. The sensor signals may include data from ultrasonic sensors, steering angle sensors, gear position sensors, proximity sensors, turn indicators, and acceleration pedal sensors. The images may be captured from multiple angles around the vehicle, such as front, rear, left, and right, and may be synchronized with the sensor data to ensure temporal consistency.

[0133] The method comprises generating a three-dimensional (3D) scene based on the received images and sensor data. This step may be performed using a machine learning model, such as a neural radiance field (NeRF), which may reconstruct a volumetric representation of the environment. The 3D scene may encode spatial and appearance information, allowing for the rendering of arbitrary viewpoints and the extraction of object-centric views.

[0134] The method comprises generating a bird’s eye view (BEV) image from the 3D scene. The BEV image may provide a top-down perspective of the vehicle’s surroundings and may be constructed using inverse perspective mapping or a learned projection model. The BEV image may serve as a spatially intuitive representation for object detection and trajectory analysis.

[0135] The method comprises detecting one or more objects surrounding the vehicle based on the generated BEV image. A deep learning-based object detection model may be applied to the BEV image to identify traffic participants such as pedestrians, cyclists, and vehicles. The model may output bounding boxes and class labels for each detected object, along with confidence scores.

[0136] The method comprises predicting one or more future trajectories for each detected object by considering multiple plausible views of each object. This may involve rendering multiple views from the 3D scene, extracting features using a shared convolutional neural network, and applying a collaborative correlation filter for 202405595

[0137] 24 tracking. If tracking confidence is low, a fallback trajectory prediction network may be triggered to estimate the object’s future path.

[0138] The method comprises updating the vehicle sensor signals values based on the predicted trajectories. For example, if a pedestrian is predicted to cross behind the vehicle, the rear proximity sensor value may be adjusted to reflect the anticipated distance at a future time. This fusion of predicted data with real-time sensor signals may enable the system to anticipate potential hazards and respond proactively.

[0139] The method comprises recommending one or more viewpoints based on the updated sensor data and the BEV image. A viewpoint recommendation model may analyze the spatial and temporal context to determine the most appropriate viewpoint for the current driving scenario. The model may output a viewpoint class label corresponding to a predefined camera configuration.

[0140] In embodiments, the method may comprise generating a recommended view image based on the selected viewpoint. The image may be rendered from the 3D scene using the associated camera parameters and may provide a high-fidelity, occlusion-aware perspective of the environment.

[0141] In embodiments, the method may comprise displaying the recommended view image to the driver or using it to assist in vehicle operation. The system may issue a driver alert, initiate an automated braking or steering maneuver, or modify the display interface to emphasize a potential hazard. This step ensures that the system delivers a technical effect beyond mere information display, enhancing safety and situational awareness.

[0142] For example, consider a scenario in which the vehicle is reversing out of a parking space while a cyclist is approaching from the left. The system may detect the cyclist, predict their trajectory, update the left-side sensor data, and recommend a rear-left diagonal view. The rendered image may be displayed to the driver, and if the cyclist’s path intersects with the vehicle’s projected path, the system may issue an alert or initiate braking. 202405595

[0143] 25

[0144] The method disclosed herein may be implemented in real time and may be adaptable to various driving scenarios, including manual, assisted, and autonomous modes. It may provide a robust framework for integrating visual and sensor data to deliver intelligent viewpoint recommendations that enhance vehicle safety and driver awareness.

[0145] Fig. 6 shows a flow diagram illustrating a computer-implemented method 700 for recommending a viewpoint for vehicle control assistance. Fig. 7 shows a schematic illustrating an embodiment of the system used for implementing the methods disclosed herein.

[0146] Step 610 of method comprises receiving, by a processing unit, a plurality of images captured by one or more cameras mounted on a vehicle and one or more vehicle sensor signals. The cameras may be surround-view cameras or any type of cameras. The cameras may be positioned to provide overlapping fields of view around the vehicle, such as front, rear, left, and right perspectives. The images may be captured in real time and may be synchronized with vehicle sensor data to ensure temporal alignment. The vehicle sensor signals may include, but are not limited to, steering angle, gear position, wheel direction, proximity, ultrasonic, acceleration pedal, and turn indicator signals. In alternative embodiments, the sensor data may also include GPS, inertial measurement unit (IMU) data, or radar inputs. The processing unit may be implemented using an embedded system, a GPU-accelerated computing platform, or a distributed architecture.

[0147] For example, when a vehicle is preparing to reverse out of a parking space, the system may receive images from four cameras and sensor data indicating that the gear is in reverse, the steering angle is -15°, and the rear proximity sensor reads 1 .5 meters. The effect of this step is to establish a synchronized and comprehensive dataset that reflects the vehicle’s current state and environment, forming the basis for subsequent scene reconstruction and decision-making. 202405595

[0148] 26

[0149] Step 620 of method 700 comprises generating a three-dimensional (3D) scene of surroundings of the vehicle based on the plurality of images. In one embodiment, a machine learning model may be used to generate the 3D scene, said model may comprise a neural radiance field (NeRF) model trained to reconstruct volumetric scenes from multi-view images. The model may encode the scene as a continuous function of spatial coordinates and viewing direction, allowing for photorealistic rendering from arbitrary viewpoints. In alternative embodiments, the 3D scene may be generated using stereo vision, structure-from-motion (SfM), or depth estimation networks. The 3D scene may be stored as a voxel grid, point cloud, or neural representation, depending on the implementation.

[0150] For instance, the system may generate a 3D scene covering a 5-meter radius around the vehicle, including a pedestrian walking behind the vehicle and a cyclist approaching from the left. The effect of this step is to create a spatially rich and occlusion-aware representation of the environment, enabling downstream modules to reason about object positions and movements in three dimensions.

[0151] Step 630 of method 600 comprises generating, from the 3D scene, a bird’s eye view (BEV) image. The BEV image may be constructed by projecting the 3D scene onto a top-down plane using inverse perspective mapping or a learned projection model. The BEV image may provide a spatially intuitive layout of the environment, showing the relative positions of objects and obstacles around the vehicle. In some embodiments, the BEV image may be enhanced with semantic segmentation, lane markings, or dynamic overlays. The resolution and coverage area of the BEV image may be configurable based on the driving context.

[0152] For example, the BEV image may show a pedestrian at (x=1.2 m, y=-0.5 m) and a cyclist at (x=-2.5 m, y=1 .0 m), relative to the vehicle’s center. The effect of this step is to provide a compact and interpretable representation of the scene that facilitates object detection, trajectory prediction, and viewpoint reasoning.

[0153] Step 640 of method 600 comprises detecting one or more objects surrounding the vehicle based on the BEV image. The detection may be performed using a deep 202405595

[0154] 27 learning-based object detection model, such as YOLO, Faster R-CNN, or SSD, trained to identify traffic participants in top-down imagery. The model may output bounding boxes, class labels, and confidence scores for each detected object. In some embodiments, the detection may be augmented with temporal tracking or fused with radar / lidar data for improved robustness. The detection step may be executed periodically or triggered by changes in the scene.

[0155] For instance, the system may detect a pedestrian with 0.91 confidence and a cyclist with 0.88 confidence in the BEV image. The effect of this step is to identify dynamic elements in the environment that may pose a risk or require attention, enabling the system to focus computational resources on relevant objects.

[0156] Step 650 of method 600 comprises predicting, by a multi-view trajectory model, one or more future trajectories for each of the one or more objects by rendering multiple views of each object from the 3D scene and extracting features from each view. The multi-view trajectory model may be configured to generate object-centric renderings from the 3D scene using virtual camera parameters that simulate different angles and distances. Each rendered view may be processed by a shared convolutional neural network (CNN) to extract spatial-aware features. These features may be passed to a collaborative correlation filter that tracks object motion across views. In some embodiments, if the tracking confidence falls below a predefined threshold, a fallback trajectory prediction network (TPN), such as a long short-term memory (LSTM) network or a transformer-based model, may be triggered to estimate the object’s future path. The predicted trajectories may be represented as sequences of spatial coordinates over a defined time horizon, such as 1 to 3 seconds into the future.

[0157] For example, if a cyclist is detected approaching from the left at a speed of 3.5 meters per second, the system may render five views of the cyclist from different angles, extract features from each view, and track the cyclist’s motion. If the cyclist briefly disappears behind a parked car, the TPN may predict that the cyclist will re-emerge and continue along a curved path. The effect of this step is to enable the 202405595

[0158] 28 system to anticipate the future positions of dynamic objects, allowing for proactive decision-making and improved safety.

[0159] Step 660 of method 600 comprises updating one or more values of the vehicle sensor signals based on the one or more predicted trajectories. This step may involve projecting the predicted future positions of the detected objects into the coordinate frame of the vehicle’s sensor system. For example, the predicted position of a pedestrian at time t+2 seconds may be used to adjust the rear ultrasonic sensor reading to reflect a future proximity value. In some embodiments, the updated sensor data (i.e. signals values) may be stored in a temporal buffer or fused with real-time sensor readings using a weighted average or Kalman filter. This fusion may allow the system to maintain consistency between real-time and predicted data, enhancing the reliability of downstream modules.

[0160] For instance, if a pedestrian is predicted to cross behind the vehicle in 2.5 seconds, the system may update the rear proximity sensor value from 1.5 meters to 0.5 meters. This updated value may then be used to trigger a warning or influence viewpoint selection. The effect of this step is to enrich the vehicle’s perception system with predictive information, enabling it to respond to future events rather than merely reacting to current conditions.

[0161] Step 670 of method 600 comprises recommending, by a viewpoint recommendation model, at least one viewpoint based on the updated vehicle sensor signals values and the BEV image. The viewpoint recommendation model may be implemented using a deep learning architecture trained to classify driving scenarios and select the most contextually appropriate viewpoint. The model may take as input the BEV image and the updated sensor vector, and may output a viewpoint class label corresponding to a predefined camera configuration. In some embodiments, the model may be trained using supervised learning on a dataset of labeled driving scenarios, or using reinforcement learning to optimize for safety and visibility metrics. 202405595

[0162] 29

[0163] For example, if the updated sensor data indicates that a cyclist is approaching from the left and a pedestrian is crossing behind the vehicle, the model may recommend a rear-left diagonal view to maximize visibility of both objects. The effect of this step is to automate the selection of the most informative viewpoint, reducing driver distraction and improving situational awareness.

[0164] The method may further comprise rendering a recommended view image corresponding to the at least one viewpoint using the 3D scene. The rendering may be performed by querying the 3D scene with the camera parameters associated with the selected viewpoint class. In embodiments where the 3D scene is represented using a neural radiance field, the rendering may involve sampling rays through the scene and integrating color and density values to produce a photorealistic image. The rendered image may be generated in real time and may be displayed at a resolution suitable for the vehicle’s human-machine interface (HMI).

[0165] For instance, if the selected viewpoint is a rear-left diagonal view, the system may render an image showing the pedestrian and cyclist in their predicted positions relative to the vehicle. The effect of this step is to provide the driver or vehicle system with a high-fidelity, occlusion-aware visualization of the environment, tailored to the current and anticipated context.

[0166] The method may further comprise using the recommended view image to assist in vehicle operation by at least one of: issuing a driver alert, initiating an automated braking or steering maneuver, or modifying a display interface to emphasize a potential hazard. This step may involve analyzing the rendered image or its associated metadata to determine whether a safety-critical event is likely to occur. If so, the system may trigger an appropriate response, such as displaying a visual warning, sounding an audible alert, or engaging the vehicle’s braking or steering systems. In some embodiments, the system may also adjust the HMI layout to highlight the relevant object or area of concern. 202405595

[0167] 30

[0168] For example, if the rendered view shows a pedestrian predicted to cross behind the vehicle within 1.5 seconds, the system may overlay a red bounding box on the pedestrian and issue an audible alert. If the driver does not respond, the system may initiate soft braking to prevent a collision. The effect of this step is to translate visual information into actionable outcomes, thereby enhancing safety and reducing the cognitive load on the driver.

[0169] In one embodiment, the one or more vehicle sensor signals may comprise at least one of: a steering angle signal, a gear position signal, a proximity sensor signal, a turn indicator signal, a wheel direction signal, an acceleration pedal signal, or an ultrasonic sensor signal. These signals may be generated by respective sensors integrated into the vehicle’s control and perception systems. The signals may be sampled at a fixed frequency and encoded into a feature vector for use by downstream modules. In alternative embodiments, additional signals such as brake pressure, yaw rate, or GPS velocity may also be included to enrich the contextual understanding of the vehicle’s state.

[0170] For example, when the vehicle is reversing with the left turn indicator activated and the steering angle at -15°, the system may interpret this as a maneuver into a leftward parking space. The proximity sensor may detect an obstacle 1.2 meters behind the vehicle. The effect of incorporating these signals is to provide the system with a real-time understanding of the vehicle’s motion intent and spatial constraints, which may be critical for accurate viewpoint selection and hazard anticipation.

[0171] In one embodiment, generating the BEV image may comprise applying inverse perspective mapping (IPM) to transform camera images into a top-down view. IPM may involve projecting image pixels from the camera’s perspective into a bird’s eye coordinate frame using known camera intrinsics and extrinsics. In alternative embodiments, a deep learning-based BEV generation model may be used, which may be trained on annotated datasets to infer top-down spatial layouts from raw images or 3D scenes. The BEV image may be enhanced with semantic overlays, object masks, or temporal information. 202405595

[0172] 31

[0173] For instance, a BEV image generated using IPM may show a pedestrian crossing behind the vehicle and a cyclist approaching from the left. A deep learning-based model may further annotate these objects with class labels and motion vectors. The effect of this step is to produce a spatially structured representation that simplifies object detection and trajectory prediction, while also enabling intuitive visualization for the driver or control system.

[0174] In one embodiment, detecting the one or more objects may comprise applying a convolutional neural network (CNN)-based object detection model that may be selected from the group consisting of YOLO, Faster R-CNN, or SSD. These models may be trained on BEV or top-down datasets to detect traffic participants and static obstacles. The detection model may output bounding boxes, class labels, and confidence scores. In alternative embodiments, the model may be fused with radar or lidar data to improve detection robustness under poor lighting or occlusion conditions.

[0175] For example, the system may detect a pedestrian with 0.91 confidence and a cyclist with 0.88 confidence in the BEV image. The bounding boxes may be used to crop object-centric views for trajectory prediction. The effect of this step is to enable the system to identify and localize dynamic elements in the environment, which are essential for predictive modeling and viewpoint reasoning.

[0176] In one embodiment, predicting the one or more future trajectories may comprise rendering a plurality of views of each object from the 3D scene using virtual camera parameters centered on the object. Each view may be processed using a shared convolutional neural network to extract spatial-aware features. These features may be passed to a collaborative correlation filter that tracks object motion across views. In alternative embodiments, the views may be rendered at different time intervals to capture temporal dynamics, or the feature extractor may be replaced with a transformer-based encoder.

[0177] For example, the system may render five views of a cyclist from different angles and extract features that capture the cyclist’s motion direction and speed. The 202405595

[0178] 32 correlation filter may track the cyclist’s movement across views and predict a curved trajectory. The effect of this step is to enable robust and accurate trajectory prediction, even in the presence of occlusions or erratic motion, thereby improving the reliability of downstream decisions.

[0179] In one embodiment, the method may further comprise triggering a trajectory prediction network when a tracking confidence score for one or more views falls below a predefined threshold. The confidence score may be computed based on feature similarity, motion consistency, or detection stability. The fallback network may be implemented using a long short-term memory (LSTM) network, a gated recurrent unit (GRU), or a transformer model trained to predict future positions from partial observations.

[0180] For instance, if a pedestrian briefly disappears behind a parked vehicle, the correlation filter may lose track of the object. The fallback network may then predict that the pedestrian will reappear and continue walking along a straight path. The effect of this step is to maintain continuity in trajectory prediction under uncertainty, ensuring that the system remains proactive and responsive in dynamic environments.

[0181] In embodiments, the viewpoint recommendation model may comprise a deep learning model selected from the group consisting of a transformer network, a diffusion model, a three-dimensional convolutional neural network (3D CNN), or a long short-term memory (LSTM) network. The viewpoint recommendation model may be configured to receive as input a bird’s eye view (BEV) image and a vector of updated vehicle sensor signals values, and to output a viewpoint class label corresponding to a predefined or dynamically generated camera configuration. A transformer network may be employed to capture long-range dependencies and contextual relationships between detected objects and vehicle state indicators. A diffusion model may be utilized to iteratively refine viewpoint predictions through a denoising process, particularly in uncertain or ambiguous scenarios. A 3D CNN may be used to process spatiotemporal sequences of BEV images, enabling the model to learn motion patterns and environmental dynamics. An LSTM network may be 202405595

[0182] 33 applied to sequential sensor data and image features to model temporal dependencies and predict viewpoint transitions over time. In alternative embodiments, the viewpoint recommendation model may comprise a hybrid architecture combining attention mechanisms with convolutional or recurrent layers, or may be implemented using a graph neural network (GNN) to reason over spatial relationships among detected objects.

[0183] For example, in a scenario where a pedestrian is detected crossing behind the vehicle and a cyclist is approaching from the left, a transformer-based viewpoint recommendation model may attend to both objects and prioritize a rear-left diagonal viewpoint that maximizes visibility of both hazards. In another example, a 3D CNN may process a sequence of BEV images showing a vehicle reversing into a narrow alley, and recommend a top-down rear view to assist with alignment. The technical effect of employing such deep learning models is to enable context-aware, data-driven viewpoint selection that adapts to dynamic environments, thereby improving driver situational awareness and reducing cognitive load.

[0184] In embodiments, the at least one viewpoint may be selected from a predefined set of viewpoint classes, each associated with a specific camera configuration. The viewpoint classes may include, but are not limited to, front view, rear view, left-side view, right-side view, rear-left diagonal view, rear-right diagonal view, and top-down view. Each class may be defined by a set of virtual or physical camera parameters, including position, orientation, field of view, and rendering resolution. The use of a predefined set of viewpoint classes may simplify the training and deployment of the viewpoint recommendation model by constraining the output space to a finite and interpretable set. In alternative embodiments, the viewpoint classes may be dynamically generated based on scene geometry, user preferences, or task-specific constraints, such as maximizing visibility of occluded regions or minimizing blind spots.

[0185] For instance, when the vehicle is reversing with the left turn indicator activated and a pedestrian is detected behind the vehicle, the system may select the “rear-left diagonal” viewpoint class to provide a comprehensive view of the pedestrian’s 202405595

[0186] 34 trajectory. In another example, during highway lane changes, the system may select a “left-side view” to monitor adjacent lanes. The technical effect of using predefined viewpoint classes is to ensure consistent and interpretable visual outputs, reduce computational complexity, and facilitate integration with existing camera systems and human-machine interfaces (HMIs).

[0187] In embodiments, rendering the recommended view image may comprise using the machine learning model used for 3D scene generation from the obtaining plurality images, and camera parameters associated with the at least one viewpoint. The rendering process may involve querying a three-dimensional (3D) scene representation, such as a neural radiance field (NeRF), with the selected viewpoint’s camera intrinsics and extrinsics to generate a photorealistic image. The rendering may be performed in real time and may account for occlusions, lighting conditions, and object motion. In alternative embodiments, the rendering may be performed using mesh-based reconstruction, voxel grids, or depth-based image warping techniques. The rendering engine may be implemented on a GPU-accelerated platform to meet real-time performance requirements.

[0188] For example, if the selected viewpoint corresponds to a rear-left diagonal view, the system may render an image showing a pedestrian and a cyclist in their predicted positions relative to the vehicle. The rendered image may be displayed on the vehicle’s HMI or used by an automated control module. The technical effect of this step is to provide a high-fidelity, occlusion-aware visualization of the environment that enhances driver perception and supports informed decision-making.

[0189] In embodiments, the method may further comprise determining whether any objects are detected in the BEV image, and activating the multi-view trajectory model only if at least one object is detected, and otherwise deactivating the multi-view trajectory model. The object detection step may be performed using a convolutional neural network (CNN)-based model trained on top-down imagery. The activation logic may be implemented as a conditional trigger within the processing pipeline, allowing the system to conserve computational resources when no dynamic elements are present. In alternative embodiments, the activation may be based on motion 202405595

[0190] 35 detection, sensor anomalies, or a confidence threshold derived from the object detection model.

[0191] For instance, if the BEV image shows no pedestrians, cyclists, or vehicles within a predefined radius, the system may bypass the trajectory prediction step. If a pedestrian is detected near the rear of the vehicle, the trajectory model may be activated to predict future motion. The technical effect of this conditional activation is to optimize computational efficiency, reduce power consumption, and prioritize processing resources for safety-critical scenarios.

[0192] In embodiments, updating one or more values of the vehicle sensor signals may comprise projecting the predicted future positions of the detected objects into the coordinate frame of the vehicle sensors, and modifying at least one proximity or ultrasonic sensor value to reflect the projected future distance to the object. In embodiments, updating the one or more values of the vehicle sensor signals may comprise: projecting the predicted future trajectories of the detected objects into a coordinate frame of the vehicle sensors, thereby obtaining projected future trajectories; computing a projected future distance from the vehicle to the detected objects based on the projected future trajectories; and modifying at least one value of the vehicle sensor signals to reflect the projected future distance to the object, wherein the at least one value optionally comprises proximity or ultrasonic sensor signals values.

[0193] The projection may be performed using a transformation matrix derived from the vehicle’s calibration parameters. The updated sensor signals values may be stored in a temporal buffer or fused with real-time sensor readings using a Kalman filter, weighted average, or Bayesian fusion technique. In alternative embodiments, the sensor updates may include synthetic signals or confidence scores that reflect the likelihood of future object interactions.

[0194] For example, if a pedestrian is predicted to cross behind the vehicle in 2.5 seconds, the system may update the rear ultrasonic sensor value from 1.5 meters to 0.5 meters. This updated value may be used to trigger a visual or auditory alert. The technical effect is to enhance the temporal awareness of the vehicle’s perception 202405595

[0195] 36 system, enabling proactive responses to anticipated hazards rather than reactive responses to current conditions.

[0196] In embodiments, the BEV image may comprise a temporal sequence of BEV images generated from current and previous timestamps, and the viewpoint recommendation model may be trained to process temporal sequences to improve prediction accuracy. The temporal sequence may be constructed using a sliding window approach, with each frame aligned to a common coordinate system. The model may be implemented using a recurrent neural network (RNN), a temporal convolutional network (TCN), or a transformer-based architecture capable of modeling temporal dependencies. In alternative embodiments, the temporal sequence may be augmented with optical flow, motion vectors, or object tracking data to enhance temporal coherence.

[0197] For instance, a sequence of BEV images showing a cyclist approaching from the left may enable the model to anticipate the cyclist’s trajectory and recommend a left-side viewpoint. The technical effect is improved robustness and accuracy of viewpoint recommendations in dynamic environments, particularly when objects exhibit non-linear or occluded motion patterns.

[0198] In embodiments, the machine learning model that may be used to generate the 3D scene may comprise a three-dimensional neural radiance field (NeRF) model configured to learn a volumetric representation of the scene from the plurality of images. The NeRF model may encode the scene as a continuous function of spatial coordinates and viewing direction, and may be trained to synthesize novel views by integrating color and density values along sampled rays. The model may be trained using supervised or self-supervised learning on multi-view image datasets. In alternative embodiments, the 3D scene may be represented using point clouds, voxel grids, or mesh reconstructions generated via stereo vision, structure-from-motion (SfM), or depth estimation networks.

[0199] For example, a NeRF model may reconstruct a 3D scene around the vehicle, capturing fine-grained details such as the curvature of a cyclist’s path or the 202405595

[0200] 37 occlusion of a pedestrian behind a parked car. The technical effect is to enable photorealistic, occlusion-aware rendering of arbitrary viewpoints, thereby enhancing the fidelity and interpretability of the recommended view images.

[0201] In embodiments, a system for recommending a viewpoint for vehicle control assistance may comprise one or more cameras configured to capture a plurality of images of surroundings of a vehicle. These cameras may be surround-view cameras or any camera types. These cameras may be mounted at strategic locations such as the front grille, rear bumper, and side mirrors, and may provide overlapping fields of view to enable 360-degree environmental coverage. The system may further comprise one or more vehicle sensors configured to generate vehicle sensor signals, which may include steering angle, gear position, wheel direction, proximity, ultrasonic, acceleration pedal, brake pressure, and turn indicator signals. A machine learning model may be configured to generate a three-dimensional (3D) scene based on the plurality of images, wherein the model may include a neural radiance field (NeRF), stereo vision network, or depth estimation model. A BEV generation model may be configured to generate a bird’s eye view (BEV) image from the 3D scene, using either inverse perspective mapping or a deep learning-based projection network. An object detection model may be configured to detect one or more objects in the BEV image, such as pedestrians, cyclists, or vehicles, using CNN-based architectures like YOLO or Faster R-CNN. A multi-view trajectory model may be configured to predict future trajectories of the one or more objects by rendering multiple object-centric views and analyzing motion patterns. A trajectory aggregation module may be configured to update the vehicle sensor signals values based on the predicted trajectories, and a viewpoint recommendation model may be configured to recommend at least one viewpoint based on the updated vehicle sensor signals and the BEV image.

[0202] For example, the system may detect a cyclist approaching from the left and a pedestrian crossing behind the vehicle. The trajectory model may predict their future paths, and the trajectory aggregation module may update the rear proximity sensor to reflect a predicted distance of 0.5 meters. The viewpoint recommendation model may then select a rear-left diagonal view to maximize visibility. The technical effect 202405595

[0203] 38 is a fully integrated perception and decision-making system that enhances driver awareness and supports proactive vehicle control in complex environments.

[0204] In embodiments, the multi-view trajectory model of claim 16 may comprise a rendering engine configured to generate object-centric views from the 3D scene. These views may be rendered using virtual camera parameters centered on each detected object, simulating different angles and distances. A feature extraction module may be configured to extract spatial-aware features from each view using a shared convolutional neural network or transformer-based encoder. A correlation filter module may be configured to track object motion across views by comparing feature similarity and motion consistency. A fallback trajectory prediction network may be configured to estimate trajectories when tracking confidence is below a threshold, and may be implemented using a long short-term memory (LSTM) network, gated recurrent unit (GRU), or transformer model trained on partial observation sequences.

[0205] For instance, if a pedestrian briefly disappears behind a parked vehicle, the correlation filter may lose track of the object. The fallback network may then predict that the pedestrian will reappear and continue walking along a straight path. The technical effect is to maintain continuity in trajectory prediction under uncertainty, ensuring that the system remains proactive and responsive in dynamic environments.

[0206] In embodiments, the system may further comprise a rendering module configured to generate a recommended view image based on the at least one viewpoint. The rendering module may query the 3D scene using the camera parameters associated with the selected viewpoint class and may produce a photorealistic image using volumetric rendering techniques such as ray sampling in a NeRF model. The system may also comprise a control module configured to use the recommended view image to assist in vehicle operation by at least one of: issuing a driver alert, initiating an automated braking or steering maneuver, or modifying a display interface to emphasize a potential hazard. The control module may analyze the 202405595

[0207] 39 rendered image or its associated metadata to determine whether a safety-critical event is likely to occur and may trigger appropriate responses.

[0208] For example, if the rendered view shows a pedestrian predicted to cross behind the vehicle within 1.5 seconds, the control module may overlay a red bounding box on the pedestrian and issue an audible alert. If the driver does not respond, the system may initiate soft braking. The technical effect is to translate visual information into actionable outcomes, thereby enhancing safety and reducing the cognitive load on the driver.

[0209] In embodiments, the control module may be further configured to prioritize rendering of the recommended view image on a user interface when a predicted trajectory intersects with a predefined safety zone around the vehicle. The safety zone may be defined as a dynamic or static region surrounding the vehicle, such as a 2-meter radius in all directions or a directional buffer based on current motion. The control module may monitor predicted trajectories and compare them against the safety zone boundaries. If an intersection is detected, the system may prioritize the corresponding viewpoint for display, override user-selected views, or trigger additional alerts.

[0210] For instance, if a cyclist’s predicted path intersects with the vehicle’s rear-left safety zone within 2 seconds, the system may immediately render and display the rear-left diagonal view on the HMI. The technical effect is to ensure that the most safety-relevant information is presented to the driver in a timely manner, thereby improving hazard awareness and reducing reaction time.

[0211] In embodiments, a machine learning model may be configured to generate a three-dimensional (3D) scene from a plurality of images may comprise a neural radiance field (NeRF) model. This model may be trained using supervised learning on datasets comprising multi-view image sequences of real-world driving environments, where each image is associated with known camera intrinsics and extrinsics. The training data may include urban, suburban, and highway scenes captured under varying lighting and weather conditions. During training, the model 202405595

[0212] 40 may learn to map spatial coordinates and viewing directions to color and density values by minimizing a photometric reconstruction loss between rendered and ground-truth images. For example, the model may be trained on a dataset where a vehicle-mounted camera rig captures 360° imagery of a parking lot, and the NeRF learns to reconstruct the volumetric geometry of surrounding vehicles and pedestrians.

[0213] For example, suppose the training dataset includes 1000 image sets, each with 6 camera views and known poses. For a given pixel ray in one view, the model samples 64 points along the ray and predicts RGB values. The loss is computed as the mean squared error between the predicted and actual pixel color, e.g., if the ground truth RGB is (0.8, 0.7, 0.6) and the predicted is (0.75, 0.72, 0.58), the loss is 0.0017. Over time, the model learns to synthesize consistent views from unseen angles.

[0214] In embodiments, the BEV generation model may comprise a convolutional neural network or transformer-based encoder-decoder architecture trained to infer top-down spatial layouts from multi-view or monocular images. The training data may consist of annotated top-view datasets where each input image or image set is paired with a corresponding BEV semantic map, including lane markings, object masks, and free space. The model may be trained using supervised learning with pixel-wise segmentation loss or adversarial training to improve realism. For example, the model may be trained on a dataset where front and side camera images are paired with LiDAR-projected BEV maps, allowing the network to learn spatial transformations and semantic segmentation jointly.

[0215] For example, assume the model is trained on 10,000 samples, each with a front-view image and a 200x200 pixel BEV map. If the model predicts a lane marking at pixel (100, 120) with 0.9 confidence and the ground truth label is 1 (lane), the binary cross-entropy loss for that pixel may be — log(0.9) « 0.105. The model minimizes this loss across all pixels to learn accurate spatial projections. 202405595

[0216] 41

[0217] In embodiments, the object detection model applied to the BEV image may comprise a convolutional neural network (CNN) architecture such as YOLO (You Only Look Once), Faster R-CNN, or SSD (Single Shot MultiBox Detector). These models may be trained on datasets containing BEV images annotated with bounding boxes and class labels for traffic participants such as pedestrians, cyclists, and vehicles. The training process may involve supervised learning using classification and localization loss functions. For example, the model may be trained on a dataset derived from the nuScenes or Argoverse datasets, where BEV images are generated from camera or LiDAR data and annotated with object positions.

[0218] For example, suppose a BEV image contains a cyclist labeled with a bounding box at (x=50, y=60, width=20, height=40). The model predicts a box at (x=52, y=58, width=19, height=42) with a confidence of 0.88. The loU (Intersection over Union) is 0.78, and the classification loss is -log(0.88) « 0.13. The model is trained to minimize both localization and classification errors.

[0219] In embodiments, the multi-view trajectory model may be trained to predict future object trajectories by analyzing object-centric views rendered from a 3D scene. The model may include a rendering engine, a shared CNN-based feature extractor, and a collaborative correlation filter. The training data may consist of sequences of object-centric images rendered from real or synthetic 3D scenes, along with ground-truth future trajectories obtained from tracking datasets. The model may be trained using a combination of contrastive loss (for feature consistency across views) and trajectory prediction loss (e.g., mean squared error between predicted and actual positions).

[0220] For example, suppose the model renders 5 views of a cyclist and extracts features from each. The ground truth trajectory is [(1.0, 1.0), (1.5, 1.2), (2.0, 1.5)] over 3 seconds. The model predicts [(1.1 , 1.0), (1.6, 1.3), (2.1 , 1.6)]. The average Euclidean error per step is A / ((0.1 )2+ 02) = 0.1 , A / ((0.1 )2+ (0.1 )2) « 0.14, and A / ((0.1 )2+ (0.1 )2) « 0.14. The model minimizes this trajectory error during training. 202405595

[0221] 42

[0222] In embodiments, the fallback trajectory prediction network may comprise a recurrent neural network such as a long short-term memory (LSTM) network, a gated recurrent unit (GRU), or a transformer model. This model may be activated when the tracking confidence of the multi-view model falls below a threshold. The training data may include sequences of object positions over time, annotated with future trajectories. The model may be trained using supervised learning to minimize trajectory prediction error.

[0223] For example, suppose the LSTM receives a sequence of past positions [(0.5, 0.5), (0.7, 0.6), (0.9, 0.7)] and predicts the next position as (1.1 , 0.8). The ground truth is (1 .0, 0.85), resulting in an error of A / ((0.1 )2+ (-0.05)2) « 0.11 . The model is trained to minimize this prediction error across thousands of such sequences.

[0224] In embodiments, the viewpoint recommendation model may comprise a deep learning model selected from the group consisting of a transformer network, a diffusion model, a 3D convolutional neural network (3D CNN), or a long short-term memory (LSTM) network. The model may be trained to classify driving scenarios and recommend optimal viewpoints based on BEV images and vehicle sensor signals. The training data may include labeled driving scenarios where each sample is associated with a preferred viewpoint class, determined by expert annotation or simulation-based optimization.

[0225] For example, suppose the input is a BEV image showing a pedestrian at (x=1.2, y=-0.5) and a cyclist at (x=-2.5, y=1 .0), with sensor inputs indicating the vehicle is reversing. The ground truth label is “rear-left diagonal view.” The model predicts this class with 0.92 confidence. The cross-entropy loss is -log(0.92) « 0.083. The model is trained to minimize this loss across a dataset of 50,000 labeled scenarios.

[0226] In the present disclosure, any feature described in an embodiment, example, method (e.g., method 600), method step, and / or system (e.g., systems 100 to 500, and 700), may be combined with any feature of another embodiment, example, method, method step, and / or system unless otherwise specified. 202405595

[0227] 43

[0228] Fig. 7 schematically illustrates a block diagram of an embodiment of an apparatus 700 comprising one or more interfaces 710 for communication and one or more processors or data processing circuits 720 configured to execute any one of the methods described herein.

[0229] In embodiments, the apparatus may further comprise a memory or a storage medium 730. A storage medium 730 (or a data carrier, or a computer-readable data carrier, or a computer-readable medium) may comprise, stored thereon, a computer program or computer-executable instructions for performing one of the methods described herein when it is performed by one or more processors 720. The processors 720 may communicate with the storage medium 730 directly and / or via the one or more interfaces 710.

[0230] In embodiments, the one or more interfaces 710 may comprise wired and / or wireless interfaces for transmitting and / or receiving communication signals in connection with the execution of the proposed concept. In practice, the interfaces, e.g., comprise pins, wires, antennas, and / or the like. As well, the interfaces may comprise means for (analog and / or digital) signal or data processing in connection with the communication, e.g., filters, samples, analog-to-digital converters, signal acquisition and / or reconstruction means as well as signal amplifiers, compressors and / or any encryption / decryption means.

[0231] The data processing circuit 720 may correspond to or comprise any type of programable hardware. So, examples of the data processing circuit 720, e.g., comprise a memory, microcontroller, field programmable gate arrays, one or more central and / or graphical processing units. To execute the proposed method, the data processing circuit 720 may be configured to access or retrieve an appropriate computer program for the execution of the proposed method from a memory of the data processing circuit 720 or a separate memory which is communicatively coupled to the data processing circuit 720.

[0232] Some or all of the method steps described above with regard to Figures 1 -2 may be implemented by a computer in that they are executed by (or using) a processor, a 202405595

[0233] 44 microprocessor, an electronic circuit, or the data processing circuit 720. For example, the implementation can be performed using a non-transitory storage medium such as a computer-readable storage medium or data carrier. Such computer-readable media include any media that can be accessed by either a general-purpose or a specialized computer system.

[0234] Furthermore, any component of apparatus 700 may be embedded locally on the autonomous system or located in the cloud 740. For instance, the data processing circuit 720 can be located in an edge device of the autonomous system, in the cloud 740 (e.g., a digital twin), or partially in both. Edge devices, equipped with edge Al capabilities, can perform real-time data processing and decision-making close to the source of data, reducing latency and improving response times. This is particularly helpful for autonomous systems that require immediate reactions to dynamic environments.

[0235] The cloud component 740 can leverage high-performance computing (HPC) resources and advanced machine learning models for more complex computations that are not time-sensitive. By utilizing cloud-based digital twins 740, which are virtual replicas of physical systems, the autonomous system can simulate various scenarios and optimize its performance based on real-time data and predictive analytics. This hybrid approach ensures that the system can handle both real-time processing and complex computations efficiently.

[0236] Additionally, the integration of 5G / 6G technology (e.g. in communication interfaces 710) can further enhance this setup by providing high-speed, low-latency communication between the edge devices and the cloud 740. This ensures seamless data transfer and synchronization, enabling the autonomous system to make informed decisions quickly and accurately.

[0237] In summary, by combining edge Al, HPC, digital twins, and 5G / 6G technology, the implementation of these methods becomes more robust, scalable, and efficient. This hybrid architecture not only enhances the performance and decision-making 202405595

[0238] 45 capabilities of the autonomous system but also ensures that it can adapt to various dynamic environments and scenarios effectively.

[0239] The apparatus 700 may also include a cloud implementation 740, which enhances its capabilities by leveraging cloud-based resources. This cloud implementation comprises several key components. The cloud implementation may comprise cloud interfaces 710’ designed for communication between the local apparatus and the cloud. These interfaces enable data transfer, remote access, and integration with cloud services, ensuring seamless connectivity.

[0240] Additionally, the cloud implementation 740 may include cloud processors or data processing circuits 720’. These may be processing units located in the cloud, configured to execute any of the methods described for the apparatus. By utilizing cloud processors, the system can handle more complex computations and larger datasets without being limited by the local hardware's capabilities. This allows for greater flexibility and efficiency in processing tasks, as well as the ability to dynamically scale resources based on demand.

[0241] Furthermore, the cloud implementation 740 may include a cloud memory or storage medium 730’ that may be responsible for storing data, datasets, and program / method code in the cloud. Cloud storage offers scalability, redundancy, and accessibility, ensuring that data is securely stored and can be accessed from anywhere. This enhances the overall reliability and availability of the system, providing robust data protection and disaster recovery options.

[0242] The cloud implementation 740 allows for flexibility in how the methods are executed. Specifically, any of the methods described herein can be partially executed in the cloud, where some parts of the method may be processed locally while others are offloaded to the cloud. This hybrid approach optimizes performance and resource utilization by balancing the load between local and cloud resources. Alternatively, the entire method can be fully executed in the cloud, leveraging the cloud's extensive computational power and storage capabilities. This dual implementation (local and cloud) ensures that the apparatus can efficiently handle a wide range of 202405595

[0243] 46 tasks, from simple local processing to complex cloud-based computations, providing a robust and scalable solution that can adapt to varying workloads and operational requirements.

Claims

20240559547Claims:

1. A method for recommending a viewpoint for vehicle control assistance, the method comprising:• obtaining (610) a plurality of images captured by one or more cameras mounted on the vehicle, and one or more vehicle sensor signals;• generating (620) a three-dimensional, 3D, scene of surroundings of the vehicle based on the plurality of images;• generating (630), from the 3D scene, a bird’s eye view, BEV, image;• detecting (640) one or more objects surrounding the vehicle based on the BEV image;• predicting (650), by a multi-view trajectory model, one or more future trajectories for each of the one or more objects by rendering multiple views of each object from the 3D scene and extracting features from each view;• updating (660) one or more values of the vehicle sensor signals based on the one or more predicted trajectories; and• recommending (670), by a viewpoint recommendation model, at least one viewpoint based on the updated vehicle sensor signals values and the BEV image.

2. The method of claim 1 , further comprising rendering a recommended view image corresponding to the at least one viewpoint using the 3D scene.

3. The method of 2, further comprising using the recommended view image to assist in vehicle operation by at least one of: issuing a driver alert, initiating an automated braking or steering maneuver, or modifying a display interface to emphasize a potential hazard.

4. The method of any previous claim, wherein the one or more vehicle sensor signals comprise at least one of: a steering angle signal, a gear position signal, a proximity sensor signal, a turn indicator signal, a wheel direction signal, an acceleration pedal signal, or an ultrasonic sensor signal.202405595485. The method of any previous claim, wherein generating the BEV image comprises applying inverse perspective mapping, IPM, to transform camera images of the plurality of images into a top-down view, or using a deep learning-based BEV generation model trained on annotated top-view datasets to infer spatial layouts from the 3D scene.

6. The method of any previous claim, wherein predicting, by the multi-view trajectory model, the one or more future trajectories comprises:• rendering a plurality of views of each object from the 3D scene using virtual camera parameters centered on the object;• extracting spatial-aware features from each view using a shared convolutional neural network; and• applying a collaborative correlation filter to track object movement across the views.

7. The method of any previous claim, wherein the viewpoint recommendation model comprises a deep learning model selected from the group consisting of a transformer network, a diffusion model, a 3D convolutional neural network, or a long short-term memory, LSTM, network.

8. The method of any previous claim, further comprising:• prior to the step of detecting (640), determining whether any objects are present in the BEV image; and, if no objects are detected, omitting steps (650) through (670) and repeating from the obtaining step (610).

9. The method of any previous claim, wherein updating the one or more values of the vehicle sensor signals comprises:• projecting the predicted future trajectories of the detected objects into a coordinate frame of the vehicle sensors, thereby obtaining projected future trajectories;• computing a projected future distance from the vehicle to the detected objects based on the projected future trajectories; and20240559549• modifying at least one value of the vehicle sensor signals to reflect the projected future distance to the detected objects, wherein the at least one value optionally comprises proximity or ultrasonic sensor signals values.

10. The method of claim 9, wherein generating the three-dimensional scene of surroundings of the vehicle further comprises using the one or more vehicle sensor signals to refine camera pose estimation or spatial alignment of the plurality of images.

11. The method of any previous claim, wherein generating the 3D scene of surroundings of the vehicle based on the plurality of images comprises using a machine learning model, wherein said model comprises a three-dimensional neural radiance field, NeRF, model configured to learn a volumetric representation of the scene from the plurality of images.

12. A system for recommending a viewpoint for vehicle control assistance, the system comprising:• one or more cameras configured to capture a plurality of images of surroundings of a vehicle;• one or more vehicle sensors configured to generate vehicle sensor signals;• a module configured to generate a 3D scene based on the plurality of images;• a BEV generation model configured to generate a BEV image from the 3D scene;• an object detection model configured to detect one or more objects in the BEV image;• a multi-view trajectory model configured to predict future trajectories of the one or more objects;• a trajectory aggregation module configured to update one or more values of the vehicle sensor signals based on the predicted trajectories; and• a viewpoint recommendation model configured to recommend at least one viewpoint based on the updated vehicle sensor signals and the BEV image.2024055955013. A computer program comprising instructions which, when the computer program is executed by a computer, cause the computer to carry out a method of any one of claims 1 to 11 .

14. A computer-readable data carrier having stored thereon the computer program of claim 13.

15. An apparatus (700) comprising: one or more interfaces (710) for communication; a memory (730); and a data processing circuit (720) configured to carry out the method of any one of claims 1 to 11 .

Images