Managing vehicle navigation via machine learning and vehicle-to-vehicle (V2V) communication

IN597967BActive Publication Date: 2026-08-05INDIAN INST OF TECH MADRAS
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
IN202541075861
Authority / Receiving Office
IN · IN
Patent Type
Patents
Current Assignee / Owner
Filing Date
2025-08-08
Publication Date
2026-08-05
Estimated Expiration
2045-08-08
Patent Text Reader

Abstract

ABSTRACT MANAGING VEHICLE NAVIGATION VIA MACHINE LEARNING AND VEHICLE-TO-VEHICLE (V2V) COMMUNICATION The present subject matter discloses techniques for managing vehicle navigation via machine learning and V2V communication. The 5 method involves acquiring real-time visual data (224) of the external environment from a monocular camera. A Machine Learning Model Ensemble (MLME) (214) processes this data to generate distinct environmental outputs: identifying roadway elements, determining control conditions, and estimating traffic conditions. These three outputs are then 10 integrated into a comprehensive unified output. This unified output, indicative of the external environment, is provided to the vehicle's navigation dashboard for real-time driver assistance and broadcast to other externally connected vehicles via the V2V communication network. Additionally, unified outputs from external connected vehicles are received via the V2V 15 network and subsequently provided to the vehicle's navigation dashboard. <>
Need to check novelty before this filing date? Find Prior Art

Description

TECHNICAL FIELD

[0001] The present invention relates, in general, to driver-assistancesystems and autonomous driving systems, and particularly but notexclusively, to techniques for managing vehicle navigation via machinelearning and vehicle-to-vehicle (V2V) communication, within a connectedvehicle ecosystem, which facilitates communication and data exchangeamong vehicles creating an interconnected environment.BACKGROUND

[0002] In modern times, the landscape of vehicular transportation isundergoing a substantive transformation, driven by a relentless pursuit ofenhanced safety and intelligence. Advancements in various technologicaldomains have significantly contributed to making driving experiencesprogressively safer and smarter. These innovations encompass a widearray of sensors, sophisticated processing units, and highly refinedalgorithms that works together to perceive, interpret, and react to thedynamic environment surrounding a vehicle.

[0003] One such technology is Advanced Driver-Assistance Systems(ADAS). They are also a key underlying technology in emergingautonomous vehicles. Conventional ADASs are primarily based on lightdetection and ranging (lidar), radio detection and ranging (radar), and otheradvanced-sensing technologies designed to assist drivers in various drivingtasks, aiming to improve safety and convenience. These systems typicallyutilize a large number of sensors to gather data about the vehicle'ssurroundings, offering functionalities such as, but not limited to adaptivecruise control, lane keeping assist, and automatic emergency braking,thereby contributing to a more secure and intuitive driving experience.

[0004] Complementing the advancements in individual vehicle ADAS,the emergence of a connected vehicle ecosystem represents anothertransformative leap in vehicular safety and intelligence. In this, vehicles areno longer isolated entities but become integral nodes within a dynamicnetwork. This connectivity enables vehicles to communicate with eachother, with infrastructure, and even with pedestrians and other road users.For instance, if one vehicle senses a sudden hazard, such as a largepothole, an unexpected obstruction, or an icy patch on the road, its onboardsensors and ADAS can immediately detect this anomaly, and instead ofkeeping this information localized, the connected vehicle can give this realtimeenvironmental data to other vehicles in its vicinity or along its projectedpath. This proactive sharing of localized hazard information allows nearbyvehicles to be alerted well in advance, thereby enhancing collective roadsafety and traffic efficiency.

[0005] Furthermore, the real-time environmental data exchanged withinthis connected vehicle ecosystem serves as a fundamental building blockfor the development and operation of autonomous vehicles. For self-drivingcars, this shared information encompassing roadway elements, controlconditions, traffic dynamics, and detected hazards provides a richer andmore comprehensive understanding of the driving environment. Thisenhanced environmental awareness is crucial for robust perception,enabling autonomous vehicles to accurately map their surroundings, predictthe behavior of other road users, and identify potential risks far beyond theirimmediate line of sight, thereby representing a critical use case for thisadvanced communication paradigm.SUMMARY

[0006] Various embodiments of systems and methods for managingvehicle navigation within a connected vehicle ecosystem are describedherein.

[0007] The details of some embodiments of the invention described inthis specification are set forth in the accompanying drawings and thedescription below. Other features, aspects, and advantages of the inventionwill become apparent from the description, the drawings, and the claims.

[0008] This summary is not intended to identify essential features of theclaimed subject matter nor is it intended for use in determining or limitingthe scope of the claimed subject matter.

[0009] In accordance with an example embodiment of the presentsubject matter, a system for managing vehicle navigation within a connectedvehicle ecosystem is provided. The system comprises a processor and adata collection module coupled to the processor. The data collection moduleis to acquire real-time visual data of an external environment surroundingthe vehicle, from a monocular camera. The system also comprises a dataprocessing module, coupled to the processor. The data processing moduleis to process the visual data, using a machine learning model ensemble(MLME), to identify one or more roadway elements to generate a first set ofenvironmental data output, the roadway elements indicative of obstructionson a path that the vehicle is traversing. The data processing module is toalso identify one or more control conditions to generate a set of second setof environmental data output. The control conditions is to govern movementof the vehicle relative to a location of the vehicle and are identifiable basedon at least on visual indicator present at the location. Additionally, the dataprocessing module is to identify traffic conditions to generate a set of thirdenvironmental data output. Further, the system comprises a data fusionmodule coupled to the processor. The data fusion module is to integrate thefirst set of environmental data output, the second set of environmental dataoutput, and the third set of environmental data output to generate a unifiedoutput. The unified output is indicative of the external environmentalconditions, including one or more of identified roadway elements, controlconditions, and estimated traffic conditions. The system also comprises acommunication module coupled to the processor. The communicationmodule is to provide the unified output to a vehicle's navigation dashboardfor real-time driver assistance, wherein the unified output is to indicate theexternal environmental conditions along with their respective attributes andspatial relationships relative to the vehicle. The communication module is tobroadcast the unified output to one or more other connected vehicles via avehicle-to-vehicle (V2V) communication network, when the one or moreexternal connected vehicles are within a predefined communication range.The communication module is to further receive unified outputs from one ormore external connected vehicles via the V2V communication network, andbroadcast the unified outputs to the vehicle's navigation dashboard, whenthe vehicle is within the predefined communication range with the one ormore external connected vehicles.

[0010] In accordance with another embodiment of the present subjectmatter, a method for managing vehicle navigation within a connectedvehicle ecosystem is provided. The method includes acquiring real-timevisual data of an external environment surrounding a vehicle from amonocular camera. Once visual data is acquired, one or more roadwayelements are identified by a machine learning model ensemble (MLME), byprocessing the visual data, to generate a first set of environmental dataoutput, wherein a roadway element is a physical feature comprising asurface condition, or obstruction present on a road surface ahead of thevehicle. Further, one or more control conditions are determined, by theMLME, based on processing the visual data, to generate a second set ofenvironmental data output, wherein the control conditions comprise a visualindicator or regulatory device on the road surface to govern movement ofthe vehicle. Further, traffic conditions are estimated, by the MLME, basedon processing the visual data, to generate a third set of environmental dataoutput. Subsequently, the first, second, and third sets of environmental dataoutput are integrated to generate a unified output. This unified output is thenprovided to the vehicle's navigation dashboard, the unified data beingindicative of the one or more of identified roadway elements, controlconditions, and estimated traffic conditions, along with their respectiveattributes and spatial relationships relative to the vehicle. Furthermore, thisunified output is broadcast to other external connected vehicles via aVehicle-to-Vehicle (V2V) communication network. Concurrently, the unifiedoutputs from these external connected vehicles may also be receivedthrough the same V2V communication network.

[0011] The techniques for managing vehicle navigation described hereinimproves both the efficiency and accuracy of environmental understandingin vehicular operations. Leveraging a MLME to process real-time visual dataand identify distinct roadway elements, control conditions, and trafficconditions, followed by the integration of these diverse data streams into aunified output, significantly enhances situational awareness for both humandrivers and autonomous systems. Additionally, the broadcasting this unifiedoutput to other connected vehicles provides a collective awareness acrossthe transportation network, which supports large-scale traffic optimization,and enhances overall road safety, all while benefiting from the practicaladvantages of a cost-effective monocular camera for easier implementationand maintenance.

[0012] Additional features and advantages are realized through theconcepts of the present invention. Other embodiments and aspects of theinvention are described in detail herein and are considered a part of theclaimed invention.BRIEF DESCRIPTION OF DRAWINGS

[0013] The detailed description is described with reference to theaccompanying figures. In the figures, the left-most digit(s) of a referencenumber identifies the figure in which the reference number first appears.The same numbers are used throughout the drawings to reference featuresand components.

[0014] Figure 1 illustrates a network environment for implementingexample techniques for managing vehicle navigation within a connectedvehicle ecosystem, in accordance with an example implementation of thepresent subject matter.

[0015] Figure 2 illustrates a system managing vehicle navigation withina connected vehicle ecosystem, in accordance with an exampleimplementation of the present subject matter.

[0016] Figures 3A-3C illustrate a visual data of a crack at various stagesof a crack detection process, in accordance with an exampleimplementation of the present subject matter.

[0017] Figure 4 illustrates a method of managing vehicle navigationwithin a connected vehicle ecosystem, in accordance with an exampleimplementation of the present subject matter.

[0018] In the figures, the left-most digits of a reference number identifythe figure in which the reference number first appears. The same numbersare used throughout the drawings to reference like features andcomponents.DETAILED DESCRIPTION

[0019] The present subject matter relates to techniques for managingvehicle navigation within a connected vehicle ecosystem. These techniquesmay have various applications, including but not limited to providing real15time driver assistance and facilitating vehicle-to-vehicle (V2V)communication.

[0020] While traditional ADAS have undeniably guided in a new era ofautomotive safety and convenience, their widespread implementation andfull potential are significantly hampered by several limitations. A primaryconcern revolves around the substantial cost associated with many existingADAS implementations. These systems usually rely on an array of highend,specialized sensors such as high-resolution LiDAR units, radar arrays,and advanced stereo cameras, each contributing significantly to the overallbill of materials. Beyond the raw sensor costs, the integration of thesetechnologies, coupled with complex proprietary hardware and highlyspecialized processing units, further escalates manufacturing expenses.This high-cost barrier often restricts the deployment of comprehensiveADAS features only to premium vehicle segments, making theminaccessible to a larger demographic of drivers and limiting their pervasiveadoption across the entire vehicle fleet.

[0021] Furthermore, the performance of conventional ADAS solutionscan exhibit considerable variability, particularly when operated within thehighly dynamic nature of real-world driving environments. Traditionalsystems may struggle with the nuanced interpretation of diverse trafficconditions, such as highly congested urban settings with unpredictablepedestrian and cyclist movements, or rapidly changing highway conditions.Their effectiveness can be compromised by atypical or poorly maintainedroad elements, including faded lane markings, unmarked constructionzones, or unpaved surfaces. Ambiguous or partially obscured traffic signsand signals, which are common occurrences in various global regions,particularly in developing regions such as the Indian sub-continent, alsopose significant challenges, leading to potential misinterpretations ordelayed responses that can undermine safety.

[0022] Another critical limitation arises from the inherent data processingcapabilities and the scope of environmental understanding in manyconventional ADAS. While adapting to specific tasks like adaptive cruisecontrol or basic lane keeping, they often lack a holistic, real-timeunderstanding of the broader traffic ecosystem and subtle roadwaynuances. They may not efficiently process and integrate data from varioussources to form a comprehensive situational awareness that goes beyondimmediate collision avoidance. For instance, anticipating the flow of trafficbased on aggregated data, identifying potential hazards beyond direct lineof-sight, or understanding complex intersections with multi-modal trafficremains a significant hurdle. This limited scope of perception andinterpretation can result in reactive rather than truly proactive safetyinterventions, reducing their overall efficacy in preventing complexaccidents.

[0023] Therefore, there is a need for a system that overcomes thelimitations of current ADAS technologies by providing a cost-effectivesolution. Such a system must be capable of automatically identifying,processing, and disseminating a comprehensive array of real-time roadway,traffic, and control conditions, thereby enhancing vehicle safety, all withoutincurring the high costs that currently restricts broader adoption.

[0024] In accordance with the present subject matter, a system formanaging vehicle navigation within a connected vehicle ecosystem isprovided. In example implementations, the system provides to solve theabove-mentioned problems.

[0025] In accordance with an implementation, the system comprises aprocessor and a data collection module coupled to the processor. The datacollection module is to acquire real-time visual data of an externalenvironment surrounding the vehicle, from a monocular camera. Thesystem also comprises a data processing module, coupled to the processor.The data processing module is to process the visual data, using a machinelearning model ensemble (MLME), to identify one or more roadwayelements to generate a first set of environmental data output, the roadwayelements indicative of obstructions on a path that the vehicle is traversing.The data processing module is to also identify one or more controlconditions to generate a set of second set of environmental data output. Thecontrol conditions is to govern movement of the vehicle relative to a locationof the vehicle and are identifiable based on at least on visual indicatorpresent at the location. Additionally, the data processing module is toidentify traffic conditions to generate a set of third environmental dataoutput. The system also comprises a data fusion module coupled to theprocessor. The data fusion module is to integrate the first set ofenvironmental data output, the second set of environmental data output,and the third set of environmental data output to generate a unified output,the unified output is indicative of the external environmental conditions,including one or more of identified roadway elements, control conditions,and estimated traffic conditions. The system also comprises acommunication module coupled to the processor. The communicationmodule is to provide the unified output to a vehicle's navigation dashboardfor real-time driver assistance, wherein the unified output is to indicate theexternal environmental conditions along with their respective attributes andspatial relationships relative to the vehicle and broadcast the unified outputto one or more other connected vehicles via a vehicular communicationnetwork, when the one or more external connected vehicles are within apredefined communication range. The communication module is to furtherreceive unified outputs from one or more external connected vehicles viathe V2V communication network, and provide the unified outputs to thevehicle's navigation dashboard, when the vehicle is within the predefinedcommunication range with the one or more external connected vehicles.

[0026] By utilizing a machine learning model ensemble (MLME) toprocess real-time visual data and identify distinct environmental conditionsand their attributes, the system can capture and leverage the uniquecharacteristics and patterns associated with various driving scenarios. Thisapproach allows for a more nuanced and accurate understanding of theexternal environment across diverse traffic paradigms, road elements, andcontrol conditions, thereby helping the vehicle to adapt to rapidly changingor complex real-world driving situations. Furthermore, through theintegration of distinct environmental data outputs into a unified output, thesystem can effectively adapt to real-time changes in roadway, control, andtraffic conditions. This adaptive capability significantly enhances thesystem's resilience to environmental variations and its ability to maintainaccurate perception over time, even in the face of dynamic drivingsituations. Additionally, the present system allows for the continuous andeffective utilization of real-time visual data, addressing the shortcoming ofconventional systems that may underutilize or discard valuable currentenvironmental information.

[0027] Further, by using a single monocular camera, the system offerssubstantial economic benefits, as it significantly reduces hardware costscompared to multi-sensor ADAS. This design also makes the system easierto implement and maintain, simplifying installation and calibration, leadingto faster deployment and reduced long-term operational expenses. Thiscost-effectiveness and ease of use facilitate wider adoption of advancedsafety features across the vehicular landscape.

[0028] The above-mentioned implementations are further describedherein with reference to the accompanying figures. It should be noted thatthe description and figures relate to exemplary implementations and shouldnot be construed as a limitation to the present subject matter. It is also to beunderstood that various arrangements may be devised that, although notexplicitly described or shown herein, embody the principles of the presentsubject matter. Moreover, all statements herein reciting principles, aspects,and embodiments of the present subject matter, as well as specificexamples, are intended to encompass equivalents thereof.

[0029] Figure 1 illustrates a network environment 100 for implementingexample techniques for managing vehicle navigation within a connectedvehicle ecosystem, in accordance with an example implementation of thepresent subject matter.

[0030] Accurate and comprehensive understanding of the externalvehicular environment is essential for various stakeholders within theconnected vehicle and autonomous driving landscape. For human drivers,such insights provide crucial real-time assistance, enhancing situationalawareness and enabling more informed and safer driving decisions. Forautonomous driving systems, precise environmental data is fundamental forrobust perception, planning, and control, facilitating reliable and safenavigation. Furthermore, sharing enriched environmental informationamong connected vehicles and with traffic management infrastructure cansignificantly optimize traffic flow, reduce congestion, and prevent accidentsacross the entire transportation network. This collective awarenesscontributes to overall road safety and efficiency, fostering the developmentof intelligent transportation systems.

[0031] In accordance with an example of the present subject matter, thenetwork environment comprises a vehicle 102, the vehicle being situatedwithin a connected vehicle ecosystem comprising a plurality of othervehicles. Examples of the vehicles may include, but may not be limited to,two-wheelers, three-wheelers, four-wheeled vehicles, and heavy vehiclessuch as, but not limited to, buses and trucks. In an example, the vehicleecosystem may enable communication between the plurality of vehicles viavarious network connections, such as, but not limited to, Vehicle-to-Vehicle(V2V).

[0032] In an example, the connected vehicle ecosystem may enablesophisticated communication between the plurality of vehicles via variousnetwork connections, extending beyond just Vehicle-to-Vehicle (V2V) toinclude Vehicle-to-Infrastructure (V2I) and Vehicle-to-Everything (V2X)communication. This allows for the real-time exchange of critical data, suchas speed, location, braking events, and detected road hazards. In anexample, the communication may be facilitated through establishedprotocols designed for vehicular environments, such as Dedicated Short-Range Communication (DSRC), based on the IEEE 802.11p standard, maybe utilized for direct, low-latency communication between vehicles andbetween vehicles and roadside units. Alternatively, or in conjunction,Cellular Vehicle-to-Everything (C-V2X) technology, that utilizes cellularnetworks (e.g., 4G LTE or 5G), may provide communication capabilities thatinclude direct communication mode (PC5 interface) and network-basedcommunication mode. This protocol may enable a dynamic information flow,allowing vehicles to share their perceived environmental conditions, therebycreating a collective awareness that significantly enhances safety,optimizes traffic flow, and supports cooperative driving maneuvers withinthe ecosystem.

[0033] In an example, the vehicle 102 comprises a system 104, alsoreferred to as "driving assistance system 104". The driving assistancesystem 104 provides enhanced real-time awareness of the externalenvironment, thereby significantly improving driver safety and overalloperational efficiency.

[0034] In an example, the driving assistance system 104 may be anElectronic Control Unit (ECU) (not illustrated) of the vehicle 104 or a partthereof. As will be understood by one skilled in the art, the ECU is a systemof a vehicle that controls and manages one or more of its electrical systemsor subsystems, such as, but not limited to, engine management, braking,infotainment systems, and safety features. In an example, the drivingassistance system 104 may be further connected to one or more remoteservers via a network (not illustrated) to enable the system 104 to accessand integrate external data sources for comprehensive environmentalunderstanding and operational enhancement. For example, the system 104may connect to remote servers to obtain real-time location data for precisepositioning, detailed weather information (e.g., precipitation, temperature,visibility) that can influence road conditions, and updated mapping data. Thedriving assistance system 104 may comprise one or more processors forexecuting the process for managing vehicle navigation.

[0035] In an example, the processor may be implemented asmicroprocessors, microcomputers, microcontrollers, digital signalprocessors, central processing units, state machines, logic circuitries,and / or any devices that manipulate signals based on operationalinstructions. The driving assistance system 104 may comprise a memoryfor storing the instructions executable by the one or more processors. Thememory may include any computer-readable medium known in the artincluding, for example, volatile memory (e.g., RAM), and / or non-volatilememory (e.g., EPROM, flash memory, etc.). The memory may also be anexternal memory unit, such as a flash drive, a compact disk drive, anexternal hard disk drive, or the like.

[0036] In an example, the vehicle 104 may comprise a monocularcamera (not shown) that is configured to capture real-time visual data of anexternal environment surrounding the vehicle. This monocular camera,being a single-lens optical sensor, offers a cost-effective andcomputationally efficient means of acquiring rich visual information. In anexample, the camera may be positioned, for example, at a center of thewindshield of the vehicle 104, typically behind the rearview mirror. Thisplacement may provide an unobstructed and expansive view of the vehicle'ssurroundings, crucially including the path directly ahead of the vehicle 104,as well as adjacent lanes, road shoulders, and overhead structures. Inoperation, the real-time visual data captured by the camera may comprisea continuous stream of images or video frames, providing dynamicinformation about the changing environment.

[0037] In an example, the communication between the system 104 andthe camera may be established through a wired or wireless connection. Insome embodiments, the communication may be facilitated by standardvehicle communication protocols that may include but are not limited to aController Area Network (CAN) bus, and a Local Interconnect Network(LIN), utilizing a high-speed data interface to access the raw or pre15processed video feed, for subsequent data analysis.

[0038] In operation, the system 104 acquires real-time visual data of theexternal environment surrounding the vehicle 102 from the monocularcamera. Once the visual data is acquired, the system 104 analyses theexternal environment, using a MLME, for identification of variouscomponents to provide for managing vehicle navigation in the vehicle.

[0039] In an embodiment, the system 104 identifies one or moreroadway elements, the roadway elements indicative of obstructions on apath that the vehicle 102 is traversing. In an example, the roadway elementsmay include, but may not be limited to, potholes, speed bumps, and differenttype of cracks including longitudinal, transverse, or alligator cracks. Theseelements may represent immediate physical anomalies or structural defectson the road surface that can impact vehicle performance, safety, or drivercomfort.

[0040] In an example, the system 104 may also identify other roadwayelements such as road debris, manholes, and gratings. Additionally, thesystem 104 may identify various surface characteristics like the presence ofwater puddles, ice patches, gravel, or changes in road surface type, forinstance, from asphalt to concrete. This identification of roadway elementsand surface characteristics may provide a detailed understanding of thedriving environment.

[0041] In an example, the system 104 also identifies one or more controlconditions, wherein the control conditions govern movement of the vehicle104 relative to its location or, in other words, dictate conditions formovement of the vehicle 104 at a location, and are identifiable based on atleast on visual indicator present at the location. These control conditionsmay include, but are not limited to traffic signs. In an example, the system104 identifies traffic signs by processing the visual data to determine theirposition, classifying them by type such as regulatory, warning, orinformational, and by ascertaining their associated meaning. This mayfurther include traffic signals, by identifying their current state, such as theirilluminated color (e.g., red, yellow, or green), flashing status, or specificpatterns. Other regulatory indicators like temporary construction zone signs,school zone indicators, and lane usage signs are also identified.

[0042] Furthermore, the system 104 also identifies traffic conditions,which include traffic density and the presence and attributes of other roadusers, such as other vehicles, pedestrians, and cyclists, along with theirrelative positions, speeds, and trajectories. This may also encompassincidents like accidents, stalled vehicles, or unexpected obstructionsaffecting traffic movement.

[0043] Based on the one or more external roadway elements, controlconditions, and traffic conditions identified, the system 104 generatesdistinct datasets for each category. Specifically, a first set of environmentaldata may be generated for the identified roadway elements, comprisingattributes such as, but not limited to, the precise type of element (e.g.,pothole, longitudinal crack, speed bump, solid lane line), its exact location(e.g., coordinates relative to the vehicle), its dimensions (e.g., depth, width,length for a pothole; length and width for a crack), and its severity orcondition. Similarly, a second set of environmental data may be generatedto indicate the identified control conditions, detailing attributes like the typeof traffic sign, its regulatory or informational meaning, its position on theroadway, and the current state of traffic signals. Furthermore, a third set ofenvironmental data may be representative of the identified traffic conditions,including metrics such as traffic density, average vehicle speeds within adefined zone, and the presence, types, and trajectories of other road users(e.g., cars, trucks, pedestrians, cyclists). In an example, each of thesegenerated datasets may also include the respective attributes and precisespatial relationships of these identified elements, conditions, and trafficpatterns relative to the vehicle 102, providing a comprehensive and dynamicunderstanding of the immediate external environment.

[0044] In an example, the system 104 comprises a fusion module 106configured to integrate the first set of environmental data output, the secondset of environmental data output, and the third set of environmental dataoutput to generate a unified output. The unified output is indicative of theexternal environmental conditions, including one or more of identifiedroadway elements, control conditions, and estimated traffic conditions. Inan example, this integration process may involves correlating andsynthesizing diverse data streams, which may originate from differentmachine learning models and may represent varied aspects of the externalenvironment, into a cohesive and comprehensive representation. Theresulting unified output may be a holistic and real-time representationindicative of the complete external environmental conditions, enhancing theaccuracy and reliability of driver assistance and autonomous navigationfunctions by presenting a complete picture of the vehicle's immediatesurroundings.

[0045] In an example, the system 104 may provide this unified output tothe driver of the vehicle 102, through the vehicle's navigation dashboard, forreal-time driver assistance. In an example, this unified output may bepresented in various formats, including visual displays overlaid on anavigational map or as audible notifications. For instance, visual cues onthe map may highlight detected roadway elements like potholes, display thecurrent state of traffic signals, or indicate areas of high traffic density. In anexample, the unified output may be provided as audible alerts which maybe used to warn the driver of immediate hazards or critical changes inconditions.

[0046] Simultaneously, the system 104 may also broadcast this unifiedoutput to other connected vehicles within the connected vehicle ecosystemvia a vehicular communication network, such as, but not limited to,Dedicated Short-Range Communications (DSRC) or Cellular Vehicle-to-Everything (C-V2X) technologies. This offers an advantage by enabling acollective awareness among vehicles, facilitating cooperative maneuvers,early warning dissemination for hazards, and optimized traffic flow acrossthe network, thereby contributing to a safer and more efficient transportationsystem for all participants. For an explanation of the implementation andoperation of the system 104 to perform the function of enhancing theoperation of the vehicle 102, reference is made to Figure 2.

[0047] Figure 2 illustrates the driving assistance system 104 formanaging vehicle navigation within a connected vehicle ecosystem,according to example implementation of the present subject matter. Thedriving assistance system 104 implements a technical process for real-timeenvironmental awareness within a connected vehicle ecosystem. Thesystem 104 as described herein addresses the technical limitations ofconventional driving assistance systems, which are often expensive due totheir reliance on multiple, high-cost sensor modalities and complexprocessing architectures.

[0048] In an example implementation, the system 104 may include atleast one processor 202 and a memory 204 coupled to the processor 202.In an example, the processor 202 may be implemented as microprocessors,microcomputers, microcontrollers, digital signal processors, centralprocessing units, state machines, logic circuitries, and / or any devices thatmanipulate signals based on operational instructions. The memory 204 mayinclude any computer-readable medium known in the art including, forexample, volatile memory (e.g., RAM), and / or non-volatile memory (e.g.,EPROM, flash memory, etc.). The memory 204 may also be an externalmemory unit, such as a flash drive, a compact disk drive, an external harddisk drive, or the like.

[0049] As also depicted in Figure 2, in an example implementation,interface(s) 206 may be coupled to the processor 202. The interface(s) 206may include a variety of software and hardware interfaces that allowinteraction of the driving assistance system 104 with other communicationand computing devices, such as network entities, external repositories, andperipheral devices. The interface(s) 206 may also enable the coupling ofcomponents of the system 104 with each other.

[0050] The driving assistance system 104 may also comprise module(s)208 and data 222 coupled to the processor 202. In one example, themodule(s) 208 and data 222 may reside in memory 204.

[0051] In an example, the data 222 may comprise visual data 224, firstoutput data 226, second output data 228, third output data 230, unifiedoutput data 232, and other data 234. The module(s) 208 may includeroutines, programs, objects, components, data structures, and the like,which perform particular tasks or implement particular abstract data types.The module(s) 208 may further include modules that supplementapplications on the driving assistance system 104 for enhancing vehicleoperations, for example, modules of an operating system. The data 222serves, amongst other things, as a repository for storing data that may befetched, processed, received, or generated by one or more of the module(s)208. The module(s) 208 may include a data collection module 210, a dataprocessing module 212, a data fusion module 216, a communicationmodule 218, and other module(s) 220. The other module(s) 220 may includeprograms or coded instructions that supplement applications and functions,for example, programs in the operating system of the driving assistancesystem 104. In accordance with an example of the present subject matter,the data processing module 212 may incorporate a Machine Learning ModelEnsemble (MLME) 214.

[0052] To initiate the process, the driving assistance system 104 mayacquire visual data comprising the real-time visual data of the externalenvironment surrounding the vehicle captured using the monocular camera(not shown) installed on the vehicle 102. The monocular camera isstrategically positioned in the vehicle 102. In an example, this camera maybe a dedicated automotive-grade camera, optimized for robust performancein varying lighting and weather conditions. Alternatively, the camera couldbe a commercially available dashcam camera, adapted to provide thenecessary real-time video feed to the data collection module 210. In anexample, the camera may be positioned to provide a wide field of view,typically encompassing the road ahead, adjacent lanes, and surroundingareas, to ensure comprehensive environmental data capture.

[0053] The visual data captured by the camera may be in the form of acontinuous stream of image frames or video. In an example, this visualinformation data may be stored as visual data 224 within the data 222. Thisvisual data 224 serves as the fundamental input for the system'senvironmental understanding and situational awareness.

[0054] The data collection module 210 of the system 104 retrieves thevisual data 224 to acquire the real-time visual data of an externalenvironment surrounding the vehicle as captured by the camera. The dataprocessing module 212 of the system 104 is operable to process the visualdata, using the MLME 214. In an example, the data processing module 212may implement a plurality of trained artificial intelligence (AI) models as apart of the MLME 214 to process the visual data to identify one or moreroadway elements, control conditions and traffic conditions.

[0055] In an example, the one or more roadway elements may includevarious physical features and surface conditions crucial for safe vehicleoperation. The data processing module 212, utilizing the MLME 214, maybe configured to identify these elements. In an example, the MLME 214 maycomprise a first model 214-1 to detect hazardous obstructions such aspotholes and speed bumps, which can significantly impact vehicle stabilityand tire integrity. Furthermore, a second model 214-2 within the MLME 214may be trained to identify diverse types of cracks on the path. In an example,the first model 214-1 and the second model 214-2 may each comprise oneor more individual models. Furthermore, each of the first model 214-1 andsecond model 214-2 may be implemented as a single trained ArtificialIntelligence (AI) model or as a combination of multiple trained AI modelsworking in conjunction to perform their specific identification tasks.

[0056] In an example, the MLME 214 may also comprise other models214-8 to detect other roadway elements such as road debris, manholes, andgratings. These other models 214-8 may identify dynamic surfacecharacteristics, such as the presence of water puddles, ice patches, orgravel, which may directly affect tire traction and pose potential risks for thevehicle 102. In an example, the models may also detect transitions of thevehicle 102 from one road surface type to another, such as from asphalt toconcrete. In an example, once the one or more roadway elements areidentified and characterized by the respective models, a first set ofenvironmental output may be generated.

[0057] In an example, the first set of environmental data output maycomprise detailed information regarding the identified roadway elements.This includes the classification of each element, such as "pothole,""longitudinal crack," "speed bump", or "gravel patch." For each identifiedelement, the output may further specify its spatial coordinates relative to thevehicle, its dimensions (e.g., depth and width for a pothole, length andorientation for a crack), and its current condition or severity. Thiscomprehensive output may enable the system 104 to understand thephysical state and layout of the road ahead, crucial for navigation andsafety. This first set of environmental output may then be stored in the data222 as first output data 226.

[0058] In an example, the first model 214-1 from the MLME 214 foridentifying potholes and speed bumps may utilize a compact single-shotdetector. This detector may be trained on an extensive augmented roadwayimagery dataset to achieve high accuracy and efficiency in identifying thesecritical obstructions. Once identified, each detected anomaly, whether apothole, or speed bump may be characterized by its specific type, itsestimated location, and a calculated severity score, all of which are thenfed into dynamic hazard maps for comprehensive environmentalrepresentation.

[0059] In an example implementation, once the roadway element, suchas a pothole or speed bump is identified, using the first model 214-1, aheadof the vehicle 102, its location may be estimated by projecting a short offsetahead of the vehicle's current GPS position, based on the context of thedetection within the visual data. In an example, each detected anomaly,characterized by its specific type, such as pothole or speed bump, itsprecise estimated location in vehicle coordinates, and a calculated severityscore, is then fed into dynamic hazard maps for comprehensiveenvironmental representation. Upon detection of such an anomaly, thesystem immediately triggers an alert for the driver of the vehicle 102. Thisalert may be presented through various intuitive modalities, including visualcues on the navigation dashboard highlighting the hazard's estimatedlocation and audible warnings. Concurrently, the system 104 may broadcastthis critical information, including the type of anomaly, its location on themap, and severity to other connected vehicles approaching these markedhazard hotspots via the V2V communication network. This allows nearbyvehicles to be informed well in advance, enabling their drivers orautonomous systems to take timely evasive actions, adjust speed, orreroute, thereby significantly enhancing collective road safety andpreventing potential accidents across the connected vehicle ecosystem.

[0060] The training of the first model 214-1 within the MLME 214 may beperformed using selected parameters and datasets to ensure optimalperformance. For instance, the first model 214-1 for pothole and speedbump detection may be achieved using a deep learning-based objectdetection model, such as, but not limited to YOLOv8 (YOLO: You Only LookOnce) model. This model is trained on a custom open dataset comprising6,442 total images, each meticulously annotated to highlight potholes andspeed bumps. The training process for this model utilizes a batch size of 16,a learning rate of 0.01, and is conducted over 50 epochs, as further detailedin Table 1 below.Training parameter ValueModel YOLOv8Dataset Custom open dataset with 6,442total images annotated for potholesand speed bumpsBatch size 16Learning rate 0.01Number of epochs 50Table: 1 (Pothole and Speed Bump Detection)

[0061] In an example, the second model 214-2 within the MLME 214may be trained to identify cracks on the path. For this, the second model214-2 may employ techniques such as, but not limited to, adaptivethresholding and morphological processing. Adaptive thresholding allowsthe model to dynamically adjust the pixel intensity threshold based on localimage characteristics, effectively separating crack patterns from varyingbackground lighting and road textures. Morphological processing, includingoperations like erosion and dilation, further refines the segmented crackregions by removing noise, filling small gaps, and enhancing theconnectivity of crack networks, thereby making the identification andisolation of crack patterns from the raw visual data highly effective androbust.

[0062] In an example, the process of identifying cracks on the path fromthe acquired visual data may involves a series of image processing steps toeffectively segment and isolate crack patterns. For this explanation,reference is made to Figures 3A to 3C.

[0063] Initially, upon receiving an image of a crack, as depicted in Figure3A showing an original image, the second model 214-2 may performIntensity Normalization & Pixel Saturation, as illustrated in Figure 3B. Thisstep adjusts the brightness of the image by applying a normalization factor,for instance, by multiplying darker images by a factor:Factor = μ μ + 10-6where μ represents the mean intensity of the image. Thisenhances the visibility of features in low-light conditions. Concurrently, pixelsaturation is applied to prevent pixels from becoming excessively bright,ensuring that normalized pixel values, denoted as Normalized(i, j),remainwithin a defined range. Specifically:Normalized(i, j)= {Normalized(i, j) if Normalized(i, j) ≤ Saturation ThresholdSaturation Threshold(i, j) if Normalized(i, j) ≤ Saturation Threshold

[0064] Following intensity normalization and pixel saturation, theprocessed image may undergo Gaussian Blur. A Gaussian kernel may beapplied over each pixel to smooth the image, effectively reducing noise andminor variations that could interfere with crack detection. A larger sigma (σ)value in the Gaussian kernel, defined by the function:G(x, y) = 1 2πσ2 exp (- x2 + y2 2σ2 )This results in more significant blurring. This smoothing step prepares theimage for subsequent thresholding by emphasizing larger structuralelements, such as cracks, while diminishing irrelevant fine details.

[0065] Finally, to segment the cracks from the background, the secondmodel 214-2 may then apply 'Otsu's Thresholding', as illustrated in Figure3C. This technique calculates an optimal global threshold value, denoted asT, by minimizing the intra-class variance of the pixel intensities. Thiseffectively divides the pixels into two distinct classes: cracks (foreground)and background. The objective function for minimizing intra-class varianceis given by:σω 2 (T) = ω1(T)σ1 2(T) + ω2(T)σ2 2(T)

[0066] In an example, the output of this thresholding step may be abinary image where cracks are clearly segmented, enabling their preciseidentification and characterization as part of the first set of environmentaldata output.

[0067] In an example, the MLME 214 may also include other AI modelswhich are capable of identifying any foreign objects present on the roadway.These additional models may be trained on diverse data sets specificallycurated to encompass a wide range of foreign objects, such as fallen debris,discarded items, or unexpected obstacles, enabling the system 104 todetect and classify these potential hazards. To achieve this, different typesof foreign objects, including but not limited to debris, construction cones,and other typical obstacles, may be identified and annotated in road images,following a similar methodology of dataset creation for potholes and speedbumps. In an example, the deep learning-based object detection modelused for detecting potholes and speed bumps, may be modified to predictnew object classes in addition to its original targets. Subsequently, thenewly trained model may be fine-tuned using a combined dataset thatintegrates both the original pothole / speed bump annotations and the newlylabeled foreign object data, thereby allowing the system 104 to detect otherforeign objects alongside potholes and speed bumps.

[0068] In an example, the data processing module 212 may be furtherconfigured to identify one or more control conditions pertaining to theexternal environment of the vehicle 102. This may involve the recognitionand interpretation of visual indicators designed to govern vehicle movement.In an example, the data processing module 212 may utilize a third model214-3 to identify lane markings for accurately detecting drivable lanes onthe path, which is essential for lane-keeping assistance and autonomousnavigation. For traffic signs, the data processing module 212, utilizing afourth model 214-4 may identify their position, such as whether they arepole-mounted, gantry-mounted, or roadside placement. The fifth model 214-5 may be capable of classifying them by type, including regulatory signs likestop, yield, or speed limit signs; warning signs indicating curves orpedestrian crossings; and informational signs providing guidance, and mayascertain their content. For traffic signals, the data processing module 212,utilizing a sixth model 214-6 from the MLME 214, may identify their currentoperational state, including their illuminated color, such as red, yellow, orgreen, their flashing status, or specific patterns like arrows for turn lanes.Additionally, other critical regulatory indicators such as temporaryconstruction zone signs, school zone indicators, and lane usage signs mayalso be identified by the relevant models within the MLME 214. In anexample, the third model 214-3, the fourth model 214-4, the fifth model 214-5, and the sixth model 214-6 may each comprise one or more individualmodels. Furthermore, each of these models may be implemented as asingle trained Artificial Intelligence (AI) model or as a combination of multipletrained AI models working in conjunction to perform their specific tasks.

[0069] In an example, once the one or more control conditions areidentified and characterized, a second set of environmental data output maybe generated. In an example, the second set of environmental output maycomprise detailed information regarding the identified control conditions.This may include the precise type of traffic sign or signal, its exact locationrelative to the vehicle, its orientation, and the specific regulatory orinformational meaning it conveys. For traffic signals, it would further specifythe illuminated color, flashing status, and any directional patterns.Additionally, this output may also include information derived from theidentification of lane markings, such as the type of lane line (e.g., solid,dashed, double-solid), its color, and its precise boundaries, all of which arecrucial for defining the vehicle's permissible path and guiding its movement.This data may allow the system 104 to understand and react appropriatelyto all governing directives on the roadway. In an example, this second setof environmental data output may be stored in the data 222 as secondoutput data 228.

[0070] In an example, the third model 214-3 within the MLME 214 maybe trained to identify lane markings on the path. In an example, lanemarkings may be detected using a vision-based segmentation approach,specifically employing a deep learning-based segmentation model, such asbut not limited to YOLOPv2 (panoptic driving-perception) model. Thisapproach may involve processing the visual data to precisely delineate theboundaries of drivable lanes. The YOLOPv2 model may utilize an E-LANencoder, which consists of efficient, multi-scale convolutional blocksdesigned to extract rich semantic features from the input images. Thesefeatures may then be processed by a Spatial Pyramid Pooling module,which aggregates context at different scales, enhancing the model'sunderstanding of the scene. Furthermore, a Feature Pyramid Network(FPN) combined with a Path Aggregation Network (PAN) may be employedto fuse high-level and low-level features, enabling robust performance forboth object detection and segmentation tasks. The model may then utilizespecialized decoder heads, including a dedicated lane-detection head thatoutputs instance-aware lane line masks, allowing the data processingmodule 212 to identify and interpret lane markings accurately, even inchallenging scenarios such as faded, inconsistent, or partially obscuredmarkings, or under varying lighting and weather conditions. Once identified,this information regarding lane boundaries and current lane positioning maybe utilized to guide the host vehicle's lane keeping and maneuvering.

[0071] In an example, the third model 214-3, which may be a YOLOv8-seg or YOLOPv2 (panoptic driving-perception) model, that may be trainedon BDD100K dataset, which provides a rich collection of diverse drivingscenarios. The training parameters for the YOLOPv2 model include alearning rate of 0.001 and a significantly higher number of epochs,specifically 300, to ensure high precision in identifying and delineating lanemarkings for accurate lane detection on the path, as further detailed in Table2 below.Training parameter ValueModel YOLOPv2Dataset BDD100KLearning rate 0.001Number of epochs 300Table: 2 (Lane detection)

[0072] In an example, the fourth model 214-4 of the MLME 214, trainedfor identifying the position of traffic signs on the path, may utilize acombination of deep learning and computer vision (CV) techniques. Thefourth model 214-4 may be a lightweight detection model, and may betrained to efficiently isolate regions of interest within each real-time visualframe. This isolation, or color-based Region of Interest (ROI) proposal, maybe achieved by applying color and shape-based filters on-the-fly.Specifically, each incoming input frame may be converted from Red, Green,Blue (RGB) to Hue, Saturation, Value (HSV) color space, and two distincthue ranges (for example, [0°-12°] and [165°-180°]) may be extracted toisolate red pixels, thereby producing an initial binary mask. This mask maythen undergo morphological cleanup, where an erosion operator may beapplied, followed by dilation, to effectively remove small speckles and closeminor gaps within the identified red regions. Subsequently, contour filteringmay be performed to find connected components in the cleaned mask,discarding any components whose area falls outside a plausible traffic-signsize range (for example, less than 100 pixels or greater than 6000 pixels).This multi-step process may efficiently narrow down the potential locationsof traffic signs within the current frame, generating a set of candidate signregions of interest (ROIs) for further analysis.

[0073] Subsequently, a deep-learning detection network, specificallyoptimized for real-time inference, may be operated on these isolated ROIsto classify common traffic signs, such as speed limits, stop signs, and yieldsigns. In an example, this deep-learning network typically may comprise abackbone and neck component, which may function as a convolutionalfeature extractor to compute multi-scale feature maps from the input ROI. Adetection head may then process these feature maps to predict, per spatialcell and per anchor, bounding-box offsets, objectness scores, and classlogits for the potential signs. Following these predictions, non-maximumsuppression may be applied to prune overlapping bounding boxes, therebyproducing the final, distinct traffic sign detections for the current frame. Forthe fusion of the color-based ROI proposals with the deep-learningdetections, the overlap, specifically the Intersection over Union (IoU), foreach detected box may be computed against the color-segmentation ROImask. If the calculated IoU is below a predetermined threshold, thedetection confidence may be down-weighted (for example, halved), whichmay effectively reduce false positives on non-sign red objects such astaillights or brake lights. The precise position of each identified traffic sign,along with its detected type and associated confidence score, may then beforwarded to the data processing module 212 for integration into the secondset of environmental data output.

[0074] In an example, the fourth model 214-4 may be a deep learningbasedobject detection model, such as, but not limited to YOLOv5 model.This model is trained on the well-established German Traffic Sign DetectionBenchmark (GTSDB) dataset, which comprises 592 samples, providing arobust foundation for recognizing a wide variety of traffic signs. The trainingparameters for present model include a batch size of 16, a learning rate of0.01, and a total of 30 epochs, as further detailed in Table 3 below.Training parameter ValueDetection model (used in conjunctionwith CV techniques for detecting theposition of the traffic sign in theimage)YOLOv5Dataset German Traffic Sign DetectionBenchmark (GTSDB) with 592samplesBatch size 16Learning rate 0.01Number of epochs 30Table: 3 (Traffic Sign Detection-position)

[0075] Once the fourth model 214-4 detects the presence of traffic signsin the image, it works in conjunction with additional computer visiontechniques. These techniques may be to precisely determine the position ofthe traffic sign detected within the image frame, allowing for accurate spatiallocalization relative to the vehicle. This combined approach ensures bothhigh accuracy in identifying traffic signs and precise determination of theirlocation.

[0076] In an example, the fifth model 214-5 from the MLME 214 may betrained to recognize and classify an identified traffic sign as a regulatory,warning, or informational type. In an example, fifth model 214-5 mayincorporate a dedicated processing methodology to localize and classifytraffic signs, enhancing the robustness of the recognition process. This mayinvolve per-frame classification, where each detected bounding boxcorresponding to a potential traffic sign is cropped and subsequently passedthrough a specialized classification network. This may output a SoftMaxprobability distribution over the 11 target sign classes (e.g., various speedlimits, Stop, Yield, etc.), indicating the likelihood of the sign belonging toeach class. To further enhance reliability and reduce erroneous singleframemisclassifications, a temporal consensus may be employed. This mayinvolve maintaining a sliding window of the last 10 classification results foreach tracked traffic sign bounding box. The sign may be declared"identified" only if a consensus is reached, specifically when at least 7 outof those 10 frames agree on the same label, and each of those individualframes has a confidence score equal to or greater than 0.4. This temporalaggregation mechanism may significantly reduce flicker and preventerroneous interpretations arising from transient visual noise or momentaryocclusions, ensuring a more stable and accurate traffic sign classification.

[0077] In an example, the fifth model 214-5, for recognizing andclassifying identified traffic signs, may be a deep learning-based objectdetection model, such as, but not limited to YOLOv8 model. This model isspecifically trained for the classification of detected signs, distinguishingbetween regulatory, warning, and informational types. The training of thismodel is performed on a modified version of the German Traffic SignRecognition Benchmark (GTSRB) dataset, which has been adapted toinclude 2,799 samples with classes particularly relevant to Indian roadconditions. The training process for this model utilizes a batch size of 16, alearning rate of 0.01, and is conducted over 10 epochs, as further detailedin Table 4 below. This focused training ensures the model's high accuracyand efficiency in classifying traffic signs in real-time, thereby contributingcrucial information to the second set of environmental data output.Training parameter ValueRecognition model (for classificationof detected signs)YOLOv8Dataset Modified version of German TrafficSign RecognitionBenchmark (GTSRB) with 2,799samples withclasses relevant to Indian conditionsBatch size 16Learning rate 0.01Number of epochs 10Table: 4 (Traffic Sign Classification)

[0078] In an example, the sixth model 214-6 from the MLME 214 may betrained to identify state of the traffic signal, the state being indicative ofspecific illuminated colour, pattern, or operational status the traffic signaldisplays. This identification may be achieved by using an object detectorcalibrated to perform accurately across varying distances from the vehicle.In an example, the detection range of object detector may be continuouslyvalidated against stopping-sight-distance metrics to ensure that timely alertscan be generated, providing the driver with sufficient reaction time. In anexample, each identified traffic signal's current state and its estimateddistance from the vehicle may be included into the second set ofenvironmental data output, contributing to the comprehensiveunderstanding of control conditions for subsequent transmission anddisplay.

[0079] In an example, the sixth model 214-6, which functions as anobject detector for identifying the state of traffic signals, may be a deeplearning-based object detection model, such as, but not limited to YOLOv5model. This model is trained to recognize the specific illuminated color,pattern, or operational status that a traffic signal displays. The training ofthis model is performed using the Cintav2 dataset, which comprises 2,097samples relevant to traffic signal recognition. The training process for thesixth model 214-6 utilizes a batch size of 8, a learning rate of 0.01, and isconducted over 50 epochs, as further detailed in Table 5 below. Thisensures the model's accuracy and efficiency in real-time traffic signal stateidentification, contributing vital information to the second set ofenvironmental data output.Training parameter ValueModel YOLOv5Dataset Cintav2 dataset with 2,097samplesBatch size 8Learning rate 0.01Number of epochs 50Table: 5 (Traffic Signal Detection)

[0080] Furthermore, the data processing module 212 is also configuredto identify traffic conditions. This may include calculating traffic density,which may be performed by a seventh model 214-7 within the MLME 214,providing an understanding of the concentration of vehicles within a givenarea. In an example, the data processing module 212 may also identify thepresence and attributes of other road users, which encompasses varioustypes of vehicles (e.g., cars, trucks, motorcycles, buses, emergencyvehicles), as well as vulnerable road users like pedestrians and cyclists. Inan example, for each identified road user, their relative positions, currentspeeds, and predicted trajectories may be determined, enabling the hostvehicle 102 to anticipate their movements. In another example, the dataprocessing module 212 may also be configured to detect and characterizeincidents such as accidents, stalled vehicles, or unexpected obstructionsthat impact traffic flow and safety. In an example, the seventh model 214-7may comprise one or more individual models. Furthermore, the seventhmodel 214-7 may be implemented as a single trained Artificial Intelligence(AI) model or as a combination of multiple trained AI models working inconjunction to perform its specific tasks.

[0081] In an example, once the traffic conditions are identified andcharacterized, a third set of environmental data output may be generated.In an example, the third set of environmental data output may compriseinformation regarding the identified traffic conditions. This may include, butnot limited to, traffic density metrics, average speeds of surroundingvehicles, the presence and classification of other road users such as cars,trucks, motorcycles, pedestrians, and cyclists, along with their precise realtimepositions, velocities, and predicted trajectories relative to the hostvehicle. In an example, it may also include the detection andcharacterization of incidents like accidents, stalled vehicles, or unexpectedobstructions, noting their location and impact on traffic flow. This data mayallow the system 104 to understand and react appropriately to differenttraffic conditions on the roadway, enabling proactive adjustments to speed,lane positioning, and overall driving strategy. In an example, the third set ofenvironmental data output may be stored in the data 222 as third outputdata 230.

[0082] In an example, the seventh model 214-7, which may be an objectdetection model, may be trained to identify traffic density by analyzing theoutputs it generates when applied to front-facing road images. While thisobject detection provides bounding box coordinates and class labels forvarious vehicles, a first step may involve filtering out irrelevant detectionsthat do not contribute to the actual forward-moving traffic stream, such asparked vehicles or those traveling in the opposite direction. This may beachieved by defining a specific Region of Interest (RoI) as a trapeziumwithin the image, representing the drivable road area directly ahead of theego vehicle. A rule-based filtering algorithm may then process the boundingbox coordinates to ensure only vehicles falling within this RoI areconsidered. Furthermore, vehicles located too far from the vehicle 102 mayalso be excluded, as their contribution to near-field traffic density is minimal.To accurately estimate distance without heavy computational overhead, alightweight geometric model may be employed. This technique calculatesvehicle distance by utilizing the known physical width of different vehicleclasses, the pixel width of their bounding boxes, and the camera's calibratedintrinsic parameters, specifically its horizontal focal length. This detailedfiltering process ensures that only valid and relevant vehicle detectionsproceed to the next stage of analysis.

[0083] After filtering irrelevant and distant vehicles, the remaining validdetections are utilized to construct a comprehensive traffic density feature.This may involve counting the number of vehicles for each identified class(e.g., cars, trucks, two-wheelers) and subsequently combining these countswith their corresponding Passenger Car Unit (PCU) factors. PCU is astandard traffic engineering measure that quantifies the relative space andflow interference caused by different vehicle types, with larger or slowervehicles typically assigned higher PCU values. The final traffic densityfeature is computed as a weighted sum, where each valid vehicle count ismultiplied by its respective PCU factor. The result is a single scalar valuethat accurately quantifies the traffic density within a defined range in front ofthe ego vehicle, providing crucial input for the third set of environmental dataoutput.

[0084] In an example, the seventh model 214-7, which may be trained toidentify various traffic conditions, may incorporate an underlying objectdetection component. In an example, this object detection component,which may be a deep learning-based object detection model, such as, butnot limited to SqueezeDet model, may be trained on the Indian DrivingDataset (IDD), which is specifically curated with relevant classes for Indianroad conditions, including two-wheelers, three-wheelers, light motorvehicles (LMV), heavy motor vehicles (HMV), and pedestrians. The trainingparameters for this model include a batch size of 8, a learning rate of 0.0001,and a total of 240 epochs. During inference, a non-max suppressionthreshold of 0.10 and a confidence score threshold of 0.50 are applied, andthe top 64 bounding boxes are retained to ensure accurate and relevantvehicle detections, as further detailed in Table 6 below:Object Detection Parameter ValueModel SqueezeDetDataset Indian Driving Dataset (IDD) withrelevant classes for Indian conditionssuch as two-wheeler, three-wheeler,light motor vehicle (LMV), heavymotor vehicle (HMV), pedestrianBatch size 8Learning rate 0.0001Number of epochs 240Non-max suppression threshold 0.10Confidence score threshold 0.50Keep top bounding boxes 64Table: 6: (Vehicle Density Estimation)

[0085] Following the object detection and filtering, the seventh model214-7 may perform vehicle density estimation by applying equivalentPassenger Car Unit (PCU) factors to the detected and classified vehicles.These PCU factors, which quantify the relative space and flow interferenceof different vehicle types, are defined as follows in an example: Two-Wheelers are assigned a PCU factor of 0.75; Three-Wheelers are assigned2.00; Light Motor Vehicles (LMV) are assigned 1.00; and Heavy MotorVehicles (HMV) are assigned 3.70, as detailed in table 7 below. By weighingthe count of each vehicle type by its respective PCU factor, the model mayaccurately calculate the overall traffic density, contributing to the third set ofenvironmental data output.Vehicle Density EstimationParameter(Equivalent PCU Factor)ValueTwo-Wheeler 0.75Three-Wheeler 2.00Light Motor Vehicle 1.00Heavy Motor Vehicle 3.70Table: 7

[0086] The driving assistance system 104 comprises a data fusionmodule 216, which is coupled to the processor 202. In an example, the datafusion module 106 may be similar to the data fusion module 216. The datafusion module 216 is to integrate the first set of environmental data output,the second set of environmental data output, and the third set ofenvironmental data output to generate a unified output, the unified outputmay be indicative of the external environmental conditions, including one ormore of identified roadway elements, control conditions, and estimatedtraffic conditions. As explained previously, in an example, this integrationprocess may involves correlating and synthesizing diverse data streams,which may originate from different machine learning models and mayrepresent varied aspects of the external environment, into a cohesive andcomprehensive representation.

[0087] In an example, the fusion module 216 may be configured totimestamp and package the first set of environmental data output, thesecond set of environment data output, and the third set of environmentaldata output into a standardized message, ensuring data synchronizationand facilitating seamless integration. The resulting unified output may be aholistic and real-time representation indicative of the complete externalenvironmental conditions, enhancing the accuracy and reliability of driverassistance and autonomous navigation functions by presenting a completepicture of the vehicle's immediate surroundings. In an example, the unifiedout may be stored as unified output data 232 within the data 222.

[0088] In an example, the system 104 comprises a data communicationmodule coupled to the processor 202. The data communication module isto provide this unified output to the driver of the vehicle 102, through thevehicle's navigation dashboard, for real-time driver assistance.Simultaneously, the communication module 218 of the system 104 may alsobroadcast this unified output to other connected vehicles within theconnected vehicle ecosystem via a vehicular communication network, suchas, but not limited to, Dedicated Short-Range Communications (DSRC) orCellular Vehicle-to-Everything (C-V2X) technologies.

[0089] In an example, the communication module (218) is furtherconfigured to receive unified outputs from other connected vehicles withinthe connected vehicle ecosystem via the V2V communication network. Thisreception capability allows the system 104 to gather real-time environmentaldata including roadway elements, control conditions, and traffic conditionsfrom a broader network of vehicles. Upon receiving such unified outputs,the communication module 218 may process this external data and provideit to the vehicle's 102 navigation dashboard. This integration of externalvehicle data with the vehicle's 102 own sensor information significantlyenhances the driver's overall situational awareness, providing a morecomprehensive view of the surrounding environment and enabling moreinformed decisions for real-time driver assistance.

[0090] In an example, the communication module 218 may be furtherconfigured to generate and provide real-time alerts to the driver of thevehicle. These alerts may be designed to be immediate and actionable,indicative of the detection of critical roadway elements, significant variationsin control conditions, or adverse traffic conditions along the roadway. Forinstance, an alert could be triggered upon detecting a large pothole directlyin the vehicle's path, a sudden change in a traffic signal's state, or rapidlydecelerating traffic ahead. In an example, these alerts may be presented tothe driver through a dual modality to maximize effectiveness, as a prominentvisual display on the vehicle dashboard (e.g., a flashing icon, a textualwarning, or an overlay on the navigation screen) and may also as an audiblenotification (e.g., a chime, a spoken warning like "hazard ahead," or aspecific tone) to ensure the driver's attention is captured regardless of theirimmediate visual focus. In an example, the nature and urgency of the alertcan be dynamically adjusted based on the severity and proximity of thedetected condition, providing timely and relevant warnings to enhance driversafety.

[0091] Furthermore, the communication module 218 may be furtherconfigured to seamlessly integrate and display the identified roadwayelements, control condition changes, and prevailing traffic conditionsdirectly onto a navigational map accessible through the vehicle dashboard.This advanced functionality allows the driver to visualize the detectedenvironmental information in a spatial context, complementing the real-timealerts. For example, the map may graphically represent upcoming speedbumps, highlight areas with detected cracks, show the current status oftraffic lights at intersections, or visually depict congested zones and thepositions of other vehicles. This visual overlay provides a richer, moreintuitive understanding of the driving environment, enabling the driver toanticipate upcoming challenges, plan routes more effectively, and makeinformed decisions based on a comprehensive, real-time view of the roadahead, thereby significantly enhancing both driver assistance and overallnavigation experience.

[0092] By enabling the real-time detection and characterization ofroadway elements, control conditions, and traffic conditions, the system 104significantly enhances situational awareness for both human drivers andautonomous driving systems. This comprehensive environmentalunderstanding, derived from a single monocular camera, offers a costeffectiveyet robust solution compared to systems relying on multiple, moreexpensive sensor modalities. Furthermore, the system's ability to fuse thesediverse data streams into a unified output and then disseminate thisinformation through both in-vehicle alerts and vehicle-to-vehicle (V2V)communication channels provides a critical advantage. This multi-facetedapproach not only empowers individual drivers with timely warnings andenriched navigational context but also provides a collective awarenessacross the connected vehicle ecosystem, paving the way for improved trafficflow, reduced congestion, and a substantial increase in overall road safetyby enabling proactive responses to dynamic environmental challenges.

[0093] Reference is made to Figure 4 that illustrates a method formanaging vehicle navigation within a connected vehicle ecosystem, inaccordance with an example implementation of the present subject matter.

[0094] The order in which the method 400 is described is not intended tobe construed as a limitation, and any number of the described methodblocks may be combined in any order to implement method 400, or analternative method. Furthermore, the method 400 may be implemented byprocessor(s) or computing device(s) through any suitable hardware, nontransitorymachine-readable instructions, or combination thereof.

[0095] It may be understood that blocks of the method 400 may beperformed, for example, by the above-described driving assistance system104, as illustrated in Figures 1 and 2. In an example, the driving assistancesystem 104 may be installed in a network environment, such as the networkenvironment 100 described in reference to Figure 1.

[0096] At block 402, the method 400 comprises acquiring real-time visualdata of an external environment surrounding a vehicle 104 from amonocular camera. As explained previously, the monocular camera, beinga single-lens optical sensor, may offer a cost-effective and computationallyefficient means of acquiring rich visual information. In an example, thiscamera may be positioned, for example, at the center of the windshield ofthe vehicle 102, typically behind the rearview mirror. This placement mayprovide an unobstructed and expansive view of the vehicle's surroundings,crucially including the path directly ahead of the vehicle 102, as well asadjacent lanes, road shoulders, and overhead structures. In operation, thereal-time visual data captured by the camera may comprise a continuousstream of images or video frames, providing dynamic information about thechanging environment.

[0097] At block 404, the method 400 comprises identifying, by a machinelearning model ensemble (MLME) 214, by processing the visual data 224,one or more roadway elements to generate a first set of environmental dataoutput, wherein a roadway element is a physical feature comprising asurface condition, or obstruction present on a road surface ahead of thevehicle. As explained previously, the one or more roadway elements maycomprise hazardous obstructions such as potholes and speed bumps,which may be detected by a first model 214-1 within the MLME 214.Furthermore, diverse types of cracks on the path may be identified by asecond model 214-2.

[0098] In an example, the roadway elements may also include roaddebris, manholes, gratings, the presence of foreign objects, dynamicsurface characteristics such as water puddles, ice patches, or gravel, andchanges in road surface type, for instance, from asphalt to concrete, whichmay be identified by other models within the MLME 214, such as model 214-8. Upon identification and characterization of these roadway elements, thefirst set of environmental data output may be generated.

[0099] At block 406, the method 400 comprises determining, by theMLME, based on processing the visual data 224, one or more controlconditions to generate a second set of environmental data output, whereinthe control conditions comprise a visual indicator or regulatory device on theroad surface to govern movement of the vehicle 102. As explainedpreviously, the one or more control conditions may include lane markingson the path, which may be identified by a third model 214-3 of the MLME214. The one or more control conditions may also include traffic signs, whichmay be identified by a fourth model 214-4 of the MLME 214. Once theposition of the traffic sign is identified, a fifth model 214-5 may be employedto recognize and classify the identified traffic sign by ascertaining its type,such as regulatory, warning, or informational, and its associated meaning.

[00100] In an example, determining one or more control conditions mayalso include identifying a state of traffic signals, the state being indicative ofspecific illuminated colour, pattern, or operational status the traffic signaldisplays, using a sixth model 214-6 of the MLME 214. This may be identifiedby determining their current operational state, such as illuminated color(e.g., red, yellow, or green), flashing status, or specific patterns.Additionally, other regulatory indicators like temporary construction zonesigns, school zone indicators, and lane usage signs may also be identified.Upon determination and characterization of these control conditions, thesecond set of environmental data output may be generated.

[00101] At block 408, the method 400 comprises estimating, by theMLME, based on processing the visual data 224, traffic conditions togenerate a third set of environmental data output. As explained previously,this may include calculating traffic density, which may be performed by aseventh model 214-7 within the MLME 214, providing an understanding ofthe concentration of vehicles within a given area.

[00102] In an example, the seventh model 214-7 may also identify thepresence and attributes of other road users, which encompasses varioustypes of vehicles (e.g., cars, trucks, motorcycles, buses, emergencyvehicles), as well as vulnerable road users like pedestrians and cyclists. Foreach identified road user, their relative positions, current speeds, andpredicted trajectories may be determined, enabling the host vehicle 102 toanticipate their movements. In another example, the MLME 214 may alsobe configured to detect and characterize incidents such as accidents, stalledvehicles, or unexpected obstructions that impact traffic flow and safety.Upon identification and characterization of these traffic conditions, the thirdset of environmental data output may be generated.

[00103] At block 410, the method 400 comprises integrating the first setof environmental data output, the second set of environmental data output,and the third set of environmental data output to generate a unified output.As explained previously, in an example, this integration process may involvea centralized onboard fusion engine that timestamps and packages allmodule outputs into a standardized message. This may enable correlationand synthesizing diverse data streams, which may originate from differentmachine learning models and may represent varied aspects of the externalenvironment, into a cohesive and comprehensive representation.

[00104] At block 412, the method 400 comprises providing the unifiedoutput to the vehicle's navigation dashboard, wherein the unified output isindicative of the one or more of identified roadway elements, controlconditions, and estimated traffic conditions, along with their respectiveattributes and spatial relationships relative to the vehicle 102. As explainedpreviously, this unified output may be presented to the driver throughvarious intuitive formats. For instance, visual cues may be overlaid directlyonto a navigational map display, highlighting the precise locations ofidentified roadway elements such as potholes, speed bumps, or cracks.Simultaneously, the current state of traffic signals (e.g., green, yellow, red)or the meaning of detected traffic signs (e.g., speed limits, stop signs) maybe prominently displayed.

[00105] Furthermore, real-time traffic conditions, such as areas of hightraffic density or detected incidents like stalled vehicles, may be visuallyrepresented. In addition to visual displays, audible notifications may beprovided to alert the driver to critical or immediate hazards, such as anapproaching obstacle or a sudden change in traffic light status and may alsoinclude directional cues or distance estimates. This comprehensive andintegrated presentation of environmental information, emphasizing spatialrelationships, may enhance the driver's situational awareness, enablingmore informed and safer driving decisions in dynamic real-world scenarios.

[00106] At block 414, the method 400 comprises broadcasting the unifiedoutput to one or more external connected vehicles via a vehicularcommunication network. As explained previously, this vehicularcommunication network may leverage various technologies such as but notlimited to Dedicated Short-Range Communication (DSRC) based on IEEE802.11p, or Cellular Vehicle-to-Everything (C-V2X), utilizing 4G LTE or 5Gcellular networks. The predefined communication range may varydepending on the specific communication technology and environmentalfactors, but it ensures that the unified output is shared with relevant vehiclesin the immediate vicinity or along the projected path. This broadcasting ofreal-time environmental information, including identified roadway elements,control conditions, and traffic conditions, allows for the creation of acollective situational awareness among connected vehicles. For example, ifthe vehicle 102 detects a newly formed pothole or a sudden change in atraffic light's state, this critical information can be instantly disseminated toother vehicles approaching the same location. This enables surroundingvehicles to anticipate hazards, adjust their driving behavior, or even reroute,thereby significantly enhancing overall road safety, optimizing traffic flow,and contributing to the development of intelligent transportation systems.

[00107] At block 416, the method 400 comprises receiving unified outputsfrom one or more external connected vehicles via the V2V communicationnetwork. In an example, this reception occurs when these externalconnected vehicles are within a predefined communication range of thevehicle 102, using the V2V communication network to facilitate real-timedata exchange. Upon successful reception, unified outputs originating fromother vehicles, may be processed and provided to the vehicle's 102navigation dashboard, for example, for displaying the information visually(e.g., overlaid on a map, as icons, or text alerts) or by conveying it audibly(e.g., through voice prompts or specific alert sounds).This integration ofexternal environmental data of other connected vehicles with vehicle's ownenvironmental understanding significantly enhances the driver's situationalawareness, offering a more comprehensive and dynamic view of thesurrounding road conditions and potential hazards.

[00108] By identifying and integrating diverse environmental datastreams, including roadway elements, control conditions, and trafficconditions, the method 400 may significantly enhance situationalawareness for both human drivers and autonomous systems. By leveraginga machine learning model ensemble for nuanced perception and fusingthese distinct data types into a unified output, the method may enable moreinformed and safer driving decisions, allowing vehicles to proactively adaptto dynamic and complex real-world scenarios. Furthermore, bybroadcasting this unified environmental intelligence to other connectedvehicles, the method may foster a collective awareness across thetransportation network, contributing to optimized traffic flow, reducedcongestion, and a substantial improvement in overall road safety. Thisapproach, particularly with its reliance on a cost-effective monocularcamera, may also offer practical advantages in terms of ease ofimplementation and maintenance, making advanced driving assistancemore accessible.

[00109] Although the subject matter has been described in considerabledetail with reference to certain examples and implementations thereof, otherimplementations are possible. As such, the present disclosure should notbe limited to the description of the preferred examples and implementationscontained therein.

Claims

1. A system (104) for managing vehicle navigation within a connected vehicle ecosystem, the system (104) comprising: a processor; a data collection module (210) coupled to the processor (202), to acquire real-time visual data (224) of an external environment surrounding the vehicle (102), from a monocular camera; a data processing module (212), coupled to the processor (202), to process the visual data (224), using a machine learning model ensemble (MLME) (214), to identify: one or more roadway elements to generate a first set of environmental data output, the roadway elements indicative of obstructions on a path that the vehicle (102) is traversing; one or more control conditions to generate a set of second environmental data output, wherein the control conditions are to govern movement of the vehicle (102) relative to a location of the vehicle (102) and are identifiable based on at least on visual indicator present at the location; and traffic conditions to generate a set of third environmental data output; a data fusion module (216), coupled to the processor (202), and configured to integrate the first set of environmental data output, the second set of environmental data output, and the third set of environmental data output to generate a unified output, the unified output is indicative of the external environmental conditions, including one or more of identified roadway elements, control conditions, and estimated traffic conditions; and a communication module (218), coupled to the processor (202), to: provide the unified output to a vehicle's navigation dashboard for real-time driver assistance, wherein the unified output is to indicate the external environmental conditions along with their respective attributes and spatial relationships relative to the vehicle (102); broadcast the unified output to one or more external connected vehicles via a vehicle-to-vehicle (V2V) communication network, when the one or more connected vehicles are within a predefined communication range; and receive unified outputs from one or more external connected vehicles via the V2V communication network and provide the unified outputs to the vehicle's (102) navigation dashboard, when the vehicle (102) is within the predefined communication range with the one or more external connected vehicles.

2. The system (104) as claimed in claim 1, wherein the MLME (214), for identifying one or more roadway elements from the acquired visual data (224), comprises at least: a first model (214-1) to identify potholes and speed bumps on the path; and a second model (214-2) to identify cracks on the path.

3. The system (104) as claimed in claim 1, wherein the MLME (214), for identifying one or more control conditions from the acquired visual data (224), comprises at least: a third model (214-3) to identify lane markings for detecting lanes on the path; a fourth model (214-4) to identify position of traffic signs on the path; a fifth model (214-5) to recognise and classify an identified traffic sign as a regulatory, warning, or informational type and to ascertain their content; and a sixth model (214-6) to identify a state of the traffic signal, the state being indicative of specific illuminated colour, pattern, or operational status the traffic signal displays.

4. The system (104) as claimed in claim 1, wherein the MLME (214), for identifying one or more traffic conditions, includes at least a seventh model (214-7) for detecting traffic conditions on the roadway by calculating traffic density from the acquired real-time visual data (224).

5. The system (104) as claimed in claim 1, wherein the communication module (218) is further configured to provide a real-time alert to a driver of the vehicle (102), the alert being indicative of detection of one or more of roadway elements, variations in control conditions, and traffic conditions along the roadway, wherein the alerts are presented as at least one of a visual display on the vehicle dashboard and as an audible notification to the driver.

6. The system (104) as claimed in claim 5, wherein the communication module (218) is further configured to provide the identified roadway elements, control condition changes, and traffic conditions present on the roadway for displaying on a navigational map through the vehicle dashboard.

7. A method (400) for managing vehicle navigation within a connected vehicle ecosystem, the method (400) comprising: acquiring (402) real-time visual data (224) of an external environment surrounding a vehicle (102) from a monocular camera; identifying (404), by a machine learning model ensemble (MLME), by processing the visual data (224), one or more roadway elements to generate a first set of environmental data output, wherein a roadway element is a physical feature comprising a surface condition, or obstruction present on a road surface ahead of the vehicle; determining (406), by the MLME, based on processing the visual data (224), one or more control conditions to generate a second set of environmental data output, wherein the control conditions comprise a visual indicator or regulatory device on the road surface to govern movement of the vehicle (102); estimating (408), by the MLME, based on processing the visual data (224), traffic conditions to generate a third set of environmental data output; integrating the first set of environmental data output, the second set of environmental data output, and the third set of environmental data output to generate a unified output; providing (412) the unified output to the vehicle's navigation dashboard, wherein the unified output is indicative of the one or more of identified roadway elements, control conditions, and estimated traffic conditions, along with their respective attributes and spatial relationships relative to the vehicle (102); broadcasting (414) the unified output to one or more external connected vehicles via a vehicle-to-vehicle (V2V) communication network; and receiving (416) unified outputs from one or more external connected vehicles via the V2V communication network.

8. The method as claimed in claim 7, wherein identifying one or more roadway elements comprises at least: identifying potholes and speed bumps on the road surface by a first model (214-1) of the MLME (214); and identifying cracks on the road surface by a second model (214-2) of the MLME (214).

9. The method (400) as claimed in claim 7, wherein determining one or more control conditions comprises at least: identifying lane markings for detecting lanes on the road surface using a third model (214-3) of the MLME (214); identifying position of traffic signs on the road surface using a fourth model (214-4) of the MLME (214); recognizing and classifying an identified traffic sign as a regulatory, warning, or informational type using a fifth model (214-5) of the MLME (214); and identifying a state of the traffic signal, the state being indicative of specific illuminated colour, pattern, or operational status the traffic signal displays, using a sixth model (214-6) of the MLME (214).

10. The method (400) as claimed in claim 7, wherein identifying traffic conditions comprises at least detecting traffic conditions by calculating traffic density from the acquired visual data (224) using a seventh model (214-7) of the MLME (214).