Crowdsourcing road conditions from unusual vehicle events
By installing sensors and cameras on the vehicle, reporting abnormal events to the road condition monitoring system in real time, the problem of difficult to effectively utilize abnormal traffic events in the prior art to improve road conditions is solved, and road and driver safety is improved.
Patent Information
- Application Number
- CN202180016282.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2020-02-07
- Filing Date
- 2021-02-04
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2041-02-04
AI Technical Summary
The prior art is difficult to effectively utilize abnormal transportation incidents to report and improve road conditions in real time, resulting in road safety and driver safety being affected.
Detect abnormal events such as sudden braking, sharp turns and pothole impacts by installing multiple sensors and cameras on the vehicle, and transmit relevant data to the Road Condition Monitoring System (RCMS) to identify road patterns and provide driver advice Or directives to improve the automation control capabilities of vehicles.
Real-time reporting and handling of abnormal traffic accidents is realized, road safety and driver safety are improved, traffic accidents caused by human errors are reduced, and road maintenance efficiency and accuracy are improved.
Smart Images

Figure CN115516539B_ABST
Abstract
Description
[0001] Related Applications
[0002] This application claims priority to U.S. Patent Application No. 16 / 784,554, filed on February 7, 2020, and entitled “Crowdsourcing Road Conditions from Abnormal Vehicle Events,” the entire disclosure of which is hereby incorporated by reference herein. Technical Field
[0003] At least some embodiments disclosed herein relate to crowdsourcing reporting of road conditions from unusual vehicle events. Background Art
[0004] Crowdsourcing is an outsourcing model in which an entity can obtain services from a large, growing and evolving group of Internet users. Crowdsourcing delegates work or processes between participants to achieve a cumulative result. The key advantage of crowdsourcing is that a large number of users can perform continuous tasks in parallel.
[0005] Crowdsourcing has been used to improve navigation information and driving. For example, crowdsourcing has been used to improve the business-accumulated information found in navigation applications. In such instances, the crowdsourcing participants may be vehicle drivers. Crowdsourcing is just one of many techniques to improve driving.
[0006] Another way to improve driving is through advanced driver assistance systems (ADAS). ADAS is an electronic system that helps vehicle drivers while driving. ADAS can provide improved car safety and road safety. ADAS can use electronic technology, such as electronic control units and power semiconductor devices. Most road accidents occur due to human error; therefore, ADAS that automates some of the controls of vehicles can reduce human error and road accidents. Such systems have been designed to automate, adapt and enhance vehicle systems to achieve safety and improved driving. The safety features of ADAS are designed to avoid collisions and accidents by providing technology to alert drivers to potential problems, or to avoid collisions by implementing protective measures and taking over vehicles. Adaptive features can automate lighting, provide adaptive cruise control and collision avoidance, provide pedestrian collision avoidance mitigation (PCAM), alert drivers to other vehicles or dangerous factors, provide lane departure warning systems, provide automatic lane centering, display the field of view in blind spots, or connect to a navigation system. BRIEF DESCRIPTION OF THE DRAWINGS
[0007] The present disclosure will be more fully understood from the detailed description given below and from the accompanying drawings of various embodiments of the present disclosure.
[0008] Figures 1 to 3 An exemplary networked system according to some embodiments of the present disclosure is described, which includes at least a mobile device and a vehicle and a road condition monitoring system (RCMS) and is configured to implement crowdsourcing reporting of road conditions from abnormal vehicle events.
[0009] Figures 4 to 6 The following describes some embodiments of the present disclosure. Figures 1 to 3 Flowchart of example operations performed by aspects of the networked system depicted in . DETAILED DESCRIPTION
[0010] At least some embodiments disclosed herein relate to crowdsourcing reporting of road conditions from abnormal vehicle events. Abnormal vehicle events (e.g., sudden braking, sharp turns, evasive maneuvers, pothole strikes, etc.) may be detected and reported to one or more servers of a road condition monitoring system (RCMS). For example, the RCMS may include a server in a cloud computing environment, and may identify patterns in reported road conditions to generate advisory information or instructions to vehicles and vehicle users. For example, suspicious obstacles may be identified and used to instruct the driver or vehicle to gradually slow down to avoid sudden braking and sharp turns. In some embodiments, vehicles with cameras may upload images of suspicious obstacles (e.g., potholes) to allow for active identification of road problems, enabling the RCMS to dispatch road services to resolve the problem.
[0011] The vehicle may be equipped with multiple sensors that can detect unusual vehicle events, such as sudden braking, sharp turns, evasive maneuvers, and pothole strikes. The vehicle may transmit the corresponding information along with precise geographic location information to the cloud or another type of computer group working together (e.g., via peer-to-peer computing). Each vehicle's transmission of this data may be one or more data points for determining a road condition or hazard. For example, the determination may be made in the cloud or via a peer-to-peer computing environment. The determination may be used to generate recommendations that are reported to the vehicles participating in the system. The recommendations may be presented by the UI of the vehicle or the mobile device of the user in the vehicle. The recommendations may be distributed based on the geographic location of the determined condition or hazard and the matching geographic location of vehicles that are close to the geographic location of the determined condition or hazard.
[0012] In addition to or as an alternative to suggestions, instructions or data regarding a road condition or hazard may be sent to the vehicle when the vehicle is approaching the geographical location of the condition or hazard. And, the vehicle may adjust its components according to the distributed instructions or data regarding the road condition or hazard. This is beneficial for many reasons. For example, a user cannot see a series of potholes in time when driving fast on a highway, but the corresponding notification or instruction can be used as a basis for guiding the vehicle to automatically slow down in a safe and reasonable manner when the vehicle approaches the road condition or hazard. In addition, the system may also provide the corresponding suggestion to the user via the UI. Therefore, the driver will not be confused by the vehicle slowing down.
[0013] Generally speaking, road conditions are sensed by vehicles and sent to a central computing system, such as one or more servers of the RCMS. The corresponding data is processed and suggestions and / or instructions are generated and distributed accordingly. Drivers can receive such suggestions and the vehicles themselves can receive this information and automatically make corresponding adjustments. Vehicles driving through or approaching road conditions or hazards can then provide feedback to the central computing system (or in other words, the RCMS). The feedback can be used to train and improve the centralized computing system and the subsequent generation of suggestions and instructions for automatic vehicle adjustments.
[0014] Other actions may also be taken from the crowdsourcing provided by participating vehicles. Images of road conditions and hazards may be recorded by cameras on vehicles, and the redundancy of the images and other such data may legitimize the actions provided by the RCMS. For example, images of road conditions and hazards may be recorded by cameras on vehicles, and the redundancy of the images and other such data may legitimize the credibility of calling service or other types of response actions, such as dispatching maintenance or cleaning services.
[0015] One or more servers of the RCMS may aggregate information from reporting vehicles, vehicles reporting events, and vehicles using information from the servers. The server(s) do not need to receive raw data and diagnose abnormal conditions. The server(s) may receive already processed information, which is processed by the vehicle. The vehicle may process raw data from sensors and cameras and perform diagnosis. The vehicle may then send the diagnosis to the server(s) of the RCMS for further analysis and to generate recommendations. The server(s) of the RCMS are more than just routers that broadcast information received from one vehicle to another. The server(s) may synthesize reports from a large number of reporting vehicles to make their distributed information more reliable and meaningful.
[0016] Furthermore, the reporting vehicle may have a certain level of intelligence in diagnosing abnormal conditions and thus reducing data traffic in the report. If a condition does not require notification, then there is no need to send a report about the condition from the vehicle to the server(s) of the RCMS. The information sent from the server to the receiving vehicle may be instructional, advisory and / or informational. The receiving vehicle may have a certain level of intelligence in using the information, rather than simply receiving the information and acting accordingly (e.g., merely sounding an alarm after receiving the information).
[0017] In some embodiments, a vehicle may include a body, a transmission system, and a chassis and at least one sensor attached to at least one of the body, transmission system, or chassis or any combination thereof. The at least one sensor may be configured to detect at least one sudden movement of the vehicle or at least one component of the vehicle, and to send movement data derived from the detected at least one sudden movement. The vehicle may also include a global positioning system (GPS) device, the GPS device being configured to detect the geographic location of the vehicle during the detection of at least one sudden movement, and to send location data derived from the detected geographic location. The vehicle may also include a computing system, the computing system being configured to receive movement data and location data, and to associate the received movement data with the received location data. The computing system may also be configured to determine whether the detected at least one sudden movement in the received movement data exceeds a sudden movement threshold. In some embodiments, the determination may be made according to artificial intelligence (AI). Furthermore, the computing system may be configured to train the AI using machine learning. For example, the AI may include an artificial neural network (ANN) and the computing system may be configured to train the ANN. The computing system may also be configured to send the correlated data or a derivative thereof to the road condition monitoring system in response to determining that at least one sudden movement exceeds the sudden movement threshold. The correlated data or a derivative thereof may be sent by the computing system via a wide area network.
[0018] In such embodiments and other embodiments, the vehicle may also include at least one camera configured to record at least one image of an area within a preselected distance of the vehicle during detection of at least one sudden movement. And, the at least one camera may also be configured to send image data derived from the at least one recorded image. And, in such instances, the computing system may be configured to receive the image data and associate the received image data with the received movement data and the received position data. In response to a determination that the at least one sudden movement exceeds a sudden movement threshold, the computing system may also be configured to send the associated image data or a derivative thereof along with the associated movement and position data to a road condition monitoring system via a wide area network.
[0019] In such and other embodiments, a road condition monitoring system may include at least one processor and at least one non-transitory computer-readable medium having instructions executable by at least one processor to perform a method, such as a method for providing crowdsourcing reporting of road conditions from abnormal vehicle events. This method may include receiving movement data and geolocation data from a corresponding computing system in a vehicle that moves suddenly. This method may also include determining the geolocation of a dangerous condition in a road based on the received movement data and the received geolocation data. In some embodiments, the determination of the geolocation of the dangerous condition may be based on an AI. And, the method may include using machine learning to train the AI. For example, the AI may include an ANN and the method may include training the ANN.
[0020] The method may also include generating hazard information based on at least the received movement data and the received geographic location data (e.g., the hazard information may include instructional data). The generation of the hazard information may also be based on an AI; and the method may include training such an AI using machine learning. For example, the AI may include an ANN and the method may include training the ANN. The method may also include sending a portion of the hazard information to a computing system in the approaching vehicle when the approaching vehicle is approaching a location in the determined geographic locations of the hazardous condition and within a preselected distance of the location.
[0021] In such and other embodiments, the computing system of the vehicle may be configured to receive and process data (e.g., including instructional data) from a road condition monitoring system via a wide area network. Also, the data may include information derived from at least correlated movement and position data sent by other vehicles in the geographic location that the vehicle is approaching. The computing system of the vehicle may be configured to process the received data via an AI; and this AI may be trained by the computing system. Also, the AI may include an ANN, and the ANN may be trained by the computing system. Also, based on the received and processed data, the driver or vehicle may take corrective action for the vehicle.
[0022] In summary, a system for crowdsourcing reporting of road conditions from abnormal vehicle events is described herein. Abnormal vehicle events (e.g., sudden braking, sharp turns, evasive maneuvers, pothole strikes, etc.) may be detected and reported to the RCMS. The RCMS may identify patterns in reported road conditions to generate advisory information or instructions for vehicles and vehicle users. For example, suspicious obstacles may be identified and used to instruct the driver or vehicle to gradually slow down to avoid sudden braking and sharp turns. In some instances, the vehicle may have a camera that can upload images of suspicious obstacles (e.g., potholes) to allow for active identification of road problems. This gives the RCMS more confidence to take corrective action, such as automatically calling a road maintenance service.
[0023] Figures 1 to 3 An example networked system 100 is illustrated according to some embodiments of the present disclosure, which includes at least a RCMS and mobile devices and vehicles (e.g., see mobile devices 140-142 and 302 and vehicles 102, 202 and 130-132) and is configured to implement crowdsourcing reporting of road conditions from abnormal vehicle events.
[0024] The networked system 100 is networked via one or more communication networks 122. The communication networks described herein, such as (several) communication networks 122, may include at least a local to device network (e.g., Bluetooth, etc.), a wide area network (WAN), a local area network (LAN), an intranet, a mobile wireless network (e.g., 4G or 5G), an extranet, the Internet, and / or any combination thereof. The nodes of the networked system 100 (e.g., see mobile devices 140, 142, and 302, vehicles 102, 130, 132, and 202, and one or more RCMS servers 150) may each be part of a peer-to-peer network, a client-server network, a cloud computing environment, etc. Moreover, any of the devices, computing devices, vehicles, sensors, or cameras, and / or user interfaces described herein may include some kind of computer system (e.g., see computing systems 104 and 204). Moreover, this computer system may include a network interface to other devices in a LAN, an intranet, an extranet, and / or the Internet. The computer system may also operate in the capacity of a server or a client machine in a client-server network environment, as a peer machine in a peer-to-peer (or distributed) network environment, or as a server or a client machine in a cloud computing infrastructure or environment.
[0025] like Figure 1, the networked system 100 may include at least a vehicle 102, the vehicle 102 including a vehicle computing system 104 (including a client application 106 of the RCMS - also referred to herein as the RCMS client 106), a body and controllable components of the body (not depicted), a transmission and controllable components of the transmission (not depicted), a body control module 108 (which is a type of ECU), a transmission control module 110 (which is a type of ECU), and a power steering control unit 112 (which is a type of ECU). The vehicle 102 also includes a plurality of sensors (e.g., see sensors 114a-114b), a GPS device 116, a plurality of cameras (e.g., see cameras 118a-118b), and a controller area network (CAN) bus 120, which connects at least the vehicle computing system 104, the body control module 108, the transmission control module 110, the power steering control unit 112, the plurality of sensors, the GPS device 116, and the plurality of cameras to each other. Also, as shown, vehicle 102 is connected to network(s) 122 via vehicle computing system 104. Also, as shown, vehicles 130-132 and mobile devices 140-142 are connected to network(s) 122. And, thus, are communicatively connected to vehicle 102.
[0026] RCMS client 106 included in computing system 104 may communicate with RCMS server(s) 150. RCMS client 106 may be part of, include, or be connected to an ADAS; and thus, the ADAS may also communicate with RCMS server(s) 150 (not depicted).
[0027] In some embodiments, the vehicle 102 may include a vehicle body, a transmission system, and a chassis, and at least one sensor (e.g., see sensors 114a to 114b). The at least one sensor may be attached to at least one of the vehicle body, transmission system, or chassis, or any combination thereof. The at least one sensor may be configured to: detect at least one sudden movement of the vehicle 102 or at least one component of the vehicle, and send movement data derived from the detected at least one sudden movement. The sudden movement may include a change in speed, acceleration, angular velocity, or angular acceleration, or any combination thereof, that exceeds a predetermined threshold. For example, the sudden movement may include a change in speed, acceleration, angular velocity, or angular acceleration, or any combination thereof, that exceeds a corresponding predetermined threshold for the one or more directions in one or more directions.
[0028] As shown, the vehicle 102 also includes a GPS device 116. The GPS device 116 may be configured to detect the geographic location of the vehicle 102 during the detection of at least one sudden movement, and to send location data derived from the detected geographic location. The vehicle 102 also includes a computing system 104 (which includes a RCMS client 106), and the computing system (e.g., via the RCMS client 106) may be configured to receive movement data and location data. The computing system 104 (e.g., via the RCMS client 106) may also be configured to associate the received movement data with the received location data, and determine whether the at least one sudden movement detected in the received movement data exceeds a sudden movement threshold. In some embodiments, the determination may be made according to artificial intelligence (AI). Furthermore, the computing system 104 (e.g., via the RCMS client 106) may be configured to train the AI using machine learning. For example, the AI may include an ANN and the computing system 104 (e.g., via the RCMS client 106) may be configured to train the ANN. The computing system 104 (e.g., via the RCMS client 106) may also be configured to send the associated data or a derivative thereof to the RCMS in response to a determination that at least one sudden movement exceeds a sudden movement threshold. For example, the associated data or a derivative thereof may be sent by the computing system 104 to the RCMS server(s) 150 via a portion of the network(s) 122.
[0029] In such and other embodiments, the vehicle 102 may include at least one camera (e.g., see cameras 118a-118b). The at least one camera may be configured to record at least one image of an area within a preselected distance of the vehicle 102 during detection of at least one sudden movement, and to send image data derived from the at least one recorded image. Moreover, the at least one camera may be configured to record at least one image of an area within a preselected distance of the vehicle 102 during a predetermined time period after at least one sudden movement, and to send image data derived from the at least one recorded image recording at least one sudden movement. In such and other embodiments, the computing system 104 (e.g., via the RCMS client 106) may be configured to receive the image data, and to associate the received image data with the received movement data and the received position data. Also, in response to a determination that at least one sudden movement exceeds the sudden movement threshold, computing system 104 (e.g., via RCMS client 106) may be configured to send the associated image data, or a derivative thereof, along with the associated movement and position data, to RCMS via a wide area network (e.g., see network(s) 122). For example, the associated data, or a derivative thereof, may be sent by computing system 104 (e.g., via RCMS client 106) via a portion of network(s) 122 to RCMS server(s) 150.
[0030] In such and other embodiments, computing system 104 (e.g., via RCMS client 106) may be configured to receive and process data (e.g., data including instructive data) from RCMS via a wide area network. For example, data may be received by computing system 104 (e.g., via RCMS client 106) from (several) RCMS server(s) 150 via a portion of (several) network(s) 122, and the received data may then be processed. The received data may include information derived from at least associated movement and position data sent by other vehicles (e.g., see vehicles 130 to 132) in the geographic location that vehicle 102 is approaching. In some embodiments, the derivation of received data and / or subsequent processing of received data is performed according to AI, and the AI may be trained by the RCMS and / or the computing system of the vehicle.
[0031] In such and other embodiments, the vehicle 102 may include a user interface (e.g., a graphical user interface) configured to provide at least a portion of the received and processed data to a user of the vehicle (e.g., see Figure 2 202, which may include a GUI).
[0032] Moreover, the vehicle 102 may include an ECU configured to receive at least a portion of the received and processed data via the computing system 104 (e.g., via the RCMS client 106). The ECU may also be configured to control the steering of the vehicle based on at least a portion of the received and processed data via at least one electrical system in the vehicle (e.g., see the power steering control unit 112). The ECU may also be configured to control the deceleration of the vehicle based on at least a portion of the received and processed data via at least one electrical system in the vehicle 102 (e.g., see the transmission control module 110). The ECU may also be configured to control the acceleration of the vehicle based on at least a portion of the received and processed data via at least one electrical system in the vehicle 102 (e.g., see the transmission control module 110). Moreover, the vehicle 102 may include one or more ECUs configured to receive at least a portion of the received and processed data via the computing system 104 (e.g., via the RCMS client 106). The ECU(s) may also be configured to control at least one of the steering of the vehicle, the deceleration of the vehicle, or the acceleration of the vehicle, or any combination thereof based on at least a portion of the received and processed data via at least one electrical system in the vehicle 102 (e.g., see the body control module 108, the transmission control module 110, and the power steering control unit 112).
[0033] In such and other embodiments, a system (e.g., RCMS) may include at least one processor and at least one non-transitory computer-readable medium having instructions executable by the at least one processor to perform a method (e.g., see RCMS server(s) 150). The method performed may include receiving movement data and geolocation data from a computing system in a vehicle that is suddenly moving (e.g., see computing systems 104 and 204 of vehicles 102 and 202, respectively). In some examples, the method may include determining a geolocation of a hazardous condition in a road based on the received movement data and the received geolocation data; and, this determination may be made based on an AI and the AI may be trained via machine learning and may include an ANN. The method may include generating hazard information (e.g., hazard information including instructive data) based at least on the received movement data and the received geolocation data. In some examples, the information may be related to the determined geolocation of the hazardous condition. The generation of the information may be made based on an AI and the AI may be trained via machine learning and may include an ANN. The method may also include sending a portion of the hazard information to a computing system in the approaching vehicle (eg, see computing systems 104 and 204) when the approaching vehicle is approaching one of the determined geographic locations of the hazardous condition and is within a preselected distance of the one location.
[0034] In such and other embodiments, the portion of the hazard information may be configured to at least provide a basis for alerting a user of the approaching vehicle of the hazard via a user interface in the approaching vehicle. Also, the portion of the hazard information may be configured to at least provide a basis for controlling steering, deceleration, and acceleration of the approaching vehicle via at least one electrical system in the approaching vehicle.
[0035] Furthermore, the received movement data may include corresponding movement data sent from the corresponding suddenly moving vehicle. The corresponding movement data may be derived from the sensed sudden movement of the suddenly moving vehicle. The received position data may include corresponding position data sent from the suddenly moving vehicle. Furthermore, the corresponding position data may be associated with the position of the suddenly moving vehicle when the sudden movement is sensed.
[0036] In addition, the method performed by the system (e.g., RCMS) may include receiving image data from a computing system in a vehicle that moves suddenly. And, determining the geographic location of the dangerous condition may be performed based on the received image data, the received movement data, and the received geographic location data. Determining the geographic location of the dangerous condition may also be performed based on AI and the AI may be trained via machine learning and may include ANN. Moreover, the image data may include corresponding image data derived from at least one image of an area within a preselected distance of the vehicle that moves suddenly, and at least one image may be recorded when the sudden movement is sensed or within a predetermined time period after the sudden movement is sensed. Part of the danger information may be corresponding image data and may be configured to at least provide a basis for alerting a user of a vehicle approaching the danger via a user interface in the vehicle approaching the danger and displaying a danger image reproduced from the corresponding image data.
[0037] The vehicle 102 includes vehicle electronics, which include at least electronic devices for controllable components of the vehicle body, controllable components of the transmission system, and controllable components of the power steering. The vehicle 102 includes controllable components of the vehicle body and such components and subsystems are connected to the body control module 108. The vehicle body includes at least a frame for supporting the transmission system. The chassis of the vehicle can be attached to the frame of the vehicle. The vehicle body can also include an interior for at least one driver or passenger. The interior can include seats. The controllable components of the vehicle body can also include one or more power doors and / or one or more power windows. The vehicle body can also include any other known components of the vehicle body. And, the controllable components of the vehicle body can also include a convertible top, a sunroof, a power seat, and / or any other type of controllable components of the vehicle body. The body control module 108 can control the controllable components of the vehicle body.
[0038] Furthermore, the vehicle 102 also includes controllable components of a transmission system. The controllable components of the transmission system and its components and subsystems are connected to the transmission system control module 110. The controllable components of the transmission system may include at least an engine, a transmission, a drive shaft, a suspension and steering system, and a transmission system electrical system. The transmission system may also include any other known components of a vehicle transmission system and the controllable components of the transmission system may include any other known controllable components of a transmission system. Furthermore, the controllable power steering components may be controlled via a power steering control unit 112.
[0039] The UI elements described herein, such as UI elements of a mobile device or a vehicle, may include any type of UI. The UI element may be, be part of, or include an automobile control. For example, the UI may be a gas pedal, a brake pedal, or a steering wheel. Moreover, the UI may be part of or include an electronic device and / or an electromechanical device, and may be part of or include a tactile UI (touch), a visual UI (vision), an auditory UI (sound), an olfactory UI (smell), a balanced UI (balance), or a taste UI (taste), or any combination thereof.
[0040] Multiple sensors (e.g., see sensors 114a to 114b) and / or multiple cameras (e.g., see cameras 118a to 118b) of the vehicle 102 may include one or more features or characteristics respectively configured to sense and / or record multiple UI elements or their outputs or any other component of the vehicle 102 or its surroundings. The sensors or cameras of the vehicle 102 may also be configured to generate one or more features or characteristics corresponding to the multiple UI elements or their outputs or any other component of the vehicle 102 or its surroundings according to the (several) sensed and / or recorded features or characteristics. The sensors or cameras of the vehicle 102 may also be configured to output generated data corresponding to the one or more features or characteristics. Any of the multiple sensors or cameras may also be configured to send the generated data corresponding to the one or more features or characteristics to the computing system 104 or other electronic circuit systems of the vehicle 102 (e.g., the body control module 108, the transmission control module 110, and the power steering control unit 112), for example, via the CAN bus 120.
[0041] A set of mechanical components for controlling the driving of vehicle 102 may include: (1) braking mechanisms on the wheels of the vehicle (for stopping the rotation of the wheels), (2) a throttle mechanism on the engine or motor of the vehicle (for regulating the amount of gas entering the engine or the current entering the engine), which determines the speed at which the drive shaft can rotate and therefore the speed at which the vehicle can travel, and (3) a steering mechanism for the direction of the front wheels of the vehicle (e.g., so that the vehicle travels in the direction the wheels are pointing). These mechanisms can control the braking (or deceleration), acceleration (or throttling), and steering of vehicle 102. The user controls the vehicle 102 through UI elements that can be operated by the user (e.g., see Figure 2These mechanisms are indirectly controlled by other components 216 of the vehicle 202 shown in the figure, which are typically brake pedals, accelerator pedals and steering wheels. The pedals and steering wheels are not necessarily mechanically connected to the drive mechanisms for braking, acceleration and steering. Such components may have or be close to sensors that measure the degree to which the driver depresses the pedals and / or turns the steering wheel. The sensed control input is transmitted to the control unit via wires (and therefore may be controlled by wire). Such control units may include a body control module 108 or 220, a transmission control module 110 or 222, a power steering control unit 112 or 224, a battery management system 226, etc. This output may also be sensed and / or recorded by the sensors and cameras described herein (e.g., see sensors 114a to 114b or 217a to 217b and cameras 118a to 118b or 219a to 219b). In addition, the output of the sensors and cameras may be further processed, for example, by the RCMS client 106, and then reported to the (several) servers 150 of the RCMS for cumulative data processing.
[0042] In some embodiments, the vehicle 102 or 202 may include a body, a transmission, and a chassis. The vehicle 102 or 202 may also include a plurality of electronic control units (ECUs) configured to control the driving of the vehicle (e.g., see body control module 108 or 220, transmission control module 110 or 222, and power steering control unit 112 or 224). The vehicle 102 or 202 may also include a plurality of user UI elements configured to be manipulated by the driver to indicate the degree of control applied by the driver (e.g., see Figure 2 202 shown in FIG. 21 ). Multiple UI elements may be configured to measure signals indicative of the degree of control applied by the driver. Multiple UI elements may also be configured to electronically transmit signals to multiple ECUs. ECUs (e.g., see body control module 108 or 220, transmission control module 110 or 222, and power steering control unit 112 or 224) may be configured to generate control signals for driving vehicle 102 or 202 based on measured signals received from multiple UI elements.
[0043] In a vehicle such as vehicle 102 or 202, a driver may control the vehicle via physical control elements (e.g., steering wheel, brake pedal, accelerator pedal, paddle shifters, etc.) that interface with drive components via mechanical linkages and some electromechanical linkages. However, more and more vehicles now interface control elements with mechanical drivetrain elements (e.g., brake system, steering mechanism, drive train, etc.) via electronic control elements or modules (e.g., electronic control units or ECUs). The electronic control elements or modules may be part of a drive-by-wire technology.
[0044] Drive-by-wire technology may include electrical or electromechanical systems for performing vehicle functions traditionally accomplished by mechanical linkages. The technology may replace traditional mechanical control systems with electronic control systems that use electromechanical actuators and human-machine interfaces, such as pedals and steering feel simulators. Components such as steering columns, intermediate shafts, pumps, hoses, belts, coolers, and vacuum servos and master cylinders may be eliminated from the vehicle. There are varying degrees and types of drive-by-wire technology.
[0045] Vehicles with drive-by-wire technology, such as vehicles 102 and 202, may include a modulator (e.g., a modulator including or as part of an ECU and / or ADAS) that receives input from a user or driver (e.g., via more conventional controls or via drive-by-wire controls or some combination thereof). The modulator may then use the driver's input to modulate the input or transform the input to match the input of a "safe driver." The input of the "safe driver" may be represented by a model of the "safe driver."
[0046] The vehicles 102 and 202 may also include ADAS (not depicted). And, as mentioned herein, the RCMS client 106 may be part of, include or be connected to the ADAS. And, therefore, the ADAS may also communicate with (several) RCMS servers 150 (not depicted). The ADAS may be configured to identify the mode in which the driver interacts with the UI element (e.g., see other components 216 including the UI element). The ADAS may also be configured to determine the deviation of the mode from a predetermined model (e.g., a predetermined conventional driver model, a predetermined safe driving model, etc.). In such embodiments and other embodiments, the predetermined model may be derived from the relevant model of the preselected safe driver. Moreover, the predetermined model may be derived from the relevant model of the driver with the preselected driver ability level. The predetermined model may also be derived from the relevant model of the driver with the preselected driving habits. The predetermined model may also be derived from the relevant model of the driver with the preselected driving style. And, the predetermined model may also be derived from its combination.
[0047] The ADAS may also be configured to adjust multiple ECUs (e.g., see body control module 108 or 220, powertrain control module 110 or 222, and power steering control unit 112 or 224) according to the deviation when converting signals measured by the UI elements into control signals for driving the vehicle 102 or 202. For example, the ADAS may be configured to change the transfer function used by the ECUs to control the driving of the vehicle based on the deviation.
[0048] In such and other embodiments, the ADAS may be further configured to adjust multiple ECUs (e.g., body control module 108, transmission control module 110, and power steering control unit 112) according to sensor data indicating environmental conditions of the vehicle when converting signals measured by the UI elements into control signals for driving the vehicle 102 or 202. And, the ADAS may be further configured to determine the response difference between the measured signals generated by the multiple UI elements and the driving decisions autonomously generated by the ADAS according to a predetermined model and sensor data indicating environmental conditions of the vehicle 102 or 202 or its surroundings (e.g., see Figure 1 and 2 In some embodiments, the ADAS may be configured to train the ANN to identify deviations based on the response differences. In such embodiments and other embodiments, to determine the deviations, the ADAS may be configured to input a transmitted signal indicating the degree of control into the ANN. Furthermore, the ADAS may be configured to determine at least one characteristic of the deviation based on the output of the ANN. Furthermore, to train the determination of the deviations, the ADAS may be configured to train the ANN. To train the ANN, the ADAS may be configured to adjust the ANN based on the deviations.
[0049] In such embodiments and others, the multiple UIs may include steering controls (e.g., a steering wheel or a GUI or another type of UI equivalent, such as a voice input UI for steering). Also, the multiple UIs may include braking controls (e.g., a brake pedal or a GUI or another type of UI equivalent, such as a voice input UI for braking). The multiple UIs may also include throttle controls (e.g., an accelerator pedal or a GUI or another type of UI equivalent, such as a voice input UI for accelerating the vehicle). Also, the degree of control applied by the driver may include a detected user interaction with at least one of the steering controls, the braking controls, or the throttle controls, or any combination thereof. In such embodiments and others, the ADAS may be configured to change the transfer function used by the ECU (e.g., the body control module 108 or 220, the transmission control module 110 or 222, and the power steering control unit 112 or 224) to control the driving of the vehicle 102 or 202 based on the deviation. Furthermore, the transfer function may include or be derived from at least one transfer function for controlling a steering mechanism of the vehicle 102 or 202, a throttle mechanism of the vehicle, or a brake mechanism of the vehicle, or any combination thereof. Furthermore, the plurality of UIs may include transmission controls (e.g., a manual transmission and a clutch operated by the driver or a GUI or another type of UI equivalent, such as a voice input UI for changing the gear of the vehicle). Furthermore, the degree of control applied by the driver may include detected user interactions with the transmission controls. The transfer function may include or be derived from a transfer function for controlling a transmission mechanism of the vehicle 102 or 202.
[0050] In some embodiments, the electronic circuit system of a vehicle (e.g., see vehicles 102 and 202) (which may include or be part of the vehicle's computing system) may include at least one of engine electronics, transmission electronics, chassis electronics, passenger environment and comfort electronics, in-vehicle entertainment electronics, in-vehicle safety electronics, or navigation system electronics, or any combination thereof (e.g., see Figure 1 and Figure 2 2, power steering control units 112 and 224, battery management system 226, and infotainment electronics 228 are shown in FIG. 1 . In some embodiments, the electronic circuit system of the vehicle may include an electronic device for an autonomous driving system.
[0051] Adjustable aspects for driving the vehicle 102 or 202 may include driving configurations and preferences that may be adjusted from a controller via the vehicle electronics (e.g., adjusting the transmission, engine, chassis, passenger environment, and safety features via the corresponding vehicle electronics). Driving aspects may also include typical drive-by-wire aspects, such as controlling the steering, braking, and acceleration of the vehicle (e.g., see body control module 108, transmission control module 110, and power steering control unit 112). Aspects for driving the vehicle may also include settings that control different levels of automation according to SAE, such as settings to control no automation preference / configuration (level 0), driver assistance preference / configuration (level 1), partial automation preference / configuration (level 2), conditional automation preference / configuration (level 3), high automation preference / configuration (level 4), or full preference / configuration (level 5). Aspects for driving the vehicle may also include settings that control driving modes, such as sport or performance mode, fuel economy mode, traction mode, all-electric mode, hybrid mode, AWD mode, FWD mode, RWD mode, and 4WD mode.
[0052] In some embodiments, a computing system of a vehicle (e.g., computing system 104 or 204) may include a central control module (CCM), a central timing module (CTM), and / or a general electronic module (GEM). Also, in some embodiments, a vehicle may include an ECU, which may be any embedded system in an automotive electronic device that controls one or more electrical systems or subsystems in the vehicle. Types of ECUs may include an engine control module (ECM), a powertrain control module (PCM), a transmission control module (TCM), a brake control module (BCM or EBCM), a CCM, a CTM, a GEM, a body control module (BCM), a suspension control module (SCM), a door control unit (DCU), etc. Types of ECUs may also include a power steering control unit (PSCU), one or more human-machine interface (HMI) units, a powertrain control module (PCM)—which may function as at least an ECM and a TCM, a seat control unit, a speed control unit, a telematics control unit, a transmission control unit, a brake control module, and a battery management system.
[0053] like Figure 2 , networked system 100 may include at least vehicles 130-132 and vehicle 202, which includes at least vehicle computing system 204, a vehicle body (not depicted) having an interior (not depicted), a transmission system (not depicted), a climate control system (not depicted), and an infotainment system (not depicted). Vehicle 202 may also include other vehicle components.
[0054] The computing system 204, which may have a similar structure and / or functionality to the computing system 104, may be connected to (several) communication networks 122, which may include at least a local to device network (e.g., Bluetooth, etc.), a wide area network (WAN), a local area network (LAN), an intranet, a mobile wireless network (e.g., 4G or 5G), an extranet, the Internet, and / or any combination thereof. The computing system 204 may be a machine capable of executing a set of instructions (sequential or otherwise) that specify actions to be taken by that machine. Moreover, although a single machine is illustrated for the computing system 204, the term "machine" should also be understood to include any collection of machines that individually or collectively execute a set (or multiple sets) of instructions to perform a methodology or operation. And, it may include at least a bus (e.g., see bus 206) and / or a motherboard, one or more controllers (e.g., one or more CPUs, e.g., see controller 208), a main memory (e.g., see memory 210) that may include a temporary data storage device, at least one type of network interface (e.g., see network interface 212), a storage system (e.g., see data storage system 214) that may include a permanent data storage device, and / or any combination thereof. In some multi-device embodiments, one device may complete some parts of the methods described herein, and then send the completion results to another device over a network so that the other device can continue other steps of the methods described herein.
[0055] Figure 2Also illustrated are example components of a computing system 204 that may include and implement a RCMS client 106. The computing system 204 may be communicatively coupled to the network(s) 122 as shown. The computing system 204 includes at least a bus 206, a controller 208 (e.g., a CPU) that may execute instructions of the RCMS client 106, a memory 210 that may hold instructions of the RCMS client 106 for execution, a network interface 212, a data storage system 214 that may store instructions for the RCMS client 106, and other components 216—which may be any type of component found in a mobile or computing device, such as a GPS component, I / O components (e.g., a camera and various types of user interface components) (which may include one or more of the plurality of UI elements described herein), and sensors (which may include one or more of the plurality of sensors described herein). Other components 216 may include one or more user interfaces (e.g., GUIs, auditory user interfaces, tactile user interfaces, automotive controls, etc.), displays, different types of sensors, tactile, audio and / or visual input / output devices, additional dedicated memory, one or more additional controllers (e.g., GPUs), or any combination thereof. Computing system 204 may also include a sensor and camera interface configured to interface with sensors and cameras of vehicle 202 (which may be one or more of any sensors or cameras described herein (e.g., see sensors 217a-217b and cameras 219a-219b)). In some embodiments, bus 206 communicatively couples controller 208, memory 210, network interface 212, data storage system 214, other components 216, sensors and cameras, and sensor and camera interfaces. The computing system 204 includes a computer system that includes at least a controller 208, a memory 210 (e.g., read-only memory (ROM), flash memory, dynamic random access memory (DRAM) (e.g., synchronous DRAM (SDRAM) or Rambus DRAM (RDRAM)), static random access memory (SRAM), cross-point memory, cross-type memory, etc.) and a data storage system 214 that communicate with each other through a bus 206 (which may include multiple buses).
[0056] In some embodiments, the computing system 204 may include a set of instructions that, when executed, cause the machine to perform any one or more of the methodologies discussed herein. In such embodiments, the machine may be connected (e.g., via a network interface 212) to other machines in a LAN, an intranet, an extranet, and / or the Internet (e.g., network(s) 122). The machine may operate as a server or a client machine in a client-server network environment, as a peer machine in a peer-to-peer (or distributed) network environment, or as a server or a client machine in a cloud computing infrastructure or environment.
[0057] Controller 208 represents one or more general purpose processing devices, such as a microprocessor, a central processing unit, etc. More specifically, the processing device may be a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, a single instruction multiple data (SIMD), a multiple instruction multiple data (MIMD), or a processor implementing other instruction sets, or a processor implementing a combination of instruction sets. Controller 208 may also be one or more special purpose processing devices, such as an ASIC, programmable logic (such as an FPGA), a digital signal processor (DSP), a network processor, etc. Controller 208 is configured to execute instructions for performing the operations and steps discussed herein. Controller 208 may further include a network interface device, such as a network interface 212, to communicate through one or more communication networks (such as (several) networks 122).
[0058] The data storage system 214 may include a machine-readable storage medium (also referred to as a computer-readable medium) on which is stored one or more sets of instructions or software embodying any one or more of the methodologies or functions described herein. The data storage system 214 may have execution capabilities, for example, it may at least partially execute instructions residing in the data storage system. The instructions may also reside completely or at least partially within the memory 210 and / or within the controller 208 during execution by the computer system, the memory 210 and the controller 208 also constituting machine-readable storage media. The memory 210 may be or include the main memory of the system 204. The memory 210 may have execution capabilities, for example, it may at least partially execute instructions residing in the memory.
[0059] The vehicle 202 may also have a vehicle body control module 220 for the vehicle body, a transmission control module 222 for the transmission system, a power steering control unit 224, a battery management system 226, an infotainment electronics 228 for the infotainment system, and a CAN bus 218 that connects at least the vehicle computing system 204, the vehicle body control module, the transmission control module, the power steering control unit, the battery management system, and the infotainment electronics. Also, as shown, the vehicle 202 is connected to the network(s) 122 via the vehicle computing system 204. Also, as shown, the vehicles 130-132 and the mobile devices 140-142 are connected to the network(s) 122. And, therefore, communicatively coupled to the vehicle 202.
[0060] The vehicle 202 is also shown with multiple sensors (e.g., see sensors 217a-217b) and multiple cameras (e.g., see cameras 219a-219b), which may be part of the computing system 204. In some embodiments, the CAN bus 218 may connect the multiple sensors and multiple cameras, the vehicle computing system 204, the vehicle body control module, the powertrain control module, the power steering control unit, the battery management system, and the infotainment electronics to at least the computing system 204. The multiple sensors and multiple cameras may be connected to the computing system 204 via its sensor and camera interfaces.
[0061] like Figure 3 , the networked system 100 may include at least a mobile device 302 and mobile devices 140 to 142. The mobile device 302, which may have a somewhat similar structure and / or functionality to the computing system 104 or 204, may be connected to the communication network(s) 122. And, therefore, connected to the vehicles 102, 202, and 130 to 132 and the mobile devices 140 to 142. The mobile device 302 (or mobile device 140 or 142) may include one or more of the multiple sensors mentioned herein, one or more of the multiple UI elements mentioned herein, a GPS device, and / or one or more of the multiple cameras mentioned herein. Therefore, the mobile device 302 (or mobile device 140 or 142) may function similarly to that of the computing system 104 or 204 and may host and run the RCMS client 106.
[0062] Depending on the embodiment, the mobile device 302 may be or include a mobile device, such as a smart phone, a tablet computer, an IoT device, a smart TV, a smart watch, glasses or other smart home appliances, an in-vehicle information system, a wearable smart device, a game console, a PC, a digital camera, or any combination thereof. As shown, the mobile device 302 may be connected to (several) communication networks 122, which include at least a local device network (such as Bluetooth, etc.), a wide area network (WAN), a local area network (LAN), an intranet, a mobile wireless network (such as 4G or 5G), an extranet, the Internet, and / or any combination thereof.
[0063] Each of the mobile devices described herein may be or be replaced by a personal computer (PC), a tablet PC, a set-top box (STB), a personal digital assistant (PDA), a cellular phone, a network appliance, a server, a network router, a switch or a bridge, or any machine capable of executing a set of instructions (sequential or otherwise) that specify actions to be taken by that machine. The computing system of the vehicle described herein may be a machine capable of executing a set of instructions (sequential or otherwise) that specify actions to be taken by that machine.
[0064] Moreover, although a single machine is described for the computing systems and mobile devices described herein, the term "machine" should also be understood to include any collection of machines that individually or collectively execute a set (or multiple sets) of instructions to perform any one or more of the methodologies or operations discussed herein. Moreover, each of the illustrated mobile devices may each include at least a bus and / or motherboard, one or more controllers (e.g., one or more CPUs), a main memory that may include a temporary data storage device, at least one type of network interface, a storage system that may include a permanent data storage device, and / or any combination thereof. In some multi-device embodiments, one device may complete some parts of the methods described herein, and then send the completion results to another device over a network so that the other device can continue with other steps of the methods described herein.
[0065] Figure 3 Also illustrated are example components of a mobile device 302 according to some embodiments of the present disclosure. The mobile device 302 may be communicatively coupled to the network(s) 122, as shown. The mobile device 302 includes at least a bus 306, a controller 308 (e.g., a CPU), a memory 310, a network interface 312, a data storage system 314, and other components 316 (which may be any type of component found in a mobile or computing device, such as a GPS component, an I / O component (e.g., various types of user interface components), and a sensor (e.g., a biometric sensor), and one or more cameras). Other components 316 may include one or more user interfaces (e.g., a GUI, an auditory user interface, a tactile user interface, etc.), a display, different types of sensors, tactile (e.g., a biosensor), audio and / or visual input / output devices, additional dedicated memory, one or more additional controllers (e.g., a GPU), or any combination thereof. The bus 306 communicatively couples the controller 308, the memory 310, the network interface 312, the data storage system 314, and the other components 316. The mobile device 302 includes a computer system that includes at least a controller 308, a memory 310 (e.g., read-only memory (ROM), flash memory, dynamic random access memory (DRAM) (e.g., synchronous DRAM (SDRAM) or Rambus DRAM (RDRAM), static random access memory (SRAM), cross-point memory, cross-type memory, etc.) and a data storage system 314 that communicate with each other through a bus 306 (which may include multiple buses).
[0066] in other words, Figure 331 is a block diagram of a mobile device 302 having a computer system in which embodiments of the present disclosure may operate. In some embodiments, the computer system may include a set of instructions that, when executed, cause the machine to perform some of the methodologies discussed herein. In such embodiments, the machine may be connected (e.g., via a network interface 312) to other machines in a LAN, an intranet, an extranet, and / or the Internet (e.g., network(s) 122). The machine may operate as a server or a client machine in a client-server network environment, as a peer machine in a peer-to-peer (or distributed) network environment, or as a server or a client machine in a cloud computing infrastructure or environment.
[0067] Controller 308 represents one or more general purpose processing devices, such as a microprocessor, a central processing unit, etc. More specifically, the processing device may be a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, a single instruction multiple data (SIMD), a multiple instruction multiple data (MIMD), or a processor implementing other instruction sets, or a processor implementing a combination of instruction sets. Controller 308 may also be one or more special purpose processing devices, such as an ASIC, programmable logic (such as an FPGA), a digital signal processor (DSP), a network processor, etc. Controller 308 is configured to execute instructions for performing the operations and steps discussed herein. Controller 308 may further include a network interface device, such as a network interface 312, to communicate through one or more communication networks (such as (several) networks 122).
[0068] The data storage system 314 may include a machine-readable storage medium (also referred to as a computer-readable medium) on which is stored one or more sets of instructions or software embodying any one or more of the methodologies or functions described herein. The data storage system 314 may have execution capabilities, for example, it may at least partially execute instructions residing in the data storage system. The instructions may also reside completely or at least partially within the memory 310 and / or within the controller 308 during execution by the computer system, the memory 310 and the controller 308 also constituting machine-readable storage media. The memory 310 may be or include the main memory of the device 302. The memory 310 may have execution capabilities, for example, it may at least partially execute instructions residing in the memory.
[0069] Although the memory, controller, and data storage components are shown as each being a single component in the example embodiments, each component should be understood to include a single component or multiple components that can store instructions and perform their corresponding operations. The term "machine-readable storage medium" should also be understood to include any medium that is capable of storing or encoding a set of instructions for execution by a machine and causing the machine to perform any one or more of the methodologies of the present disclosure. Therefore, the term "machine-readable storage medium" should be understood to include, but not limited to, solid-state memory, optical media, and magnetic media.
[0070] like Figure 3 As shown in , the mobile device 302 may include a user interface (e.g., see other components 316). The user interface may be configured to provide a graphical user interface (GUI), a tactile user interface, or an auditory user interface, or any combination thereof. For example, the user interface may be or include a display connected to at least one of a wearable structure, a computing device, or a camera, which may also be part of the mobile device 302, or any combination thereof, and the display may be configured to provide a GUI. Moreover, the embodiments described herein may include one or more user interfaces of any type, including a tactile UI (touch), a visual UI (vision), an auditory UI (sound), an olfactory UI (smell), a balanced UI (balance), and a taste UI (taste).
[0071] Figure 4 The following describes some embodiments of the present disclosure. Figures 1 to 3 400 is a flowchart of an example operation of the method 400 performed by aspects of the networked system depicted in FIG. Figures 1 to 3 The computing system and / or other components of any vehicle and / or mobile device depicted may be executed.
[0072] like Figure 4 , method 400 begins at step 402 with detecting, by at least one sensor, at least one sudden movement of a vehicle or at least one component of a vehicle. The sudden movement may include a change in speed, acceleration, angular velocity, or angular acceleration, or any combination thereof, that exceeds a predetermined threshold. For example, the sudden movement may include a change in speed, acceleration, angular velocity, or angular acceleration, or any combination thereof, in one or more directions that exceeds a corresponding predetermined threshold for the one or more directions.
[0073] At step 404, the method 400 continues with sending, by the sensor(s), movement data derived from the detected at least one sudden movement. At step 406, the method 400 continues with recording, by at least one camera, at least one image of an area of the vehicle within a preselected distance of the vehicle during or after the detection of the at least one sudden movement. At step 408, the method 400 continues with sending, by the camera(s), image data derived from the recorded at least one image. At step 410, the method 400 continues with detecting, by the GPS device, a geographic location of the vehicle during the detection of the at least one sudden movement. At step 412, the method 400 continues with sending, by the GPS device, location data derived from the detected geographic location. At step 414, the method 400 continues with receiving, by the computing system, movement data, location data, and image data. At step 416, the method 400 continues with associating, by the computing system, the received movement data, the received location data, and the received image data. At step 418, method 400 continues with determining by the computing system whether at least one sudden movement detected in the received mobile data exceeds the sudden movement threshold. In some embodiments, the determination may be performed according to AI and the AI may be trained via machine learning. In response to determining at 418 that at least one sudden movement exceeds the sudden movement threshold, method 400 continues (at step 420) by sending the associated data or its derivatives to the road condition monitoring system via a wide area network. Otherwise, method 400 may return to the sudden movement of at least one component of the sensing vehicle or vehicle at step 422 and return to step 402 when the sudden movement is sensed. In this way, if the sudden movement is not significant enough, the resources used to process and send the sensed or recorded data are not used. In other words, this allows efficient crowdsourcing of reporting road conditions to RCMS from abnormal vehicle events.
[0074] Figure 5 The following describes some embodiments of the present disclosure. Figures 1 to 3 Flowchart of example operations of method 500 performed by aspects of the networked system depicted in FIG. For example, method 500 may be performed by Figures 1 to 3, and / or other components of any vehicle and / or mobile device depicted in . As shown, method 500 may begin after method 400, and step 502 may be dependent on the occurrence of step 420 of method 400. At step 502, method 500 begins with a road condition monitoring system receiving movement data, image data, and geographic location data from a computing system in a suddenly moving vehicle. At step 504, method 500 continues with generating hazard information based at least on the received movement data, image data, and geographic location data. At step 506, method 500 continues with sending a portion of the hazard information to a computing system in a vehicle approaching the hazard when the vehicle approaching the hazard is approaching one of the determined geographic locations of the hazard condition and is within a preselected distance of the one of the locations. Also, as shown, Figure 6 The method 600 depicted in FIG. 5 may also occur after the method 500 .
[0075] Figure 6 The following describes some embodiments of the present disclosure. Figures 1 to 3 600 is a flowchart of an example operation of the method 600 performed by aspects of the networked system depicted in FIG. Figures 1 to 3 , and the like. The method 600 may be performed by the computing system and / or other components of any vehicle and / or mobile device depicted in . As shown, the method 600 may start after the method 500, and step 602 may depend on the occurrence of step 506 of the method 500. At step 602, the method 600 begins with receiving and processing data sent from the road condition monitoring system via the wide area network by the computing system. Then, at step 604, the method 600 continues with receiving at least part of the received and processed data by the UI. At step 606, the method 600 continues with providing at least part of the received and processed data to the driver by the UI. Moreover, at step 608, the method 600 continues with receiving the first part of the received and processed data by the first ECU. At step 610, the method 600 continues with controlling the acceleration or deceleration of the vehicle according to the first part of the data by the first ECU. And, at step 612, the method 600 continues with receiving another part of the received and processed data by another ECU. At step 614, method 600 continues with another ECU controlling the steering of the vehicle based on another portion of the data. As shown, there may be more than two ECUs, and more than two portions of the received and processed data. Thus, other components of the vehicle may be controlled based on other portions of the received and processed data. For example, although not depicted, method 600 may continue with a second ECU receiving a second portion of the received and processed data. And, then, method 600 may continue with the second ECU controlling the transmission of the vehicle based on the second portion of the data.
[0076] In some embodiments, it should be understood that the steps of method 400, 500, or 600 can be implemented as a continuous process, for example, each step can be independently operated by monitoring input data, performing operations, and outputting data to subsequent steps. Moreover, such steps of each method can be implemented as a discrete event process, for example, each step is triggered on the event it should trigger and produces a certain output. It should also be understood that Figures 4 to 6 Each of the diagrams in the figure represents a minimum method within a possible larger method of a computer system, the larger method being Figures 1 to 3 The method partially presented in is more complex. Figures 4 to 6 The steps depicted in each of the figures in the diagram may be combined with other steps that feed into and out of other steps from larger methods associated with more complex systems.
[0077] It should be understood that the vehicle described herein may be any type of vehicle unless otherwise specified. The vehicle may include automobiles, trucks, ships, and airplanes, as well as vehicles or vehicle equipment for military, construction, agricultural, or recreational purposes. Electronic devices used by vehicles, vehicle components, or drivers or passengers of vehicles may be considered vehicle electronics. Vehicle electronics may include electronic devices for engine management, ignition, radio, car computers, telematics, in-vehicle entertainment systems, and other components of vehicles. Vehicle electronics may be used with or by ignition and engine and transmission controls, which may be found in vehicles with internal combustion engine power machinery, such as gasoline-powered cars, trucks, motorcycles, ships, airplanes, military vehicles, forklifts, tractors, and excavators. Moreover, vehicle electronics may be used by or with related elements to control electrical systems found in hybrid and electric vehicles (such as hybrid or electric vehicles). For example, electric vehicles may use power electronics for main propulsion engine control and manage battery systems. Furthermore, autonomous vehicles rely almost entirely on vehicle electronics.
[0078] Some portions of the foregoing detailed description have been presented in terms of algorithms and symbolic representations of operations on data bits within a computer memory. These algorithmic descriptions and representations are the means by which those skilled in the art of data processing most effectively convey the substance of their work to others skilled in the art. An algorithm is herein and generally conceived to be a self-consistent sequence of operations leading to a desired result. The operations are those requiring physical manipulation of physical quantities. Typically, although not necessarily, these quantities take the form of electrical or magnetic signals capable of being stored, combined, compared, and otherwise manipulated. At times, it has proven convenient, primarily for common reasons, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, or the like.
[0079] It should be remembered, however, that all of these and similar terms are associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. The present disclosure may be directed to the actions and processes of a computer system or similar electronic computing device that manipulates and transforms data represented as physical (electronic) quantities within the computer system's registers and memories into other data similarly represented as physical quantities in the computer system's memories or registers or other such information storage systems.
[0080] The present disclosure also relates to an apparatus for performing the operations described herein. This apparatus may be specially constructed for the intended purpose, or it may comprise a general purpose computer selectively activated or reconfigured by a computer program stored in the computer. This computer program may be stored in a computer-readable storage medium, such as any type of disk (including floppy disks, optical disks, CD-ROMs, and magneto-optical disks), read-only memory (ROM), random access memory (RAM), EPROM, EEPROM, magnetic or optical cards, or any type of media suitable for storing electronic instructions, each coupled to a computer system bus.
[0081] The algorithms and displays presented herein are not inherently related to any particular computer or other device. Various general purpose systems may be used with the programs according to the teachings herein, or it may prove convenient to construct more specialized equipment to perform the methods. The structures of a variety of these systems will appear as described in the following description. In addition, the present disclosure is not described with reference to any particular programming language. It will be appreciated that a variety of programming languages may be used to implement the teachings of the present disclosure as described herein.
[0082] The present disclosure may be provided as a computer program product or software that may include a machine-readable medium having instructions stored thereon, which instructions may be used to program a computer system (or other electronic device) to perform a process according to the present disclosure. A machine-readable medium includes any mechanism for storing information in a form readable by a machine (e.g., a computer). In some embodiments, a machine-readable (e.g., computer-readable) medium includes a machine (e.g., computer) readable storage medium, such as a read-only memory ("ROM"), a random access memory ("RAM"), a magnetic disk storage medium, an optical storage medium, a flash memory component, etc.
[0083] In the foregoing description, embodiments of the present disclosure have been described with reference to specific example embodiments of the present disclosure. It will be apparent that various modifications may be made to the specific embodiments of the present disclosure without departing from the broader spirit and scope of the embodiments of the present disclosure as set forth in the appended claims. Accordingly, the description and drawings are to be regarded in an illustrative rather than a restrictive sense.
Claims
1. A device comprising: at least one camera configured to record at least one image and provide image data; processor; a memory coupled to the processor; and at least one interface coupled to the memory and the processor and configured to: Receiving danger information wirelessly transmitted from a remote server of a road condition monitoring system RCMS; detecting movement of a first vehicle; receiving first position data indicating a first position of the first vehicle and first movement data indicating a first movement of the first vehicle exceeding a threshold value associated with at least one component of the first vehicle; associating the received first position data with the received first movement data; recording an image when the first movement of the first vehicle is detected to exceed the threshold; receiving the image data; associating received image data with the received first movement data and the received first position data; in response to determining that the first movement of the first vehicle exceeds the threshold, sending the associated image data and the associated first movement data and the first position data to the remote server via a wide area network; inputting the associated image data received by the remote server into artificial intelligence AI; sending danger information from the AI to a second vehicle proximate to the first location, the danger information being generated by the AI based on the first movement data; detecting, via at least one sensor of the second vehicle, a second movement associated with an acceleration characteristic of the second vehicle and second position data indicative of a second position of the second vehicle; deriving second movement data from the detected second movement via the at least one sensor of the second vehicle; associating the second location data with the second movement data; determining that a second movement of the second vehicle exceeds an associated threshold; In response to the determining, sending the associated second movement data and the second location data to the remote server; and The first movement data, the second movement data and the second location data are synthesized to provide more reliable and meaningful hazard information.
2. The apparatus according to claim 1, wherein: the at least one camera being configured to record at least one image of an area within a preselected distance of the first vehicle during a predetermined time period after determining that the first movement of the first vehicle exceeds the threshold; and The at least one interface is configured to send the correlated image data, or a derivative thereof, to the road condition monitoring system.
3. An apparatus according to claim 1, wherein at least one of the interfaces is configured to receive and process first data from the remote server that performs road condition monitoring, and wherein the first data includes information derived from movement and position data sent from other vehicles when located in a geographic location that the first vehicle is approaching. 4 . The apparatus of claim 3 , comprising a user interface configured to provide at least a portion of the first data to a user of the first vehicle.
5. The apparatus of claim 3, comprising an electronic control unit (ECU), the ECU being configured to: receiving at least part of the first data; and Steering of the first vehicle is controlled based on the at least a portion of the first data via at least one electrical system in the first vehicle.
6. The apparatus of claim 3, comprising an electronic control unit (ECU), the ECU being configured to: receiving the first data; and A deceleration of the first vehicle is controlled based on the first data via at least one electrical system in the first vehicle.
7. The apparatus of claim 3, comprising an electronic control unit (ECU), the ECU being configured to: receiving at least part of the first data; and Acceleration of the first vehicle is controlled based on the at least a portion of the first data via at least one electrical system in the first vehicle.
8. A method comprising: detecting, by at least one sensor of a first vehicle, a first movement associated with an acceleration characteristic of the first vehicle; deriving, by the at least one sensor, first movement data from the detected first movement; upon detecting the first movement, recording at least one image by at least one camera of the first vehicle; deriving image data from at least one image recorded via the at least one camera; detecting, by a global positioning system (GPS) device, a geographic location of the first vehicle during detection of the first movement; deriving location data from the detected geographic location by the GPS device; associating, by a computing system of the first vehicle, the first movement data with the location data; associating the image data with the first movement data and the position data; determining, by the computing system based on the first movement data, whether the detected first movement exceeds a predetermined threshold; In response to determining that the first movement exceeds the predetermined threshold, sending the correlated data to a server of a road condition monitoring system RCMS via a wireless communication network; Inputting the correlated data received by the server into the artificial intelligence AI; sending hazard information from the AI to a second vehicle proximate to the detected geographic location, the hazard information being generated by the AI based on the first movement data; detecting, via at least one sensor of the second vehicle, a second movement associated with an acceleration characteristic of the second vehicle and second position data indicative of a second position of the second vehicle; deriving second movement data from the detected second movement via the at least one sensor of the second vehicle; associating the second location data with the second movement data; determining that a second movement of the second vehicle exceeds an associated threshold; In response to the determining, sending the associated second movement data and second location data to the server; and The first movement data, the second movement data and the second location data are synthesized to provide more reliable and meaningful hazard information.
9. The method according to claim 8, wherein: Image data is associated with at least one image of an object located external to the first vehicle.
10. The method according to claim 8, comprising: receiving and processing data from the server by the computing system; Wherein the data from the server comprises information derived from correlated movement and position data received by the server from other vehicles.
11. The method of claim 10, comprising providing at least a portion of the processed data to a user of the second vehicle by a user interface.
12. The method according to claim 10, comprising: receiving, by an electronic control unit (ECU), at least a portion of the processed data; and At least one of steering of the second vehicle, deceleration of the second vehicle, or acceleration of the second vehicle, or any combination thereof, is controlled by the ECU based on the at least portion of the processed data.
13. A method comprising: receiving movement data and geographic location data from a computing system in a plurality of vehicles, wherein movement of each vehicle has been determined to exceed a predetermined threshold, the movement being associated with an acceleration characteristic of the respective vehicle, and the movement being detected by at least one sensor of the respective vehicle; recording an image when the movement of the corresponding vehicle is detected to exceed a predetermined threshold; associating the image with the movement data and the geographic location data; generating, by a remote server, danger information regarding a geographic location of the danger condition based at least on the received movement data and the received geographic location data; sending a portion of the hazard information to a computing system in a first vehicle when the first vehicle is approaching a location in the geographic locations of the hazard condition and is within a preselected distance of the location; detecting a first movement and a first geographic location of the first vehicle; receiving, via the remote server, data associated with the first movement and the geographic location; inputting said data associated with said first movement and said geographic location into artificial intelligence AI; transmitting hazard information from the AI to a second vehicle proximate to the geographic location, the hazard information being generated by the AI based on the data associated with the first movement and the geographic location; detecting, via at least one sensor of the second vehicle, a second movement associated with an acceleration characteristic of the second vehicle and second position data indicative of a second position of the second vehicle; exporting data associated with the second movement via the at least one sensor of the second vehicle; associating the second position data with the second movement data; determining that a second movement of the second vehicle exceeds an associated threshold; In response to the determination, sending the associated second movement data and second location data to the remote server; and The data associated with the first movement, the data associated with the second movement, and the second location data are synthesized to provide more reliable and meaningful hazard information.
14. The method of claim 13, wherein the portion of the hazard information is configured for use in providing an alert to a user of the first vehicle via a user interface in the first vehicle.
15. The method of claim 13, wherein the portion of the hazard information is configured for controlling steering, deceleration, or acceleration of the first vehicle via at least one electrical system in the first vehicle.
16. The method of claim 13, wherein: The received movement data includes respective movement data sent from each vehicle, and the respective movement data is derived from sensed movement of each vehicle; and The received position data includes respective position data sent from each vehicle and the respective position data is associated with the position of each vehicle when the movement is sensed.
17. The method of claim 16, comprising receiving image data from the computing systems in the plurality of vehicles, wherein: determining the geographic location of the hazardous condition based on received image data, the received movement data, and the received geographic location data; The image data includes respective image data derived from at least one image of an area within a preselected distance of each vehicle; and The at least one image is recorded when a respective movement of the respective vehicle is sensed.
18. A method according to claim 17, wherein the portion of the hazard information includes first image data associated with the one location and is configured to at least provide a basis for alerting a user of the first vehicle via a user interface in the first vehicle and presenting an image of the hazard reproduced from the first image data.
