Vehicle control in geo-control area
By developing a geographic control area control system in self-driving vehicles, generating predicted video frames and analyzing GNSS information, monitoring and predicting unsafe behaviors of moving objects in real time, the problem of insufficient risk aversion capabilities of self-driving vehicles in complex environments is solved, and the problem of self-driving vehicles is improved. Safety and accuracy of arrival time.
Patent Information
- Application Number
- CN202411077897.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2019-10-29
- Filing Date
- 2020-10-27
- Publication Date
- 2024-11-19
AI Technical Summary
When self-driving vehicles encounter shading structures, intersections or sensor failures, it is difficult to avoid collisions, overturns and traffic jams in real time, and the limited field of view and dirty sensors may affect occupants' safety and arrival time accuracy.
A vehicle control system for use in geographic control areas has been developed to monitor and predict unsafe behavior of moving objects in real time by generating predicted video frames and analyzing GNSS information, speed, motion and traffic rules, suggest alternative routes and prioritize vehicles safe movement, using drones to navigate in invisible areas.
Improves the real-time risk aversion capability of self-driving vehicles in complex environments, ensures occupants' safety, reduces traffic jams and collision risks, and keeps the vehicle safe in severe weather or low visibility.
Smart Images

Figure CN118977732A_ABST
Abstract
Description
[0001] This application is a divisional application of the invention patent application with application number 202080039825.8, application date October 27, 2020, and name “Vehicle Control in Geographic Control Areas”. Technical Field
[0002] Various embodiments of the present disclosure relate to self-driving technology. More specifically, various embodiments of the present disclosure relate to systems and methods for vehicle control in a geographically controlled area and for predicting future scenarios in real time. Background Art
[0003] Recent advances in self-driving technology have led to the development of various technologies to provide driving assistance for self-driving vehicles. Typically, self-driving vehicles can rely on various onboard sensors to learn and determine the surrounding environment and make driving decisions in real time or near real time. Such sensors (such as image sensors, light detection and ranging (LiDAR) or radar) have a limited field of view. This may be relevant, especially when the surrounding environment includes obstructing structures or intersections. Moreover, in the event of dirty or faulty sensors, self-driving vehicles may have to rely on either user input or other operating sensors on the vehicle. For example, this may potentially affect the riding experience of the occupants, the risk safety of the occupants, or may result in performance loss, such as a large deviation from the set estimated time of arrival (ETA).
[0004] Further limitations and disadvantages of conventional and traditional approaches will become apparent to those skilled in the art by comparing the described systems with some aspects of the present disclosure, as set forth in the remainder of this application and with reference to the accompanying figures. Summary of the invention
[0005] There is substantially provided a system and method for vehicle control in a geographic control area, as shown in at least one of the figures and / or described in conjunction with at least one of the figures, as more fully set forth in the claims.
[0006] These and other features and advantages of the present disclosure may be understood by reading the following detailed description of the disclosure and the accompanying drawings, in which like reference numerals refer to like parts throughout. BRIEF DESCRIPTION OF THE DRAWINGS
[0007] Figure 1 is a diagram illustrating an exemplary environment for vehicle control in a geographic control area according to an embodiment of the present disclosure.
[0008] Figure 2 is a block diagram illustrating an example control system for vehicle control in a geographic control zone according to an embodiment of the present disclosure.
[0009] Figure 3 is a diagram illustrating an exemplary operation for vehicle control in a geographic control area according to an embodiment of the present disclosure.
[0010] Figure 4 is a diagram illustrating an exemplary scenario for vehicle control in a geographic control area according to an embodiment of the present disclosure.
[0011] Figure 5 is a diagram illustrating an exemplary scenario for vehicle control in a geographic control area according to an embodiment of the present disclosure.
[0012] Figure 6 is a diagram illustrating an exemplary scenario for vehicle control in a geographic control area according to an embodiment of the present disclosure.
[0013] Fig. 7A and 7B is a flow chart illustrating exemplary operations for vehicle control in a geographic control zone according to an embodiment of the present disclosure.
[0014] Figure 8 is a flow chart illustrating exemplary operations for generating control information for vehicles in a geographic control area according to an embodiment of the present disclosure.
[0015] Fig. 9 is a flowchart illustrating an exemplary operation for controlling a first vehicle according to an embodiment of the present disclosure. DETAILED DESCRIPTION
[0016] The embodiments described below can be understood in the disclosed system and method for vehicle control in a geographic control area. An exemplary aspect of the present disclosure provides a control system. The disclosed control system acts as a predictive vehicle movement system that helps vehicles with different levels of automation avoid certain situations in real time or near real time, such as collisions, rollovers, unexpected delays, traffic jams, and violations of traffic rules. Such situations may occur due to unsafe behaviors of certain moving objects. For vehicles with a certain level of automation, the disclosed control system can generate a predicted video frame to indicate the possible location of (one or more) moving objects and analyze other information, such as GNSS information / positioning information, speed, movement, traffic rules, etc. of (one or more) moving objects. This can help obtain improved predictions of unsafe behaviors of (one or more) moving objects in advance, especially for (one or more) moving objects that are out of the field of view of a registered vehicle (needing assistance). With the generation of predicted video frames, the disclosed control system can provide reduced time redundancy through continuous image frame analysis and can predict vehicle movement (e.g., unsafe behavior) without trying to reconstruct (one or more) moving objects through real pixel information.
[0017] The disclosed control system can measure the traffic conditions on the route in real time and notify the vehicle of the traffic conditions on the route. In addition, based on the prediction of unsafe behavior, the disclosed control system can suggest alternative routes to such vehicles to avoid certain situations, such as collisions, rollovers, undesirable delays, traffic jams, violations of traffic regulations, etc. The disclosed control system can also prioritize the movement of such vehicles in the geographic control area for such vehicles based on the prediction of unsafe behavior of (one or more) moving objects, traffic conditions, traffic violations, etc.
[0018] The disclosed control system can monitor the movement of objects (such as vehicles, animals, pedestrians, debris, sharp objects and weapons). Based on this movement monitoring, the disclosed control system can predict the safe or unsafe behavior of such objects and take preemptive maneuvers in advance to ensure the safety of occupants in the vehicle. For the safety of the occupants, the disclosed control system can also monitor the movement of objects nearby (within the field of view) or far away (at an intersection or outside the field of view) for certain unsafe behaviors (e.g., violations of traffic rules). The disclosed control system can also track the behavior of the driver or occupants of the vehicle and can override the driver's intentions in the event of social problems (terrorist intent for mass destruction, insurance claims for damage to own cars, etc.). The disclosed control system also utilizes drones mounted on the vehicle to enable such vehicles to navigate around areas / moving objects that are not visible to such vehicles, especially in situations such as road turns, severe weather conditions, low visibility or incorrect trajectories or routes.
[0019] Figure 1 is a diagram illustrating an exemplary environment for vehicle control in a geographic control area according to an embodiment of the present disclosure. Figure 1 , an exemplary environment 100 is shown. The exemplary environment 100 may include a group of mobile objects 102 and a first vehicle 104 in a geographic control area 106. The exemplary environment 100 may also include a control system 108 and a server 110. The control system 108 may include a first electronic device 112. Additionally, in some embodiments, the control system 108 may include a collection of electronic devices 114. The exemplary environment 100 may also include a communication network 116 that may be established between the first vehicle 104, the server 110, the first electronic device 112, and the collection of electronic devices 114.
[0020] The moving object group 102 may be objects that appear within a detectable range of at least one electronic device in the set of electronic devices 114. Additionally or alternatively, from the objects detected in the geographic control area 106, the moving object group 102 may be selected as objects that move within an area of interest of the geographic control area 106. The area of interest may be, for example, a road segment, a network of interconnected roads, a specific lane, or an intersection of roads. Additionally or alternatively, from the objects detected in the geographic control area 106, the moving object group 102 may be selected as objects of at least one object type in a specific group of object types. The specific group of object types may include, for example, vehicles, pedestrians, debris, etc. Examples of the moving object group 102 may include, but are not limited to, moving vehicles, pedestrians, animals, scattered objects, or a combination thereof.
[0021] The first vehicle 104 may be configured to operate according to a particular level of automation, e.g., as one of the different levels of automation (e.g., 0, 1, 2, 3, 4, and 5) defined by the National Highway Traffic Safety Administration (NHTSA). Based on the level of automation, the first vehicle 104 may be one of a semi-autonomous vehicle, a fully autonomous vehicle, or a non-autonomous vehicle. Examples of the first vehicle 104 may include, but are not limited to, a two-wheeled vehicle, a three-wheeled vehicle, a four-wheeled vehicle, a hybrid vehicle, or a vehicle with autonomous driving capabilities using one or more different renewable or non-renewable energy sources. Additional examples of the first vehicle 104 may include, but are not limited to, a fossil fuel-based vehicle, an electric propulsion-based vehicle, a hydrogen fuel-based vehicle, a solar-powered vehicle, and / or a vehicle powered by other forms of alternative energy.
[0022] In some cases, the first vehicle 104 may include functionality or the ability to drive itself from a starting point to a destination point based on various vehicle technologies and sensors, including adaptive cruise control, active steering, anti-lock braking systems (brake-by-wire), global navigation satellite system (GNSS) navigation technology, lasers, cameras, radar, or (one or more) light detection and ranging (LIDAR) systems.
[0023] The geographic control area 106 may be established as a portion of land with defined boundaries. For example, the portion of land may include a road, intersection, crosswalk, pedestrian walkway, highway, interchange, bridge, turnout, or other form of road or driveway. The geographic control area 106 may be established to monitor the mobile object group 102 for unsafe behavior (or potential unsafe behavior) to provide driving assistance to the first vehicle 104 in the geographic control area 106. In some cases, the geographic control area 106 may be established when the first vehicle 104 is detected to be within the communication range of the electronic device group 114.
[0024] The control system 108 may include suitable logic, circuit systems, and interfaces that may be configured to provide driving assistance to the first vehicle 104 as the first vehicle 104 passes through one or more geographic control zones. The control system 108 may be configured to monitor unsafe behavior of at least one of the mobile object groups 102 in the geographic control zone 106. In addition, in some cases, when the first vehicle 104 is in the geographic control zone 106, the control system 108 may be configured to assist the first vehicle 104 in reaching a destination point and / or ensure the safety of the first vehicle 104, the (one or more) occupants of the first vehicle 104, or other objects near the first vehicle 104. Examples of the control system 108 may include, but are not limited to, a vehicle or a network of vehicles, a traffic management system, a server, or a roadside computing infrastructure. In some embodiments, the control system 108 may be part of an advanced driver assistance system (ADAS) of the first vehicle 104.
[0025] The server 110 may include suitable logic, circuitry, and interfaces that may be configured to serve as a data store for traffic information for the group of moving objects 102. Additionally, as a proactive measure, the server 110 may be configured to collect real-time or near real-time information about events that may occur or may have occurred in the geographic control area 106 or anywhere along the current travel route of the first vehicle 104. Such events may potentially affect the ride experience or may cause imminent harm to the occupants of the first vehicle 104. Examples of such events may include, for example, blockades, accidents, terrorist attacks, natural disasters, riots, or any form of violence.
[0026] Additionally or alternatively, the server 110 may be configured to train a neural network (NN) model on a set of test images of the test object group to generate an image predicting an unsafe behavior of at least one test object in the test object group. The server 110 may be implemented as a cloud server that can perform operations through a web application, a cloud application, an HTTP request, a repository operation, a file transfer, etc. Other examples of the server 110 may include, but are not limited to, an in-vehicle server, a database server, a file server, a web server, a media server, an application server, a mainframe server, a cloud server, or other types of servers.
[0027] In some embodiments, the server 110 may be implemented as a plurality of distributed cloud-based resources using several techniques well known to those of ordinary skill in the art. It will be appreciated by those of ordinary skill in the art that the scope of the present disclosure may not be limited to implementing the server 110 and the first electronic device 112 as separate entities. In some embodiments, the functionality of the server 110 may be incorporated in whole or at least in part into the first electronic device 112 without departing from the scope of the present disclosure.
[0028] The first electronic device 112 may include suitable logic, circuit systems, and interfaces, which may be configured to receive traffic information of the mobile object group 102 and predict unsafe behavior of the first mobile object based on the received traffic information. In addition, the first electronic device 112 may be configured to generate first control information including an alternative route for the first vehicle 104 based on the predicted unsafe behavior of the first mobile object. Examples of the first electronic device 112 may include, but are not limited to, a vehicle, an onboard electronic control unit (ECU), an onboard camera, a vehicle external camera, a roadside unit (RSU), a vehicle-to-infrastructure (V2I) communication device, or an aerial drone. In some embodiments, the first electronic device 112 may be part of the ECU or ADAS of the first vehicle 104.
[0029] Each electronic device in the set of electronic devices 114 may include suitable logic, circuit systems, and interfaces that may be configured to collect traffic information, the traffic information including multiple image frames of the group of mobile objects 102 in the geographic control area 106. In addition, in some cases, each electronic device in the set of electronic devices 114 may form a peer-to-peer (P2P) network and share the collected traffic information with the first electronic device 112 and between one or more electronic devices in the set of electronic devices 114. The first vehicle 104 may be controlled based on the generated first control information. In some embodiments, the set of electronic devices 114 may be actively deployed as a traffic monitoring device in the geographic control area 106. In some other embodiments, at least one of the set of electronic devices 114 may be part of a mobile vehicle that enters the geographic control area 106 and acts as a traffic monitoring device for other vehicles (e.g., the first vehicle 104). In this case, when the mobile vehicle leaves the geographic control area 106, such electronic devices may be removed from the set of electronic devices 114.
[0030] Examples of the electronic device set 114 may include, but are not limited to, an onboard ECU, an onboard camera, a vehicle external camera, an RSU, a V2I communication device, and an aerial photography drone. Additional examples of the electronic device set 114 may include, but are not limited to, traffic signals, roadside beacons, and wireless access points.
[0031] The communication network 116 may include a communication medium through which the first vehicle 104, the server 110, the first electronic device 112, and the collection of electronic devices 114 can communicate with each other. The communication network 116 may be a communication medium through which different internal and peripheral electronic components of the first vehicle 104 can communicate with the server 110, the first electronic device 112, and the collection of electronic devices 114. The communication network 116 may be established based on one or more communication standards / protocols such as a controller area network (CAN) protocol, a local interconnect network (LIN) protocol, a media-oriented systems transport (MOST) protocol, a vehicle area network (VAN), or a FlexRay standard. Examples of the communication network 116 may include, but are not limited to, the Internet, an Internet-based mobile ad hoc network (IMANET), a cellular network (such as a 3G, 4G, or 5G network), a cloud network, and / or a wide area network (WAN).
[0032] Various devices in the exemplary environment 100 may connect to the communication network 116 according to various wireless communication protocols. Examples of such wireless communication protocols may include, but are not limited to, IEEE 802.11, 802.11x, 802.15, 802.16, 1609, Worldwide Interoperability for Microwave Access (Wi-MAX), Wireless Access in Vehicular Environments (WAVE), cellular communication protocols, Transmission Control Protocol and Internet Protocol (TCP / IP), User Datagram Protocol (UDP), Hypertext Transfer Protocol (HTTP), LTE, File Transfer Protocol (FTP), ZigBee, EDGE, Li-Fi, and / or other wireless communication protocols.
[0033] In operation, each electronic device in the set of electronic devices 114 may be configured to collect traffic information, including but not limited to a plurality of image frames of the group of mobile objects 102 in the geographic control area 106. As an example, for an intersection of two roads, there may be two electronic devices, namely, a V2I device on the first road and a vehicle on the second road. While the V2I device may collect a first portion of traffic information of the mobile objects on the first road, the vehicle parked on the second road may collect the remaining portion of traffic information of the mobile objects on the second road.
[0034] In some embodiments, the collected traffic information may also include GNSS information of multiple locations in the geographic control area 106. The GNSS information may include, for example, vehicle locations, specific routes, specific intersections, traffic conditions on specific routes, turns on specific routes, accidents on specific routes, etc. In addition, for a mobile vehicle that is one of the mobile objects 102 group, such a mobile vehicle shares frequent updates of time, location, speed, direction, or current travel route to one or more of the set 114 of electronic devices. Traffic information for a specific route may be provided by a specific electronic device in the set 114 of electronic devices. In some cases, the collected traffic information may be shared with a master node (e.g., the first electronic device 112).
[0035] In some embodiments, the control system 108 may include a drone as one of the electronic devices in the set 114 of electronic devices on the first vehicle 104. The first electronic device 112 may be configured to dispatch the drone to the location of the first moving object in the group of moving objects 102. Additionally or alternatively, the first electronic device 112 may be configured to detect a user discomfort level for the first vehicle 104 and dispatch the drone further based on the detected user discomfort level. For example, the first moving object may not be within the field of view (FOV) of the first vehicle 104. The drone may be configured to capture traffic information of the first moving object and share the captured traffic information with the first electronic device 112 via the communication network 116.
[0036] At any given time in the current travel route of the first vehicle 104, the first vehicle 104 may enter the communication range of the control system 108 (e.g., the first electronic device 112 and / or the collection of electronic devices 114). The first electronic device 112 may be configured to detect the first vehicle 104 based on the first vehicle 104 being within the communication range of the control system 108. The first electronic device 112 may also be configured to receive input from the detected first vehicle 104 corresponding to acceptance of setting the geographic control zone 106 across a given distance. Such input may be, for example, a user input, a handshake, a packet signed by a uniquely recognizable signature, an authentication token, or an access credential authenticating the identity of the user (or occupant) of the first vehicle 104. The first electronic device 112 may also be configured to establish the geographic control zone 106 around the detected first vehicle 104 based on the received input.
[0037] The first electronic device 112 may also be configured to receive traffic information (e.g., collected traffic information) from one or more electronic devices in the set of electronic devices 114. The traffic information may include a plurality of image frames of the group of moving objects 102 in the geographic control area 106. In some cases, the traffic information may also include, for example, location information of different moving and / or non-moving objects, 3D models or 3D scan data of the environment around one or more electronic devices, safety or event information associated with certain locations in the current driving route of the first vehicle 104, and one or more of the above.
[0038] For example, the group of moving objects 102 may include vehicles moving in the same or different lanes of a road or different roads, or different directions on a road relative to the first vehicle 104. At any time, there may be several vehicles in the direct field of view of the first vehicle 104, while other vehicles may not be in the field of view of the first vehicle 104 or may be blocked by it. In order to collect traffic information for the group of moving objects 102, one or more electronic devices in the collection 114 of electronic devices may be deployed or assigned to monitor and collect partial traffic information from different areas around the corresponding locations of one or more electronic devices in the geographic control area 106. Once collected, the one or more electronic devices may share the collected partial traffic information with each other and / or with the master node.
[0039] The first electronic device 112 may also be configured to generate a set of image frames of a first moving object in the moving object group 102 based on applying the trained NN model to the received traffic information. The first moving object may correspond to one of a vehicle (autonomous or non-autonomous), a pedestrian, an animal, an aircraft, flying objects, etc. The generated set of image frames may correspond to a set of possible positions of the first moving object at a future time. A plurality of image frames may be provided as input to an initial layer of a trained NN model, which may be stored on the first electronic device 112. The trained NN model may generate a set of image frames as an output of a final NN layer of the trained NN model.
[0040] The first electronic device 112 may also be configured to predict an unsafe behavior of the first moving object based on a set of generated image frames. When the first moving object is not within the field of view (FOV) of the first vehicle, the unsafe behavior of the first moving object may be predicted. In addition or alternatively, when the first moving object is within the field of view (FOV) of the first vehicle, the unsafe behavior of the first moving object may be predicted. For example, in the case of a pedestrian or a vehicle as the first moving object, the predicted unsafe behavior may correspond to a violation of traffic rules (e.g., a pedestrian crossing the road illegally, i.e., crossing the road when the traffic light is still red for pedestrians) or a situation that may cause damage to the first vehicle 104 (e.g., a crash or collision).
[0041] The predicted unsafe behavior may correspond to a violation of a traffic rule by the first moving object in the geographic control area 106. For example, the first moving object may violate a traffic rule because the first moving object may be supposed to travel on the right side of the road. Additionally or alternatively, the predicted unsafe behavior may correspond to, for example, a sudden flash of light, unrecognized behavior of an animal, a noisy and distracting object, etc. It should be noted that the prediction of unsafe behavior may change from one geographic control area to another and from time to time based on changes in traffic rules in different control areas and updates to the traffic rules for the geographic control area 106 when appropriate.
[0042] In some embodiments, the first electronic device 112 may also be configured to determine the motion information of the mobile object group 102 based on the received traffic information. The motion information of the mobile object group 102 may include, for example, one or more of the moving direction, moving speed, driving mode, driving lane, driving side of the road, etc. In this case, the unsafe behavior of the first mobile object may also be predicted based on the determined motion information.
[0043] The first electronic device 112 may also be configured to generate the first control information based on the predicted unsafe behavior. The first control information may include an alternative route for the first vehicle 104. The alternative route may lead to the same destination point that the first vehicle 104 intends to reach. In addition, the first control information may include, for example, a control signal of different vehicle control parameters of the first vehicle 104 to allow the first vehicle 104 to switch to an alternative route and / or avoid a collision with the first moving object. The control signal may correspond to, for example, a steering control, a braking control, an acceleration control, a route change, a safety measure, etc. for the first vehicle 104. The first vehicle 104 may be controlled based on the generated first control information. The first electronic device 112 may apply braking, acceleration, deceleration and / or steering to the first vehicle 104 based on the generated first control information to avoid, for example, a collision with one of the moving objects in the moving object group 102. Alternatively, in some embodiments, the first electronic device 112 may be configured to directly control the first vehicle 104 based on the generated first control information. For example, in this case, the first electronic device 112 may be either part of the ADAS of the first vehicle 104 , part of another vehicle, or a roadside electronic infrastructure in the geographic control area 106 .
[0044] According to an embodiment, the first electronic device 112 may be configured to register a set of vehicles on a control network (e.g., a communication network 116), including the first vehicle 104. The set of registered vehicles may exist in the geographic control area 106. The first electronic device 112 may also be configured to generate second control information for each vehicle in the set of registered vehicles and transmit the generated second control information to each vehicle in the set of registered vehicles. The second control information may be generated based on the predicted unsafe behavior of the first moving object.
[0045] In addition or alternatively, the first electronic device 112 may also be configured to generate third control information including a set of safety measures for the first mobile object, and transmit the generated third control information to the first mobile object. The third control information may be generated based on determining that the first mobile object is a vehicle registered on the control network.
[0046] According to an embodiment, the first electronic device 112 may be configured to receive safety information associated with a subsequent geographic control zone that may be located in the current driving route of the first vehicle 104. Such safety information may correspond to, for example, an event associated with the subsequent geographic control zone. The first electronic device 112 may also be configured to predict a safe route for the first vehicle 104 based on the received safety information and generate fourth control information including the predicted safe route. The first vehicle 104 may also be controlled based on the generated fourth control information.
[0047] Additionally or alternatively, the first electronic device 112 may be configured to determine one or more safe driving zones of the first vehicle 104, and predict a safe route for the first vehicle 104 based on the determined one or more safe driving zones. The first electronic device 112 may also be configured to generate fourth control information including the predicted safe route. The first vehicle 104 may be controlled based on the generated fourth control information while overriding the current driving route preset for the first vehicle 104.
[0048] Figure 2 is a block diagram illustrating an example control system for vehicle control in a geographic control area according to an embodiment of the present disclosure. Figure 1 To illustrate the elements in Figure 2 . refer to Figure 2, a block diagram 200 of a control system 108 is shown. The control system 108 may include a first electronic device 112 and a collection of electronic devices 114. The first electronic device 112 may include a circuit system 202 that may perform operations for vehicle control in a geographic control area, such as the geographic control area 106. The first electronic device 112 may also include a memory 204 and an input / output (I / O) device 206, an image capture device 208, and a network interface 210. The circuit system 202 may be communicatively coupled to the memory 204, the I / O device 206, the image capture device 208, and the network interface 210.
[0049] The circuit system 202 may include suitable logic, circuit systems, and interfaces, which may be configured to execute instructions stored in the memory 204. The instructions may correspond to operations, which may include, but are not limited to, receiving traffic information of the mobile object group 102, generating a set of image frames of the first mobile object based on applying the trained NN model to the received traffic information, predicting an unsafe behavior of the first mobile object based on the generated set of image frames, and generating first control information including an alternative route for the first vehicle 104 based on the predicted unsafe behavior. The circuit system 202 may be implemented based on a variety of processor technologies known in the art. Examples of the circuit system 202 may include, but are not limited to, a graphics processing unit (GPU), a processor, a coprocessor, a central processing unit (CPU), an x86-based processor, a reduced instruction set computing (RISC) processor, an application specific integrated circuit (ASIC) processor, a complex instruction set computing (CISC) processor, and combinations thereof.
[0050] The memory 204 may include suitable logic, circuit systems, and interfaces, which may be configured to store instructions to be executed by the circuit system 202. The memory 204 may also be configured to store traffic information, other event-related information of the first vehicle 104, and / or a trained NN model. Examples of implementations of the memory 204 may include, but are not limited to, a random access memory (RAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a hard disk drive (HDD), a solid-state drive (SSD), a CPU cache, and / or a secure digital (SD) card.
[0051] The I / O device 206 may include suitable logic, circuit systems, and / or interfaces that may be configured to act as an I / O channel / interface between a user and the first electronic device 112 and / or the set of electronic devices 114. The I / O device 206 may be configured to receive input and display output based on the received input. The I / O device 206 may include various input and output devices that may be configured to communicate with different operating components of the first electronic device 112. Examples of the I / O device 206 may include, but are not limited to, a touch screen, a keyboard, a mouse, a joystick, a microphone, and a display screen.
[0052] The image capture device 208 may include suitable logic, circuitry, and / or interfaces that may be configured to capture a plurality of image frames of the moving object group 102 within a field of view (FOV) of the image capture device 208. The image capture device 208 may be implemented as, for example, one of an active pixel sensor (such as a complementary metal oxide semiconductor (CMOS) sensor), a passive pixel sensor (such as a charge coupled device (CCD) sensor), an oversampled binary image sensor, a planar Fourier capture array (PFCA), a backside illuminated (BSI or BI) sensor, and the like.
[0053] The network interface 210 may include suitable logic, circuitry, and / or interfaces that may be configured to facilitate communication between the control system 108 and the server 110 via the communication network 116. The network interface 210 may be configured to facilitate communication between the first electronic device 112, the set of one or more electronic devices 114, and other nodes via the communication network 116. The network interface 210 may be implemented using various known techniques to support wired or wireless communication between the control system 108 and the communication network 116. The network interface 210 may include, but is not limited to, an antenna, a radio frequency (RF) transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a codec (CODEC) chipset, a subscriber identity module (SIM) card, or a local buffer circuitry.
[0054] The network interface 210 may be configured to communicate with a network such as the Internet, an intranet, or a wireless network such as a cellular telephone network, a wireless local area network (LAN), and a metropolitan area network (MAN) via wireless communication. The wireless communication may use one or more of a variety of communication standards, protocols, and technologies such as Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), Wideband Code Division Multiple Access (W-CDMA), Long Term Evolution (LTE), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, Wireless Fidelity (Wi-Fi) such as IEEE 802.11a, IEEE 802.11b, IEEE 802.11g, or IEEE 802.11n, Voice over Internet Protocol (VoIP), Light Fidelity (Li-Fi), Worldwide Interoperability for Microwave Access (Wi-MAX), email protocols, instant messaging, and short message service (SMS).
[0055] like Figure 1 As described in , the functions or operations performed by the control system 108 may be performed by the circuit system 202 of the first electronic device 112. For example, in Figure 3 , 4 , 5, 6, 7 and 8 describe in detail the operations performed by the circuit system 202.
[0056] Figure 3 FIG. 1 illustrates an exemplary operation for vehicle control in a geographic control area according to an embodiment of the present disclosure. Figure 1 and Figure 2 To illustrate the elements in Figure 3 . refer to Figure 3 , a block diagram 300 is shown of exemplary operations for vehicle control in a geographic control area, such as geographic control area 106 .
[0057] In stage 1, the first electronic device 112 may be configured to receive traffic information from one or more electronic devices in the set 114 of electronic devices in the geographic control zone 106. Additionally or alternatively, the traffic information may be received from a first control zone master device (not shown). The first electronic device 112 may be configured to select a second electronic device from the set 114 of electronic devices as the first control zone master device for the geographic control zone 106. This selection may be based on one or more factors, such as, but not limited to, the proximity or distance of the first electronic device 112 to the first vehicle 104, or a communication range with the lowest latency.
[0058] For example, the first control area master device may be on the same road, lane, or within a certain radius around the first vehicle 104 as the first vehicle 104, and may collect traffic information of the group of moving objects 302 in the geographic control area 106. The first control area master device may be configured to receive a retrieval request for the collected traffic information from one or more electronic devices in the set 114 of electronic devices, and share the collected traffic information with one or more electronic devices in the set 114 of electronic devices based on the received retrieval request.
[0059] The traffic information may include a plurality of image frames 304 of the group of moving objects 302 in the geographic control area 106. The plurality of image frames 304 may be captured via the image capture device 208 or an image capture device integrated with the set of electronic devices 114. Such an image capture device may capture the plurality of image frames 304 and transmit the captured plurality of image frames 304 to the first electronic device 112 directly via the communication network 116 or via a control area master device. In some embodiments, one or more electronic devices in the set of electronic devices 114 may also capture GNSS information of a plurality of locations in the geographic control area 106, such as landmarks, intersections, traffic conditions, etc. The captured GNSS information may be included in the traffic information and transmitted to the first electronic device 112 either directly or via a control area master device.
[0060] In stage 2, the first electronic device 112 may be configured to process the received traffic information, which includes the captured multiple image frames 304. The processing of the multiple image frames 304 may generate a set of image frames 306A...306N based on applying a (trained) NN model to the captured multiple image frames 304. The generated set of image frames 306A...306N may correspond to a set of possible positions of the first moving object 308 in the moving object group 302 at a future time.
[0061] The NN model can be pre-trained on a training data set that includes input-output image pairs of moving objects. For example, in the input-output image pair of the moving object, the input image can represent the initial position as the initial state of the moving object, and the output image can represent the next position as the next state of the moving object. The NN model can be pre-trained to predict multiple discrete distributions of the first moving object 308 in the moving object group 302. The NN model can be pre-trained to predict the movement of the first moving object 308 (e.g., vehicle movement) without attempting to reconstruct the first moving object 308 through real pixel information. The trained NN model can apply the predicted multiple discrete distributions separately to the first image frame of the first moving object 308 from the multiple image frames 304. The trained NN model can use a convolution mechanism (e.g., using fully connected weights) to calculate the expected value of the motion distribution of each pixel of the first image frame, and apply the motion distribution to each pixel of the first image frame, thereby generating a first predicted image frame of the set of image frames 306A...306N. As an example, the first image frame in the multiple image frames 304 can be obtained based on the equation (1) given as follows: To generate a set of predicted image frames
[0062]
[0063] where k can be the normalized spatial size of the predicted convolution kernel, and It can be applied to the first image frame To generate a collection of image frames Multiple transformations (or motion distributions) of .
[0064] In some cases, the set of generated image frames 306A...306N may include a specific number of image frames, which may be the optimal result of the trained NN model within a given time frame. In addition or alternatively, the trained NN may also output a prediction result as an indication of an unsafe behavior of the first moving object 308 in the set of generated image frames 306A...306N. In some cases, the unsafe behavior of the first moving object 308 may be determined based on motion information of the group of moving objects 302, and the motion information of the group of moving objects 302 is determined based on the received traffic information. The motion information may include, for example, one or more of a moving direction, a moving speed, a driving mode, a driving lane, a driving side of a road, etc.
[0065] The predicted unsafe behavior may correspond to the first moving object 308 violating a traffic rule. Additionally or alternatively, the predicted unsafe behavior may correspond to, for example, sudden flashes of light, unrecognized behavior of animals, noisy and distracting objects, and the like. In some embodiments, the trained NN model may also utilize GNSS information in the traffic information to predict unsafe behavior. As an example, GNSS information may be helpful in the case of turns and intersections in the road. For example, the trained NN model may determine that the moving object will turn right in "2" seconds, and therefore will not recognize the behavior of the first moving object 308 as unsafe at a future moment.
[0066] The trained NN model may output a set 306A...306N of image frames corresponding to a set of possible positions of the first moving object 308 at a future time. The trained NN model may output a likelihood score for each image frame in the set 306A...306N of generated image frames. The likelihood score may determine the probability that the first moving object 308 is at one of the positions determined by the trained NN model at a future time. In some embodiments, the trained NN model may output a prediction result as an indication of unsafe behavior of the first moving object 308 based on a comparison between the likelihood scores of each image frame in the set 306A...306N of generated image frames.
[0067] Some examples of trained NN models may include, but are not limited to, convolutional neural networks (CNNs), fast region-adaptive (R) CNNs, very fast RCNNs, artificial neural networks (ANNs), or multi-step CNNs. In some cases, the trained neural network model may be a hybrid network of CNNs and other deep NN types, such as recurrent neural networks (RNNs), RNNs based on long short-term memory (LSTM), RNNs based on connectionist temporal classification (CTC), or generative adversarial networks (GANs).
[0068] In some embodiments, the trained NN model may be implemented on a dedicated neural network circuit system of the first electronic device 112. In some other embodiments, the trained NN model may be stored as a program code, a neural model, and / or an artificial intelligence (AI)-based application on the first electronic device 112 or the server 110. Additionally or alternatively, the trained NN may be implemented as a combination of program code and a dedicated neural network circuit system.
[0069] The first electronic device 112 may be configured to predict an unsafe behavior of the first moving object 308 based on the set 306A…306N of generated image frames. As an example, the unsafe behavior may be predicted based on the prediction result of the trained NN model. In addition, in some cases, as described above, the first electronic device 112 may utilize motion information to determine the unsafe behavior of the first moving object 308. In some embodiments, the first electronic device 112 may also be configured to determine the possibility of a collision between the first vehicle 104 and the first moving object 308 based on the set 306A…306N of generated image frames, and predict the unsafe behavior of the first moving object 308 based on the determined possibility of a collision.
[0070] At stage 3, a final image frame 310 corresponding to the possible position of the first moving object 308 may be selected from the set 306A ... 306N of generated image frames. The final image frame 310 may be selected as the image frame with the highest likelihood score in the set 306A ... 306N of generated image frames. The final image frame 310 may include, for example, a directional arrow 310A and a directional arrow 310B to indicate the safe behavior and unsafe behavior of the first moving object 308 at a future moment, respectively. For example, in the final image frame 310, two vehicles may be colliding. The directional arrow 310A may indicate that if the first moving object 308 changes its moving direction to the left, the first moving object 308 may remain safe; and the directional arrow 310B may indicate that if the first moving object 308 changes its moving direction to the right, the first moving object 308 may remain unsafe. The directional arrow 310A and the directional arrow 310B may also be used to provide a recommendation to select the moving direction of the first vehicle 104.
[0071] The first electronic device 112 may also be configured to generate first control information for the first vehicle 104. The first control information may include an alternative route for the first vehicle 104 in the geographic control area 106. In addition or alternatively, the first control information may include information about the alternative route. Such information may include, for example, traffic conditions on the alternative route and an estimated time of arrival (ETA) to reach the destination via the alternative route. In addition or alternatively, the first control information may also include control signals for braking control, steering control, acceleration control, clutch control, and / or gear control of the first vehicle 104.
[0072] In some embodiments, the first electronic device 112 may be configured to predict a first driving route of the first moving object 308 based on the predicted unsafe behavior of the first moving object 308, and determine the current driving route of the first vehicle 104. The first electronic device 112 may also be configured to determine the possibility of a collision between the first vehicle 104 and the first moving object 308 based on the predicted first driving route and the determined current driving route. In this case, the first control information (including an alternative route for the first vehicle 104) may be generated based on the determined possibility of a collision between the first vehicle 104 and the first moving object 308.
[0073] The first vehicle 104 may be controlled based on the first control information. Alternatively, the first electronic device 112 may be configured to control the first vehicle 104 based on the generated first control information. The current driving route of the first vehicle 104 may be changed according to the alternative route as specified in the first control information. The alternative route may be a safe route for the first vehicle 104, and by moving along the alternative route, the first vehicle 104 may take a preemptive maneuver to avoid a collision with the first moving object 308.
[0074] In some scenarios, it may be determined that a collision of the first vehicle 104 with the first moving object 308 is inevitable based on the current speed of either or both of the first vehicle 104 and the first moving object 308. In some other scenarios, it may be determined that the first vehicle 104 does not have enough time to change the current driving route to an alternative route, and the likelihood of the first vehicle 104 colliding with the first moving object 308 is greater than a threshold. Additionally or alternatively, the first electronic device 112 may predict the impact of a collision of the first vehicle 104 with the first moving object 308. In such a scenario, as a preemptive maneuver, the first electronic device 112 may be configured to control the safety hardware of the first vehicle 104 to mitigate the effect of the impact on the first vehicle 104. For example, if the predicted impact is small, the first electronic device 112 may be configured to control the inflation of the airbags in the first vehicle 104 before the collision. Alternatively, if the predicted impact is large (e.g., a rollover situation), the first electronic device 112 may be configured to control the first vehicle 104 to perform a preemptive counter rollover maneuver to ensure that the first vehicle does not roll over before, during, or after the collision with the first moving object 308. For example, such a preemptive counter rollover maneuver may include controlling the axle damping position in the first vehicle 104 to balance the center of gravity, thereby avoiding rollover of the first vehicle 104.
[0075] In some scenarios, the traffic information may include multiple image frames of flying debris (e.g., metal sheets) caused by a tornado in the geographic control area 106. The first electronic device 112 may be configured to process the multiple image frames and predict unsafe behavior of the flying debris. The first electronic device 112 may also be configured to generate first control information including an alternative route for the first vehicle 104 based on the predicted unsafe behavior of the flying debris. The first vehicle 104 may be controlled to change the current driving route to an alternative route based on the first control information. The alternative route may be a safe route in which the probability of the flying debris colliding with the first vehicle 104 is lower than a threshold (e.g., very low (<0.01) or close to zero).
[0076] Figure 4 FIG. 1 is a diagram illustrating an exemplary scenario for vehicle control in a geographic control area according to an embodiment of the present disclosure. Figure 1 , Figure 2 and Figure 3 To illustrate the elements in Figure 4 . refer to Figure 4 , a block diagram 400 is shown for an exemplary scenario of vehicle control in a geographic control area. Figure 4 , a plurality of routes 402A, 402B, 402C, and 402D in a geographic control area 404 are shown. Also shown are groups of moving objects 406A and 406B and a first vehicle 408. The geographic control area 404 may also include a plurality of control area masters, such as a first control area master 410A and a second control area master 410B. The first control area master 410A and the second control area master 410B may be, for example, V2I devices installed near an intersection on the route 402A and may be part of the control system 108.
[0077] Initially, the first vehicle 408 may be configured to follow a current driving route 412, which may lead to a destination location that the first vehicle 408 intends to reach. At some point, the first vehicle 408 may enter a first communication range of a first control zone master device 410A, which may be part of the control system 108. The first communication range of the first control zone master device 410A may correspond to a range in which the first control zone master device 410A may assist the first vehicle 408.
[0078] The first control zone master device 410A may be configured to detect the first vehicle 408 within the first communication range by using an image capture device associated with the first control zone master device 410A. In addition or alternatively, the first vehicle 408 may be detected by using a radar sensor (not shown) or a roadside beacon (not shown) or a LiDAR (not shown). The first control zone master device 410A may also be configured to transmit a query to the detected first vehicle 408 via the communication network 116 to establish a geographic control zone 404 across a given distance. The first vehicle 408 may be configured to receive the query and respond to the query by input corresponding to the acceptance of the geographic control zone 404 set to the given distance. The first control zone master device 410A may also be configured to establish a geographic control zone 404 around the first vehicle 408 and notify the second control zone master device 410B to establish a geographic control zone 404 around the first vehicle 408. The geographic control zone 404 established in this way may also include other control zone masters, including but not limited to the second control zone master device 410B.
[0079] The first control area master device 410A can be configured to collect traffic information of mobile object groups 406A and 406B from one or more electronic devices within the first communication range of the first control area master device 410A. Thereafter, the first control area master device 410A can be configured to share the collected traffic information with other electronic devices (including but not limited to the second control area master device 410B). Similarly, the second control area master device 410B can be configured to collect traffic information from one or more electronic devices within the second communication range of the second control area master device 410B. The second control area master device 410B can also be configured to share the collected traffic information with other electronic devices (including but not limited to the first control area master device 410A). Mobile object groups 406A and 406B can include registered vehicles and / or unregistered vehicles. Registered vehicles can include vehicles that can use services (such as driving assistance) provided by the control system 108. And unregistered vehicles can include vehicles that cannot use services provided by the control system 108.
[0080] In some embodiments, the first control area master device 410A can be configured to receive traffic information from the second control area master device 410B. The first control area master device 410A can also be configured to determine that the first mobile object 406A in the mobile object groups 406A and 406B belongs to a set of unregistered vehicles. The first control area master device 410A can also be configured to process the received traffic information and predict the unsafe behavior of the first mobile object 406A. The processing of traffic information may include applying the trained NN model to the received traffic information. The unsafe behavior of the first mobile object 406A can correspond to, for example, a violation of traffic rules, which may be different from one geographic area to another.
[0081] The first control zone master device 410A may also be configured to generate first control information including an alternative route 414 for the first vehicle 408. The first control information may be transmitted to the first vehicle 408. The first vehicle 408 may process the first control information and change its route from the current driving route 412 to the alternative route 414 based on the first control information. In some cases, the first control zone master device 410A may be configured to control the first vehicle 408 to change its route from the current driving route 412 to the alternative route 414 based on the first control information. In some other cases, if the first control zone master device 410A determines that the probability of a collision between the first vehicle 408 and the first moving object 406A is very low, the current driving route 412 of the first vehicle 408 may remain unchanged.
[0082] In some embodiments, the first control zone master device 410A may be configured to register a set of vehicles including but not limited to the first vehicle 408 on the control network. The set of registered vehicles may be in the geographic control zone 404. The first control zone master device 410A may also be configured to generate second control information for each vehicle in the set of registered vehicles and transmit the generated second control information to each vehicle in the set of registered vehicles. The second control information may be generated based on the predicted unsafe behavior of the first mobile object 406A. For example, the second control information may include an alternative path for each vehicle in the set of registered vehicles that may have the possibility of colliding with the first mobile object 406A. The second control information may be directly transmitted to each vehicle in the set of registered vehicles by the first control zone master device 410A via the communication network 116. Alternatively, the first control zone master device 410A may be configured to share the second control information with each electronic device in the set of electronic devices in the geographic control zone 404. Each electronic device in the set of electronic devices may also be configured to share the second control information with the corresponding vehicle within their respective communication ranges.
[0083] In certain embodiments, the first control zone master device 410A may also be configured to generate third control information based on a determination that the first mobile object is a vehicle registered on the control network.
[0084] The third control information may include a set of safety measures for the first mobile object 406A. The set of safety measures may include, for example, an alarm message, a brake control, a change of lane, a steering control, etc. The first control zone master device 410A may be configured to transmit the third control information to the first mobile object 406A. In addition, in some cases, the first control zone master device 410A may also be configured to control the first mobile object 406A based on the third control information.
[0085] It should be noted that the above operations of the first control zone master device 410A may be performed by the second control zone master device 410B or any other electronic device in the set of electronic devices that may be part of the geographic control zone 404. For example, the first control zone master device 410A may communicate with Figure 1 Corresponding to the first electronic device 112.
[0086] In some scenarios, there may be only one control zone master device in the geographic control zone 404. Additionally or alternatively, there may be one control zone master device and multiple control zone slave devices in the geographic control zone 404. The control zone slave devices may only collect traffic information and pass it to the control zone master device, which may predict unsafe behavior of the first moving object 406A.
[0087] In some other scenarios, the control zone master device may be a collection of vehicles moving on a route (e.g., route 402A). Each vehicle in the collection of vehicles may collect traffic information within the FOV area of (one or more) vehicle cameras and share the traffic information with each other within the communication range. Each vehicle in the collection of vehicles may predict an unsafe behavior of a second vehicle within the communication range and notify other vehicles of the predicted unsafe behavior of the second vehicle.
[0088] In certain other scenarios, a first vehicle in a set of vehicles moving on a route may predict an unsafe behavior of a second vehicle (i.e., a first moving object) when the second vehicle is within the FOV of a vehicle camera mounted on the first vehicle. In some cases, the second vehicle may not be within the field of view (FOV) of the vehicle camera of the first vehicle. The first vehicle may be configured to predict an unsafe behavior of the second vehicle based on traffic information from other vehicles in the set of vehicles in the geographic control area 404. In certain other scenarios, the first vehicle may be configured to predict an unsafe behavior of the second vehicle based on traffic information collected by a vehicle, an onboard ECU, an onboard camera, a vehicle external camera, an RSU, a V2I communication device, an aerial drone, or a combination thereof.
[0089] Figure 5 FIG. 1 is a diagram illustrating an exemplary scenario for vehicle control in a geographic control area according to an embodiment of the present disclosure. Figure 1 , Figure 2 , Figure 3 and Figure 4 To illustrate the elements in Figure 5 . refer to Figure 5 , a block diagram 500 is shown for an exemplary scenario for vehicle control in a geographic control area.
[0090] exist Figure 5, multiple routes are shown, such as a first route 502A and a second route 502B. A first control zone master 504A and a second control zone master 504B are also shown. The first control zone master 504A or the second control zone master 504B can provide driving assistance to a collection of registered vehicles in the geographic control zone 508, which includes the first vehicle 506. Moving object groups 510A and 510B and the first vehicle 506 are also shown. In some embodiments, both the first control zone master 504A and the second control zone master 504B can provide driving assistance to the first vehicle 506 from time to time based on the proximity of one of the first control zone master 504A or the second control zone master 504B to the first vehicle 506.
[0091] At a certain moment, the first vehicle 506 can be detected by the first control area master device 504A. The first control area master device 504A can be configured to establish a geographic control area 508 around the detected first vehicle 506 based on the input from the first vehicle 506. The first control area master device 504A can be configured to collect traffic information from one or more electronic devices in the set of electronic devices within the communication range of the first control area master device 504A. The collected traffic information can include multiple image frames of mobile object groups 510A and 510B on the first route 502A (same as the route of the first vehicle 506). The traffic information can be sufficient to predict an alternative route to the intersection 512 (if any unsafe behavior of the mobile object is predicted). In addition, in some cases, the first control area master device 504A can be configured to request traffic information from the second control area master device 504B. The second control area master device 504B can collect traffic information and transmit it to the first control area master device 504A. The traffic information received from the second control area master device 504B can correspond to the second route 502B.
[0092] The first control area master device 504A may be configured to process traffic information to predict an unsafe behavior of the first mobile object 510A in the mobile object groups 510A and 510B. For example, when the first vehicle 506 arrives at the intersection 512, the unsafe behavior of the first mobile object 510A may be associated with the possibility of a collision with the first vehicle 506. The processing of traffic information may correspond to applying a trained NN model to the traffic information. Then, the first control area master device 504A may generate first control information, which may include an alternative route for the first vehicle 506.
[0093] In some embodiments, the second control zone master device 504B may be part of a subsequent geographic control zone located in the current driving route of the first vehicle 506. The second control zone master device 504B may collect safety information corresponding to the second route 502B from a collection of electronic devices within the communication range of the second control zone master device 504B. The second control zone master device 504B may determine events registered or likely to occur in the subsequent geographic control zone based on the safety information. Such events may correspond to, but are not limited to, accidents, terrorist attacks, crash routes, severe weather conditions (such as tornadoes or heavy rains), etc. The second control zone master device 504B may transmit safety information and / or determined events associated with the subsequent geographic control zone to the first control zone master device 504A.
[0094] The first control zone master device 504A may be configured to receive and process the safety information and predict a safe route for the first vehicle 506 based on the processed safety information. The safe route may be a preemptive measure to avoid the first vehicle 506 from encountering such an event and to ensure that the first vehicle 506 remains safe during the journey. The first control zone master device 504A may be configured to generate fourth control information, which may include the predicted safe route for the first vehicle 506. The first vehicle 506 may also be controlled based on the generated fourth control information. In some embodiments, the first control zone master device 504A may also be configured to share the safety information with all vehicles in the corresponding control zone via the communication network 116.
[0095] In some embodiments, based on the received safety information, it can be determined that the first vehicle 506 may not remain safe if the first vehicle 506 enters a subsequent geographic area, for example, a traffic collision. In this case, the first control area master device 504A can be configured to determine one or more safe driving areas (not shown) for the first vehicle 506. Such safe driving areas can guide the first vehicle 506 to the intended destination. The first control area master device 504A can also be configured to predict the safe route of the first vehicle 506 based on the determined one or more safe driving areas. The first control area master device 504A can also be configured to generate a fourth control information including the predicted safe route of the first vehicle 506. The first vehicle 506 can be controlled based on the generated fourth control information. For example, the first vehicle 506 can be controlled by using the predicted safe route to override the current driving route preset for the first vehicle 506.
[0096] In some embodiments, the first control zone master 504A and the second control zone master 504B may be vehicles present in the geographic control zone 508. The vehicle may be a mobile vehicle and may therefore leave the geographic control zone after a period of time. In this scenario, the control zone master may be selected from a set of electronic devices in the newly established geographic control zone 508 around the first vehicle 506. In some other embodiments, a priority value may be associated with each electronic device in the set of electronic devices. In this scenario, the priority values of each electronic device currently present in the newly established geographic control zone may be compared, and the control zone master may be selected based on the comparison of the priority values.
[0097] Figure 6 FIG. 1 is a diagram illustrating an exemplary scenario for vehicle control in a geographic control area according to an embodiment of the present disclosure. Figure 1 , Figure 2 , Figure 3 , Figure 4 and Figure 5 To illustrate the elements in Figure 6 . refer to Figure 6 , a block diagram 600 is shown for an exemplary scenario for vehicle control in a geographic control area.
[0098] exist Figure 6 , multiple routes 602A and 602B are shown. Also shown are groups of moving objects 604A and 604B and a first vehicle 606. Also shown are a first control zone master 608A and a second control zone master 608B. The first control zone master 608A can provide driving assistance to registered vehicles (such as the first vehicle 606) in the geographic control zone 610. Also shown is a drone 612, which can initially be docked at a docking location (not shown) on the first vehicle 606.
[0099] At some point, the first vehicle 606 may enter the communication range of the first control zone master 608A. The first control zone master 608A may be configured to detect the first vehicle 606 and establish a geographic control zone 610 around the detected first vehicle 606 based on input from the first vehicle 606.
[0100] In some scenarios, the first moving object 604A in the moving object groups 604A and 604B may not be within the FOV of the first vehicle 606. In addition or alternatively, the user associated with the first vehicle 606 may feel uncomfortable. This discomfort may be due to stress caused by, for example, the driving patterns of other vehicles, medical emergencies, news of any events on the road, etc. The first control zone master device 608A may be configured to detect the user discomfort level for the first vehicle based on certain parameters (e.g., user biosignals, user movement, or data collected from various other sensors in the first vehicle 606). The first control zone master device 608A may also be configured to dispatch a drone 612 to the location of the first moving object 604A on the second route 604B. The drone 612 may be dispatched based on the determination that the first moving object 604A is not within the FOV of the first vehicle 606. In addition or alternatively, the drone 612 may be dispatched based on the detection of the user discomfort level.
[0101] The drone 612 may be configured to capture traffic information of the first moving object 604A and transmit the captured traffic information to the first control zone master device 608A. The first control zone master device 608A receives the captured traffic information from the drone 612 and generates dynamic motion-based point cloud data for the landing path of the drone 612. The captured traffic information may be used to predict unsafe behavior of the first moving object 604A and further control the first vehicle 606 along an alternative route to avoid collision with the first moving object 604A.
[0102] In some scenarios, the image capture device 208 of the first vehicle 606 may experience a malfunction. In such a scenario, the first vehicle 606 may be configured to detach and dispatch the drone 612 to capture traffic information of the first moving object 604A, which may or may not be within the FOV of the first vehicle 606.
[0103] In some other scenarios, the drone 612 may capture traffic information and transmit it to the first vehicle 606 or the first control zone master device 608A until the first control zone master device 608A predicts a safe route for the first vehicle 606. After predicting a safe route, the drone 612 may land at a docking location on the first vehicle 606. In some other scenarios, the drone 612 may capture traffic information within a limited time frame and then dock at a docking location on the first vehicle 606.
[0104] Fig. 7A and 7B is a flow chart illustrating an exemplary operation for vehicle control in a geographic control area according to an embodiment of the present disclosure. Figure 1 , 2, 3, 4, 5 and 6 to illustrate Fig. 7A and 7B . refer to Fig. 7A and 7B , a flowchart 700 is shown. Operations from 702 to 720 may be performed in a system (e.g., Figure 2 Operations may begin at 702 and proceed to 704.
[0105] At 704, an operation of acquiring traffic information of the moving object group 102 may be initiated. According to an embodiment, the first electronic device 112 may be configured to initiate an operation of acquiring traffic information of the moving object group 102. Such information may include, for example, a plurality of image frames of the moving object group 102, GNSS information or positioning information of the moving object group 102, its speed, acceleration, object type, or other information.
[0106] At 706, it may be determined whether the user of the first vehicle 104 is comfortable. According to an embodiment, the first electronic device 112 may be configured to determine whether the user of the first vehicle 104 is comfortable. If the user is comfortable, control may be transferred to 708. Otherwise, control may be transferred to 710.
[0107] At 708, it may be determined whether the first moving object in the group of moving objects 102 is present within the FOV of the first vehicle 104. According to an embodiment, the first electronic device 112 may be configured to determine whether the first moving object is present within the FOV of the first vehicle 104. In the event that the first moving object is present within the FOV, control may proceed to 714. Otherwise, control may proceed to 710.
[0108] At 710 , the drone 612 may be dispatched. According to an embodiment, the first electronic device 112 may be configured to dispatch the drone 612 .
[0109] At 712, the drone 612 may be controlled to capture traffic information. According to an embodiment, the first electronic device 112 may control the drone 612 to capture traffic information.
[0110] At 714, traffic information may be received, the traffic information including a plurality of image frames of the group of moving objects 102 in the geographic control area 106. According to an embodiment, the first electronic device 112 may be configured to receive the traffic information including, but not limited to, a plurality of image frames of the group of moving objects 102 in the geographic control area 106. The traffic information may be received from one or more electronic devices in the set of electronic devices 114.
[0111] At 716, a set of image frames of the first moving object of the moving object group 102 may be generated based on applying the trained NN model to the received traffic information. According to an embodiment, the first electronic device 112 may be configured to generate a set of image frames of the first moving object based on applying the trained NN model to the received traffic information.
[0112] At 718, an unsafe behavior of the first moving object may be predicted based on the set of generated image frames. According to an embodiment, the first electronic device 112 may be configured to predict an unsafe behavior of the first moving object based on the set of generated image frames.
[0113] At 720, first control information including an alternative route for the first vehicle 104 based on the predicted unsafe behavior may be generated. According to an embodiment, the first electronic device 112 may be configured to generate first control information including an alternative route for the first vehicle 104 based on the predicted unsafe behavior. Control may be passed to end.
[0114] Figure 8 is a flow chart illustrating exemplary operations for generating control information for vehicles in a geographic control area according to an embodiment of the present disclosure. Figure 1 , 2 , 3, 4, 5, 6, 7A and 7B to illustrate Figure 8 . refer to Figure 8 , a flowchart 800 is shown. Operations from 802 to 816 may be performed by a system (e.g., Figure 2 The operation may start at 802 and proceed to 804.
[0115] At 804, traffic information may be received, the traffic information including a plurality of image frames of the group of moving objects 102 in the geographic control area 106. According to an embodiment, the first electronic device 112 may be configured to receive the traffic information including, but not limited to, a plurality of image frames of the group of moving objects 102 in the geographic control area 106. The traffic information may be received from one or more electronic devices in the set 114 of electronic devices.
[0116] At 806, an unsafe behavior of the first moving object may be predicted based on the set of generated image frames. According to an embodiment, the first electronic device 112 may be configured to predict an unsafe behavior of the first moving object based on the set of generated image frames. Figure 3 , 4 Details of the operations for predicting unsafe behaviors are provided in , 5 , and 6 .
[0117] At 808, it may be determined whether the first mobile object is registered in the geographic control zone 106. According to an embodiment, the first electronic device 112 may be configured to determine whether the first mobile object is registered in the geographic control zone 106. If the first mobile object is registered, control may be passed to 810. Otherwise, control may be passed to 812.
[0118] At 810, second control information may be generated for each vehicle in the set of registered vehicles. According to an embodiment, the first electronic device 112 may be configured to generate second control information for each vehicle in the set of registered vehicles. The second control information may be generated based on the predicted unsafe behavior of the first moving object, and may include, for example, an alternative path for each vehicle in the set of registered vehicles that may have a possibility of colliding with the first moving object.
[0119] At 812, third control information including a set of safety measures for the first mobile object may be generated. According to an embodiment, if the first mobile object is registered in the geographic control area 106, the first electronic device 112 may be configured to generate third control information for the first mobile object. The set of safety measures may include, for example, an alarm message, a brake control, a lane change, a steering control, etc.
[0120] At 814, the second control information may be transmitted to each vehicle in the set of registered vehicles. According to an embodiment, the first electronic device 112 may be configured to transmit the second control information to each vehicle registered in the geographic control area. Each vehicle in the set of registered vehicles may change their current driving route to their corresponding alternative route based on the transmitted second control information.
[0121] At 816, the third control information may be transmitted to the first mobile object. According to an embodiment, the first electronic device 112 may be configured to transmit the third control information to the first mobile object. The first mobile object may be controlled based on the transmitted third control information. Control may be passed to the end.
[0122] Fig. 9 is a flowchart illustrating an exemplary operation for controlling a first vehicle according to an embodiment of the present disclosure. Figure 1 , 2 , 3, 4, 5, 6, 7A, 7B and 8 to illustrate Fig. 9 . refer to Fig. 9 , a flowchart 900 is shown. Operations from 902 to 910 may be performed by a system (e.g., Figure 2 The operation may start at 902 and proceed to 904.
[0123] At 904, safety information may be received from other control zone master devices (one or more). The safety information may be associated with subsequent geographic control zones (one or more) that may be located in the current travel route of the first vehicle 104. According to an embodiment, the first electronic device 112 may be configured to receive the safety information. The safety information may correspond to, for example, an event associated with the subsequent geographic control zone. Such events may correspond to, but are not limited to, accidents, terrorist attacks, crash routes, severe weather conditions (such as tornadoes or heavy rains), and the like.
[0124] At 906, a safe route for the first vehicle 104 may be predicted based on the received safety information. According to an embodiment, the first electronic device 112 may be configured to predict a safe route for the first vehicle 104 based on the received safety information. The safe route may be a preemptive measure to avoid the first vehicle 104 from encountering such an event and ensure that the first vehicle 506 remains safe during the journey.
[0125] At 908 , the predicted safe route may be transmitted to the first vehicle 104 . According to an embodiment, the first electronic device 112 may be configured to transmit the predicted safe route to the first vehicle 104 .
[0126] At 910, the first vehicle 104 may be controlled based on the predicted safe route. According to an embodiment, the first electronic device 112 may be configured to control the first vehicle 104 based on the predicted safe route. For example, the predicted safe route may override the current driving route of the first vehicle 104, and the first vehicle 104 may adjust its self-driving control to move along the predicted safe route. Control may be passed to the end.
[0127] Various embodiments of the present disclosure may provide a non-transitory computer-readable medium and / or storage medium on which instructions that can be executed by a machine and / or computer to operate a control system are stored. The instructions may cause the machine and / or computer to perform operations including receiving traffic information from one or more electronic devices in a set of electronic devices, the traffic information including multiple image frames of a group of mobile objects in a geographic control area. The operation may also include generating a set of image frames of a first mobile object in the group of mobile objects based on applying a trained NN model to the received traffic information. The set of generated image frames corresponds to a set of possible positions of the first mobile object at a future moment. The operation may also include predicting an unsafe behavior of the first mobile object based on the set of generated image frames, and generating first control information including an alternative route for the first vehicle in the geographic control area based on the predicted unsafe behavior. The first vehicle may be controlled based on the generated first control information.
[0128] Exemplary aspects of the present disclosure may include a control system such as Figure 1A control system 108 comprising a first electronic device (such as Figure 1 The first electronic device 112). The first electronic device may include a circuit system (such as the circuit system 202). The circuit system may also be configured to receive traffic information, the traffic information including multiple image frames of a group of mobile objects in the geographic control area. The traffic information may also include, for example, GNSS information of multiple locations in the geographic control area. The circuit system may also be configured to generate a set of image frames of a first mobile object in the group of mobile objects based on applying a trained NN model to the received traffic information. The set of generated image frames may correspond to a set of possible positions of the first mobile object at a future moment. The circuit system may also be configured to predict an unsafe behavior of the first mobile object based on the set of generated image frames, and generate first control information based on the predicted unsafe behavior, the first control information including but not limited to a first vehicle (such as Figure 1 The first vehicle may be controlled based on the generated first control information.
[0129] According to an embodiment, the circuit system can be configured to determine motion information of the mobile object group based on the received traffic information, such as one or more of the moving direction, moving speed, driving mode, driving lane, and driving side. The unsafe behavior of the first mobile object can also be predicted based on the determined motion information. The predicted unsafe behavior can correspond to, for example, the first mobile object violating traffic regulations.
[0130] According to an embodiment, the circuit system may be configured to determine a likelihood of a collision between the first vehicle and the first moving object based on the set of generated image frames, and predict an unsafe behavior of the first moving object based on the determined likelihood of collision.
[0131] According to an embodiment, the circuit system can be configured to detect the first vehicle based on the first vehicle being within the communication range of the control system. The circuit system can be configured to receive input corresponding to acceptance of setting a geographic control zone, which can be set to span a predetermined distance from the detected first vehicle. The circuit system can also be configured to establish a geographic control zone around the detected first vehicle based on the received input.
[0132] According to an embodiment, the circuit system may also be configured to register a set of vehicles in a geographic control area, the set of vehicles including but not limited to a first vehicle on a control network, and generate second control information for each vehicle in the set of registered vehicles based on the predicted unsafe behavior. The circuit system may also be configured to transmit the generated second control information to each vehicle in the set of registered vehicles.
[0133] According to an embodiment, the circuit system may also be configured to generate third control information based on the determination that the first mobile object is a vehicle registered on the control network, the third control information including but not limited to a set of safety measures for the first mobile object. The circuit system may also be configured to transmit the third control information to the first mobile object.
[0134] According to an embodiment, the control system may include a set of electronic devices (such as the set of electronic devices 114) that are communicatively coupled to the first electronic device. The circuit system may be configured to select a second electronic device from the set of electronic devices as a first control area master device in the geographic control area. The first control area master device may be configured to collect traffic information of a group of mobile objects in the geographic control area. The first control area master device may be configured to collect traffic information of a group of mobile objects in the geographic control area and share the collected traffic information with at least one electronic device in the set of electronic devices based on a retrieval request.
[0135] According to an embodiment, the circuit system may be configured to predict an unsafe behavior of the first moving object when the first moving object is not within the field of view (FOV) of the first vehicle. Alternatively, the circuit system may be configured to predict an unsafe behavior of the first moving object when the first moving object is within the field of view (FOV) of the first vehicle.
[0136] According to an embodiment, the circuit system may be configured to predict a first driving route of the first moving object based on the predicted unsafe behavior. The predicted unsafe behavior may correspond to the first moving object violating at least one traffic rule. The circuit system may also be configured to determine the current driving route of the first vehicle and the possibility of the first vehicle colliding with the first moving object based on the predicted first driving route and the determined current driving route. First control information including an alternative route for the first vehicle may also be generated based on the determined possibility.
[0137] According to an embodiment, the circuit system may also be configured to receive safety information associated with a subsequent geographic control area located in the current driving route of the first vehicle. The safety information may correspond to an event associated with the subsequent geographic control area. The circuit system may be configured to predict a safe route for the first vehicle based on the received safety information. The circuit system may also be configured to generate fourth control information including the predicted safe route. The first vehicle may be further controlled based on the generated fourth information.
[0138] According to an embodiment, the circuit system may be further configured to determine one or more safe driving zones of the first vehicle. The circuit system may be further configured to predict a safe route for the first vehicle based on the determined one or more safe driving zones. The circuit system may be further configured to generate fourth control information including the predicted safe route. The first vehicle may be further controlled based on the generated fourth information while overriding the current driving route preset for the first vehicle.
[0139] According to an embodiment, the set of electronic devices may include a drone on a first vehicle. The circuit system may be configured to dispatch the drone to the location of a first mobile object in the group of mobile objects. The first mobile object may not be within the field of view (FOV) of the first vehicle, and the drone may be configured to capture traffic information of the first mobile object. The circuit system may also be configured to receive the captured traffic information from the drone.
[0140] According to an embodiment, the set of electronic devices may include a drone on a first vehicle. The circuit system may be configured to detect a user discomfort level for the first vehicle and also dispatch the drone based on the detected user discomfort level.
[0141] The present disclosure may be implemented in hardware or in a combination of hardware and software. The present disclosure may be implemented in a centralized manner or in a distributed manner in at least one computer system, wherein different elements may be distributed on several interconnected computer systems. A computer system or other device suitable for performing the methods described herein may be suitable. The combination of hardware and software may be a general-purpose computer system with a computer program that, when loaded and executed, can control the computer system so that it performs the methods described herein. The present disclosure may be implemented in hardware including a part of an integrated circuit that also performs other functions.
[0142] The present disclosure may also be embedded in a computer program product, which comprises all the features enabling the implementation of the methods described herein and which, when loaded into a computer system, is capable of carrying out these methods. In this document, a computer program refers to any expression of a set of instructions in any language, code or notation, which is intended to cause a system with information processing capabilities to perform specific functions directly, or after one or both of the following: a) conversion into another language, code or notation; b) reproduction in a different material form.
[0143] Although the present disclosure is described with reference to certain embodiments, it will be appreciated by those skilled in the art that various changes may be made and equivalents may be substituted without departing from the scope of the present disclosure. In addition, many modifications may be made to adapt specific circumstances or materials to the teachings of the present disclosure without departing from the scope of the present disclosure. Therefore, it is intended that the present disclosure is not limited to the specific embodiments disclosed, but that the present disclosure will include all embodiments falling within the scope of the appended claims.
Claims
1. A control system, comprising: A first electronic device includes a circuit system, wherein the circuit system is configured to: receiving traffic information, the traffic information comprising a plurality of image frames of a group of moving objects in a geographic control area; generating a set of image frames of a first moving object in the group of moving objects based on applying a trained neural network (NN) model to the received traffic information, wherein the set of image frames generated corresponds to a set of possible positions of the first moving object at a future time; predicting an unsafe behavior of the first moving object based on the generated set of image frames; and generating first control information including an alternative route for a first vehicle in a geographic control area based on the predicted unsafe behavior, The first vehicle is controlled based on the generated first control information.
2. The control system according to claim 1, wherein the first moving object is a moving vehicle. 3 . The control system according to claim 1 , wherein the first moving object is one of a pedestrian, an animal, an aircraft, flying objects or a person.
4. The control system of claim 1 , wherein the circuit system is further configured to: detecting the first vehicle based on the first vehicle being within a communication range of the control system; receiving input from the detected first vehicle corresponding to acceptance of setting a geographic control zone spanning a predetermined distance; and Based on the input received, a geographic control zone is established around the detected first vehicle.
5. The control system of claim 1 , further comprising a set of electronic devices communicatively coupled to the first electronic device, wherein the circuit system is configured to receive the traffic information from one or more electronic devices in the set of electronic devices, and The collection of electronic devices includes vehicles, on-board electronic control units (ECUs), on-board cameras, vehicle external cameras, roadside units (RSUs), vehicle-to-infrastructure (V2I) communication equipment, and aerial photography drones.
6. The control system of claim 1, wherein the traffic information further includes Global Navigation Satellite System (GNSS) information for a plurality of locations in a geographic control area.
7. The control system of claim 1, wherein the circuit system is configured to: determining motion information of the group of moving objects based on the received traffic information; and An unsafe behavior of the first moving object is also predicted based on the determined motion information. 8 . The control system of claim 7 , wherein the motion information includes one or more of a moving direction, a moving speed, a driving mode, a driving lane, and a driving side of a road.
9. The control system of claim 1, wherein the predicted unsafe behavior corresponds to a violation of a traffic regulation by the first moving object.
10. The control system of claim 1, wherein the circuit system is further configured to register a set of vehicles including the first vehicle on a control network, The set of registered vehicles is within a geographically controlled area.
11. The control system of claim 10, wherein the circuit system is further configured to: generating second control information for each vehicle in the set of registered vehicles based on the predicted unsafe behavior; and The generated second control information is transmitted to each vehicle in the set of registered vehicles.
12. The control system of claim 10, wherein the circuit system is further configured to: generating third control information including a set of safety measures for the first mobile object based on determining that the first mobile object is a vehicle registered on a control network; and The generated third control information is transmitted to the first mobile object.
13. The control system of claim 1, further comprising a set of electronic devices communicatively coupled to the first electronic device, wherein the circuit system is further configured to select a second electronic device from the set of electronic devices as a first zone master device of the geographic control zone, and The first zone master device is configured as: collecting traffic information of the group of mobile objects in the geographic control area; receiving a retrieval request for the collected traffic information from at least one electronic device in the set of electronic devices; and Based on the received retrieval request, the collected traffic information is shared with at least one electronic device in the set of electronic devices.
14. The control system of claim 1, wherein the circuit system is further configured to predict unsafe behavior of the first moving object when the first moving object is not within a field of view (FOV) of the first vehicle.
15. The control system of claim 1, wherein the circuit system is further configured to predict unsafe behavior of the first moving object when the first moving object is within a field of view (FOV) of the first vehicle.
16. The control system of claim 1, wherein the circuit system is further configured to: determining a likelihood of a collision between the first vehicle and the first moving object based on the generated set of image frames; and An unsafe behavior of the first moving object is predicted based on the determined likelihood.
17. The control system of claim 1, wherein the circuit system is further configured to: predicting a first driving route of the first moving object based on the predicted unsafe behavior, wherein the predicted unsafe behavior corresponds to the first moving object violating at least one traffic rule; determining a current travel route of the first vehicle; determining a possibility of a collision between the first vehicle and the first moving object based on the predicted first driving route and the determined current driving route; and The first control information including an alternative route for the first vehicle is generated based on the determined likelihood.
18. The control system of claim 1, wherein the circuit system is further configured to: receiving safety information associated with a subsequent geographic control zone located in a current travel route of the first vehicle, wherein the security information corresponds to an event associated with the subsequent geographic control area; and predicting a safe route for the first vehicle based on the received safety information; and generating fourth control information including the predicted safe route, The first vehicle is also controlled based on the generated fourth control information.
19. The control system of claim 1, wherein the circuit system is further configured to: determining one or more safe driving zones for the first vehicle; predicting a safe route for the first vehicle based on the determined one or more safe driving zones; and generating fourth control information including the predicted safe route, While overriding the current driving route preset for the first vehicle, The first vehicle is controlled based on the generated fourth control information.
20. The control system of claim 1, further comprising a drone on the first vehicle, wherein the circuit system is further configured to: dispatching the drone to the location of the first mobile object in the group of mobile objects, wherein the first moving object is not within a field of view (FOV) of the first vehicle, and wherein the drone is configured to capture traffic information of the first moving object; and The captured traffic information is received from the drone.
21. The control system of claim 20, wherein the circuit system is further configured to: detecting a user discomfort level for the first vehicle; and The drone is also dispatched based on the detected discomfort level of the user.
22. A method comprising: receiving traffic information, the traffic information comprising a plurality of image frames of a group of moving objects in a geographic control area; generating a set of image frames of a first moving object in the group of moving objects based on applying a trained neural network (NN) model to the received traffic information, wherein the set of image frames generated corresponds to a set of possible positions of the first moving object at a future time; predicting an unsafe behavior of the first moving object based on the generated set of image frames; and generating first control information including an alternative route for a first vehicle in a geographic control area based on the predicted unsafe behavior, The first vehicle is controlled based on the generated first control information.
