Method, apparatus and computer program product for threat notification

Abstract

The present disclosure relates to a method, an apparatus and a computer program product for threat notification. The method includes receiving an image indicative of a first object including at least one of a vehicle and a vulnerable road user (VRU). The method further includes detecting a first threat associated with the first object based on the processing of the image, and in response to detecting the first threat, sending a notification regarding the first threat to at least one of the first object and a second object associated with the first object. Thus, once a threat is detected, notifications can only be sent to specific road participants such as the first object and the second object, so that the notifications can be sent and processed more accurately, the bandwidth used is less, and the communication efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of intelligent transportation, and more particularly, to a method, device and computer program product for threat notification. Background Art

[0002] Multi-access edge computing (MEC) is a concept of edge computing for mobile communication networks proposed by the European Telecommunications Standards Institute (ETSI). MEC aims to support the development of mobile use cases for edge computing, which allows application developers and content providers to access computing power and IT service environments in a dynamic setting at the edge of the network. The ETSIMEC industry specification group (ISG) provides four types of use cases for intelligent transportation systems: safety, convenience, advanced driver assistance, and vulnerable road users (VRU). Among these use cases, VRUs, including pedestrians and cyclists (such as cyclists, electric bike riders, and motorcyclists, etc.), are considered to be one of the key use cases, as more than 50% of all road traffic deaths involve VRUs. Therefore, it is important to detect threats related to VRUs and send notifications to other road participants so that other road participants have enough time to react. Summary of the invention

[0003] Embodiments of the present disclosure propose a method, device, and computer program product for threat notification. In embodiments of the present disclosure, an image indicating a first object can be received, and a first threat can be detected based on processing of the image, and in response to detecting the first threat, a notification about the first threat can be sent to the first object or a second object associated with the first object. In this way, once a threat is detected, a notification can be sent only to specific road participants in the scene (such as the first object and the second object associated with the first object), so that the notification can be sent and processed more accurately, and because the notification is only sent to specific road participants, less bandwidth is used, which improves communication efficiency.

[0004] In a first aspect of an embodiment of the present disclosure, a method for threat notification is provided. The method includes receiving an image indicating a first object, the first object including at least one of a vehicle and a VRU. The method also includes detecting a first threat associated with the first object based on processing the image, and in response to detecting the first threat, sending a notification about the first threat to at least one of the first object and a second object associated with the first object.

[0005] In a second aspect of an embodiment of the present disclosure, an electronic device is provided. The electronic device includes at least one processor; and a storage device coupled to the at least one processor and having instructions stored thereon, the instructions causing the electronic device to perform an action when executed by the at least one processor, the action including receiving an image indicating a first object, wherein the first object includes at least one of a vehicle and a VRU. The action also includes detecting a first threat associated with the first object based on processing of the image, and in response to detecting the first threat, sending a notification about the first threat to at least one of the first object and a second object associated with the first object.

[0006] In a second aspect of an embodiment of the present disclosure, a computer program product is provided. The computer program product is tangibly stored on a non-volatile computer-readable medium and includes machine executable instructions that, when executed, cause a machine to perform an action, the action including receiving an image indicating a first object, wherein the first object includes at least one of a vehicle and a VRU. The action also includes detecting a first threat associated with the first object based on processing the image, and in response to detecting the first threat, sending a notification about the first threat to at least one of the first object and a second object associated with the first object.

[0007] It should be understood that the contents described in the summary of the invention are not intended to limit the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

[0009] Figure 1 A schematic diagram showing an example cellular C-V2X system in which various embodiments of the present disclosure may be implemented is shown;

[0010] Figure 2 A schematic diagram showing a process for threat notification according to some embodiments of the present disclosure;

[0011] Figure 3 A flowchart of a method for threat notification according to some embodiments of the present disclosure is shown;

[0012] Figure 4 A schematic diagram showing a process for implementing threat notification in a first application scenario according to some embodiments of the present disclosure is shown;

[0013] Figure 5 A schematic diagram showing area division and corresponding driving strategies in a first application scenario according to some embodiments of the present disclosure is shown;

[0014] Figure 6 A schematic diagram showing a method for implementing threat notification in a second application scenario according to some embodiments of the present disclosure is shown;

[0015] Figure 7 A schematic diagram showing another application scenario in which threat notification is implemented according to some embodiments of the present disclosure;

[0016] Figure 8 A schematic diagram showing a process of detecting objects in an image and performing multi-level classification on the objects according to some embodiments of the present disclosure;

[0017] Fig. 9A , 9B Schematic diagrams of processes for object detection and for marking an identity (ID) of an object in an image according to some embodiments of the present disclosure are respectively shown;

[0018] Fig.10 A schematic diagram showing a process for mapping an ID of an object in an image with an ID of the object in a communication layer according to some embodiments of the present disclosure;

[0019] Fig.11 A flowchart illustrating another method for threat notification according to some embodiments of the present disclosure; and

[0020] Fig.12 A block diagram of a device in which various embodiments of the present disclosure may be implemented is shown. DETAILED DESCRIPTION

[0021] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0022] In the description of the embodiments of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc. may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0023] Threat detection and notification for vehicles are usually achieved using Cellular Vehicle-to-Everything (C-V2X), which is an interconnected mobile platform that allows vehicles to interact with the surrounding environment (e.g., pedestrians, vehicles, road infrastructure, networks, etc.) using cellular networks.

[0024] However, conventional cellular V2X typically uses broadcast or multicast protocols in the communication layer, such as 5G networks, to send threat-related notifications based on cell coverage, i.e., the same notification is sent to all road participants within the cell. Distinguishing broadcast messages can confuse irrelevant users. One solution can use scrambling codes to distinguish between different OEM manufacturers, but this defense may not be able to determine which specific vehicle in which lane should receive the message. These two notification delivery methods are not precise enough to send relevant notifications to specific road participants. Therefore, a more precise notification delivery solution is needed.

[0025] As used herein, road participants include various types of vehicles and VRUs.

[0026] To this end, an embodiment of the present disclosure proposes a scheme for threat notification. In an embodiment of the present disclosure, an image indicating a first object may be received, the first object including at least one of a vehicle and a VRU. A first threat associated with the first object may also be detected based on processing of the image, and in response to detecting the first threat, a notification about the first threat may be sent to at least one of the first object and a second object associated with the first object.

[0027] In this way, threat detection is based on a specific first object, and once a threat is detected, notification is sent only to at least one of the first object in the scene and the second object associated with the first object, so that notifications can be sent to specific road participants more accurately. For example, for a possible right-turn collision of a vehicle, a notification related to the right-turn collision can be sent only to the VRU in front of the right side of the vehicle or the vehicle in the right-turn lane, without being sent to the vehicle in the left lane of the vehicle. By sending notifications only to specific objects, bandwidth usage can be reduced, thereby improving communication efficiency.

[0028] Figure 1 Schematic diagram of an example C-V2X system 100 is shown in which various embodiments of the present disclosure may be implemented. Figure 1As shown, the C-V2X system 100 includes vehicles 110a-110c (collectively referred to as vehicles 110), road infrastructure 120, network (or base station) 130 and pedestrians (or VRU) 140. For example, the vehicle 110 may include various types of motor vehicles, such as cars, trucks and engineering vehicles. For example, the road infrastructure 120 may include traffic lights and roadside units (RSUs), etc. For example, pedestrians 140 may include riders, such as cyclists, motorcycle riders and electric bike riders. It should be noted that pedestrians and VRUs have the same meaning and are interchangeable in this disclosure. In the C-V2X system 100, the vehicle 110 can be connected to the road infrastructure 120, the network 130 and the VRU 140 via a cellular connection. These connections can be grouped into 4 types: V2V (vehicle to vehicle), V2I (vehicle to road infrastructure), V2P (vehicle to pedestrian) and V2N (vehicle to network), and V2X refers to the combination of all these connections.

[0029] like Figure 1 As shown, the vehicle 110 or the road infrastructure 120 can collect image data. When any one of the vehicle 110 and the VRU 140 is present in the image, the image is transmitted to the electronic device. In some embodiments, the electronic device can be at the near edge. In other embodiments, the electronic device can be at the far edge. In the edge computing architecture, the near edge is closer to the data processing center (or cloud), while the far edge is farther away from the data processing center (or cloud).

[0030] The electronic device receives an image indicating any one of the vehicle 110 and the VRU 140. The electronic device may also detect a threat associated with any one of the vehicle 110 and the VRU 140 based on processing the image. In some embodiments, the threat associated with any one of the vehicle 110 and the VRU 140 may include a road safety threat caused by the vehicle 110 or the VRU 140. In other embodiments, the threat associated with any one of the vehicle 110 and the VRU 140 may also include a road threat caused by other road participants or other objects to the vehicle 110 or the VRU 140. In some embodiments, detecting a threat may include: detecting that the vehicle 110 or the VRU 140 is located in a specific area, such as an emergency stop strip; detecting that the vehicle 110 and the VRU 140 are in a specific relative position, such as being close in a specific direction and having a risk of collision; or detecting a change in the motion state of the vehicle 110 or the VRU 140, such as exceeding a safe speed, sudden deceleration, sudden braking, turning, or a pedestrian or cyclist falling down. In other embodiments, detecting a threat may include detecting other road participants that pose a potential threat to the vehicle 110 and the VRU 140, such as speeding vehicles or large vehicles, such as engineering vehicles or heavy trucks, etc. The electronic device may also send a notification about the threat to the vehicle 110 or the VRU 140 or other road participants associated with the vehicle 110 or the VRU 140 in response to detecting the threat.

[0031] Figure 2 FIG. 2 is a schematic diagram of a process 200 for threat notification according to some embodiments of the present disclosure. Figure 2 As shown, process 200 includes receiving an image indicating a first object 261, the first object 261 including at least one of a vehicle 2611 and a VRU 2612. Process 200 includes, at block 230, processing the image, and, at block 240, detecting a first threat associated with the first object 261 based on the processing of the image at block 230. Process 200 also includes, at block 250, sending a notification (or message) regarding the first threat to at least one of the first object 261 or a second object 262 associated with the first object 261.

[0032] It should be understood that in the embodiment of the present disclosure, the first object 261 is depicted as a VRU and the second object 262 is depicted as a vehicle, which is merely an example, but the present disclosure is not limited thereto, and both the first object 261 and the second object 262 can be either a vehicle or a VRU, and both can be either vehicles or both can be VRUs.

[0033] like Figure 2As shown, in some embodiments, the processing of the image at box 230 includes tracking the first object 261 at box 231 to determine the trajectory data of the first object 261, and the trajectory data may include the position, speed and direction of travel of the first object 261. In some embodiments, the image may include a plurality of consecutive image frames (image sets), and the plurality of image frames are processed to determine the trajectory data of the first object 261. In some embodiments, the trajectory data may be used for short-term position prediction. In some embodiments, at box 231, it may also include classifying the first object 261 to track a specific type of the first object 261.

[0034] like Figure 2 As shown, in some embodiments, processing the image further includes, at box 231, marking an ID of the first object 261 in the image. In some embodiments, sending a notification about the first threat to at least one of the first object 261 or a second object associated with the first object 261 includes, at box 251, mapping between the ID of the first object 261 in the image and the ID of the first object 261 in the communication layer (or transport layer), and sending a notification about the first threat to the first object 261 and at least one of the second objects 262 based on the mapping. In some embodiments, the ID of the first object 261 in the communication layer is obtained by retrieving a user equipment (UE) ID from the MEC system. In some embodiments, the ID mapping at box 251 requires establishing a connection between the application layer (the object ID marked in the image) and the transport layer (the UE ID retrieved from the MEC system), and a set of methods can be used to synthesize the mapping between the two layers. Note that the process of detecting the first object 261 in the image and marking the ID of the first object 261 in the image will be described in detail in the following sections. Fig. 9A , 9B The mapping method between IDs is described in detail in Fig.10 Described in detail in.

[0035] In some embodiments, detecting the first threat at block 240 includes detecting the first threat in a plurality of application scenarios associated with the motion state of the object and the surrounding environment. In some embodiments, detecting the first threat associated with the first object 261 may include determining potential threats to the VRU with different urgency levels and reaction times for the corresponding application scenarios, and notifications about the first threat may be classified into corresponding notification types: presence notification for early altering and collision warning for potential immediate collision.

[0036] In some embodiments, detecting the first threat at block 240 includes determining, in the first application scenario 241, a first region where the first threat exists, the first region being associated with the second object 262. In some embodiments, the first region is in the direction of travel of the second object 262 and is a sector-shaped region centered on the second object 262. In some embodiments, the first region is a collision region. Note that the threat detection process associated with the first application scenario 241 will be described in detail in the following sections. Figure 4 , 5 Described in detail in.

[0037] In some embodiments, detecting the first threat at block 240 includes, in a second application scenario 242, determining a second area associated with the road where the first threat indicative of a threatening road participant exists. In some embodiments, the second area is a monitoring area. In some embodiments, the first threat includes at least one of a large vehicle and a speeding vehicle. Note that the threat detection process associated with the first application scenario 242 will be described in detail in the following sections. Figure 6 Described in detail in.

[0038] In some embodiments, the sending notification at box 250 may include, in box 252, subscribing (or unsubscribing) notifications. In one embodiment, the subscription or unsubscription notification may be actively performed by a road participant. In another embodiment, the subscription notification may be passive, for example, when a road participant enters a specific area, the notification is automatically subscribed, and when it leaves the specific area, the subscription notification is automatically unsubscribed. In some embodiments, messages may be sent to specific objects in the form of unicast or multicast for different application scenarios or different areas. In some embodiments, when vehicles or VRUs enter a relevant area, they will automatically subscribe to a MEC system for relevant message filtering and unsubscribe when leaving the area. In some embodiments, a message queue will be maintained for message distribution, and the message will be destroyed after a certain period of time. In some embodiments, the processes of boxes 230, 240, and 250 may be performed near the edge.

[0039] In one embodiment, process 200 may receive an image from a far edge that is pre-processed to indicate first object 261. Figure 2As shown, in some embodiments, the process 200 includes, at block 210, collecting an image. In some embodiments, the image may be collected by a sensor of the second object 262. In other embodiments, the image may be collected by an RSU, which may include any device capable of collecting road data, such as a camera, a closed-circuit television monitoring system (CCTV), intelligent transportation, etc., or a radar, etc. In some embodiments, collecting the image at block 210 may be performed by a far edge.

[0040] In some embodiments, preprocessing the image to indicate the first object 261 includes detecting the object in the image. Figure 2 As shown, in some embodiments, process 200 also includes, at box 220, performing object detection to determine the presence of a first object 261 in the image. In some embodiments, the object detection at box 220 can be performed by the far edge. In some embodiments, once the first object 261 is detected in the image, the image is sent to the near edge to process the image at box 230. In some embodiments, the far edge can detect moving objects, such as vehicles and pedestrians, in images associated with intersections or other areas. If an object is detected, a VRU detection scene is triggered, and the collected image data is then sent to the near edge for image processing. In some embodiments, the image can be cropped to remove irrelevant areas in the image to optimize computing performance and save transmission bandwidth.

[0041] Figure 3 FIG. 3 is a flowchart of a method 300 for threat notification according to some embodiments of the present disclosure. Figure 3 As shown, at block 310, method 300 receives an image indicating a first object, the first object including at least one of a vehicle and a VRU. Figure 2 In the illustrated process 200 , the method 300 may receive an image indicating the first object after determining the presence of the first object in the image at block 220 to process the image at block 230 .

[0042] At block 320, method 300 detects a first threat associated with a first object based on processing the image. Figure 2 In the illustrated process 200 , the method 300 may detect, at block 240 , a first threat associated with the first object based on trajectory data of the first object obtained by processing the image at block 230 .

[0043] At block 330, in response to detecting the first threat, method 300 sends a notification about the first threat to at least one of the first object and a second object associated with the first object. Figure 2In the process 200 shown in FIG. 3 , the method 300 may send a notification about the first threat to the first object 261 or the second object 262 at box 250 in response to detecting the first threat at box 240. In this way, once a threat is detected, a notification may be sent only to specific road participants in the scene (such as the first object and the second object associated with the first object), so that the notification may be sent and processed more accurately, and less bandwidth is used, thereby improving communication efficiency.

[0044] Figure 4 A schematic diagram of a process for implementing threat notification in a first application scenario 241 according to some embodiments of the present disclosure is shown. In the process, in the first application scenario 241, a first area (collision area) 2410 where a first threat exists is determined, and the first area 2410 is associated with a second object 262, wherein the first area 2410 is in the travel direction x2 of the second object 262 and is a fan-shaped area with the second object 262 as the center.

[0045] like Figure 4 As shown, in some embodiments, the first object 261 is a VRU with a travel direction x1, and the second object 262 is a vehicle with a travel direction x2. In some embodiments, the first area 2410 is a collision area, where a collision may occur between the vehicle and the VRU based on the current movement trends of the two, that is, a threat associated with the first object 261 is detected. For example, Figure 4 As shown, a collision may occur at the intersection of the travel direction x1 of the vehicle and the VRU and the travel direction x2 of the vehicle. In some embodiments, the first area 2410 is a sector area with the second object 262 as the center. Using a wider sector area instead of a line area can sense a wider area to give an early warning, especially in an area with obstacles, which is more effective. Figure 4 As shown, in addition to covering the field of view in front of the vehicle, the first area 2410 will also cover the rear field of view and a small part of the side field of view of the vehicle, because vehicles should be careful approaching their VRUs and avoid accidents caused by emergency braking, turning, pulling over (pullover), and even opening of the door. In one example, the C-V2X system can use sensors and computer vision to achieve non-line-of-sight perception for VRU discovery.

[0046] In some embodiments, a notification about a potential collision is sent to the first object 261 or the second object 262. For example, the notification can be sent not only to the vehicle but also to the VRU, which can prevent the VRU from being in a traffic accident caused by looking at a mobile phone while walking or riding, thereby providing protection for the VRU. For example, a notification can be sent to the wearable device of the VRU for easy viewing, and the VRU can be notified faster than a mobile phone. It should be understood that although in Figure 4 , the first object 261 is shown as a VRU, but it may also be a vehicle.

[0047] Figure 5 A schematic diagram of the area division of 241 and the corresponding notification or driving strategy in the first application scenario according to some embodiments of the present disclosure is shown. In some embodiments, the first area 2410 is divided into a plurality of sub-areas according to the distance from the second object 262, and sending a notification about the first threat to at least one of the first object 261 and the second object 262 associated with the first object 261 includes: sending a corresponding notification to the first object 261 according to detecting that the first threat is in the corresponding sub-area. In some embodiments, different driving strategies may be adopted for notifications of different emergency levels.

[0048] In some embodiments, the plurality of sub-areas include a first sub-area 2411, a second sub-area 2412, and a third sub-area 2413, wherein the distance between the boundary between the first sub-area 2411 and the second sub-area 2412 and the second object 262 is determined according to the braking distance of the second object 262, and the distance between the boundary between the second sub-area 2412 and the third sub-area 2413 and the second object 262 is determined according to the speed and reaction time of the second object 262. In some embodiments, the second object 262 is a vehicle, and the distance between the boundary between the first sub-area 2411 and the second sub-area 2412 and the vehicle is D1, which is equal to the braking distance of the vehicle, that is, Where v is the speed of the vehicle, μ is the friction coefficient, and g is the acceleration due to gravity. In some embodiments, the distance between the boundary between the second sub-area 2412 and the third sub-area 2413 and the vehicle is s, s=n*v, where v is the speed of the vehicle and n is the driver's reaction time. In some embodiments, n can be 2 seconds to 4 seconds. In some embodiments, v can be equal to the average speed of the vehicle over a period of time or in multiple consecutive image frames. In some embodiments, as Figure 5 As shown, the angle between the boundary of the first sector area 2410 and the vehicle body is θ, and the angle θ is, for example, related to the length and width of the vehicle, the speed of the vehicle and the weather conditions. For example, a large engineering vehicle or a heavy truck with a larger size can have a wider danger space and thus have a larger angle θ. For example, in windy weather, the angle θ should be set larger to obtain a larger detection range. In some embodiments, the distance from the outer boundary of the third sub-area 2413 to the vehicle is s multiplied by (1+margin), for example, the margin can be 50%-200%.

[0049] like Figure 5As shown, the boundary between the first sub-region 2411 and the second sub-region 2412 is defined as the first boundary, the boundary between the second sub-region 2412 and the third sub-region 2413 is defined as the second boundary, and the outer boundary of the third sub-region 2413 is defined as the third boundary, then the distance between the first boundary and the second object 262 is D1, the distance between the first boundary and the second boundary is D2, and the distance between the second boundary and the third boundary is D3. For example, And D3 = 0.5*s~2*s.

[0050] In some embodiments, in response to detecting that the first threat is in the first sub-area 2411, a notification of the first emergency level may be sent to the second object 262; in response to detecting that the first threat is in the second sub-area 2412, a notification of the second emergency level may be sent to the second object 262; in response to detecting that the first threat is in the third sub-area 2413, a notification of the third emergency level may be sent to the second object 262. For example, the urgency of the first emergency level, the second emergency level, and the third emergency level gradually decreases. In some embodiments, the first sub-area 2411 corresponds to a notification of the first emergency level, which may correspond to an urgent warning. For example, since its maximum radius D1 is the braking distance of the vehicle, a collision will become inevitable within this distance, so the first sub-area 2411 is a situation that the vehicle must avoid. For example, when the first object 261 is detected in the second sub-area 2312, close to the first boundary but not yet falling into the first sub-area 2411, an urgent warning (notification of the first emergency level) should be issued to remind the second object 262 in advance to avoid it from continuing to approach the first object 261.

[0051] In some embodiments, the second sub-area 2412 corresponds to a notification of a second emergency level, which corresponds to a collision warning. In the travel direction x2 of the second object 262, the minimum distance of the second sub-area 2412 from the second object 262 corresponds to the first boundary, which is the braking distance of the second object 262, and the maximum distance corresponds to the second boundary. For example, when the first object 261 is detected in the second sub-area 2412, a collision warning (a notification of a second emergency level) may be issued to remind the driver of the second object 262 to be alert to avoid a collision. For example, D2 may also be related to the length and width of the vehicle, the speed of the vehicle, and weather conditions.

[0052] In some embodiments, the second sub-area 2413 corresponds to a notification of a third emergency level, which corresponds to an existence reminder. For example, the length of D3 can be much greater than D2, for example, D3 can be equal to 0.5-2 times of D2. For example, D3 can be equal to providing a rough warning, and an early warning can be given to a specific VRU within the distance, which can include pregnant women, pedestrians with limited mobility, pedestrians with children, or pedestrians pushing strollers, etc.

[0053] In some embodiments, the near edge continuously updates the first boundary and the second boundary as the vehicle moves to dynamically adjust the three sub-areas to provide the latest and accurate information. For the collision area, the near edge dynamically calculates the braking distance of each vehicle and derives the fan-shaped area corresponding to the collision area based on the first, second, and third boundaries D1, D2, D3 and angle θ of each vehicle. The shape of the fan-shaped area can vary for different vehicle categories (e.g., ordinary vehicles and trucks), and can also be dynamically adjusted based on environmental changes (e.g., weather and time).

[0054] Figure 6 A schematic diagram of a method for threat notification implemented in a second application scenario 242 according to some embodiments of the present disclosure is shown. In some embodiments, detecting a first threat includes determining, in the second application scenario 242, a second area (monitoring area) 2421 associated with a road, where the second area 2421 has a first threat indicating a threatening road participant. In some embodiments, the first threat may include at least one of a large vehicle and a speeding vehicle. For example, the first threat may include a large vehicle, such as a crane, a mixer, an excavator, a tractor, etc. For example, the first threat may also include a speeding vehicle or a speeding motorcyclist, an electric bike rider, etc.

[0055] In some embodiments, sending a notification about the first threat to at least one of the first object 261 and the second object 262 associated with the first object 261 includes: in response to detecting the first threat, sending a notification about the first threat to at least one of the first object 261 and the second object 262 located in the second area 2421.

[0056] For example, the monitoring area corresponds to the area where the vehicle or VRU should be notified. Figure 6In the embodiment of the present invention, the square area at the intersection represents a high risk area for traffic accidents and can be considered as a monitoring area. It should be understood that the monitoring area can be an area of ​​various shapes covering accident-prone areas, such as square, rectangular, circular, elliptical areas near intersections, ramps, sharp turns or other accident-prone sections. It should be understood that any detected dangerous objects or events in or near the monitoring area, such as a mixer truck or a speeding motorcyclist, should be notified to all road participants in the monitoring area for protection. In some embodiments, such as Figure 6 As shown, the RSU 2422 collects images of the monitoring area and the vicinity of the monitoring area, and detects and tracks the objects in the image. Once it is determined that the vehicle 2423 exceeding the safe speed is approaching the second area 2421, and the braking distance of the vehicle 2423 is greater than its distance from the boundary of the second area 2421, in other words, the vehicle 2423 is a threat associated with the first object 261 and the second object 262, then a notification about the vehicle 2423 is sent to the first object 261 and / or the second object 262 in the second area 2421. In some embodiments, when any one of the first object 261 and the second object 262 is a VRU, the notification about the vehicle 2423 can be sent to the wearable device of the VRU for easy viewing, and the notification sending speed will be faster than a mobile phone. In some embodiments, a notification can be sent to a VRU different from the first object 261 and the second object 262, where the VRU is also located in the second area. In some embodiments, in the monitoring area, common deep learning practices such as image classification, object detection, and semantic segmentation can be applied to detect and classify objects in the intersection. Please note that the process of object detection and classification will refer to Figure 8 Detailed Description In some embodiments, since a threat in or near a monitoring area is not very urgent, a notification about the threat may be issued as a presence alert.

[0057] Figure 7 FIG. 2 is a schematic diagram showing another application scenario 700 for implementing threat notification according to some embodiments of the present disclosure. It can be seen that the application scenario 700 includes both the first application scenario 241 and the second application scenario 242. Figure 4-6 All processes and steps described in relation to the first application scenario 241 and the second application scenario 242 may be applicable to the application scenario 700 .

[0058] In some embodiments, for the second object 262, the threat associated with the first object 261 is detected in the first area (collision area) 2410, so a first notification associated with the threat can be sent to the second object 262. At the same time, the threat associated with the speeding vehicle 2423 is detected in the second area 2321, so a second notification associated with the threat can be sent to the second object 262. Thus, the second object 262 can receive both the first notification and the second notification. Figure 4-5 In the embodiment shown, depending on the distance between the first object 261 and the second object 262, the first notification may be an emergency warning (a notification of the first urgency level), a collision warning (a notification of the second urgency level), or an existence reminder (a notification of the third urgency level). Figure 6 In the illustrated embodiment, the second notification may be a presence reminder. Similar to the second object 246, the first object 261 may receive both the first notification associated with the second object 262 and the second notification 2423 associated with the speeding vehicle 2423. In some embodiments, the notification may be sent to a VRU different from the first object 261 and the second object 262, where the VRU is also located in the second area. Figure 7 In the embodiment of the present invention, the first object 261 and the second object 262 may be a pedestrian holding a child, a wheelchair user, and a pedestrian using crutches.

[0059] In some embodiments, in order to detect potential threats, two types of areas are defined for different levels of warnings: monitoring areas and collision areas. Monitoring areas represent areas that road participants should pay attention to even if there are no urgent threats. Collision areas represent areas where the vehicle may potentially collide with the VRU based on the current movement trend, and in the collision area, urgent threats may exist. For example, based on the degree of urgency in the above two areas, threats are classified into two types: existence reminders and collision warnings. For example, existence reminders are mainly used to provide early warning of threats, and are applicable to situations in the monitoring area and D3 situations in the collision area. For example, collision warnings are used to provide immediate warnings of threats, and are typically used in situations where there is a potential for collision, such as D2 situations in the collision area.

[0060] In some embodiments, in the collision area, notifications about threats may be sent using unicast, and in the monitoring area, notifications about threats may be sent using multicast. It should be understood that the multicast approach takes into account specific road participants, such as vehicles and VRUs that are only located in the monitoring area, so that the notifications sent are more relevant and more accurate.

[0061] Figure 8 FIG. 1 is a schematic diagram showing a process of detecting objects in an image and performing multi-level classification on the objects according to some embodiments of the present disclosure. Figure 8 As shown, in order to detect threats associated with presence alerts and collision warnings, various machine learning models will be used for object detection, classification, or segmentation, and these models will be deployed across the far edge, near edge, and cloud based on their computing and storage capabilities. It should be understood that the near edge is closer to the cloud than the far edge.

[0062] For example, at the far edge, an object detection model would be employed to detect objects and provide a coarse category, such as a vehicle or a cyclist. Subsequently, near the near edge, a classification model could be utilized to obtain more detailed information about the object, such as its specific type (a refined category), such as "truck." If more detailed information is needed, a neural search could be performed in the cloud to query for relevant object information, resulting in a fine-grained classification, such as "mixer truck."

[0063] Furthermore, as the classification becomes more specific, the monitoring area can be dynamically updated accordingly. For example, at the near edge, if the object is classified as a truck, the monitoring angle can be set to 90 degrees. However, in the cloud, if the truck is further identified as a more dangerous mixer truck, the angle can be expanded to 135 degrees to cover a wider area.

[0064] Fig. 9A , 9B Schematic diagrams of the process for object detection and for marking the ID of an object in an image according to some embodiments of the present disclosure are respectively shown. In some embodiments, object detection and object tracking can be performed at far edges and near edges, respectively. In some embodiments, Fig. 9A The object detection shown can correspond to Figure 2 220 in block 220, and Fig. 9B The indicated ID may correspond to Figure 2 In block 232. In some embodiments, object detection is used to identify vehicles and VRUs on the road, and if any of the vehicles and VRUs are found in the image, the sensor data associated with the image is sent to the near edge for further processing. For example, object detection acts as a trigger for subsequent processing or filtering of related content. To optimize bandwidth usage and computing resources, only specific image areas are retained for processing in some cases. For example, given the computing power of the far edge system, an algorithm such as YoloV3-tiny can be used for object detection.

[0065] In some embodiments, object tracking is performed in the near edge using algorithms such as depth classification. For example, object tracking includes identifying the unique ID of the object in the image and determining its motion trajectory, including position, speed, and direction of travel. Fig. 9BAs shown, each vehicle and VRU on the road is marked with a unique ID, and their driving trajectories can be easily tracked by calculating the coordination and prediction of these vehicles and VRUs at different times.

[0066] Fig.10 A schematic diagram of a process for mapping an ID of an object in an image with an ID of the object in a communication layer according to some embodiments of the present disclosure is shown. In some embodiments, Fig. 9A The object detection shown can correspond to Figure 2 Frame 251 in.

[0067] In order to notify the vehicle with a real transmission message in the computer vision system including the objects of the vehicle and the VRU, it is necessary to establish a connection between the vehicle in the computer vision system and the vehicle in the real C-V2X communication system. Fig.10 The architecture for ID mapping between the two systems using methods such as GPS and license plate recognition is shown. The MEC APP can ultimately connect the MEC UE ID (the ID of the object in the communication layer) with the ID of the vehicle or VRU identified by object detection or object tracking. Various methods can be used to comprehensively mark the UE ID.

[0068] In some embodiments, license plate recognition can be used to identify the UE ID. License plate recognition is a straightforward approach where an optical character recognition (OCR) algorithm can be used to identify the vehicle's license plate number. This information can then be used as a reference to retrieve its application attributes (UE ID, application session) in the MEC system. However, this approach has limitations, such as the license plate may be covered or blurred, and cannot be applied to pedestrians.

[0069] In some other embodiments, base station positioning can be used to mark the UE ID. In the base station positioning method, the communication system has provided sufficiently accurate positioning information for the UE, with sub-meter-level accuracy in 3GPP communication protocol version 16, and further enhanced to an accuracy of tens of centimeters and an end-to-end latency of less than 100 milliseconds in version 17.

[0070] Based on the GPS coordination of the camera and the offset of the vehicle in the image, the true position of each object in the sensor image can be calculated. Then, the nearest UE in the MEC system can be found to connect the object in the image to the UE in the MEC system. The ID connection can be determined by taking into account some deviations in trajectory data such as position, speed and direction of travel. The UE ID in the MEC system can be retrieved via the MEC service API MEC013 Location API and MEC014 UE Identity API.

[0071] Various embodiments of the present disclosure provide a comprehensive user identity mapping solution for connecting UEs in computer vision and MEC systems, achieving seamless integration between the two systems for message communication.

[0072] Fig.11 A flow chart of another method 1100 for threat notification according to some embodiments of the present disclosure is shown. In box 1110, the method 1100 includes receiving an image indicating a first object, wherein the first object includes at least one of a vehicle and a VRU. In box 1120, the method 1100 includes tracking the first object to determine trajectory data of the first object, wherein the trajectory data includes a position, a speed, and a direction of travel of the first object. In box 1130, the method 1100 includes determining a first threat based on the trajectory data of the first object. In box 1130, the method 1100 includes marking an ID of the first object in the image. In box 1140, the method 1100 includes sending a notification about the first threat to the first object and at least one of a second object associated with the first object based on a mapping between the ID of the first object in the image and the ID of the first object in the communication layer.

[0073] In some embodiments, the sending of notifications takes into account specific road participants and can be implemented in the application layer rather than in the transport layer (communication layer). The same notification will not be sent to all road participants within a cell or a predetermined channel, so the notifications sent are more accurate and flexible.

[0074] In some embodiments, notifications may be sent in two ways: using multicast and unicast for monitoring areas and collision areas, respectively. When a threat is detected in a monitoring area, a message may be sent to all road participants in the monitoring area, while for a threat detected in a collision area, a message may be sent only to specific road participants in the scene, such as at least one of a specific vehicle and a specific VRU.

[0075] In some embodiments, for multicast mode, MEC can maintain a subscription mechanism for message management and distribution. When a road participant enters a monitoring area, the road participant can actively or passively subscribe to a notification message and can automatically cancel the subscription when leaving the area. In some embodiments, different types of message categories will be defined based on vehicle and VRU types to achieve accurate and efficient processing. For example, in terms of areas, exemplary message categories may include monitoring areas, collision areas, dangerous event areas, etc. For example, in terms of use cases, exemplary message categories may include: VRU discovery, left turn assist, right turn assist. For example, in the right turn assist use case, the user can be notified only in the right turn lane.

[0076] In some embodiments, message queues can be used to manage message distribution and ensure real-time delivery. For example, expired messages can be deleted from the queue to maintain accuracy and efficiency.

[0077] Various embodiments of the present disclosure may utilize a subscription-multicast mechanism in the application layer for precise warning notifications, which allows users to subscribe to specific message categories and receive relevant notifications in a timely manner.

[0078] In addition, various embodiments of the present disclosure provide an end-to-end solution with a layered IT infrastructure that can utilize the far edge, near edge, and cloud to work collaboratively for VRU risk discovery in a C-V2X system, thereby enabling real-time response and efficient processing.

[0079] Furthermore, various embodiments of the present disclosure may perform hierarchical zone processing for threat detection for preventing VRU collisions, where dynamic zone updates may be performed near the edge and in the cloud if multi-level ultra-fine-grained classification of risks is required.

[0080] Fig.12 A schematic block diagram of an example device 1200 that can be used to implement an embodiment of the present disclosure is shown. As shown, the device 1200 includes a computing unit 1201, which can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) 1202 or loaded from a storage unit 12012 into a random access memory (RAM) 1203. In the RAM 1203, various programs and data required for the operation of the device 1200 can also be stored. The computing unit 1201, the ROM 1202, and the RAM 1203 are connected to each other via a bus 1204. An input / output (I / O) interface 1205 is also connected to the bus 1204.

[0081] A number of components in the device 1200 are connected to the I / O interface 1205, including: an input unit 1206, such as a keyboard, a mouse, etc.; an output unit 1207, such as various types of displays, speakers, etc.; a storage unit 12012, such as a disk, an optical disk, etc.; and a communication unit 1209, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1209 allows the device 1200 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0082] The computing unit 1201 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1201 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1201 performs the various methods and processes described above, such as method 300. For example, in some embodiments, the method 300 may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 12012. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 1200 via the ROM 1202 and / or the communication unit 1209. When the computer program is loaded into the RAM 1203 and executed by the computing unit 1201, one or more steps of the method 300 described above may be performed. Alternatively, in other embodiments, the computing unit 1201 may be configured to execute the method 300 in any other appropriate manner (eg, by means of firmware).

[0083] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip systems (SOCs), load programmable logic devices (CPLDs), and the like.

[0084] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0085] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In addition, although each operation is depicted in a specific order, this should be understood as requiring such operations to be performed in the specific order shown or in a sequential order, or requiring that all illustrated operations should be performed to obtain the desired result. In certain environments, multitasking and parallel processing may be advantageous. Similarly, although several specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Certain features described in the context of separate embodiments can also be implemented in a single implementation in combination. On the contrary, the various features described in the context of a single implementation can also be implemented in multiple implementations individually or in any suitable sub-combination.

[0086] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. A method for threat notification, comprising: receiving an image indicative of a first object, wherein the first object comprises at least one of a vehicle and a vulnerable road user (VRU); detecting a first threat associated with the first object based on processing the image; as well as In response to detecting the first threat, a notification regarding the first threat is sent to at least one of the first object and a second object associated with the first object.

2. The method of claim 1 , wherein detecting the first threat associated with the first object comprises: Tracking the first object to determine trajectory data of the first object, wherein the trajectory data includes a position, a speed, and a direction of travel of the first object; as well as The first threat is determined based on the trajectory data of the first object.

3. The method according to claim 2, Wherein detecting the first threat associated with the first object comprises: an identification ID indicating the first object in the image, And wherein sending the notification about the first threat to at least one of the first object and the second object associated with the first object comprises: The notification about the first threat is sent to at least one of the first object and the second object associated with the first object based on a mapping between the ID of the first object in the image and the ID of the first object in a communication layer.

4. The method of claim 1 , wherein detecting the first threat associated with the first object comprises: In a first application scenario, a first area where the first threat exists is determined, the first area is associated with the second object, wherein the first area is in a direction of travel of the second object and is a fan-shaped area with the second object as the center, and wherein the first area is a collision area.

5. The method of claim 4, wherein the first area is divided into a plurality of sub-areas according to a distance from the second object, and wherein sending a notification about the first threat to at least one of the first object and the second object associated with the first object comprises: According to detecting that the first threat is in the corresponding sub-area, a corresponding notification is sent to at least one of the first object and the second object associated with the first object.

6. The method according to claim 5, wherein the plurality of sub-areas include a first sub-area, a second sub-area, and a third sub-area, and wherein a distance from a boundary between the first sub-area and the second sub-area to the second object is determined according to a braking distance of the second object, and a distance from a boundary between the second sub-area and the third sub-area to the second object is determined according to a speed and a reaction time of the second object.

7. The method of claim 6, wherein sending the notification about the first threat to at least one of the first object and the second object associated with the first object comprises: In response to detecting that the first threat is in the first sub-area, sending a notification of a first emergency level to at least one of the first object and the second object associated with the first object; In response to detecting that the first threat is in the second sub-area, sending a notification of a second emergency level to at least one of the first object and the second object associated with the first object; In response to detecting that the first threat is in the third sub-area, sending a notification of a third emergency level to at least one of the first object and the second object associated with the first object, wherein the urgency of the first emergency level, the second emergency level, and the third emergency level gradually decreases.

8. The method according to claim 1, Wherein detecting a first threat associated with the first object comprises: In a second application scenario, determining a second area associated with a road, the second area having the first threat indicative of a threatening road participant, wherein the second area is a monitoring area, wherein the first threat includes at least one of a large vehicle and a speeding vehicle; And wherein sending the notification about the first threat to at least one of the first object and a second object associated with the first object comprises: In response to detecting the first threat, the notification regarding the first threat is sent to at least one of the first object and the second object located in the second area.

9. The method of claim 8, wherein the second object comprises a vehicle, and the method further comprises: The notification is sent to a VRU different from the first object and the second object, wherein the VRU is located in the second area. 10 . The method according to claim 1 , wherein the image is collected by at least one of a roadside unit (RSU) and a sensor of the second object, and is pre-processed to indicate the first object.

11. An electronic device, comprising: at least one processor; as well as a storage device coupled to the at least one processor and having instructions stored thereon, the instructions, when executed by the at least one processor, causing the electronic device to perform actions, the actions comprising: receiving an image indicative of a first object, wherein the first object comprises at least one of a vehicle and a vulnerable road user (VRU); detecting a first threat associated with the first object based on processing the image; and In response to detecting the first threat, a notification regarding the first threat is sent to at least one of the first object and a second object associated with the first object.

12. The apparatus of claim 11, wherein detecting the first threat associated with the first object comprises: Tracking the first object to determine trajectory data of the first object, wherein the trajectory data includes a position, a speed, and a direction of travel of the first object; as well as The first threat is determined based on the trajectory data of the first object.

13. The device according to claim 12, Wherein detecting the first threat associated with the first object comprises: an identification ID indicating the first object in the image, And wherein sending the notification about the first threat to at least one of the first object and a second object associated with the first object comprises: The notification about the first threat is sent to at least one of the first object and a second object associated with the first object based on a mapping between the ID of the first object in the image and the ID of the first object in a communication layer.

14. The device of claim 11, wherein detecting the first threat associated with the first object comprises: In a first application scenario, a first area where the first threat exists is determined, the first area is associated with the second object, wherein the first area is in a direction of travel of the second object and is a fan-shaped area with the second object as the center, and wherein the first area is a collision area.

15. The apparatus of claim 14, wherein the first area is divided into a plurality of sub-areas according to a distance from the second object, and wherein sending the notification about the first threat to at least one of the first object and the second object associated with the first object comprises: According to detecting that the first threat is in the corresponding sub-area, a corresponding notification is sent to at least one of the first object and the second object associated with the first object.

16. The apparatus according to claim 15, wherein the plurality of sub-areas include a first sub-area, a second sub-area, and a third sub-area, and wherein a distance from a boundary between the first sub-area and the second sub-area to the second object is determined based on a braking distance of the second object, and a distance from a boundary between the second sub-area and the third sub-area to the second object is determined based on a speed and a reaction time of the second object.

17. The device of claim 16, wherein sending the notification about the first threat to at least one of the first object and the second object associated with the first object comprises: In response to detecting that the first threat is in the first sub-area, sending a notification of a first emergency level to at least one of the first object and the second object associated with the first object; In response to detecting that the first threat is in the second sub-area, sending a notification of a second emergency level to at least one of the first object and the second object associated with the first object; In response to detecting that the first threat is in the third sub-area, sending a notification of a third emergency level to at least one of the first object and the second object associated with the first object, wherein the urgency of the first emergency level, the second emergency level, and the third emergency level gradually decreases.

18. The device according to claim 11, Wherein detecting a first threat associated with the first object comprises: In a second application scenario, determining a second area associated with a road, the second area having the first threat indicative of a threatening road participant, wherein the second area is a monitoring area, wherein the first threat includes at least one of a large vehicle and a speeding vehicle; And wherein sending the notification about the first threat to at least one of the first object and the second object associated with the first object comprises: In response to detecting the first threat, the notification regarding the first threat is sent to at least one of the first object and the second object located in the second area.

19. The apparatus of claim 18, wherein the second object comprises a vehicle, and the actions further comprise: The notification is sent to a VRU different from the first object and the second object, wherein the VRU is located in the second area.

20. A computer program product tangibly stored on a non-transitory computer readable medium and comprising machine executable instructions that when executed cause a machine to perform actions comprising: receiving an image indicative of a first object, wherein the first object comprises at least one of a vehicle and a vulnerable road user (VRU); detecting a first threat associated with the first object based on processing the image; as well as In response to detecting the first threat, a notification regarding the first threat is sent to at least one of the first object and a second object associated with the first object.

Images