A two-wheeled vehicle safety riding visual perception device and method based on edge computing

Abstract

The application relates to a two-wheeled vehicle safety riding visual perception device and method based on edge computing. The device is an end-to-end AI camera terminal installed on the head of a two-wheeled vehicle, and comprises an image acquisition module, an edge computing processing module, a decision and control module and a communication module. The edge computing processing module comprises a neural network processing unit, which is used for locally running a single-model multi-task network, sharing a backbone feature extraction network and parallelly outputting a human target and key point detection branch, a helmet detection branch and a multi-person riding analysis branch; the helmet detection branch detects a compliant helmet, a non-safety head-wearing article and an un-worn helmet. The decision and control module generates power control instructions according to the output results. The single-model multi-task network adopts a lightweight convolutional neural network architecture, and realizes edge end real-time inference through depth separable convolution and INT8 quantization. The application relies on an edge end local inference architecture, and realizes low-delay, low-cost and high-precision safety riding monitoring and real-time power intervention.

Description

Technical Field

[0001] This invention relates to the fields of computer vision technology and intelligent transportation. Specifically, it relates to an intelligent safety monitoring device for two-wheeled electric vehicles or bicycles, which integrates artificial intelligence algorithms, edge computing, and a high-definition wide-angle camera, as well as a corresponding real-time sensing and control method. Background Technology

[0002] In recent years, shared electric bicycles have rapidly gained popularity as an important supplement to short-distance urban travel. However, the accompanying safety issues have become increasingly prominent, mainly in two aspects: first, riders do not wear helmets as required; and second, the illegal carrying of passengers, especially children, is widespread. These unsafe behaviors greatly increase the risk of traffic accidents and the rate of injuries and fatalities.

[0003] To solve the above problems, existing technologies mainly adopt the following solutions:

[0004] 1. Mechanical or simple sensor solutions. For example, a pressure sensor or mechanical switch can be installed in the helmet lock to determine whether the helmet has been removed. The drawback of this solution is that it cannot confirm whether the user is actually wearing the helmet; the user can remove the helmet but not wear it, easily bypassing monitoring.

[0005] II. Cloud-based visual analysis solution. Video streams are collected by cameras and uploaded to cloud servers for analysis. Although this solution can identify cycling behavior, it has obvious drawbacks: (1) High latency - the data is uploaded from the terminal to the cloud, and then processed and returned by the cloud. The whole process usually takes more than 1 second, which is time-consuming and cannot meet the needs of real-time vehicle control such as power outage or power assist restoration; (2) High cost - continuous video stream upload will generate huge 4G / 5G network traffic costs; (3) Strong network dependence - in areas with poor network signal, such as underground garages and tunnels, the reliability of this solution will be greatly reduced.

[0006] III. Traditional Visual Inspection Solutions. Some existing vehicle-mounted camera solutions perform poorly in complex environments, such as at night, in backlight, or under sidelight conditions, where recognition accuracy drops significantly. Furthermore, their functionality is relatively limited; they typically only perform simple helmet presence detection, cannot distinguish helmet types, and struggle to accurately identify complex passenger-carrying scenarios.

[0007] In summary, existing technologies are insufficient in terms of real-time performance, cost, accuracy, and comprehensive functionality, making it difficult to effectively and economically solve the problem of safe riding supervision of two-wheeled vehicles. Summary of the Invention

[0008] This invention aims to overcome the problems of high latency, high cost, poor environmental adaptability, and limited functionality in existing technologies, and provides a visual sensing device and method capable of real-time, accurate, and reliable monitoring of unsafe behaviors of two-wheeled vehicle riders, such as not wearing helmets and illegally carrying passengers. Specifically, the technical problems to be solved include:

[0009] 1. Solve the high latency problem of cloud solutions by achieving a recognition speed of less than 0.5 seconds through edge computing, which meets the needs of real-time control of vehicle power systems.

[0010] 2. To address the high traffic costs associated with cloud-based solutions, local processing on the device side is implemented, uploading only necessary event results or a small amount of sample data, significantly reducing operating costs.

[0011] 3. Improve recognition accuracy in complex lighting environments, such as at night, in backlight, and under sidelight, to ensure reliable operation around the clock.

[0012] 4. It can achieve multi-tasking and refined recognition functions, not only detecting whether a helmet is being worn, but also distinguishing the type of helmet, and accurately identifying complex scenarios such as multiple riders, their locations, and age composition.

[0013] 5. Provide a highly integrated, plug-and-play end-to-end solution.

[0014] The technical solution adopted by this invention to solve its technical problem is as follows:

[0015] In a first aspect, the present invention provides a visual perception device for safe riding of a two-wheeled vehicle based on edge computing, which is an end-to-end AI camera terminal installed on the front of the two-wheeled vehicle, comprising:

[0016] The image acquisition module, including an image sensor and an ultra-wide-angle lens, is used to acquire scene images covering the area from the cyclist to the pedals in real time;

[0017] The edge computing processing module includes a neural network processing unit for running a single-model multi-task neural network locally. The single-model multi-task neural network shares a backbone feature extraction network and outputs human target and key point detection branches, helmet detection branches, and multi-person cycling analysis branches in parallel.

[0018] The decision-making and control module is used to generate control commands for the two-wheeled vehicle's power system locally based on the reasoning results;

[0019] The communication module includes a vehicle-to-vehicle communication interface and a cloud-to-cloud communication interface. The vehicle-to-vehicle communication interface is used to send the control commands to the two-wheeled vehicle's main control unit to perform assist enable or disable operations. The cloud-to-cloud communication interface is used to connect to the cloud platform to upload event logs, but does not upload real-time video streams.

[0020] Furthermore, the image sensor is a large-area CMOS image sensor, and integrates a wide dynamic range (HDR) processing unit and an infrared fill light unit; the ultra-wide-angle lens is installed at a downward tilt, and its preset angle is the angle between the lens optical axis and the horizontal plane. The preset angle is configured to ensure that the lens field of view completely covers the head and pedal area of ​​cyclists with a height of 140cm to 190cm, and to prevent rainwater from accumulating on the lens surface.

[0021] Furthermore, the single-model multi-task neural network shares a backbone feature extraction network and outputs three task branches in parallel: a human target and key point detection branch, a helmet detection branch, and a multi-person riding analysis branch. The human target and key point detection branch outputs bounding boxes of people in the image, along with the coordinates of key points including the head, shoulders, elbows, waist, knees, hips, and feet. The helmet detection branch performs multi-category detection within the head region of the bounding box, with categories including at least compliant helmets, non-safety headwear, and no helmet. Compliant helmets cover various safety helmets certified by national safety standards, including but not limited to helmets with custom markings for shared mobility platforms and commercially available standardized safety helmets. The multi-person riding analysis branch detects the seat and pedal components of the two-wheeled vehicle and, combined with the human key point coordinates, analyzes the number of riders, their position, and their age category.

[0022] Furthermore, the edge computing processing module is configured to: perform perspective distortion correction on the coordinates of the human body key points; map the key points in the image coordinate system to the world coordinate system through inverse perspective transformation based on the camera intrinsic parameter matrix and lens tilt angle to obtain the corrected key point coordinates; extract skeletal proportion features based on the corrected human body key points, wherein the skeletal proportion features are defined as the ratio of trunk length to lower limb length, used to distinguish whether the cyclist is an adult or a child.

[0023] Furthermore, the rules for distinguishing between adults and children include one of the following parallel paths: when the skeletal proportion features meet a first preset threshold range, the child is identified; when they meet a second preset threshold range, the child is identified. The thresholds are determined by fitting the statistical distribution of skeletal proportions of a large number of adult and child samples. The skeletal proportion features are input into a pre-trained lightweight fully connected classification model, and the model outputs the classification result for adults or children. The classification model is trained using the cross-entropy loss function.

[0024] Furthermore, the edge computing processing module also includes a CPU and DDR memory. The CPU is used for system scheduling, data preprocessing, and communication management, while the DDR memory is used to store model weights, intermediate feature maps, and runtime data. The neural network processing unit, the CPU, and the DDR memory are configured to ensure that the total delay from detecting a violation to the decision and control module outputting a cutoff command is less than 0.5 seconds.

[0025] Secondly, the present invention provides a visual perception method for safe riding of two-wheeled vehicles based on edge computing, comprising the following steps:

[0026] S1 acquires images of two-wheeled vehicle riding scenes in real time and performs wide dynamic range (HDR) processing and infrared illumination enhancement processing.

[0027] S2, the enhanced image is input into a single-model multi-task neural network for local single forward inference, and human target and key point features, helmet features and vehicle parts and human spatial relationship features are acquired simultaneously.

[0028] S3, based on the reasoning results, performs multi-task logical fusion judgment to determine the helmet wearing compliance status and the multi-person riding violation status based on the preset number of people threshold;

[0029] S4. Generate control commands based on the stated status and send them to the main control unit.

[0030] In step S3, determining the helmet wearing compliance status includes: if no person in a riding posture is detected, it is determined that no person is riding; if a person is detected, head features are extracted for multi-category detection; if the helmet is classified as a compliant helmet, it is compliant; if it is classified as a non-safe headwear or not worn, it is non-compliant.

[0031] In step S3, determining the violation status of multiple riders includes: triggering multiple rider analysis when the number of riders is greater than 1; matching the key points of the buttocks and feet with the seat and pedals in spatial position; determining that the number of matched riders exceeds the preset number threshold; and if the overloaded riders include children, increasing the violation level and cutting off the power assist.

[0032] Thirdly, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in the second aspect above.

[0033] Compared with existing technologies, this invention brings significant benefits through its innovative end-to-end edge computing architecture:

[0034] Firstly, it achieves real-time response with extremely low latency. All calculations are completed locally on the device, with end-to-end recognition latency of less than 0.5 seconds, far superior to the typically greater than 1 second achieved by cloud-based solutions. This makes real-time vehicle power control based on visual recognition results feasible—when the system detects that a rider is not wearing a compliant helmet or is illegally carrying passengers, it can immediately cut off power assist during the ride, enabling proactive and effective intervention against unsafe behaviors. Extremely low latency and extremely low operating costs are two sides of the same architecture: since video data does not need to be uploaded to the cloud, but only anonymized event logs are uploaded after local processing, the average daily data consumption per ride is no more than 5MB, which is two orders of magnitude lower than cloud-based video analytics solutions, fundamentally solving the operational burden of high 4G / 5G data costs.

[0035] Secondly, high precision and high environmental robustness are achieved through the synergistic optimization of hardware and algorithms. At the hardware level, a large-area CMOS image sensor, combined with wide dynamic range HDR processing and 850nm infrared illumination, enables the device to acquire high-quality images even under complex lighting conditions such as nighttime, backlighting, and rain. At the algorithm level, data augmentation training is performed for the aforementioned complex scenarios, resulting in a helmet detection accuracy of 97.1% in daytime scenarios, 95.8% in nighttime infrared illumination scenarios, and 94.6% and 94.2% in rainy and backlighting scenarios, respectively. Performance fluctuations in each scenario are within acceptable ranges, ensuring reliable operation in all weather conditions.

[0036] Third, the single-model multi-task architecture brings the dual benefits of comprehensive functionality and resource efficiency. Compared with conventional multi-model concatenation schemes, the single-model multi-task network of this invention, by sharing a backbone feature extraction network and outputting three task branches in parallel, reduces NPU utilization from 98% to 62%, peak memory usage from 480MB to 210MB, and total inference latency from 0.72s to 0.28s, while simultaneously increasing joint detection accuracy from 88.6% to 93.8%. This effect stems from the feature complementarity between the multi-task branches—human keypoint detection, helmet recognition, and multi-person cycling analysis share the same set of underlying features. The optimization directions of different tasks promote each other rather than hinder each other, enabling the network to simultaneously complete multiple refined recognition tasks under edge computing power constraints. It can not only determine whether a helmet is present or not, but also distinguish between compliant helmets and non-safety headwear, and accurately identify multi-person cycling, their location, and age composition.

[0037] Fourth, the highly integrated design of the device enables convenient plug-and-play deployment, while the localized processing mechanism protects user privacy while meeting regulatory traceability requirements—sensitive raw video data does not leave the device, and only event logs without any personally identifiable information are uploaded. Attached Figure Description

[0038] Figure 1 The diagram shows the system architecture and workflow of the device of the present invention, illustrating the data flow between the image acquisition module, the edge computing processing module, the decision and control module, and the communication module, as well as the connection relationships of the main functional units within each module.

[0039] Figure 2 This is a hardware system block diagram of the device of the present invention, showing the hardware connection relationship of the image sensor module, SoC chip, main control unit, communication interface and power management unit. Detailed Implementation

[0040] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0041] I. Data Collection Compliance and Application Scenarios

[0042] This invention's device is installed on the front of a two-wheeled vehicle. The image acquisition range is limited to the area from the rider's helmet to the pedals. The sole purpose of acquisition is to monitor riding safety in real time, including helmet wearing compliance and multiple riders violating regulations. This device does not collect or process personal privacy data such as facial recognition or identity tracking. All image analysis is performed locally on the device; raw image data is not uploaded to the cloud, only anonymized event logs are uploaded, containing no personally identifiable information. The implementation of this invention complies with the relevant provisions of the "Personal Information Protection Law of the People's Republic of China" and does not involve any violation of laws, social ethics, or harm to public interests.

[0043] II. System Overall Architecture

[0044] like Figure 1 As shown, this embodiment provides a visual perception device for safe riding of a two-wheeled vehicle based on edge computing. This device is an end-to-end AI camera terminal installed on the front of the two-wheeled vehicle. The device includes an image acquisition module, an edge computing processing module, a decision-making and control module, and a communication module.

[0045] The image acquisition module is used to acquire scene images covering the area from the rider to the pedals in real time; the edge computing processing module is used to run a single-model multi-task neural network locally to perform real-time inference analysis on the acquired images; the decision and control module is used to generate control commands for the two-wheeled vehicle's power system locally based on the inference results; the communication module includes a vehicle-to-vehicle communication interface and a cloud-to-cloud communication interface. The vehicle-to-vehicle communication interface is used to send control commands to the two-wheeled vehicle's main control unit to perform assist enable or disable operations, and the cloud-to-cloud communication interface is used to connect to the cloud platform to upload event logs, but not real-time video streams.

[0046] In one embodiment, the device further includes a power management unit, which obtains DC power from the two-wheeled vehicle battery through the vehicle power input interface, and provides operating voltages to the image sensor, the SoC chip and the communication module respectively after voltage conversion and timing management.

[0047] Combination Figure 1 and Figure 2 As shown, the overall architecture and hardware composition of the device in this embodiment will be further explained. Figure 1 The device system architecture and workflow diagram provided in the embodiments of the present invention are as follows: Figure 2 This is a hardware system block diagram of the device provided in an embodiment of the present invention.

[0048] Figure 1 The overall system architecture of the device is shown, including the data flow and control flow relationships between the four main modules and their internal functional units.

[0049] Specifically, such as Figure 1 As shown, the edge computing-based visual perception device for safe riding of two-wheeled vehicles provided in this embodiment of the invention includes an image acquisition module, an edge computing processing module, a decision and control module, and a communication module.

[0050] The image acquisition module includes a large-area CMOS image sensor, a wide dynamic range (HDR) processing unit, an infrared illumination unit, and an ultra-wide-angle lens. The ultra-wide-angle lens is mounted at a preset downward tilt of 15° to 25° directly in front of the two-wheeled vehicle. The large-area CMOS image sensor is used to acquire real-time scene images covering the area from the rider's head to the pedals. The HDR processing unit is used to suppress overexposure in strong light and underexposure in dark areas. The infrared illumination unit uses an 850nm band infrared LED to enhance image acquisition quality at night or in low-light environments.

[0051] The edge computing processing module includes a neural network processing unit (NPU), a CPU, and DDR memory, and runs a single-model multi-task neural network locally. The single-model multi-task neural network constructs a backbone feature extraction network using depthwise separable convolutions, deploys it on the NPU using INT8 quantization, and has an NPU computing power of at least 1 TOPS. The backbone network output stride is 1 / 16. The single-model multi-task neural network shares the same backbone feature extraction network and outputs three parallel task branches: a human target and keypoint detection branch, a helmet detection branch, and a multi-person cycling analysis branch. The human target and keypoint detection branch outputs the bounding box of the rider and the coordinates of seven key points: head, shoulders, elbows, waist, knees, hips, and feet. The helmet detection branch performs multi-category helmet detection within the head region of the bounding box. The multi-person cycling analysis branch detects the seat and pedal components and analyzes the number of riders, rider position, and age category based on the human keypoint coordinates. The CPU is used for system scheduling, data preprocessing, and communication management, and the DDR memory is used to store model weights, intermediate feature maps, and runtime data.

[0052] The decision and control module executes decision logic based on the inference results of a single-model multi-task neural network. This decision logic includes helmet-wearing compliance assessment, multi-person riding violation assessment, and adult / child identification. Based on the inference results, the decision and control module locally generates control commands for the two-wheeled vehicle's power system, outputting commands to allow or disable power assist, and simultaneously triggering voice warnings. The control logic of the decision and control module covers the entire riding lifecycle, including the unlocking / starting phase, the riding phase, and the correction / recovery phase.

[0053] The communication module includes a vehicle-to-vehicle communication interface and a cloud-to-cloud communication interface. The vehicle-to-vehicle communication interface sends control commands to the two-wheeled vehicle's main control unit via UART or RS485 encryption protocol to perform assist-assist enable or disable operations. The cloud-to-cloud communication interface connects to the cloud platform via a Cat.1 cellular communication module, uploading only event logs and not real-time video streams; the average daily data consumption per ride does not exceed 5MB.

[0054] Figure 2 The hardware components of the device and the connection relationships between the modules are shown.

[0055] Specifically, such as Figure 2 As shown, the hardware architecture of the edge computing-based visual perception device for safe riding of two-wheeled vehicles provided in this embodiment of the invention includes an image sensor module, a SoC chip, a main control unit (MCU), and a communication interface, with a power management unit providing power to each module.

[0056] The image sensor module includes an ultra-wide-angle lens and an image sensor, and transmits the acquired real-time scene images to the SoC chip via the MIPI-CSI interface.

[0057] The SoC chip integrates a Neural Processing Unit (NPU) and a CPU, and also incorporates a DDR controller. The CPU uses an ARM Cortex-A53 quad-core architecture and is responsible for system scheduling, data preprocessing, and communication management. The DDR controller connects to off-chip DDR memory, which has a capacity of 2GB LPDDR4 and is used to store model weights, intermediate feature maps, and runtime data. The SoC chip runs a single-model multi-task neural network locally. This single-model multi-task neural network uses deep separable convolutions to construct the backbone feature extraction network and employs an INT8 quantization scheme deployed on the NPU to achieve end-to-end real-time inference.

[0058] The SoC chip is connected to the main control unit (MCU) via a high-speed bus. The main control unit integrates a system control and scheduling module, an I / O interface management module, and a secure storage module, which are used to coordinate the working timing of each module, manage the input and output of peripheral interfaces, and store encryption keys and digital certificates to ensure communication security.

[0059] The main control unit is connected to both a vehicle-to-vehicle communication interface and a cloud-to-cloud communication interface. The vehicle-to-vehicle communication interface is used to send control commands to the main control unit of the two-wheeled vehicle to perform assist-assist operation or disabling operation; the cloud-to-cloud communication interface is used to connect to the cloud platform, only uploading event logs and not real-time video streams.

[0060] After voltage conversion by the power management unit, the vehicle power input provides operating voltage to the image sensor module, the SoC chip, and the communication module.

[0061] III. Image Acquisition Module

[0062] The image acquisition module includes an image sensor and an ultra-wide-angle lens.

[0063] In this embodiment, the image sensor is preferably a large-area CMOS image sensor with a target area size of not less than 1 / 2.9 inches, a pixel size of not less than 2.8µm, and a resolution of not less than 1080p. It integrates a wide dynamic range (HDR) processing unit and an infrared illumination unit. The infrared illumination unit uses an 850nm band infrared LED, supporting dual-band illumination of visible light and 850nm infrared light to ensure clear images are acquired even in low-light environments such as nighttime and tunnels. The wide dynamic range (HDR) processing unit is used to suppress overexposure in strong light and underexposure in dark areas, improving image quality under complex lighting conditions.

[0064] In this embodiment, a large-area CMOS image sensor is preferably used to balance cost and low-light performance. In other optional embodiments, the image sensor can also be a CCD sensor, utilizing its global shutter characteristics to eliminate the rolling shutter effect in high-speed cycling scenarios; or an event camera, which asynchronously senses pixel-level brightness changes to capture the key trajectory of the cyclist's rapid movement with extremely high dynamic range and microsecond-level temporal resolution, suitable for extreme lighting switching scenarios such as strong light, backlight, or tunnel entrances and exits; or a ToF sensor or structured light depth sensor, which can directly acquire the three-dimensional spatial information of the cyclist and vehicle, helping to improve the accuracy of rider position determination in multi-person cycling scenarios. All of the above embodiments can achieve the function of acquiring real-time scene images covering the area from the cyclist to the pedals.

[0065] The ultra-wide-angle lens is installed at a downward tilt, with its preset angle defined as the angle between the lens optical axis and the horizontal plane. This preset angle is configured to ensure that the lens's field of view completely covers the head and pedal area of ​​a rider with a height of 140cm to 190cm, while preventing rainwater accumulation on the lens surface. In this embodiment, the preferred range of the preset angle is 15° to 25°. This angle range is determined by considering both the geometric coverage characteristics of the lens optical axis and the drainage angle requirements of the lens surface. Optical simulation and real-vehicle verification have confirmed that within this angle range, the illuminance uniformity at the four corners of the field of view is no less than 60%, and the sliding speed of water droplets under their own weight meets the requirement of a continuously clear field of vision during riding.

[0066] In one embodiment, the image sensor is connected to the edge computing processing module via an image sensor interface, which is a MIPI-CSI interface, used to transmit the acquired real-time scene images to the SoC chip via serial data transmission.

[0067] IV. Edge Computing Processing Module

[0068] The edge computing processing module includes a neural network processing unit, a CPU, and DDR memory.

[0069] In this embodiment, the neural network processing unit is preferably a SoC chip with a built-in NPU, possessing a computing power of no less than 1 TOPS, employing an ARM architecture, and integrating on-chip DDR memory to meet the requirements of low power consumption and high energy efficiency for edge-side inference. The CPU is used for system scheduling, data preprocessing, and communication management, while the DDR memory is used to store model weights, intermediate feature maps, and runtime data. The neural network processing unit, the CPU, and the DDR memory work together to ensure that the total latency from detecting a violation to outputting a cutoff command is less than 0.5 seconds.

[0070] In other alternative embodiments, the neural network processing unit can also employ an embedded GPU, leveraging its parallel computing architecture to support more complex model structures and online learning; or an FPGA, implementing deterministic low-latency inference through hardware-level pipeline customization, suitable for dynamic control scenarios with strict requirements on response timing; or a dedicated ASIC, with operator-level solidification for the single-model multi-task network structure, achieving maximum throughput under extreme power consumption constraints. All of the above embodiments can achieve the function of running a single-model multi-task neural network locally and generating dynamic control instructions.

[0071] The edge computing processing module runs a single-model multi-task neural network locally. The specific structure of this single-model multi-task neural network is as follows: the backbone feature extraction network adopts a lightweight convolutional neural network architecture, replacing standard convolutions with depthwise separable convolutions, reducing the computational cost to approximately 1 / 8 to 1 / 9 of that of standard convolutions. This backbone network contains five convolutional modules, each consisting of a depthwise separable convolutional layer, a batch normalization layer, and a ReLU activation function. The output stride of modules 1 and 2 is 2, and the output stride of modules 3 to 5 is 1, resulting in an output feature map size that is 1 / 16 of the input image.

[0072] The three task branches branch in parallel on the output feature map of module 5 of the backbone network, and each branch uses a lightweight detection head structure:

[0073] The human target and keypoint detection branch first refines features through a 3×3 convolutional layer, then feeds into two output heads: a bounding box detection head uses center-point-based regression to directly predict the center coordinates, width, and height of the human target bounding box; the keypoint detection head uses Soft-Argmax regression to output a heatmap of seven keypoints, and sub-pixel-level keypoint coordinates are obtained by calculating the spatial expectation of the heatmap. This branch employs a multi-task joint training strategy, using CIoU loss for bounding box detection and mean squared error loss for keypoint detection.

[0074] The helmet detection branch, based on the backbone features, extracts features of the head region of each person's target box through Region of Interest Alignment (RoI Alignment). These features are then fed into a sub-network containing two 3×3 convolutional layers and one fully connected classification layer, outputting the classification probability of the head region belonging to each category (compliant helmet, unsafe headwear, not wearing a helmet). This branch is trained using the cross-entropy loss function.

[0075] The multi-person riding analysis branch reuses the backbone feature map, compresses the number of channels through a 1×1 convolutional layer, and then detects the target boxes of the seat and pedals respectively (using the same center point regression method as human detection). Then, the spatial distance of the hip key points among the detected human key points is matched with the seat box, and the spatial distance of the foot key points is matched with the pedal box, thereby determining whether each person is in a riding position.

[0076] The three branches mentioned above share the weights of the backbone network during training. The loss functions of each branch are weighted and summed according to preset weights and then jointly backpropagated, thereby realizing multi-task collaborative optimization of the backbone features.

[0077] The single-model multi-task neural network is deployed on the NPU using the INT8 quantization scheme. The quantization calibration method is as follows: 500 images are randomly selected from the training dataset as a calibration set. The model is run at FP32 precision to collect the activation value distribution of each layer. Then, the KL divergence minimization method is used to determine the quantization scaling factor and zero-point offset of each layer, converting the model weights and activation values ​​from 32-bit floating-point numbers to 8-bit integers. The quantized model size is approximately one-quarter of the FP32 model, and the inference speed is improved by about 3 times. During the quantization process, the output layer of the keypoint detection branch maintains FP32 precision output to ensure that the sub-pixel accuracy of the keypoint coordinates is not affected by quantization loss.

[0078] The training data consists of a multi-scene cycling image dataset of approximately 50,000 frames, annotated with bounding boxes, seven key points on the human body, helmet categories, and seat / pedal areas. This dataset covers 12 typical cycling scenarios, including daytime, nighttime, rainy, backlighting, and sidelighting. The annotation standards for the seven key points are as follows: Head – center of the top of the head; Shoulder – midpoint of the left and right acromions; Elbow – midpoint of the outer side of the left and right elbow joints; Waist – L3 lumbar vertebra; Knee – midpoint of the left and right knee joints; Hip – ischial tuberosities; Feet – midpoint of the left and right ankles. The helmet category annotation standards are as follows: Compliant Helmet – Helmets that meet national safety standards (such as GB 811-2022 or GB 24429-2009), regardless of their external markings; Non-Safety Headwear – Hats, headscarves, and other head coverings that do not provide safety protection; Not Wearing – No obvious covering on the head area.

[0079] During the training phase, the model uses the Adam optimizer with an initial learning rate of 0.001 and a batch size of 32. It is trained for 100 rounds, and the learning rate is reduced to 0.1 times the current value in the 60th and 80th rounds, respectively. The loss function is a weighted sum of the loss functions of each task branch, where the object detection branch uses CIoU loss, the key point detection branch uses mean squared error loss, and the classification branch uses cross-entropy loss.

[0080] The single-model multi-task neural network shares a backbone feature extraction network and outputs three task branches in parallel: human target and key point detection branch, helmet detection branch, and multi-person cycling analysis branch. The specific configurations of each branch are as follows:

[0081] The human target and key point detection branch is used to output the bounding box of the person in the image and the coordinates of the human key points of 7 parts including the head, shoulders, elbows, waist, knees, hips and feet.

[0082] The helmet inspection branch is used to perform multi-category inspections within the head area of ​​the target frame. These categories include at least compliant helmets, unsafe headwear, and no helmet. Unsafe headwear includes items such as hats and headscarves; compliant helmets include all types of helmets that meet national safety standards.

[0083] The multi-person riding analysis branch is used to detect the seat and pedal components of the two-wheeled vehicle, and analyze the number of riders, their position, and age category by combining the coordinates of key human body points.

[0084] The edge computing processing module is also configured to execute the following judgment logic:

[0085] First, perspective distortion correction is performed on the coordinates of the key human body points. Specifically, based on the camera intrinsic parameter matrix K and the lens tilt angle θ, the key point coordinates (u,v) in the image coordinate system are mapped to the world coordinate system (X_w,Y_w,Z_w) through inverse perspective transformation (IPM). The mapping relationship is as follows:

[0086] Let the ground be the Z_w=0 plane in the world coordinate system, and the camera optical center be located at (0,0,h) in the world coordinate system, where h is the camera mounting height. For any keypoint p=(u,v) in the image coordinate system, its corresponding world coordinates satisfy:

[0087] s·[u,v,1]^T=K·[R|t]·[X_w,Y_w,0,1]^T

[0088] Where s is the scale factor, K is the 3×3 camera intrinsic parameter matrix, R is the camera rotation matrix relative to the world coordinate system (determined by the lens tilt angle θ), and t is the translation vector. Expanding the above equation and eliminating s, we get:

[0089] X_w=h·(u-c_x)·(sinθ / (f_y·cosθ+(v-c_y)·sinθ))

[0090] Y_w=h·(v-c_y)·(cosθ / (f_y·cosθ+(v-c_y)·sinθ))

[0091] Where (c_x, c_y) are the coordinates of the camera principal point, and f_y is the pixel equivalent of the focal length. Through the above transformation, the near-to-far scale distortion caused by the downward tilt of the lens can be eliminated, and the corrected keypoint coordinates (X_w, Y_w) can be obtained.

[0092] Secondly, the relative position of the corrected buttock key point and the seat cushion detection frame is used to determine whether the seat cushion is carrying a person. The determination method can be the Euclidean distance method, that is, when the distance between the buttock key point and the center point of the seat cushion is less than a preset threshold (the preset threshold is 0.15 meters in this embodiment), it is determined that the seat cushion is carrying a person; or the IoU method can be used, that is, when the intersection-union ratio of the rectangle centered on the buttock key point and the seat cushion detection frame is greater than a preset threshold, it is determined that the seat cushion is carrying a person.

[0093] Next, the relative position of the corrected foot key points and the pedal detection frame is used to determine whether the pedal is carrying a person, and the determination method is the same as above.

[0094] Finally, based on the corrected human keypoints, skeletal proportion features are extracted to distinguish whether the cyclist is an adult or a child. The skeletal proportion features are defined as the ratio of trunk length to lower limb length, where trunk length is the Euclidean distance from the head keypoint to the hip keypoint, and lower limb length is the Euclidean distance from the hip keypoint to the foot keypoint. The distinction rules include two parallel paths: First, when the skeletal proportion features meet a first preset threshold range, the child is identified as a child; when they meet a second preset threshold range, the child is identified as an adult. The thresholds are determined by fitting the statistical distribution of skeletal proportions from a large number of adult and child samples. Specifically, based on trunk / lower limb proportion data collected from 2000 adults and 2000 children, the mean ratio for adults is 1.25 with a standard deviation of 0.13; the mean ratio for children is 0.95 with a standard deviation of 0.11. Therefore, the first preset threshold range is set to ≤1.05 for children, and the second preset threshold range is set to ≥1.10 for adults. The intermediate region is handled by the classification model. Second, the skeletal proportion features are input into a pre-trained lightweight fully connected classification model, which outputs the classification result for adults or children. This classification model is trained using the cross-entropy loss function.

[0095] V. Decision and Control Module

[0096] The decision-making and control module is used to generate control commands for the two-wheeled vehicle's power system locally based on the output of the single-model multi-task neural network. The control logic covers the entire riding lifecycle.

[0097] During the unlocking and activation phase, if it is determined within a preset activation time after unlocking that the helmet is worn correctly and there is no unauthorized passenger carrying, a permission for assistance is issued. In this embodiment, the preset activation time is preferably 5 seconds.

[0098] During the riding process, if unsafe behaviors such as removing a helmet or illegally carrying passengers are detected, the power assist will be immediately cut off and a voice warning will be triggered. The total delay from detecting the violation to issuing the cut-off command is less than 0.5 seconds.

[0099] During the correction and recovery phase, the power assist output will automatically resume after the unsafe behavior has been corrected and the pre-set time has elapsed. Situations requiring correction of unsafe behavior include putting on a helmet again or having overloaded passengers disembark.

[0100] VI. Communication Module

[0101] The communication module includes a vehicle-to-vehicle communication interface and a cloud-to-cloud communication interface.

[0102] The vehicle-to-vehicle communication interface sends control commands to the main control unit of the two-wheeled vehicle via UART or RS485 encryption protocol, ensuring the security and reliability of command transmission.

[0103] The cloud communication interface connects to the cloud platform via a Cat.1 cellular communication module, uploading only event logs and not real-time video streams. Event logs include violation events, device status, and statistical data. This design ensures traceability for monitoring while reducing the average daily data consumption per ride to less than 5MB, a reduction of two orders of magnitude compared to traditional cloud-based video analytics solutions.

[0104] VII. Method Implementation Process

[0105] This embodiment also provides a visual perception method for safe riding of two-wheeled vehicles based on edge computing, applied to the above-mentioned device, including the following steps:

[0106] S1 acquires images of two-wheeled vehicle riding scenes in real time and performs wide dynamic range (HDR) processing and infrared illumination enhancement.

[0107] S2 inputs the enhanced image into a single-model multi-task neural network for local single-pass forward inference, simultaneously acquiring human target and key point features, helmet features, and spatial relationship features between vehicle parts and the human body.

[0108] S3 performs multi-task logic fusion judgment based on the reasoning results to determine the helmet wearing compliance status and the multi-person riding violation status based on a preset number of people threshold.

[0109] S4. Generate control commands based on the stated status and send them to the main control unit.

[0110] In step S3, determining the helmet wearing compliance status includes: if no person in a riding posture is detected, it is determined that no person is riding; if a person is detected, head features are extracted for multi-category detection; if the helmet is classified as a compliant helmet, it is compliant; if it is classified as a non-safe headwear or not worn, it is non-compliant.

[0111] Step S3, determining the violation status of multiple riders, includes: triggering multi-riding analysis when the number of riders is greater than 1; matching the key points of the buttocks and feet with the seat and pedals in spatial position; determining illegal passenger transport when the number of matched riders exceeds the preset passenger threshold; and increasing the violation level and cutting off power assist if the overloaded riders include children. The preset passenger threshold is set according to the legally approved passenger capacity of two-wheeled vehicles. For electric bicycles that comply with the new national standard, the approved passenger capacity is 1 person, so the preset passenger threshold is preferably set to 2 people, that is, when the number of riders matched on the seat or pedals reaches 2 or more, it is determined to be illegal passenger transport. For electric motorcycles with an approved passenger capacity of 2 people, the preset passenger threshold can be adjusted to 3 people accordingly. This threshold can be remotely configured and distributed through a cloud platform to adapt to differences in regulations in different regions and changes in vehicle models.

[0112] VIII. System Performance Verification

[0113] To verify the actual technical effect of the device of this invention, a real-vehicle test was conducted on an edge terminal equipped with at least 1TOPS NPU. In this embodiment, a 1TOPS platform was specifically used. The test dataset included eight typical scenes, such as daytime, nighttime, rainy days, and backlighting, totaling 12,000 labeled images. All test data and training data were independent of each other. The hardware configuration of the test environment was: 1TOPS NPU, ARM Cortex-A53 quad-core CPU, 2GB LPDDR4 memory, and an image input resolution of 640×480. The test results are shown in the table below:

[0114] Performance indicators Test Results Baseline Increase Helmet detection accuracy 96.3% 94.1% +2.2 percentage points Multi-person cycling recognition accuracy 93.8% 89.5% +4.3 percentage points Adult / Child Classification Accuracy 91.2% — The baseline does not have this function. Mean end-to-end inference delay 0.28s 1.35s -79% End-to-end inference delay (P99) 0.42s 2.80s -85% Average daily traffic consumption per ride ≤5MB ≥800MB -99.4% Functionality availability during offline status 100% 0% Available completely offline

[0115] Note: The baseline for comparison is a cloud-based video analytics solution; the helmet detection accuracy evaluation standard is IoU=0.5. The detailed performance under different lighting conditions is as follows: helmet detection accuracy is 97.1% in daytime scenes, 95.8% in nighttime infrared illumination scenes, 94.6% in rainy scenes, and 94.2% in backlight scenes. The performance fluctuations in each scene are within acceptable limits, indicating that the present invention has good robustness under complex lighting conditions.

[0116] Test results show that, while ensuring recognition accuracy superior to cloud-based solutions, this invention reduces end-to-end latency to less than 0.5 seconds, meeting the response requirements of real-time power control; at the same time, it reduces traffic costs by two orders of magnitude, verifying the technical advantages of the architecture that does not upload real-time video streams.

[0117] Furthermore, to verify the superiority of the single-model multi-task architecture over the traditional multi-model concatenation scheme, comparative experiments were conducted on the same hardware platform:

[0118] Architecture solution Total inference delay NPU utilization Memory peak Combined accuracy Three-model concatenation scheme (all deployed at the edge) 0.72s 98% 480MB 88.6% This invention features a single-model multi-task approach. 0.28s 62% 210MB 93.8%

[0119] Note: The three-model cascade scheme uses a general target detection model, a general pose estimation model, and a general helmet classification model for sequential reasoning; the joint accuracy is the combined accuracy of helmet detection and multi-person cycling recognition.

[0120] The comparison results show that the single-model multi-task architecture, by sharing a backbone feature extraction network, avoids redundant feature computation and significantly outperforms the cascaded approach in terms of latency and memory usage. Furthermore, due to the complementary effect of features between multi-task branches, the joint accuracy is actually improved. This indicates that the shared backbone network achieves effective feature reuse and complementarity across different tasks, and is not simply a model merging process, but a customized design for edge computing constraints and safe two-wheeled vehicle riding scenarios.

[0121] IX. Terminology Explanation

[0122] To facilitate understanding of this invention, the following core terms are clearly defined:

[0123] An end-to-end AI camera terminal refers to an independent hardware device that integrates image acquisition, AI inference, decision control, and communication functions. All AI inference and control decisions are completed locally on the device without relying on real-time cloud computing.

[0124] A single-model multi-task neural network refers to a neural network architecture that contains only one shared backbone feature extraction network and performs multiple detection tasks simultaneously through multiple parallel branches, which is different from a scheme in which multiple independent models are connected in series or in parallel.

[0125] A shared backbone feature extraction network refers to a common network layer in a multi-task neural network used to extract general image features. All task branches reuse the feature maps output by this layer, avoiding redundant calculations.

[0126] Depthwise separable convolution refers to a convolution operation that decomposes standard convolution into two independent steps: depthwise convolution and pointwise convolution. This significantly reduces computational cost and parameter count while maintaining feature extraction capabilities.

[0127] INT8 quantization refers to a technique that converts floating-point weights and activation values ​​in a neural network model into 8-bit integer representations to reduce model storage space and inference latency.

[0128] Human body key points refer to the seven anatomical reference points output by the human body target and key point detection branch, namely the head, shoulder, elbow, waist, knee, hip and foot.

[0129] Perspective distortion correction refers to the process of mapping key points in the image coordinate system to the world coordinate system through inverse perspective transformation based on camera intrinsic parameters and installation angle, thereby eliminating scale distortion caused by lens tilt.

[0130] Skeletal proportions, referring to the ratio of trunk length to lower limb length, are used to distinguish between adults and children.

[0131] End-to-end inference latency refers to the total time from the completion of image acquisition to the generation of control commands, including preprocessing, model inference, postprocessing, and command encapsulation.

[0132] The response requirement for real-time power control is that the end-to-end inference delay is less than 0.5 seconds, so as to ensure that the power assist can be cut off in time when the rider engages in unsafe behavior, thus ensuring riding safety.

[0133] Compliant helmets refer to various safety helmets that meet national safety standards (such as GB 811-2022 "Motorcycle Passenger Helmets" or GB 24429-2009 "Safety Requirements and Test Methods for Sports Helmets for Bicycles, Skateboards and Roller Skating"), including helmets customized by shared mobility platforms with operational logos and commercially available standardized safety helmets. Their common features are a buffer layer sufficient to absorb impact energy and a lacing system that keeps the helmet securely in place, providing effective head protection in actual riding accidents.

[0134] Non-safe headwear refers to head coverings that do not have safety protection functions, including but not limited to ordinary hats, headscarves, sun hats, etc. Such headwear cannot provide effective head protection in collision accidents.

[0135] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A visual perception device for safe riding of a two-wheeled vehicle based on edge computing, characterized in that, The device is an end-to-end AI camera terminal installed on the front of a two-wheeled vehicle, including: The image acquisition module, including an image sensor and an ultra-wide-angle lens, is used to acquire scene images covering the area from the cyclist to the pedals in real time; The edge computing processing module includes a neural network processing unit for running a single-model multi-task neural network locally. The single-model multi-task neural network shares a backbone feature extraction network and outputs human target and key point detection branches, helmet detection branches, and multi-person cycling analysis branches in parallel. The human target and key point detection branch is used to output the target bounding box of the person in the image and the coordinates of human key points including the head, shoulders, elbows, waist, knees, hips and feet; the helmet detection branch is used to perform multi-category detection in the head area of ​​the target bounding box, and the categories include at least compliant helmets, non-safety headwear and no helmet; the multi-person riding analysis branch is used to detect the seat and pedal components of the two-wheeled vehicle, and analyze the number of riders, rider position and age category in combination with the human key point coordinates; The edge computing processing module is further configured to: perform perspective distortion correction on the coordinates of the human body key points; map the key points in the image coordinate system to the world coordinate system through inverse perspective transformation based on the camera intrinsic parameter matrix and lens tilt angle to obtain the corrected key point coordinates; determine whether the seat is carrying a person based on the relative position of the corrected hip key points and the seat detection frame; determine whether the pedal is carrying a person based on the relative position of the corrected foot key points and the pedal detection frame; and extract skeletal proportion features based on the corrected human body key points, wherein the skeletal proportion features are defined as the ratio of trunk length to lower limb length, used to distinguish whether the rider is an adult or a child. The decision-making and control module is used to generate control commands for the two-wheeled vehicle's power system locally based on the reasoning results; The communication module includes a vehicle-to-vehicle communication interface and a cloud-to-cloud communication interface. The vehicle-to-vehicle communication interface is used to send the control commands to the two-wheeled vehicle's main control unit to perform assist enable or disable operations. The cloud-to-cloud communication interface is used to connect to the cloud platform to upload event logs, but does not upload real-time video streams.

2. The apparatus according to claim 1, characterized in that, The image sensor is a large-area CMOS image sensor, and it integrates a wide dynamic range (HDR) processing unit and an infrared illumination unit. The ultra-wide-angle lens is installed at a downward tilt, with its preset angle being the angle between the lens optical axis and the horizontal plane. The preset angle is configured to ensure that the lens field of view completely covers the head and pedal area of ​​a rider with a height of 140cm to 190cm, and to prevent rainwater from accumulating on the lens surface.

3. The apparatus according to claim 1, characterized in that, The rules for distinguishing between adults and children include one of the following parallel paths: When the bone proportion features meet the first preset threshold range, the child is identified; when they meet the second preset threshold range, the child is identified. The thresholds are determined by fitting the statistical distribution of bone proportions of a large number of adult and child samples. The skeletal proportion features are input into a pre-trained lightweight fully connected classification model, which outputs a classification result for adults or children. The classification model is trained using the cross-entropy loss function.

4. The apparatus according to claim 1, characterized in that, The backbone feature extraction network of the single-model multi-task neural network is constructed using depthwise separable convolutions and contains 5 convolutional modules. Each module consists of a depthwise separable convolutional layer, a batch normalization layer, and a ReLU activation function. The output stride of the first and second modules is 2, and the output stride of the third to fifth modules is 1. The final output feature map size is 1 / 16 of the input image. The single-model multi-task neural network is deployed in the neural network processing unit using the INT8 quantization scheme.

5. The apparatus according to claim 1, characterized in that, The edge computing processing module also includes a CPU and DDR memory. The CPU is used for system scheduling, data preprocessing and communication management, and the DDR memory is used to store model weights, intermediate feature maps and runtime data. The neural network processing unit, the CPU and the DDR memory are configured to ensure that the total delay from the detection of a violation to the output of a cut-off command by the decision and control module is less than 0.5 seconds.

6. The apparatus according to claim 1, characterized in that, The control logic of the decision and control module covers the entire life cycle of riding, including the unlocking and starting phase, the riding phase, and the correction and recovery phase. In the unlocking and starting phase, if it is determined that the helmet is worn correctly and there is no illegal passenger carrying within the preset start time after unlocking, the power assist command is output. In the riding phase, if the helmet is removed or illegal passenger carrying is detected, the power assist is immediately cut off and a voice warning is triggered. During the correction and recovery phase, once the unsafe behavior is corrected and continues for a preset time, the assist output will automatically resume.

7. A visual perception method for safe riding of a two-wheeled vehicle based on edge computing, applied to the device according to any one of claims 1 to 6, characterized in that, include: S1 acquires images of two-wheeled vehicle riding scenes in real time and performs wide dynamic range (HDR) processing and infrared illumination enhancement processing. S2, the enhanced image is input into a single-model multi-task neural network for local single forward inference, and human target and key point features, helmet features and vehicle parts and human spatial relationship features are acquired simultaneously. S3, based on the reasoning results, performs multi-task logical fusion judgment to determine the helmet wearing compliance status and the multi-person riding violation status based on the preset number of people threshold; S4, generate control commands based on the stated status and send them to the main control unit; The determination of multiple rider violations in S3 includes: triggering multiple rider analysis when the number of riders is greater than 1; matching the key points of the buttocks and feet with the seat and pedals in spatial position; determining that the number of matched riders exceeds the preset number threshold; and if the overloaded riders include children, increasing the violation level and cutting off the power assist.

8. The method according to claim 7, characterized in that, The determination of helmet wearing compliance status in S3 includes: If no riding posture is detected, the person is considered not to be riding; If a cyclist is detected, head features are extracted for multi-class detection. If it is classified as a compliant helmet, it is compliant; if it is classified as a non-safety headgear or not worn, it is non-compliant.

9. The method according to claim 7, characterized in that, Before determining the violation status of multiple riders in S3, the method further includes: performing perspective distortion correction on the coordinates of the human body key points; mapping the key points in the image coordinate system to the world coordinate system through inverse perspective transformation based on the camera intrinsic parameter matrix and lens tilt angle to obtain the corrected key point coordinates; and extracting skeletal proportion features based on the corrected human body key points. The skeletal proportion features are defined as the ratio of trunk length to lower limb length, which is used to distinguish whether the rider is an adult or a child.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by a processor, it implements the steps of the method as described in any one of claims 7 to 9.

Images