Riding safety control system and control method

Abstract

The invention provides a riding safety control system which is installed on a shared vehicle and comprises an image acquisition unit, a processing unit and a local server, the image acquisition unit is suitable for acquiring image data and comprises a first camera and a second camera, the first camera is suitable for capturing a first image, the first image comprises an area above the neck of a rider, and the second camera is suitable for capturing a second image; the second camera is suitable for capturing a second image, the second image comprises at least part of front wheels and the parking ground of the front wheels, the image data comprises training data and real-time data, and the training data comprises human body mask image data of a rider, helmet image data of a helmet worn by the rider and parking identifier data; the real-time data comprises a first image and a second image, the processing unit comprises an NPU processing unit, the NPU processing unit is suitable for carrying out feature extraction and analysis on the training data and generating model libraries, and the model libraries comprise a human body mask model library, a helmet image model library and a parking identifier model library; the NPU processing unit is further suitable for receiving the first image or the second image and comparing the first image or the second image with a model library so as to judge whether riding safety control over the shared vehicle is started or not or whether the shared vehicle is allowed to be parked in a specific area or not, and the local server can execute an instruction of riding safety control according to feedback of the processing unit. According to the invention, the NPU processing unit based on the deep neural network is arranged, so that standard wearing detection and overload detection of the helmet can be efficiently and accurately completed at the same time, and standard parking monitoring can also be completed.

Description

Technical Field

[0001] The present invention belongs to the technical field of riding control, and particularly relates to a riding safety control system and a control method. Background Art

[0002] In recent years, as a new type of urban transportation vehicle, shared electric bicycles have been rapidly popularized in China, greatly enriching the urban transportation methods and providing the public with convenient and flexible short-distance travel solutions. With the booming development of the shared electric bicycle industry, safety issues have increasingly become the focus of attention from all sectors of society. Especially in terms of unsafe riding behaviors such as overloading and not wearing helmets, it has attracted extensive social attention and the government's emphasis.

[0003] On the one hand, overloading not only increases the risk of accidents but also may cause damage to the electric bicycle and increase the operating cost; on the other hand, the behavior of not wearing a helmet greatly increases the risk of head injuries and poses a serious threat to personal safety. Currently, although technical means in the market have monitored and managed these unsafe riding behaviors, there are still defects in application.

[0004] Existing methods can be divided into two categories. One category is based on hardware sensors to collect signals, but hardware sensors are vulnerable to interference and have low accuracy, and cannot determine whether the helmet is worn properly. The other category is based on AI cameras to capture images for recognition, but it does not include a night processing solution. Without supplementary lighting, the images will be dark, and conventional supplementary lighting will affect the user experience. At the same time, the method based on deep learning has a slow recognition speed for edge devices such as cameras. Summary of the Invention

[0005] Aiming at the defects of the above-mentioned existing technologies, the purpose of the present invention is to provide a riding safety control system to meet the needs of users.

[0006] To achieve the above purpose, the present invention provides a riding safety control system installed on a shared vehicle, including

[0007] An image acquisition unit, which is adapted to acquire image data, including a first camera and a second camera. The first camera is adapted to capture a first image, and the first image includes the area above the neck of the rider. The second camera is adapted to capture a second image, and the second image includes at least part of the front wheel and the ground where it is parked.

[0008] The image data includes training data and real-time data. The training data includes the human mask image data of the rider, the helmet image data of the rider wearing a helmet, and the parking identifier data. The real-time data includes the first image and the second image.

[0009] A processing unit, which includes an NPU processing unit. The NPU processing unit is adapted to extract features from and parse training data, and generate a model library. The model library includes a human mask model library, a helmet image model library, and a parking identifier model library. The NPU processing unit is further adapted to receive the first image or the second image and compare it with the model library to determine whether to initiate riding safety control for the shared vehicle or whether to allow the shared vehicle to park in a specific area.

[0010] A local server, which can execute the instructions for the riding safety control according to the feedback of the processing unit.

[0011] Preferably: The NPU processing unit includes a deep convolutional neural network acceleration system, which includes

[0012] A multi-core parallel decomposition module, which is used to decompose the convolution dimension information of the convolution layer in the deep convolutional neural network according to the number of computing cores on the embedded neural network processor NPU, determine the number of convolution kernels responsible for each computing core, and perform multi-threaded calculation decomposition on the input feature maps according to the number of input feature maps for the convolution calculation process on one computing core.

[0013] An input data control module, which is used to read the starting addresses of each input feature map and the convolution kernel weight parameters to determine the input feature map data and convolution kernel positions responsible for each computing core;

[0014] A data format conversion module, which is used to take the input data of the convolution layer with the channel dimension as the lowest dimension, divide the dimension into an aligned length size, and store the input data of the convolution layer in the cache on the NPU.

[0015] A data expansion module, which is used to tile the input data of the convolution layer into a two-dimensional matrix according to the data dependency relationship in the convolution calculation process of the convolution layer. The data expansion module tiles the input feature map and the convolution kernel data according to the data tiling algorithm, and converts the spatial convolution operation into a two-dimensional matrix calculation;

[0016] A convolution block calculation module, which is used to perform block calculation on the two-dimensional matrix according to the matrix calculation unit in the NPU, summarize and combine the calculated results to obtain the convolution calculation result. The convolution block calculation module is based on the method of matrix slicing calculation, slices the two-dimensional matrix by columns, calculates one slice corresponding to the result at a time, and after finishing the calculation of the entire output result, replaces the data in the cache area L0B and starts the calculation of the next slice result.

[0017] Preferably: The first camera is installed on the handlebar of the shared vehicle through a snap structure, and the first camera is tilted upward by 45° to 60° relative to the horizontal plane to capture images above the necks of riders in the height range of 150 - 190 cm.

[0018] Preferably, the second camera is installed on the bottom surface of the basket of the shared vehicle, and the second camera is inclined downward by 30° to 45° with respect to the horizontal plane to capture the ground image in front of the front wheel when parking.

[0019] Preferably, the first camera includes a photosensitive element suitable for monitoring the light intensity and an infrared fill light suitable for providing night vision ability, so that the first camera can obtain the first image composed of grayscale images.

[0020] Preferably, the processing unit and the local server are integrated in the first camera, and the second camera is connected to the first camera through a wire, so as to realize the information transmission between the processing unit, the local server and the second camera.

[0021] Preferably, the instructions for the riding safety control include rejecting the return request and at least one of the following: power off to stop riding; idling riding with a limited maximum riding speed; riding accompanied by an alarm broadcast.

[0022] A riding safety control method, based on a riding safety control system as described above, includes the following steps:

[0023] S1: Collect image data through the first camera or the second camera, and extract the image data of the key frames to obtain the first image or the second image;

[0024] S2: Process the first image or the second image for feature extraction, and perform identification and comparison with the image data in the model library;

[0025] S3: Perform riding safety control according to the helmet wearing and overloading situations. Specifically:

[0026] S3-1: Judge whether the number of people is overloaded. If the number of people is overloaded, start the riding safety control;

[0027] S3-2: If the number of people is not overloaded, further judge whether the rider is wearing a helmet. If not, start the riding safety control;

[0028] S3-3: If the number of people is not overloaded and the helmet is worn, the judgment ends and the riding safety control is not started;

[0029] S4: Judge whether parking is allowed according to whether there is a parking identifier in a specific area.

[0030] Preferably, the NPU processing unit performs feature extraction on the image data through a deep convolutional neural network acceleration system to obtain the feature map of the image data, including the following steps:

[0031] 1) Decompose the convolution dimension information of the convolution layer according to the number of computing cores on the NPU processing unit to determine the number of convolution kernels responsible for each computing core;

[0032] 2) Based on the dimension information of the input image data, perform multi-threaded computing decomposition on the input image data;

[0033] 3) Read the starting addresses of each input image data and the convolution kernel weight parameters to determine the input image data and convolution kernel positions responsible for each computing core;

[0034] 4) Take the channel dimension of the convolution layer input data as the lowest dimension, perform dimension division, and store the data in the cache on the NPU processing unit;

[0035] 5) According to the data dependency relationship of the data in the convolution calculation process of the convolution layer, tile the input data to obtain a two-dimensional matrix, and perform block calculation through the matrix calculation unit, and summarize and combine to obtain the convolution calculation result.

[0036] Preferably: In the backbone network of the conventional object detection network, after the second convolution module, perform a secondary convolution operation on the generated feature map to obtain the offset value of the feature map, and its channel dimension is 2N, representing the offsets in the X and Y directions.

[0037] The beneficial effects of the present invention are:

[0038] (1) By setting the NPU processing unit based on the deep neural network, it can efficiently and accurately complete the detection of helmet standard wearing and overloading at the same time, and can also complete the monitoring of standard parking. At the same time, a deep convolutional neural network acceleration system is introduced in the NPU processing unit to implement depthwise separable convolution, reduce the calculation amount, and improve the recognition efficiency. In addition, convolution cheapness is introduced in the NPU processing unit to improve the recognition accuracy.

[0039] (2) By setting photosensitive elements in the first camera to monitor the ambient brightness and setting infrared fill lights to obtain grayscale images, it is compatible with the detection of night scenes and has a wide range of applications. Description of the Drawings

[0040] Figure 1 An assembly schematic diagram of a riding safety control system provided for Embodiment 1.

[0041] Figure 2 A flowchart of a riding safety control method provided for Embodiment 2. Detailed Embodiments

[0042] In order to make the purpose, technical solutions and advantages of the present invention clearer, the present invention will be described in detail below with reference to the drawings and specific embodiments.

[0043] Here, it should also be noted that in order to avoid obscuring the present invention with unnecessary details, only the structures and / or processing steps closely related to the solution of the present invention are shown in the drawings, while other details less related to the present invention are omitted.

[0044] In addition, it should be noted that the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements not only includes those elements, but also includes other elements not expressly listed, or also includes elements inherent to such process, method, article or device.

[0045] Embodiment 1

[0046] As Figure 1 described, a riding safety control system is installed in a shared vehicle and includes an image acquisition unit, a processing unit and a local server. The image acquisition unit is adapted to acquire image data, including a first camera 22 and a second camera 24. The first camera 22 is mounted to the handlebar of the shared vehicle through a snap structure, and the first camera 22 is tilted upward by 45° with respect to the horizontal plane to capture a first image, which includes an image of the upper part of the rider's neck in the height range of 150 - 190 cm. The second camera 24 is mounted on the bottom surface of the basket 23 of the shared vehicle, and the second camera 22 is tilted downward by 30° with respect to the horizontal plane to capture a second image, which includes at least part of the front wheel and the ground on which it is parked. The image data includes training data and real-time data. The training data includes human mask image data of the rider, helmet image data of the rider wearing a helmet 21, and parking identifier data. The real-time data includes the first image and the second image. The processing unit includes an NPU processing unit, which is adapted to perform feature extraction and analysis on the training data and generate a model library, including a human mask model library, a helmet image model library, and a parking identifier model library. The NPU processing unit is also adapted to receive the first image or the second image and compare it with the model library to determine whether to initiate riding safety control for the shared vehicle or whether to allow the shared vehicle to park in a specific area. The local server can execute the instructions for riding safety control according to the feedback of the processing unit. The instructions for riding safety control include rejecting the return request and powering off to stop riding.

[0047] In this embodiment, the first camera 22 includes a photosensitive element adapted to monitor the light intensity and an infrared fill light adapted to provide night vision capabilities, so that the first camera 22 can obtain a first image composed of grayscale images. The processing unit and the local server are integrated into the first camera 22, and the second camera 24 is connected to the first camera 22 through a wire to realize information transmission between the processing unit, the local server and the second camera 24.

[0048] In this embodiment, the NPU processing unit includes a deep convolutional neural network acceleration system, which includes a multi-core parallel decomposition module, an input data control module, a data format conversion module, a data expansion module, and a convolution block calculation module. The multi-core parallel decomposition module is used to decompose the convolution dimension information of the convolution layer in the deep convolutional neural network according to the number of computing cores on the embedded neural network processor NPU, determine the number of convolution kernels responsible for each computing core, and perform multi-threaded calculation decomposition on the input feature maps according to the number of input feature maps for the convolution calculation process on one computing core. The input data control module is used to read the starting addresses of each input feature map and the convolution kernel weight parameters to determine the input feature map data and convolution kernel positions responsible for each computing core. The data format conversion module is used to take the channel dimension of the input data of the convolution layer as the lowest dimension, divide the dimension into the alignment length size, and store the input data of the convolution layer in the cache on the NPU. The data expansion module is used to tile the input data of the convolution layer to obtain a two-dimensional matrix according to the data dependency relationship during the convolution calculation process of the convolution layer. The data expansion module tiles the input feature maps and convolution kernel data according to the data tiling algorithm, and converts the spatial convolution operation into a two-dimensional matrix calculation. The convolution block calculation module is used to perform block calculation on the two-dimensional matrix according to the matrix calculation unit in the NPU, summarize and combine the calculated results to obtain the convolution calculation result. The convolution block calculation module, based on the method of matrix sharding calculation, slices the two-dimensional matrix by columns, calculates one slice corresponding to the result at a time, and after completing the calculation of the entire output result, replaces the data in the buffer L0B and starts the calculation of the next slice result.

[0049] Embodiment 2

[0050] A riding safety control method, based on a riding safety control system as above, includes the following steps:

[0051] S1: Collect image data through the first camera 22 or the second camera 24, and extract the image data of the key frames to obtain the first image or the second image;

[0052] S2: Process the first image or the second image for feature extraction, and perform identification and comparison with the image data in the model library;

[0053] S3: Perform riding safety control according to the helmet 21 wearing and overloading situations. Specifically:

[0054] S3-1: Judge whether the number of people is overloaded. If the number of people is overloaded, start riding safety control;

[0055] S3-2: If the number of people is not overloaded, further judge whether the rider wears the helmet 21. If not, start riding safety control;

[0056] S3-3: If the number of people is not overloaded and the helmet 21 is worn, the judgment ends and the riding safety control is not activated.

[0057] S4: Determine whether parking is allowed based on whether there is a parking identifier in a specific area.

[0058] In this embodiment, the NPU processing unit performs feature extraction on the image data through the deep convolutional neural network acceleration system to obtain the feature map of the image data, including the following steps:

[0059] 1) Decompose the convolution dimension information of the convolution layer according to the number of computing cores on the NPU processing unit to determine the number of convolution kernels responsible for each computing core;

[0060] 2) Based on the dimension information of the input image data, perform multi-threaded calculation decomposition on the input image data;

[0061] 3) Read the starting addresses of each input image data and the convolution kernel weight parameters to determine the input image data and convolution kernel positions responsible for each computing core;

[0062] 4) Take the channel dimension of the convolution layer input data as the lowest dimension, perform dimension division, and store the data in the cache on the NPU processing unit;

[0063] 5) According to the data dependency relationship of the data in the convolution calculation process of the convolution layer, tile the input data to obtain a two-dimensional matrix, and perform block calculation through the matrix calculation unit, and summarize and combine to obtain the convolution calculation result.

[0064] In this embodiment, in the backbone network of the conventional object detection network, after the second convolution module, a secondary convolution operation is performed on the generated feature map to obtain the offset value of the feature map, and its channel dimension is 2N, which represents the offsets in the X and Y directions.

[0065] The above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical solutions of the present invention can be modified or equivalently replaced without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A riding safety control system, characterized in that: Installed in shared vehicles, including an image acquisition unit adapted to acquire image data, comprising a first camera and a second camera, wherein the first camera is adapted to capture a first image including an area above the rider's neck, and the second camera is adapted to capture a second image including at least a portion of the front wheel and a ground on which the front wheel is parked, The image data includes training data and real-time data, the training data includes human body mask image data of a rider, helmet image data of a rider wearing a helmet, and parking identifier data, and the real-time data includes the first image and the second image, The processing unit includes an NPU processing unit, wherein the NPU processing unit is adapted to extract and parse features of the training data and generate a model library, wherein the model library includes a human mask model library, a helmet image model library, and a parking identifier model library, and the NPU processing unit is further adapted to receive the first image or the second image and compare it with the model library to determine whether to start riding safety control for the shared vehicle or whether to allow the shared vehicle to be parked in a specific area. A local server can execute the riding safety control instructions according to the feedback from the processing unit.

2. A riding safety control system according to claim 1, characterized in that: The NPU processing unit includes a deep convolutional neural network acceleration system, which includes The multi-core parallel decomposition module is used to decompose the convolution dimension information of the convolution layer in the deep convolutional neural network according to the number of computing cores on the embedded neural network processor NPU, determine the number of convolution cores each computing core is responsible for, and perform multi-threaded computing decomposition of the input feature map according to the number of input feature maps for the convolution calculation process on a computing core. The input data control module is used to read the starting address of each input feature map and convolution kernel weight parameter to determine the input feature map data and convolution kernel position that each computing core is responsible for; The data format conversion module is used to take the channel dimension as the lowest dimension of the convolution layer input data, divide the dimension into alignment length sizes, and store the convolution layer input data into the NPU cache; A data expansion module is used to tile the input data of the convolution layer to obtain a two-dimensional matrix according to the dependency relationship of the data in the convolution calculation process of the convolution layer. The data expansion module tiles the input feature map and the convolution kernel data according to the data tiling algorithm, and converts the spatial convolution operation into a two-dimensional matrix calculation; The convolution block calculation module is used to perform block calculation on the two-dimensional matrix according to the matrix calculation unit in the NPU, and summarize and combine the calculated results to obtain the convolution calculation result. The convolution block calculation module slices the two-dimensional matrix by column based on the matrix slice calculation method, and calculates one slice corresponding to the result at a time. After the output of the entire slice result is completed, the data in the cache area L0B is replaced to start the calculation of the next slice result.

3. A riding safety control system according to claim 2, characterized in that: The first camera is mounted to the handlebar of the shared vehicle through a snap-on structure, and the first camera is tilted upward by 45° to 60° compared to the horizontal plane to capture images above the neck of riders in the height range of 150-190 cm.

4. A riding safety control system according to claim 3, characterized in that: The second camera is installed on the bottom surface of the basket of the shared vehicle, and the second camera is tilted downward by 30° to 45° compared to the horizontal plane to capture the ground image in front of the front wheels when parking.

5. A riding safety control system according to claim 4, characterized in that: The first camera includes a photosensitive element suitable for monitoring light intensity and an infrared fill light suitable for providing night vision capability, so that the first camera can obtain the first image composed of a grayscale image.

6. A riding safety control system according to claim 4, characterized in that: The processing unit and the local server are integrated into the first camera, and the second camera is connected to the first camera via a wire, thereby realizing information transmission among the processing unit, the local server and the second camera.

7. A riding safety control system according to claim 3, characterized in that: The instructions of the riding safety control include rejecting the request to return the vehicle and at least one of the following: cutting off the power to stop riding; idling riding with a maximum riding speed limit; riding accompanied by an alarm broadcast.

8. A riding safety control method, characterized in that: A riding safety control system according to any one of claims 2 to 7 comprises the following steps: S1: collecting image data through the first camera or the second camera, and extracting image data of a key frame to obtain a first image or a second image; S2: Process the first image or the second image to extract features, and perform recognition and comparison with image data in the model library; S3: Perform riding safety control based on helmet wearing and overloading conditions, specifically: S3-1: Determine whether the number of people is overloaded. If the number of people is overloaded, start riding safety control; S3-2: If the number of passengers is not overloaded, further determine whether the riders are wearing helmets. If not, activate riding safety control; S3-3: If the number of people is not overloaded and they are wearing helmets, the judgment ends and the riding safety control is not activated; S4: Determine whether parking is allowed based on whether the specific area has a parking identifier.

9. A riding safety control method according to claim 8, characterized in that: The NPU processing unit extracts features from the image data through a deep convolutional neural network acceleration system to obtain a feature map of the image data, including the following steps: 1) Decompose the convolution dimension information of the convolution layer according to the number of computing cores on the NPU processing unit to determine the number of convolution kernels each computing core is responsible for; 2) Based on the dimension information of the input image data, the input image data is decomposed by multi-threaded calculation; 3) Read the starting address of each input image data and convolution kernel weight parameter, and determine the input image data and convolution kernel position that each computing core is responsible for; 4) Divide the convolutional layer input data into dimensions with the channel dimension as the lowest dimension, and store the data into the cache on the NPU processing unit; 5) According to the data dependency of the convolution layer during the convolution calculation process, the input data is flattened to obtain a two-dimensional matrix, and the matrix calculation unit performs block calculations, which are summarized and combined to obtain the convolution calculation results.

10. A riding safety control method according to claim 9, characterized in that: In the backbone network of a conventional target detection network, after the second convolutional module, a secondary convolution operation is performed on the generated feature map to obtain the offset value of the feature map, whose channel dimension is 2N to represent the offset in the X and Y directions.

Images