Lightweight neural network model edge cross-identification vehicle yielding pedestrian method and system

Abstract

The application discloses a light neural network model edge end cross identification vehicle courtesy pedestrian method and system, belongs to the technical field of identifying vehicle courtesy pedestrians, and comprises the following steps: S1: a vehicle-mounted road surface video collector of a bus video acquisition system collects three groups of picture data of no zebra crossing picture D1, no pedestrian on the zebra crossing D2 and pedestrian on the zebra crossing D3 in a driving process; S2: the collected picture data is subjected to image preprocessing; S3: a zebra crossing identification model is trained by using negative samples (D2+D3) and positive samples D1, and a pedestrian identification model on the zebra crossing is trained by using positive samples D3 and negative samples D2; and S4: when a camera in the vehicle does not detect a zebra crossing on the road, only the zebra crossing identification model is kept listening, and when the zebra crossing on the road is detected; listening data is selectively collected to consider a multidimensional interaction state of a vehicle and a target traffic light area road condition, and a vehicle courtesy pedestrian behavior is quantified in a scoring mode.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of identifying vehicle courtesy to pedestrians, and in particular to a vehicle courtesy to pedestrians method and system based on a light neural network model edge cross identification. BACKGROUND

[0002] In the existing research on whether to show courtesy to pedestrians and its application, there are few special studies in the public transport field; and basically, simple image recognition processing logic is based on fixed camera snapshots near zebra crossings. Such application has the following problems:

[0003] ① When the vehicle speed is fast or the fixed camera snapshot frequency is low, it is easy to miss the shot, resulting in misjudgment;

[0004] ② Due to the hardware limitations of fixed cameras, there are problems such as snapshot delay. In real scenes, vehicles show courtesy to pedestrians, but are misjudged as not showing courtesy to pedestrians, resulting in misjudgment of law enforcement and affecting the credibility of law enforcement;

[0005] ③ When there is no car or pedestrian on the zebra crossing, the traditional fixed camera shooting will produce a large amount of useless information, increasing the unnecessary burden on the database.

[0006] ④ In previous research and application, there is only a "black and white" method for vehicle courtesy to pedestrians, i.e. the value of the behavior is limited to whether the vehicle shows courtesy to pedestrians. SUMMARY

[0007] The purpose of the present application is to provide a vehicle courtesy to pedestrians method and system based on a light neural network model edge cross identification to solve the problems raised in the background.

[0008] To achieve the above purpose, the present application provides the following technical solution: a vehicle courtesy to pedestrians method based on a light neural network model edge cross identification, comprising the following steps:

[0009] S1: The vehicle-mounted road video collector of the public transport video collection system collects three groups of picture data of no zebra crossing pictures D1, no pedestrian on the zebra crossing D2, and pedestrian on the zebra crossing D3 during driving;

[0010] S2: The collected picture data is subjected to image preprocessing, including light compensation, grayscale, and true histogram equalization processing methods;

[0011] S3: The negative samples (D2+D3) and positive samples D1 are used to train the zebra crossing identification model, and the positive samples D3 and negative samples D2 are used to train the pedestrian identification model on the zebra crossing;

[0012] S4: When the camera in the vehicle does not detect a zebra crossing on the road, only the zebra crossing recognition model is retained for listening, the road condition listening and transmission system is closed, when the road surface is detected to have a zebra crossing, the road condition listening and transmission system is started, the zebra crossing pedestrian recognition model is started, and the zebra crossing recognition model is closed, realizing double model cross identification;

[0013] And through the Internet of Vehicles technology to determine whether the zebra crossing at this position is a zebra crossing that needs to detect pedestrians, if so, the zebra crossing pedestrian recognition model is triggered immediately, and the vehicle speed, vehicle speed change, horn state are listened to until no zebra crossing is detected, and the listening is closed.

[0014] S5: The performance score of the bus to show the pedestrians is solved by listening to the data.

[0015] Whether to show the pedestrians is a new objective definition, and the degree of showing the pedestrians is evaluated in the form of 0-1.0, wherein the score of 0 represents the worst performance, and the value of 1 represents the best performance.

[0016] The comprehensive performance of showing the pedestrians is determined by whether there are pedestrians on the zebra crossing when the vehicle passes through the zebra crossing s (s takes the value of 0 or 1, when the vehicle passes through the zebra crossing, there are pedestrians on the zebra crossing s = 0, otherwise s = 1), the process of honking w, and the process of acceleration a.

[0017] When there are pedestrians on the zebra crossing, the vehicle is defined as not showing the pedestrians when passing through, and the score is 0, which is expressed by the formula as follows:

[0018]

[0019] Among them, the honking frequency index is an intermediate index, and the best value is assumed to be w best , w' is expressed as shown in the following formula:

[0020]

[0021] Among them, α, β, λ are hyperparameters, representing the weights between the three variables, and α + β + λ = 1, let α = 0.7, β = 0.1, λ = 0.2.

[0022] Solving s, w, and a

[0023] The score function of showing the pedestrians is defined, and it is known that if the values of s, w, and a are solved, the performance score of the bus to show the pedestrians can be obtained.

[0024] Further, the bus video acquisition system acquires three groups of picture data of no zebra crossing picture D1, no pedestrian on the zebra crossing D2 and pedestrian on the zebra crossing D3, the model is a balanced two-classification model, and the two groups of data pictures are collected in a ratio of 1:1.

[0025] Further, in S4, the zebra crossing recognition model listens to whether the vehicle honks when the vehicle passes through the zebra crossing.

[0026] Further, the road condition monitoring and transmission system is closed, and the real-time information of the vehicle speed, the road condition and the vehicle driving state is not returned to the cloud database.

[0027] Further, in S4, the zebra crossing pedestrian recognition model and the zebra crossing recognition model adopt a light neural network, and the zebra crossing pedestrian recognition model and the zebra crossing recognition model are cross-applied and transmitted to the cloud database in real time through the edge end.

[0028] Further, in S4, the zebra crossing pedestrian recognition model and the zebra crossing recognition model are trained, including two groups of images of whether there is a zebra crossing picture and whether there is a pedestrian on the zebra crossing.

[0029] The data obtained after step S2 is used as model input, a neural network is constructed, the model is trained, and a zebra crossing recognition model and a zebra crossing pedestrian recognition model are obtained, wherein the training processes of the two models are consistent.

[0030] Another technology provided by the application includes a vehicle courtesy pedestrian system with a light neural network model edge end cross recognition, which includes an acquisition module, a picture processing module, an identification module and a scoring module.

[0031] The acquisition module is used for acquiring two groups of picture data of no pedestrian on the zebra crossing and pedestrian on the zebra crossing, and sending the two groups of picture data to the picture processing module.

[0032] The picture processing module performs light supplementing, graying, equalization and standardization on the collected pictures to obtain a data format meeting the algorithm input, and outputs the data format to the identification module.

[0033] The identification module is used for identifying whether there is a zebra crossing, and if there is a zebra crossing, the zebra crossing pedestrian recognition model is triggered, and the zebra crossing recognition model is closed, and whether the vehicle honks or brakes is listened to, and the data is sent to the scoring module.

[0034] The scoring module scores the performance of the bus passing through the zebra crossing.

[0035] Compared with the prior art, the application has the following advantages:

[0036] The vehicle courtesy pedestrian method and system of the light neural network model edge cross identification provided by the application, multi-source terminal data fusion judges whether the bus is courteous to pedestrians; the double model cross identification and the Internet of Vehicles technology are adopted, the data is selectively collected and listened to, the data transmission load is reduced, the database load is reduced, and the edge computing performance is ensured; the multi-dimensional interaction state of the vehicle and the target traffic light area road condition is considered, the bus courtesy pedestrian function is innovatively defined, and the vehicle courtesy pedestrian behavior is quantified through the scoring method. BRIEF DESCRIPTION OF DRAWINGS

[0037] Figure 1 The courtesy pedestrian identification flowchart of the application;

[0038] Figure 2 The weak light zebra crossing of the application;

[0039] Figure 3 The strong light zebra crossing of the application;

[0040] Figure 4 The zebra crossing before light compensation preprocessing of the application;

[0041] Figure 5 The zebra crossing after light compensation preprocessing of the application;

[0042] Figure 6 The image after image grayscale processing of the application;

[0043] Figure 7 The logic diagram of the Internet of Vehicles technology judging the target zebra crossing at the position of the application;

[0044] Figure 8 The system diagram of the application. DETAILED DESCRIPTION

[0045] The technical solutions in the embodiments of the application will be clearly and completely described below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only part of the embodiments of the application, not all the embodiments. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor are within the protection scope of the application.

[0046] In the vehicle courtesy pedestrian identification scene, the double model cross application of zebra crossing real-time identification and pedestrian real-time identification on the zebra crossing is adopted, and real-time transmission to the cloud database through the edge is required, so the performance requirement of the edge is higher. If the network layer used in the design of the identification model is deep, the calculation time complexity in the identification is high, which may cause performance problems such as delay. Based on this, the application adopts a light neural network shufflenetv2;

[0047] Please refer toFigure 1 A vehicle yielding to pedestrian method based on light neural network model edge cross identification, comprising the following steps:

[0048] Step one: the vehicle-mounted road video collector of the bus video collection system collects two groups of picture data without pedestrians on the zebra crossing and with pedestrians on the zebra crossing;

[0049] Three groups of picture data are collected in the bus video collection system, including no zebra crossing pictures, no pedestrians on the zebra crossing, and pedestrians on the zebra crossing. This model is a balanced two-classification model, and the number of the two groups of data pictures collected is in the ratio of 1:1;

[0050] Step two: the collected picture data is preprocessed, including light compensation, grayscale, and true histogram equalization;

[0051] The reason for light compensation is as follows:

[0052] The weak light camera collects less electrical signals through the sensor than the strong light, that is, the weak light contains less information, which will affect the accuracy of modeling to some extent for the algorithm;

[0053] Compared with the image under strong light, the image noise under weak light is larger, which will also affect the accuracy of the model to some extent;

[0054] Comparison of images before and after light compensation, please refer to Figures 2-6 ;

[0055] The brightness of the picture before processing is 110.83, and the brightness of the picture after processing is 186.00.

[0056] Brightness calculation:

[0057] def get_lightness(src):

[0058] # Calculate brightness

[0059] hsv_image = cv2.cvtColor(src, cv2.COLOR_BGR2HSV)

[0060] lightness = hsv_image[:,:,2].mean()

[0061] return lightness.

[0062] A shuffleNet_v2 neural network is built using the pytorch framework

[0063] The shuffleNet_v2 network structure built is shown in the following table:

[0064]

[0065]

[0066] Step three: fusion multi-source public transport data source auxiliary judgment whether to give way to pedestrians, when the camera in the vehicle does not detect that there is a zebra crossing on the road, and through the vehicle networking technology to determine whether the identified zebra crossing geographical position is the zebra crossing that gives way to pedestrians;

[0067] When detecting that there is a zebra crossing on the road, the road condition monitoring and transmission system is started, the zebra crossing pedestrian recognition model is started, and the zebra crossing recognition model is closed. When the road condition monitoring and transmission system is closed, the real-time information of vehicle speed, road condition and vehicle driving state is not returned to the cloud database, and the double model cross recognition is realized.

[0068] Among them, through the vehicle networking technology, it is judged whether the zebra crossing at this position is a zebra crossing that needs to detect the courtesy of pedestrians. If so, the zebra crossing pedestrian recognition model is triggered immediately, and the vehicle speed, speed change, horn state are monitored until no zebra crossing is detected, and the monitoring is closed.

[0069]

[0070]

[0071] Please refer to Figure 7 , through the vehicle networking technology to determine the specific logic of the target zebra crossing at this position;

[0072] The zebra crossing recognition model is used to monitor whether the vehicle has a horn when passing through the zebra crossing.

[0073] The zebra crossing pedestrian recognition model and the zebra crossing recognition model adopt a light neural network, and the zebra crossing pedestrian recognition model and the zebra crossing recognition model are applied crossly and transmitted to the cloud database in real time through the edge end.

[0074] The trained zebra crossing pedestrian recognition model and zebra crossing recognition model obtained after step two include three groups of images: no zebra crossing picture, whether there is a zebra crossing picture, and whether there is a pedestrian on the zebra crossing;

[0075] The three groups of images are used as model input to construct neural network, train model, obtain zebra crossing recognition model and zebra crossing pedestrian recognition model, wherein the training process of the two models of zebra crossing recognition model and zebra crossing pedestrian recognition model is consistent.

[0076] The zebra crossing pedestrian recognition model and the zebra crossing recognition model adopt cross recognition compared with single model recognition, when the bus passes, the single model scheme cannot capture the state that there is no pedestrian on the zebra crossing. Or when the bus is very close to the zebra crossing, there is a pedestrian passing through the zebra crossing, the single model scheme only captures the vehicle and road state information for a very short time, and misses a lot of important vehicle and road interaction information. Compared with the zebra crossing pedestrian recognition single model, the double model cross recognition scheme adopted by the patent can capture more complete relationship between the vehicle state and the road change, provide important data support for subsequent recognition of whether to yield to pedestrians, and make the recognition accuracy higher.

[0077] The mode minimizes the edge data transmission pressure and the cloud database storage pressure, and only one recognition model is retained for edge recognition, which reduces the edge computing pressure, effectively guarantees the edge computing performance, reduces the data transmission load, reduces the database storage load, and guarantees the edge computing performance.

[0078] Step four: solve the bus performance score of yielding to pedestrians by listening to data:

[0079] Whether to yield to pedestrians is a new objective definition, and the degree of yielding to pedestrians is evaluated in the form of 0-1.0, wherein the score of 0 represents the worst performance, and the value of 1 represents the best performance;

[0080] The presence or absence of pedestrians on the zebra crossing when the vehicle passes through the zebra crossing s (s takes the value of 0 or 1, when the vehicle passes through the zebra crossing, there is a pedestrian on the zebra crossing s=0, otherwise s=1), the process of whistling w, and the process of acceleration a jointly determine the comprehensive performance of yielding to pedestrians;

[0081] When there is a pedestrian on the zebra crossing, the vehicle is defined as not yielding to pedestrians when passing through, and the score is 0, which is expressed by the formula as follows:

[0082]

[0083] Among them, the whistle frequency index is an intermediate index, and the best value is assumed to be w best , w' is expressed as shown in the following formula:

[0084]

[0085] Among them, alpha, beta, and lambda are hyperparameters, which represent the weights between the three variables, and alpha+beta+lambda=1, let alpha=0.7, beta=0.1, and lambda=0.2;

[0086] Solving s, w, and a

[0087] The definition of the pedestrian yielding score function can know that if the values of s, w and a are obtained, the yielding pedestrian, whether there is a pedestrian on the zebra crossing when the vehicle passes through the zebra crossing, the process of honking frequency w and the process of acceleration a can be obtained, and thus the performance score of the bus yielding pedestrian can be obtained.

[0088] Please refer to Figure 8 The present application provides another technology, a vehicle yielding pedestrian system based on a light neural network model edge cross identification, which comprises a collection module, a picture processing module, an identification module and a scoring module.

[0089] The collection module is used for collecting two groups of picture data of no pedestrian on the zebra crossing and pedestrian on the zebra crossing, and sending the picture data to the picture processing module.

[0090] The picture processing module performs light supplementing, graying, equalization and standardization on the collected pictures to obtain a data format meeting the algorithm input, and outputs the data to the identification module.

[0091] The identification module is used for identifying whether there is a zebra crossing, and if there is a zebra crossing, the zebra crossing pedestrian identification model is triggered, and the zebra crossing identification model is closed, and the data of whether the vehicle is honking and braking is listened to and sent to the scoring module; the scoring module scores the performance of the bus passing through the zebra crossing.

[0092] In summary, the vehicle yielding pedestrian method and system based on a light neural network model edge cross identification of the present application judge whether the bus yields the pedestrian through multi-source terminal data fusion.

[0093] The double model cross identification and the Internet of Vehicles technology are adopted to selectively collect and listen to the data, reduce the data transmission load, reduce the database load and ensure the edge computing performance.

[0094] The multi-dimensional interaction state of the vehicle and the target traffic light area road condition is considered, the bus yielding pedestrian function is innovatively defined, and the vehicle yielding pedestrian behavior is quantified through the scoring method.

[0095] The above is only the preferred specific implementation method of the present application, but the protection scope of the present application is not limited to this, any person skilled in the art can make equivalent replacement or change according to the technical solution and the inventive concept of the present application within the technical range disclosed by the present application, which should be covered in the protection scope of the present application.

Claims

1. A method for vehicle yielding to pedestrian by cross-identification of light-weight neural network model edge, characterized in that, It comprises the following steps: S1: The vehicle-mounted road video collector of the bus video collection system collects three groups of picture data of no zebra crossing picture D1, no pedestrian on the zebra crossing D2, and pedestrian on the zebra crossing D3 in the driving process; S2: The collected picture data is pre-processed, including light compensation, grayscale, and true histogram equalization processing mode; S3: The zebra crossing recognition model is trained with negative samples D2+D3 and positive samples D1, and the pedestrian recognition model on the zebra crossing is trained with positive samples D3 and negative samples D2; S4: When the camera in the vehicle does not detect a zebra crossing on the road, only the zebra crossing recognition model is kept listening, and the road condition monitoring and transmission system is closed. When a zebra crossing is detected on the road, the road condition monitoring and transmission system is turned on, the pedestrian recognition model on the zebra crossing is turned on, and the zebra crossing recognition model is turned off, realizing the cross recognition of the two models; And through the Internet of Vehicles technology, it is judged whether the zebra crossing at this position is the zebra crossing that needs to detect the courtesy pedestrian. If it is, the pedestrian recognition model on the zebra crossing is triggered immediately, and the vehicle speed, speed change, horn state are monitored until no zebra crossing is detected, and the monitoring is closed; S5: The courtesy pedestrian performance score of the bus is solved through the monitoring data; Whether to be courteous to pedestrians is a new objective definition, and the degree of courtesy to pedestrians is evaluated in the value range of 0-1.0, where a score of 0 indicates the worst performance, and a value of 1 indicates the best performance; Whether there is a pedestrian on the zebra crossing when the vehicle passes through the zebra crossing s, s takes the value of 0 or 1, when there is a pedestrian on the zebra crossing when the vehicle passes through the zebra crossing s=0, otherwise s=1, process horn frequency w, process acceleration a three aspects together determine the comprehensive performance of courtesy to pedestrians; Among them, when there is a pedestrian on the zebra crossing, the vehicle passing through is defined as not courteous to pedestrians, and the score is 0, which can be expressed by the formula as follows: Wherein, the number of sounding the whistle is an intermediate index, and the best value of the index is w best , and w' is expressed as the following formula: Where, α, β, λ are hyperparameters, representing the weights between the three variables, and α+β+λ=1, let α=0.7, β=0.1, λ=0.2; Solving of s, w, a After defining the courtesy pedestrian score function, it is known that if the values of s, w, and a are solved, the courtesy pedestrian, whether there is a pedestrian on the zebra crossing when the vehicle passes through the zebra crossing s, the process horn frequency w, and the process acceleration a can be solved, and thus the courtesy pedestrian performance score of the bus can be obtained.

2. The method of claim 1, wherein the method comprises: For S1, the bus video collection system collects three groups of picture data of no zebra crossing picture D1, no pedestrian on the zebra crossing D2, and pedestrian on the zebra crossing D3, which is a balanced two-class model of positive and negative samples, and the two groups of data pictures are collected in a ratio of 1:

1.

3. The method of claim 1, wherein the method further comprises: determining, by the light neural network model edge end, whether the vehicle is in a lane of the road; and determining, by the light neural network model edge end, whether the vehicle is in a lane of the road. For S4, the zebra crossing recognition model is used to monitor whether the vehicle has a horn when the vehicle passes through the zebra crossing.

4. The method of claim 1, wherein the method further comprises: For S4, the road condition monitoring and transmission system is closed, and the real-time information of vehicle speed, road condition, and vehicle driving state is not transmitted to the cloud database.

5. The light neural network model edge-end cross-identification vehicle yielding pedestrian method of claim 1, wherein, For S4, the zebra crossing pedestrian recognition model and the zebra crossing recognition model use light neural networks, and the zebra crossing pedestrian recognition model and the zebra crossing recognition model are applied crossly and transmitted to the cloud database in real time through the edge.

6. The light neural network model edge-end cross-identification vehicle yielding pedestrian method of claim 1, wherein, For S4, the zebra crossing pedestrian recognition model and the zebra crossing recognition model are trained, including three groups of images in the driving process: no zebra crossing image, whether there is a zebra crossing image, and whether there is a pedestrian on the zebra crossing; The data obtained after step S2 is used as model input, a neural network is constructed, the model is trained, and a zebra crossing recognition model and a pedestrian recognition model on the zebra crossing are obtained, wherein the training processes of the two models are consistent as a whole.

7. A vehicle yielding pedestrian system at intersection edge with light neural network model for implementing the method of claim 1-6. It comprises a collection module, a picture processing module, an identification module and a scoring module. The collection module is used to collect three groups of picture data in the driving process: no zebra crossing image, no pedestrian on the zebra crossing, and pedestrian on the zebra crossing, and send them to the picture processing module; The picture processing module performs light compensation, grayscale, equalization and normalization processing on the collected pictures to obtain data formats that meet the algorithm input, and outputs them to the identification module; The identification module is used to identify whether there is a zebra crossing. If there is a zebra crossing, the zebra crossing pedestrian recognition model is triggered, and the zebra crossing recognition model is closed. Meanwhile, the identification module listens to whether the vehicle is honking or braking data, and sends the data to the scoring module; The scoring module scores the performance of the bus passing through the zebra crossing.

Images