Blind area pedestrian early warning method, device and equipment and storage medium

Abstract

The invention discloses a blind area pedestrian early warning method, device and equipment and a storage medium, and relates to the technical field of automobile driving, and the method comprises the steps: obtaining the image data, collected by roadside end equipment, of pedestrians and vehicles; performing target detection and target tracking on the image data, and obtaining a tracking result according to the three-dimensional representation features; and based on the tracking result, adopting a first preset algorithm to carry out early warning on pedestrians in the blind area, and outputting early warning information. According to the invention, the comprehensiveness and accuracy of vehicle blind area monitoring and early warning can be improved.

Description

Technical Field

[0001] The present application relates to the field of automobile driving technology, and in particular to a blind spot pedestrian warning method, device, equipment and storage medium. Background Art

[0002] The technical background of pedestrian warning in vehicle blind spots is an important research direction in the field of traffic safety. With the acceleration of urbanization, traffic congestion and accident rates continue to rise, among which pedestrian collision accidents caused by vehicle blind spots account for a considerable proportion. Vehicle blind spots refer to areas that the driver cannot directly observe through the inside and outside rearview mirrors in the normal driving seat. These areas are particularly dangerous when the vehicle changes lanes, turns or reverses, because pedestrians, bicycles or other vehicles may be in these areas, but the driver cannot directly observe them with the naked eye, which leads to accidents. Many current vehicle blind spot warning methods usually have problems such as difficulty in observing small target groups such as young children or squatting pedestrians in the blind spots, and low accuracy when performing blind spot detection. Summary of the invention

[0003] The main purpose of this application is to provide a blind spot pedestrian warning method, device, equipment and storage medium, aiming to solve the technical problem of how to improve the comprehensiveness and accuracy of vehicle blind spot monitoring and warning.

[0004] To achieve the above objectives, the present application proposes a blind spot pedestrian warning method, the method comprising:

[0005] Obtain image data of pedestrians and vehicles collected by roadside cameras;

[0006] Performing target detection and target tracking on the image data, and obtaining tracking results according to three-dimensional representation features;

[0007] Based on the tracking result, a first preset algorithm is used to warn pedestrians in the blind spot and output warning information.

[0008] In one embodiment, the step of performing target detection and target tracking on the image data and obtaining a tracking result according to the three-dimensional representation feature comprises:

[0009] Using a first preset model to perform target detection on the image data to obtain a prediction box;

[0010] A second preset algorithm is used to track the target in the prediction frame, and a tracking result is obtained according to the three-dimensional representation feature.

[0011] In one embodiment, the step of performing target detection on the image data using the first preset model to obtain a prediction frame includes:

[0012] The anchor frame is obtained through cluster analysis;

[0013] Inputting the image data into a first preset model;

[0014] Extract features from the image data using the backbone network in the first preset model to obtain a feature map;

[0015] According to the anchor box, target detection is performed on the feature map to obtain a predicted box.

[0016] In one embodiment, the step of using the second preset algorithm to track the target on the prediction frame and obtaining the tracking result according to the three-dimensional representation feature includes:

[0017] Obtaining an initial frame according to the prediction frame;

[0018] Extracting three-dimensional representation features from the initial frame;

[0019] According to the three-dimensional representation features, a second preset algorithm is used to iteratively process the center coordinates of the prediction box to obtain a tracking result.

[0020] In one embodiment, the step of extracting three-dimensional representation features from the initial frame comprises:

[0021] Based on the initial frame, obtaining texture features through a preset relationship;

[0022] Based on the initial frame, quantize the pixel values ​​of the R and G color channels in the RGB image to obtain color features;

[0023] The color feature is combined with the texture feature to obtain a three-dimensional representation feature.

[0024] In one embodiment, the step of using a first preset algorithm to warn pedestrians in the blind spot based on the tracking result and outputting warning information includes:

[0025] Get vehicle speed and pedestrian speed;

[0026] According to the tracking result, constructing a second preset model using a first preset algorithm;

[0027] Processing the vehicle speed, the pedestrian speed and the tracking result by the second preset model to obtain a model output result;

[0028] Output warning information according to the model output result.

[0029] In one embodiment, the step of processing the vehicle speed, the pedestrian speed and the tracking result by the second preset model to obtain a model output result includes:

[0030] Inputting the vehicle speed, the pedestrian speed and the tracking result into the second preset model;

[0031] Processing the vehicle speed, the pedestrian speed, and the tracking result by using the second preset model to obtain a target subset;

[0032] Reasoning the target subset based on preset rules to obtain reasoning results;

[0033] The inference result is processed to obtain a model output result.

[0034] In addition, to achieve the above purpose, the present application also proposes a blind spot pedestrian warning device, the device comprising:

[0035] An image acquisition module is used to acquire image data of pedestrians and vehicles collected by roadside cameras;

[0036] A detection and tracking module, used to perform target detection and target tracking on the image data, and obtain tracking results according to three-dimensional representation features;

[0037] The warning output module is used to use a first preset algorithm to warn pedestrians in the blind spot based on the tracking result and output warning information.

[0038] In addition, to achieve the above-mentioned purpose, the present application also proposes a blind spot pedestrian warning device, which includes: a memory, a processor, and a computer program stored on the memory and executable on the processor, and the computer program is configured to implement the steps of the blind spot pedestrian warning method described above.

[0039] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by the processor, the steps of the blind spot pedestrian warning method described above are implemented.

[0040] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, it implements the steps of the blind spot pedestrian warning method as described above.

[0041] One or more technical solutions proposed in this application have at least the following technical effects:

[0042] By using multiple algorithms to process and detect blind spot pedestrian images collected by roadside cameras, the technical problems of difficulty in observing small target groups such as young children or squatting pedestrians in blind spots and low accuracy in blind spot detection are solved. Compared with the existing technology, the comprehensiveness and accuracy of vehicle blind spot monitoring and early warning are improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0044] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0045] Figure 1 A flowchart of the first embodiment of the blind spot pedestrian warning method of the present application is provided;

[0046] Figure 2 A flowchart of Embodiment 2 of the blind spot pedestrian warning method of this application is provided;

[0047] Figure 3 A flowchart of Embodiment 3 of the blind spot pedestrian warning method of this application is provided;

[0048] Figure 4 A schematic diagram of a fuzzy algorithm framework of a blind spot pedestrian warning method provided in Example 3 of the present application;

[0049] Figure 5 This is a schematic diagram of the module structure of the blind spot pedestrian warning device according to an embodiment of the present application;

[0050] Figure 6 Schematic diagram of the device structure of the hardware operating environment involved in the blind spot pedestrian warning method in the embodiment of the present application.

[0051] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0052] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0053] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0054] The main solution of the embodiment of the present application is: to obtain image data of pedestrians and vehicles collected by the roadside camera; to perform target detection and target tracking on the image data, and to obtain tracking results based on three-dimensional representation features; based on the tracking results, to use a first preset algorithm to warn pedestrians in the blind spot and output warning information.

[0055] In this embodiment, for the convenience of description, the following description is made by taking the internal actuator of the blind spot pedestrian warning system as the execution subject.

[0056] It is difficult for existing technologies to observe small target groups such as young children or squatting pedestrians in blind spots, and the accuracy of blind spot detection is low.

[0057] The present application provides a solution that uses multiple algorithms to process and detect blind spot pedestrian images collected by roadside cameras, thereby improving the comprehensiveness and accuracy of vehicle blind spot monitoring and early warning.

[0058] It can be seen from the above embodiments that the present application uses multiple algorithms to process and detect blind spot pedestrian images collected by roadside cameras, thereby solving the technical problems of difficulty in observing small target groups such as young children or squatting pedestrians in blind spots and low accuracy in blind spot detection, thereby improving the comprehensiveness and accuracy of vehicle blind spot monitoring and early warning.

[0059] It should be noted that the execution subject of this embodiment may be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, etc. The following takes the internal actuator of the blind spot pedestrian warning system as an example to illustrate this embodiment and the following embodiments.

[0060] Based on this, the present application embodiment provides a blind spot pedestrian warning method, referring to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the blind spot pedestrian warning method of the present application.

[0061] In this embodiment, the blind spot pedestrian warning method includes steps S10 to S30:

[0062] Step S10, obtaining image data of pedestrians and vehicles collected by the roadside camera.

[0063] It should be noted that the roadside device is the roadside camera, which is an important component of the intelligent transportation system and is mainly responsible for monitoring and collecting road traffic image information; the image data is a single-frame image captured by the camera, which can be in JPEG or other image formats, and is used for subsequent target detection, tracking and recognition and other processing.

[0064] Step S20, performing target detection and target tracking on the image data, and obtaining tracking results according to three-dimensional representation features.

[0065] It should be noted that target detection is one of the core issues in the field of computer vision. Its task is to find all targets of interest in an image and determine their categories and locations. Target tracking is an important research direction in the field of computer vision. It involves the process of continuously locating a specific target in a video sequence. The three-dimensional representation feature combines color features and texture features to obtain three-dimensional color-texture features. Color features are usually described by the color histogram of the target and depend on the distribution characteristics of the target color in the color space. Texture features describe the local arrangement pattern of pixels in the image and reflect the surface structure information of the image.

[0066] Step S30: Based on the tracking result, a first preset algorithm is used to warn pedestrians in the blind spot and output warning information.

[0067] It should be noted that the first preset algorithm is a fuzzy algorithm, also known as fuzzy logic or fuzzy reasoning. It is a mathematical method based on fuzzy set theory and is used to process fuzzy or uncertain information. Compared with traditional binary logic, fuzzy logic allows data to have ambiguity, uncertainty and partial truth.

[0068] This embodiment provides a blind spot pedestrian warning method, which uses multiple algorithms to process and detect blind spot pedestrian images collected by roadside cameras, thereby solving the technical problems of difficulty in observing small target groups such as young children or squatting pedestrians in blind spots and low accuracy in blind spot detection, thereby improving the comprehensiveness and accuracy of vehicle blind spot monitoring and warning.

[0069] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction, and will not be repeated in the following. Figure 2 , step S20 includes steps S21-S22:

[0070] Step S21: Use a first preset model to perform target detection on the image data to obtain a prediction box.

[0071] It should be noted that the first preset model is the YOLOv3 (You Only Look Once version 3) model, which is the third-generation YOLO target detection algorithm developed by Joseph Redmon. It is improved on the basis of YOLOv2, introducing multi-scale prediction and three different sizes of detection kernels, which significantly improves the detection accuracy of targets of different sizes.

[0072] In a feasible implementation manner, step S21 includes steps S211 to S214:

[0073] Step S211, obtaining an anchor frame through cluster analysis.

[0074] It should be noted that the anchor box is an important concept in the target detection algorithm. It generates multiple bounding boxes of different sizes and aspect ratios centered on each pixel of the image. These bounding boxes are called anchor boxes. On each grid cell of each feature map, YOLOv3 predicts multiple bounding boxes, each of which corresponds to an anchor box. Usually, 3 anchor boxes are defined on each cell of each feature map. The anchor box is a predefined bounding box, and its size and proportion are obtained through cluster analysis (such as k-means clustering) based on the distribution of the target size in the training data set. The anchor box helps the network predict the location and size of the target, making the detection process more efficient and accurate; Cluster Analysis is a commonly used unsupervised learning method in data mining and statistics. It aims to group objects in the data set according to similarity, so that objects in the same group have high similarity, while objects in different groups have low similarity. Cluster analysis does not rely on predefined labels or categories, but automatically discovers patterns and structures in the data through algorithms.

[0075] Cluster analysis is used to determine the appropriate anchor box size for feature maps of each scale, such as (9, 42), (12, 32), and (14, 44). These numbers represent the width and height of the anchor box used on the feature map of the new scale. For example, (9, 42) means that the width of an anchor box is 9 and the height is 42.

[0076] Step S212: input the image data into a first preset model.

[0077] The single-frame image data collected by the camera is input into the YOLOv3 model for target detection.

[0078] Step S213: extract features from the image data through the backbone network in the first preset model to obtain a feature map.

[0079] It should be noted that the backbone network is Darknet-53, which is a deep convolutional neural network composed of multiple convolutional layers and pooling layers, which is used to extract features in images; the feature map contains feature information at different levels in the image, which helps to detect objects of different sizes.

[0080] A key feature of YOLOv3 is that it makes predictions at different depths of the network, which allows the model to detect objects of different sizes. Vehicles and pedestrians are in motion, and pedestrians have different postures such as walking, standing, and squatting, which will cause the size and shape of the same object in the image to change. The network will extract different features, so a multi-scale fusion method is used to make fuller use of image information, thereby improving the detection effect of small objects. At the same time, in order to enable the model to predict more rectangular boxes and improve the recognition accuracy of small objects, a feature map of one scale, such as 39*39, is added to the original three feature maps of different scales (13*13, 26*26, 52*52).

[0081] In addition, it should be noted that the 13*13 feature map corresponds to the high-resolution area of ​​the original image, the 26*26 feature map corresponds to the medium-resolution area, and the 52*52 feature map corresponds to the low-resolution area.

[0082] Step S214: performing target detection on the feature map according to the anchor frame to obtain a predicted frame.

[0083] On the grid cell of each feature map, YOLOv3 predicts multiple bounding boxes, which are adjusted based on the anchor boxes. The model predicts the offset, confidence, and category probability of the bounding boxes, that is, the predicted boxes are obtained to determine the location and category of the target.

[0084] It should be noted that the prediction box usually includes the bounding box coordinates, confidence score and category probability; the bounding box coordinates are usually represented by four coordinate values, namely the x-coordinate (x_min) and y-coordinate (y_min) of the upper left corner, and the x-coordinate (x_max) and y-coordinate (y_max) of the lower right corner. In some cases, these coordinates can also be expressed as the coordinates of the center point plus the width and height; the confidence score is a numerical value indicating the confidence that the prediction box contains the target, which can be interpreted as the probability that the prediction box contains the target; the category probability is a probability distribution output by the model for each prediction box in the multi-category target detection task, indicating the probability that the box contains each category of targets.

[0085] By using the YOLOv3 model for target detection, the detection effect of small targets is improved.

[0086] Step S22: Use a second preset algorithm to track the predicted frame and obtain a tracking result based on the three-dimensional representation features.

[0087] The Meanshift tracking algorithm is used to track the predicted box, where the features extracted are three-dimensional representation features.

[0088] In a feasible implementation manner, step S22 includes steps S221 to S223:

[0089] Step S221, obtaining an initial frame according to the predicted frame.

[0090] The window box where the target first appears is selected from the predicted box as the initial frame.

[0091] Step S222: extracting three-dimensional representation features from the initial frame.

[0092] When object tracking is first performed, 3D representation features are extracted from the initial frame.

[0093] In a feasible implementation manner, step S222 includes steps A10 to A30:

[0094] Step A10, based on the initial frame, obtaining texture features through a preset relationship.

[0095] Roadside cameras rely on vision to complete various functions, but light has a great impact on vision. Therefore, based on the single color feature, texture features are introduced as another representation information to reduce the impact of light on the camera and improve tracking accuracy. Texture features are expressed by the following formula:

[0096]

[0097] Where (x c ,y c ) is the center point of the window, g c is the gray value of the point, the neighborhood radius is r, the number of points with the same interval in the neighborhood is p, and g c As the threshold, g c Compared with the grayscale of the neighboring points, the threshold function S(x) is expressed by the following formula:

[0098]

[0099] Step A20, based on the initial frame, quantize the pixel values ​​of the R and G color channels in the RGB image to obtain color features.

[0100] It should be noted that quantization in image processing refers to the process of mapping the pixel values ​​of an image from a continuous range or a wider discrete range to a smaller discrete range. For RGB images, the pixel value range of each color channel is usually 0 to 255, where R represents red, G represents green, and B represents blue. Quantization can reduce the number of values ​​of each channel, thereby reducing the total number of colors in the image.

[0101] Step A30, combining the color feature with the texture feature to obtain a three-dimensional representation feature.

[0102] The pixel values ​​of the R and G color channels in the quantized RGB image and the LBP texture features form a R×G×LBP three-dimensional space, that is, a three-dimensional representation feature, which is used to represent the characteristics of the target.

[0103] By combining color features and texture features, the impact of light on camera visual acquisition is reduced.

[0104] Step S223: According to the three-dimensional representation features, a second preset algorithm is used to iteratively process the center coordinates of the prediction box to obtain a tracking result.

[0105] The position of the search window is updated using the Mean Shift algorithm. The Mean Shift algorithm calculates the average moving direction of the sample points (i.e., the Mean Shift vector) and moves the search window in the direction of the highest sample point density. The window position is iteratively adjusted to move it to the area that best matches the target features. Based on the three-dimensional representation features, the center coordinates of the prediction box are iteratively processed from the initial frame to other consecutive frames using the Meanshift tracking algorithm to obtain the center coordinates of the target window, i.e., the position coordinates of the vehicle and pedestrian, thereby achieving target tracking.

[0106] It should be noted that when performing Mean Shift iterative processing, the three-dimensional representation features are updated according to the initial search box of each iteration.

[0107] The target is tracked by using Mean Shift algorithm to improve the tracking accuracy.

[0108] This embodiment provides a blind spot pedestrian warning method, which uses a first preset model to perform target detection on the image data to obtain a prediction frame; uses a second preset algorithm to track the prediction frame, and obtains a tracking result based on a three-dimensional representation feature, thereby improving the accuracy of target detection and target tracking.

[0109] Based on the first embodiment of the present application, in the third embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction, and will not be repeated in the following. Figure 3 , step S30 includes steps S31 to S34:

[0110] Step S31, obtaining vehicle speed and pedestrian speed.

[0111] Get the vehicle speed and pedestrian speed collected by sensors or vehicle controllers.

[0112] Step S32: constructing a second preset model using a first preset algorithm according to the tracking result.

[0113] According to the position coordinates and scales of vehicles and pedestrians in the tracking results, the position of vehicles and pedestrians and the size of pedestrians are taken as the main factors, and the safety model of the vehicle is determined and constructed through fuzzy algorithm. In terms of position information, the following is followed: there is enough distance S1 for the driver to brake the vehicle or perform other operations after receiving the warning signal; there is enough distance S2 for the vehicle to brake; when the vehicle stops, the vehicle and pedestrians can still maintain a certain distance.

[0114] Step S33: Process the vehicle speed, the pedestrian speed and the tracking result by using the second preset model to obtain a model output result.

[0115] The vehicle speed, the pedestrian speed and the tracking result are processed by a vehicle safety model.

[0116] In a feasible implementation manner, step S33 includes steps S331 to S334:

[0117] Step S331: input the vehicle speed, the pedestrian speed and the tracking result into the second preset model.

[0118] The vehicle speed, pedestrian speed, and the position coordinates of the vehicle and pedestrian in the tracking results are input into the vehicle's safety model.

[0119] Step S332: Process the vehicle speed, the pedestrian speed and the tracking result by using the second preset model to obtain a target subset.

[0120] The vehicle speed, pedestrian speed and tracking results are fuzzified through the vehicle safety model. Fuzzy processing requires the definition of input and output as well as related parameters and membership relationships. For each precise input data point, the corresponding membership function is used to calculate its membership value in each fuzzy set, and finally the fuzzy subset, that is, the target subset, is obtained. For example, if the input data is a temperature value, and three fuzzy sets of "low", "medium" and "high" are defined, then for each temperature value, its membership in these three sets will be calculated.

[0121] It should be noted that the membership function is a core concept in fuzzy logic, which is used to describe the degree to which an element belongs to a fuzzy set. In a fuzzy set, the membership of an element is no longer 0 or 1 in traditional binary logic, but can take any real value between 0 and 1 to represent the degree of membership of the element; a fuzzy subset is a set consisting of a series of elements and their corresponding memberships, usually represented by a symbol, such as A.

[0122] The Sigmoid membership function is used to represent the corresponding relationship between vehicle speed, pedestrian speed and tracking results:

[0123]

[0124] Among them, x is the input value, a controls the steepness of the curve. A larger a value will make the curve steeper. B controls the symmetry of the curve, that is, the midpoint position of the curve.

[0125] Step S333: inferring the target subset based on preset rules to obtain inference results.

[0126] It should be noted that the preset rules are fuzzy rules, which are a set of conditional statements in a fuzzy logic system that define the relationship between input variables and output variables.

[0127] For fuzzy rules, it is expressed by the following formula, which indicates from conditions to conclusions:

[0128] if A then B

[0129] Fuzzy reasoning is performed using the Mamdani method through fuzzy rules. The derivation process is shown in the following formula, where ∨ is the fuzzy relationship generation algorithm:

[0130] R(x,y)=(A→B)(x,y)=A(x)∨B(x,y)

[0131] B * (y) = ∨[A * (x)∧A(x)∧B(y)]∧B(y)

[0132] Use the Fuzzy toolbox in Matlab to solve and obtain the fuzzy quantity, that is, the reasoning result.

[0133] In addition, it should be noted that fuzzy quantities refer to quantities with uncertainty or ambiguity. Unlike quantities in traditional mathematics, they do not have precise numerical values, but are described by membership functions to a certain extent. The Mamdani method is a fuzzy logic reasoning method proposed by Ebrahim Mamdani and is used for the design of fuzzy control systems. The antecedents and consequents of its fuzzy rules are both fuzzy language values. In essence, fuzzy generators and fuzzy eliminators are added to the input and output parts of a pure fuzzy logic system respectively. Both the input and output are precise quantities.

[0134] Step S334, processing the inference result to obtain a model output result.

[0135] The inference results are defuzzified, and the warning results are determined based on the data obtained after defuzzification. The final prediction output is the safety level of the blind spot: safe, warning, dangerous, which is the model output result.

[0136] It should be noted that defuzzification is a key step in fuzzy logic, which converts the fuzzy set obtained after fuzzy logic reasoning into one or a group of precise data for practical application or decision-making.

[0137] By processing the input data with fuzzy algorithm, accurate warning results can be output.

[0138] Step S34: outputting warning information according to the model output result.

[0139] The warning information is output according to the output results of the model, and the DSRC technology is used to transmit the warning information to the vehicle side for warning. For example, the human-computer interaction display screen of the vehicle can display the following messages: "Safety", "Warning, there are pedestrians or vehicles in the blind spot, please pay attention to avoid them" or "Danger! There are pedestrians or vehicles close to the blind spot, please brake in time!".

[0140] It should be noted that DSRC (Dedicated Short Range Communications) is a wireless communication technology mainly used for communication between vehicles (V2V) and vehicles and infrastructure (V2I). It is based on the IEEE 802.11p standard, operates in the 5.9GHz frequency band, provides high-speed data transmission, and ensures low latency and system reliability of communication links.

[0141] This embodiment provides a blind spot pedestrian warning method, which obtains vehicle speed and pedestrian speed; based on the tracking results, a second preset model is constructed using a first preset algorithm; the vehicle speed, the pedestrian speed and the tracking results are processed by the second preset model to obtain a model output result; warning information is output based on the model output result, thereby improving the accuracy of blind spot warning.

[0142] For example, to help understand the implementation process of the vehicle safety model based on the fuzzy algorithm in the third embodiment, please refer to Figure 4 , Figure 4 A fuzzy algorithm framework schematic diagram is provided, specifically:

[0143] According to the detection and tracking results, the vehicle speed, pedestrian speed, vehicle size and position in the image, and pedestrian size and position in the image are input into the model, and the data is fuzzy processed. nAfter fuzzy reasoning and defuzzification processing, the warning result is output.

[0144] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the blind spot pedestrian warning method of the present application. More simple transformations based on this technical concept are all within the protection scope of the present application.

[0145] This application also provides a blind spot pedestrian warning device, please refer to Figure 5 , the device comprises:

[0146] The image acquisition module 10 is used to acquire image data of pedestrians and vehicles collected by roadside equipment.

[0147] The detection and tracking module 20 is used to perform target detection and target tracking on the image data and obtain tracking results according to three-dimensional representation features.

[0148] The warning output module 30 is used to use a first preset algorithm to warn pedestrians in the blind spot based on the tracking result and output warning information.

[0149] In one embodiment, the detection and tracking module 20 is further used to perform target detection on the image data using a first preset model to obtain a prediction frame; and to perform target tracking on the prediction frame using a second preset algorithm to obtain a tracking result based on a three-dimensional representation feature.

[0150] In one embodiment, the detection and tracking module 20 is also used to obtain an anchor frame through clustering; input the image data into a first preset model; extract features of the image data through a backbone network in the first preset model to obtain a feature map; perform target detection on the feature map according to the anchor frame to obtain a prediction frame.

[0151] In one embodiment, the detection and tracking module 20 is further used to obtain an initial frame based on the prediction frame; extract three-dimensional representation features from the initial frame; and according to the three-dimensional representation features, use a second preset algorithm to iteratively process the center coordinates of the prediction frame to obtain a tracking result.

[0152] In one embodiment, the detection and tracking module 20 is further used to obtain texture features based on the initial frame through a preset relationship; based on the initial frame, quantify the pixel values ​​of the R and G color channels in the RGB image to obtain color features; and combine the color features with the texture features to obtain three-dimensional representation features.

[0153] In one embodiment, the warning output module 30 is also used to obtain vehicle speed and pedestrian speed; based on the tracking results, a second preset model is constructed using a first preset algorithm; the vehicle speed, the pedestrian speed and the tracking results are processed by the second preset model to obtain a model output result; and warning information is output based on the model output result.

[0154] In one embodiment, the warning output module 30 is also used to input the vehicle speed, the pedestrian speed and the tracking result into the second preset model; process the vehicle speed, the pedestrian speed and the tracking result through the second preset model to obtain a target subset; infer the target subset based on preset rules to obtain an inference result; process the inference result to obtain a model output result.

[0155] The blind spot pedestrian warning device provided by the present application adopts the blind spot pedestrian warning method in the above embodiment, which can solve the technical problem of how to improve the comprehensiveness and accuracy of vehicle blind spot monitoring and warning. Compared with the prior art, the beneficial effects of the blind spot pedestrian warning device provided by the present application are the same as the beneficial effects of the blind spot pedestrian warning method provided by the above embodiment, and the other technical features of the blind spot pedestrian warning device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0156] The present application provides a blind spot pedestrian warning device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the blind spot pedestrian warning method in the above-mentioned embodiment one.

[0157] Reference below Figure 6 , which shows a schematic diagram of the structure of a blind spot pedestrian warning device suitable for implementing the embodiment of the present application. The blind spot pedestrian warning device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 6 The blind spot pedestrian warning device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0158] like Figure 6As shown, the blind spot pedestrian warning device may include a processing device 1001 (such as a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the blind spot pedestrian warning device are also stored. The processing device 1001, ROM1002 and RAM1004 are connected to each other via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems may be connected to the I / O interface 1006: input devices 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 1003 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 1009. The communication device 1009 may allow the blind spot pedestrian warning device to communicate wirelessly or wired with other devices to exchange data. Although the figure shows a blind spot pedestrian warning device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems may be implemented or have alternatively.

[0159] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0160] The blind spot pedestrian warning device provided by the present application adopts the blind spot pedestrian warning method in the above embodiment, which can solve the technical problem of how to improve the comprehensiveness and accuracy of vehicle blind spot monitoring and warning. Compared with the prior art, the beneficial effects of the blind spot pedestrian warning device provided by the present application are the same as the beneficial effects of the blind spot pedestrian warning method provided by the above embodiment, and the other technical features of the blind spot pedestrian warning device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0161] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0162] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0163] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the blind spot pedestrian warning method in the above-mentioned embodiment.

[0164] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0165] The computer-readable storage medium may be included in the blind spot pedestrian warning device; or may exist independently without being assembled into the blind spot pedestrian warning device.

[0166] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by the blind spot pedestrian warning device, the blind spot pedestrian warning device enables the following: to obtain image data of pedestrians and vehicles collected by the roadside terminal device; to perform target detection and target tracking on the image data, and obtain tracking results based on the three-dimensional representation features; based on the tracking results, to use the first preset algorithm to warn pedestrians in the blind spot and output warning information.

[0167] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0168] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0169] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0170] The readable storage medium provided by the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned blind spot pedestrian warning method, and can solve the technical problem of how to improve the comprehensiveness and accuracy of vehicle blind spot monitoring and warning. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the present application are the same as the beneficial effects of the blind spot pedestrian warning method provided by the above-mentioned embodiment, and will not be repeated here.

[0171] The present application also provides a computer program product, including a computer program, which implements the steps of the blind spot pedestrian warning method as described above when executed by a processor.

[0172] The computer program product provided by this application can solve the technical problem of how to improve the comprehensiveness and accuracy of vehicle blind spot monitoring and warning. Compared with the prior art, the beneficial effects of the computer program product provided by this application are the same as the beneficial effects of the blind spot pedestrian warning method provided by the above embodiment, which will not be repeated here.

[0173] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A blind spot pedestrian warning method, characterized in that: The method comprises: Obtain image data of pedestrians and vehicles collected by roadside cameras; Performing target detection and target tracking on the image data, and obtaining tracking results according to three-dimensional representation features; Based on the tracking result, a first preset algorithm is used to warn pedestrians in the blind spot and output warning information.

2. The method according to claim 1, characterized in that The step of performing target detection and target tracking on the image data and obtaining a tracking result according to the three-dimensional representation feature comprises: Using a first preset model to perform target detection on the image data to obtain a prediction box; A second preset algorithm is used to track the target in the prediction frame, and a tracking result is obtained according to the three-dimensional representation feature.

3. The method according to claim 2, characterized in that The step of using the first preset model to perform target detection on the image data to obtain a prediction frame includes: The anchor frame is obtained through cluster analysis; Inputting the image data into a first preset model; Extract features from the image data using the backbone network in the first preset model to obtain a feature map; According to the anchor box, target detection is performed on the feature map to obtain a predicted box.

4. The method according to claim 2, characterized in that The step of using the second preset algorithm to track the target in the prediction frame and obtaining the tracking result according to the three-dimensional representation feature comprises: Obtaining an initial frame according to the prediction frame; Extracting three-dimensional representation features from the initial frame; According to the three-dimensional representation features, a second preset algorithm is used to iteratively process the center coordinates of the prediction box to obtain a tracking result.

5. The method according to claim 4, characterized in that The step of extracting three-dimensional representation features from the initial frame comprises: Based on the initial frame, obtaining texture features through a preset relationship; Based on the initial frame, quantize the pixel values ​​of the R and G color channels in the RGB image to obtain color features; The color feature is combined with the texture feature to obtain a three-dimensional representation feature.

6. The method according to claim 1, characterized in that The step of using a first preset algorithm to warn pedestrians in the blind spot based on the tracking result and outputting warning information comprises: Get vehicle speed and pedestrian speed; According to the tracking result, constructing a second preset model using a first preset algorithm; Processing the vehicle speed, the pedestrian speed and the tracking result by the second preset model to obtain a model output result; Output warning information according to the model output result.

7. The method according to claim 6, characterized in that The step of processing the vehicle speed, the pedestrian speed and the tracking result by using the second preset model to obtain a model output result comprises: Inputting the vehicle speed, the pedestrian speed and the tracking result into the second preset model; Processing the vehicle speed, the pedestrian speed, and the tracking result by using the second preset model to obtain a target subset; Reasoning the target subset based on preset rules to obtain a reasoning result; The inference result is processed to obtain a model output result.

8. A blind spot pedestrian warning device, characterized in that: The device comprises: An image acquisition module is used to acquire image data of pedestrians and vehicles collected by roadside equipment; A detection and tracking module, used to perform target detection and target tracking on the image data, and obtain tracking results according to three-dimensional representation features; The warning output module is used to use a first preset algorithm to warn pedestrians in the blind spot based on the tracking result and output warning information.

9. A blind spot pedestrian warning device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the blind spot pedestrian warning method according to any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the blind spot pedestrian warning method according to any one of claims 1 to 7 are implemented.

Images