A 360-degree monitoring, prompting and warning method and device for buses

Abstract

The embodiment of the present application discloses a 360-degree monitoring prompt and early warning method and device for buses. The technical solution provided by the embodiment of the present application is to determine the boundary target in the 360-degree image data based on the boundary detection model during the driving process of the bus, and to delineate the monitoring prompt area of ​​the current driving process based on the boundary target; determine the initial target within the first set distance range based on the 360-degree image data, judge whether the initial target is in the monitoring prompt area, and determine the initial target in the monitoring prompt area as the monitoring target; mark the monitoring target on the 360-degree image screen of the bus, mark the speed information and distance information of the monitoring target on the 360-degree image screen, and issue a hidden danger warning to the monitoring target when the speed information and / or distance information reaches the set early warning index. The above-mentioned technical means can improve the accuracy of bus driving monitoring prompts, improve the safety and reliability of bus driving, and avoid unnecessary driving accidents.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of data identification technology, and in particular to a method and device for 360-degree monitoring, prompting and warning of buses. Background Art

[0002] At present, in bus operation scenarios, in order to improve the safety of bus driving, 360-degree surveillance cameras are usually arranged around the bus to capture the surrounding view of the bus in real time, and the captured video images are displayed on the display screen of the driver's seat so that the driver can view the surrounding view of the bus. In addition, by monitoring the distance between the surrounding targets and the bus, timely warning of the approach of the target can be issued, which can improve driving safety.

[0003] However, due to the large blind spots of buses, traditional 360-degree monitoring only provides early warning of approaching people, vehicles, and obstacles based on distance. In environments with complex road conditions, it is impossible to accurately distinguish whether there are hidden dangers among the people, vehicles, and obstacles around the bus, resulting in a poor early warning effect. Summary of the invention

[0004] The embodiment of the present application provides a 360-degree monitoring, prompting and warning method and device for buses, which can accurately filter out the impact of objects outside the boundary targets on driving by detecting boundary targets, thereby improving the accuracy of monitoring prompts.

[0005] In a first aspect, an embodiment of the present application provides a 360-degree monitoring and warning method for a bus, comprising:

[0006] During the driving process of the bus, 360-degree image data of the bus is acquired in real time, the 360-degree image data is input into a pre-constructed boundary detection model, boundary targets in the 360-degree image data are determined based on the boundary detection model, and a monitoring prompt area of ​​the current driving process is delineated based on the boundary targets. The boundary detection model constructs training samples based on boundary target images of the bus driving section in advance, and the boundary detection model is trained based on the training samples;

[0007] Determine an initial target within a first set distance range based on the 360-degree image data, judge whether the initial target is in the monitoring prompt area, and determine that the initial target in the monitoring prompt area is a monitoring target;

[0008] The monitoring target is marked on the 360-degree image of the bus, the speed information and the distance information of the monitoring target are collected, the speed information and the distance information are marked on the 360-degree image, and when the speed information and / or the distance information reaches the set warning index, a hidden danger warning is issued to the monitoring target.

[0009] Furthermore, after the monitoring prompt area of ​​the current driving process is demarcated based on the boundary target object, it also includes:

[0010] A second set distance range outside the monitoring prompt area is defined as an auxiliary analysis area, a designated target in the auxiliary analysis area is detected, and the designated target is marked at a corresponding position in the auxiliary analysis area.

[0011] Furthermore, after the monitoring prompt area of ​​the current driving process is demarcated based on the boundary target object, it also includes:

[0012] The historical accident records of the bus route are obtained, and the accident-frequently occurring sections are determined based on the historical accident records. The monitoring prompt area is expanded outwardly based on the boundary target object in the accident-frequently occurring sections.

[0013] Furthermore, after the monitoring prompt area of ​​the current driving process is demarcated based on the boundary target object, it also includes:

[0014] When it is detected based on the 360-degree image data that the monitoring targets in the monitoring prompt area reach a set quantity threshold, the monitoring prompt area is expanded outward based on the boundary target object.

[0015] Furthermore, after the monitoring prompt area of ​​the current driving process is demarcated based on the boundary target object, it also includes:

[0016] When it is detected based on the 360-degree image data that the monitoring target obstructs the boundary target, the monitoring prompt area is expanded outward based on the boundary target.

[0017] Further, after detecting that the monitoring target blocks the boundary target based on the 360-degree image data, the method further includes:

[0018] An occlusion image in which the monitored target occludes the boundary target is intercepted from the 360-degree image data, a corrected training sample is constructed based on the occlusion image, and the boundary detection model is trained based on the corrected training sample.

[0019] Further, after determining the boundary target in the 360-degree image data based on the boundary detection model, the method further includes:

[0020] Target image data including the boundary target object is intercepted from the 360-degree image data, and the boundary detection model is optimized and trained based on the target image data.

[0021] In a second aspect, an embodiment of the present application provides a 360-degree monitoring, prompting and warning device for a bus, comprising:

[0022] A boundary delineation module is used to obtain 360-degree image data of the bus in real time during the driving process of the bus, input the 360-degree image data into a pre-constructed boundary detection model, determine boundary targets in the 360-degree image data based on the boundary detection model, and delineate a monitoring prompt area of ​​the current driving process based on the boundary targets. The boundary detection model constructs training samples based on boundary target images of the bus driving section in advance, and trains the boundary detection model based on the training samples;

[0023] A monitoring module, configured to determine an initial target within a first set distance range based on the 360-degree image data, determine whether the initial target is within the monitoring prompt area, and determine that the initial target within the monitoring prompt area is a monitoring target;

[0024] The early warning module is used to mark the monitored target on the 360-degree image screen of the bus, collect the speed information and distance information of the monitored target, mark the speed information and the distance information on the 360-degree image screen, and issue a hidden danger early warning to the monitored target when the speed information and / or the distance information reaches the set early warning index.

[0025] In a third aspect, an embodiment of the present application provides an electronic device, including:

[0026] memory and one or more processors;

[0027] The memory is used to store one or more programs;

[0028] When the one or more programs are executed by the one or more processors, the one or more processors implement the 360-degree monitoring, prompting and warning method for buses as described in the first aspect.

[0029] In a fourth aspect, an embodiment of the present application provides a storage medium comprising computer executable instructions, which, when executed by a computer processor, are used to execute the bus 360-degree monitoring, prompting and warning method as described in the first aspect.

[0030] The embodiment of the present application obtains 360-degree image data of a bus in real time during the driving process of the bus, inputs the 360-degree image data into a pre-constructed boundary detection model, determines boundary targets in the 360-degree image data based on the boundary detection model, and delineates a monitoring prompt area of ​​the current driving process based on the boundary targets. The boundary detection model constructs training samples based on boundary target images of the bus driving section in advance, and trains the boundary detection model based on the training samples; determines an initial target within a first set distance range based on the 360-degree image data, determines whether the initial target is within the monitoring prompt area, and determines that the initial target within the monitoring prompt area is a monitoring target; identifies the monitoring target on the 360-degree image screen of the bus, collects speed information and distance information of the monitoring target, identifies the speed information and distance information on the 360-degree image screen, and issues a hidden danger warning for the monitoring target when the speed information and / or distance information reaches the set warning index. By adopting the above-mentioned technical means, by detecting boundary targets, the impact of targets outside the boundary targets on driving can be accurately filtered out, and then early warning prompts of monitored targets can be given in the monitoring prompt area, thereby improving the accuracy of bus driving monitoring prompts, improving bus driving safety and reliability, avoiding unnecessary driving accidents, and optimizing bus operation effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 This is a flow chart of a 360-degree monitoring, prompting and warning method for a bus provided in Example 1 of the present application;

[0032] Figure 2 This is a schematic diagram of the demarcation of the monitoring prompt area in Example 1 of the present application;

[0033] Figure 3 This is a flow chart of identification of auxiliary analysis areas in Example 1 of the present application;

[0034] Figure 4 This is a schematic diagram of the structure of a 360-degree monitoring, prompting and warning device for a bus provided in Example 2 of the present application;

[0035] Figure 5 It is a structural schematic diagram of an electronic device provided in Example 3 of the present application. DETAILED DESCRIPTION

[0036] In order to make the purpose, technical scheme and advantages of the present application clearer, the specific embodiments of the present application are further described in detail below in conjunction with the accompanying drawings. It is understood that the specific embodiments described herein are only used to explain the present application, rather than to limit the present application. It should also be noted that, for the convenience of description, only part of the present application is shown in the accompanying drawings, but not all of the content. Before discussing the exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow chart describes each operation (or step) as a sequential process, many of the operations therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of the operations can be rearranged. The process can be terminated when its operation is completed, but it can also have additional steps not included in the accompanying drawings. The process can correspond to a method, a function, a procedure, a subroutine, a subprogram, etc.

[0037] Embodiment 1:

[0038] Figure 1 A flowchart of a bus 360-degree monitoring, prompting and warning method provided in the first embodiment of the present application is given. The bus 360-degree monitoring, prompting and warning method provided in this embodiment can be executed by a bus 360-degree monitoring, prompting and warning device, which can be implemented by software and / or hardware. The bus 360-degree monitoring, prompting and warning device can be composed of two or more physical entities, or it can be composed of one physical entity. Generally speaking, the bus 360-degree monitoring, prompting and warning device can be a processing device such as a vehicle-mounted central control device.

[0039] The following description is made by taking the bus 360-degree monitoring prompt warning device as an example of the main body of the bus 360-degree monitoring prompt warning method. Figure 1 The bus 360-degree monitoring and warning method specifically includes:

[0040] S110. During the driving process of the bus, 360-degree image data of the bus is obtained in real time, the 360-degree image data is input into a pre-constructed boundary detection model, boundary targets in the 360-degree image data are determined based on the boundary detection model, and a monitoring prompt area of ​​the current driving process is delineated based on the boundary targets. The boundary detection model constructs training samples based on boundary target images of the bus driving section in advance, and the boundary detection model is trained based on the training samples.

[0041] The 360-degree monitoring, prompting and warning method for buses in this application aims to detect boundary targets, accurately filter out the impact of targets outside the boundary targets on driving, and then provide early warning prompts for monitored targets within the monitoring prompt area, thereby improving the accuracy of bus driving monitoring prompts.

[0042] During the driving process of the bus, 360-degree image data is acquired and processed in real time for boundary detection and demarcation of monitoring prompt areas. By installing high-resolution 360-degree panoramic cameras or multiple wide-angle cameras around the bus, it can cover all key areas around the vehicle. The real-time image data captured by the camera is transmitted to the central processing unit (such as an on-board computer or GPU) through an efficient data transmission system (such as high-speed Ethernet).

[0043] Then, the received image data is preprocessed as necessary, such as denoising, color correction, image stitching (if multiple cameras are used), etc., to ensure the data quality of the input model. By developing or integrating an interface, the preprocessed image data can be seamlessly input into the pre-built boundary detection model. A boundary detection model is built using computer vision technologies such as deep learning. The model can accurately identify boundary targets in bus driving sections, such as roadside guardrails, road edge lines, sidewalk boundaries, etc. A large number of boundary target images of bus driving sections are collected as training samples. These samples include images under different time, weather, and lighting conditions to ensure the generalization ability of the model. The boundary detection model is trained using the training samples, and the model parameters are continuously optimized to improve detection accuracy and efficiency.

[0044] After the model is implemented, the real-time image data is input into the trained boundary detection model, and the model will output the location and outline of the boundary target in the image. Based on the detected boundary targets, the monitoring prompt area of ​​the current driving process can be delineated. This area can be a buffer zone around the boundary target, which is used to monitor potential obstacles or pedestrians that may enter the bus's driving path. Through the above steps, the bus can obtain and process 360-degree image data in real time, accurately identify boundary targets, and delineate monitoring prompt areas, thereby improving driving safety and the driver's driving experience.

[0045] Specifically, during the training process of the boundary detection model, boundary target images are first collected from the road sections where the bus actually travels. These images contain scenes at different times (daytime, nighttime), in different weather conditions (sunny, rainy, foggy, etc.), and under different lighting conditions. The collected images are manually annotated to clearly indicate the location and outline of the boundary targets in the image (such as roadside guardrails, road edge lines, sidewalk boundaries, etc.). The annotated image dataset is divided into a training set, a validation set, and a test set. The training set is used for model training, the validation set is used to adjust model parameters and perform preliminary performance evaluation, and the test set is used for final performance evaluation. Optionally, in order to increase the generalization ability of the model, data augmentation operations such as rotation, scaling, cropping, flipping, etc. can be performed on the training set to generate more training samples.

[0046] Select the deep learning model architecture according to the task requirements (boundary detection). Deep learning models include convolutional neural networks (CNN), U-Net, Mask R-CNN, etc. Then determine the parameters of the model network, such as the number of layers, convolution kernel size, activation function, etc. These parameters directly affect the performance and computational efficiency of the model. In addition, by introducing an attention mechanism or a context information extraction module, the model's ability to recognize boundary targets can be improved. Furthermore, using pre-trained weights or randomly initializing model parameters, pre-trained weights can accelerate the training process and improve the performance of the model. Then select a suitable optimization algorithm (such as Adam, SGD) and loss function (such as cross entropy loss, Dice loss) to train the model.

[0047] When training the model, the training set is input into the model, the prediction results are calculated through forward propagation, and the gradients are calculated and the model parameters are updated through back propagation. At the end of each iteration cycle, the validation set is used to evaluate the performance of the model, and the hyperparameters or model structure are adjusted as needed. If the performance on the validation set does not improve after multiple consecutive iterations, an early stopping strategy is used to prevent overfitting. The performance of the model is evaluated using the test set, and the evaluation indicators include accuracy, recall, F1 score, intersection over union (IoU), etc. The prediction results are visualized to check whether the model accurately identifies the boundary targets and delineates the correct monitoring prompt area. According to the performance evaluation results and the feedback from the visualization analysis, the model is iteratively optimized, including adjusting the model structure, optimizing parameter settings, and adding data enhancement.

[0048] Through the above training process, an accurate and reliable boundary detection model can be constructed to provide strong support for monitoring prompts during bus driving, so as to achieve accurate monitoring prompt area division.

[0049] For example, refer to Figure 2 After detecting the boundary target b (fence) in the 360-degree image data based on the boundary detection model, the area within the boundary target b, that is, the area where bus A is located, can be demarcated as a monitoring prompt area. The monitoring prompt area is as follows: Figure 2 As shown in area a in FIG. By accurately identifying boundary objects, accurate division of the monitoring prompt area is achieved.

[0050] Optionally, refer to Figure 3 After defining the monitoring prompt area of ​​the current driving process based on the boundary target, it also includes:

[0051] S1101, defining a second set distance range outside the monitoring prompt area as an auxiliary analysis area;

[0052] S1102: Detect a designated target in the auxiliary analysis area, and mark the designated target at a corresponding position in the auxiliary analysis area.

[0053] After defining the monitoring prompt area of ​​the current driving process based on boundary targets, this solution improves driving safety by expanding the monitoring range and analysis capabilities.

[0054] It is understandable that although the monitoring prompt area can cover the most direct potential risk area during the bus driving, sometimes the risk may come from a farther distance, or some targets (such as pedestrians, bicycles, etc.) may suddenly change their movement trajectory and enter the monitoring prompt area. Therefore, demarcating an auxiliary analysis area can perceive and analyze these potential risks in advance.

[0055] Among them, a suitable distance range is determined as the auxiliary analysis area based on factors such as the bus's speed, road type, and traffic rules. This range covers potential targets that may enter the monitoring prompt area. Specific targets (such as pedestrians, bicycles, electric vehicles, and other vehicles) are detected and identified in the auxiliary analysis area so that the driver or the autonomous driving system can perceive and respond accordingly in advance.

[0056] By using a pre-trained target detection model to analyze the images in the auxiliary analysis area in real time, the designated targets are identified. These targets can be pre-defined categories, such as pedestrians, bicycles, cars, etc. The detected designated targets are marked at the corresponding positions in the auxiliary analysis area. The marking method can be to mark the outline of the target with different colors or shapes on the image, or to display the relevant information of the target on the display screen (such as distance, speed, motion trajectory prediction, etc.). In addition, the detected targets are prioritized according to the potential danger level of the target (such as distance from the bus, speed, direction of movement, etc.). This helps the driver or the autonomous driving system to identify the targets that need the most attention more quickly.

[0057] Through the above steps, potential risks in the surrounding environment can be more comprehensively perceived and analyzed, improving driving safety and driving experience.

[0058] Optionally, after determining the boundary target in the 360-degree image data based on the boundary detection model, the method further includes:

[0059] Target image data containing boundary objects is captured from the 360-degree image data, and the boundary detection model is optimized and trained based on the target image data.

[0060] After determining the boundary targets in the 360-degree image data based on the boundary detection model, the present application further extracts target image data containing the boundary targets from these data, and uses these target image data to optimize the training of the boundary detection model to achieve effective model iteration and performance improvement.

[0061] Among them, by cutting out only the part containing the boundary targets from the original 360-degree image data, that is, the target image data. This can reduce the amount of calculation during model training, and at the same time make the model more focused on learning the characteristics of boundary targets. Using the output of the boundary detection model (that is, the position and outline of the boundary targets), the area containing these targets is automatically cut out from the 360-degree image data. The cut area can be appropriately expanded to ensure that it contains complete information about the boundary targets. Then, the boundary detection model is optimized and trained using the newly cut target image data to improve the detection accuracy and generalization ability of the model.

[0062] Among them, the newly generated training samples (including the original training samples and the newly intercepted target image data) are input into the boundary detection model for fine-tuning training. During fine-tuning training, most of the parameters of the model can be kept unchanged, and only the parameters of some key layers can be adjusted to speed up the training process and avoid overfitting. During and after the training, the performance of the model is evaluated using an independent test set. Evaluation indicators can include accuracy, recall rate, F1 score, intersection over union (IoU), etc. Based on the evaluation results, the model structure and training parameters can be further adjusted. By continuously iterating the above process (intercepting target image data, optimizing the training model, and performance evaluation), the performance of the boundary detection model can be gradually improved.

[0063] S120, determining an initial target within a first set distance range based on the 360-degree image data, judging whether the initial target is within a monitoring prompt area, and determining the initial target within the monitoring prompt area as a monitoring target.

[0064] Furthermore, when processing 360-degree image data, in order to determine which targets need to be monitored intensively (ie, monitoring targets), the present application uses a predefined monitoring prompt area to screen initial targets based on the area.

[0065] Among them, a target detection algorithm (such as a deep learning model) is used from the 360-degree image data to identify all potential targets within the first set distance range. These targets include pedestrians, obstacles, bicycles, other vehicles, etc. For each identified initial target, its position relative to the vehicle is calculated (for example, using a method of converting image coordinates to a world coordinate system), and the position is determined by geometric correction and coordinate transformation of the 360-degree image data. The position of each initial target is compared with the monitoring prompt area to determine whether it is located in the area. By checking whether the position coordinates of the initial target fall within the boundary of the monitoring prompt area, the initial target in the monitoring prompt area is determined as a monitoring target. Monitoring targets require special attention, which may pose a direct threat to the driving safety of the vehicle.

[0066] Optionally, after defining the monitoring prompt area of ​​the current driving process based on the boundary target object, the method further includes:

[0067] Obtain historical accident records of bus routes, determine accident-frequently occurring sections based on historical accident records, and expand the monitoring and prompting area outward based on boundary targets in accident-frequently occurring sections.

[0068] After defining the monitoring and warning area of ​​the current driving process based on boundary targets, in order to further improve the driving safety and risk warning capabilities of the bus, the historical accident records of the bus route can also be obtained to expand the monitoring and warning area.

[0069] By reading the detailed historical accident records of the bus routes, including the location, time, type, cause and other information of the accident. Perform statistical analysis on the collected historical accident records and calculate the accident rate of each road section or intersection. Select a critical number of accidents as the standard. If the number of accidents on a certain road section exceeds this standard, it is determined to be an accident-prone section. According to the results of the statistical analysis, accident-prone sections are marked on the bus routes. These sections are high-risk areas with complex road conditions, large traffic volumes, and serious mixing of pedestrians and non-motor vehicles. In accident-prone sections, due to the higher risks, more stringent monitoring and early warning are required. Therefore, this application appropriately expands the monitoring and prompting area outward based on the defined boundary targets (such as road edges, intersections, etc.).

[0070] Among them, based on the boundary targets of accident-prone sections, GIS technology (such as ArcGIS) or vehicle monitoring systems (such as map fences in the Mercedes me function) are used to expand the monitoring prompt area. The expanded monitoring prompt area is input into the bus monitoring system so that the driver or the automatic driving system can be reminded of the risks in the area in real time during driving. Through the above steps, based on the historical accident records of bus driving routes, accident-prone sections can be determined, and the monitoring prompt area can be expanded outward based on the boundary targets on these sections, thereby further improving the driving safety and risk warning capabilities of buses.

[0071] Optionally, after defining the monitoring prompt area of ​​the current driving process based on the boundary target object, the method further includes:

[0072] When the number of monitoring targets detected in the monitoring prompt area based on 360-degree image data reaches a set threshold, the monitoring prompt area is expanded outward based on the boundary targets.

[0073] After the monitoring prompt area of ​​the current driving process is delineated based on the boundary targets, in order to further improve driving safety and warning efficiency, when the number of monitoring targets (such as pedestrians, vehicles, obstacles, etc.) detected in the monitoring prompt area through 360-degree image data reaches or exceeds a set number threshold, the monitoring prompt area is expanded outward based on the boundary targets.

[0074] Among them, through image processing technology and algorithms, real-time 360-degree image data is analyzed to identify and track targets in the monitoring prompt area, such as pedestrians, bicycles, and other vehicles. The number of monitoring targets in the monitoring prompt area is counted and compared with the preset number threshold. The number threshold can be pre-set based on factors such as road type, traffic flow, and vehicle speed, aiming to reflect the complex or high-risk conditions that may occur in the current driving environment. When the number of monitoring targets reaches or exceeds the number threshold, it means that the current driving environment may become more complex or dangerous. In order to provide early warning and respond to potential risks, the system will expand the monitoring prompt area outward as needed. The direction and range of expansion can be flexibly adjusted according to actual conditions. For example, if the main risk comes from the front, it will mainly expand forward; if there are more pedestrians on both sides, it will expand to both sides.

[0075] Optionally, after defining the monitoring prompt area of ​​the current driving process based on the boundary target object, the method further includes:

[0076] When it is detected based on the 360-degree image data that the monitoring target blocks the boundary target, the monitoring prompt area is expanded outward based on the boundary target.

[0077] After the monitoring prompt area of ​​the current driving process is delineated based on boundary targets, in order to cope with special situations that may occur during actual driving, such as when the monitoring target (such as pedestrians, vehicles, large obstacles, etc.) obscures the boundary target originally used to delineate the monitoring prompt area, the system can adopt a flexible adjustment strategy to expand the monitoring prompt area outward based on this occlusion situation.

[0078] Among them, the 360-degree surround view system equipped on the vehicle is used to obtain image data around the vehicle in real time, and the data is continuously monitored and analyzed. During the monitoring process, attention is paid to whether the monitoring target blocks the boundary target originally used to delineate the monitoring prompt area. Through image recognition technology and target tracking algorithms, the system can identify the shape, size, position of the monitoring target and their relative relationship with the boundary target. When the system detects occlusion, it will further evaluate the severity and duration of the occlusion. If the occlusion causes the boundary target to be unable to be clearly identified or is completely invisible, and the occlusion is expected to continue for a period of time, the monitoring prompt area needs to be adjusted.

[0079] Based on the obstruction situation, the system will expand the monitoring prompt area outward as needed. The direction and range of the expansion can be dynamically determined based on the location and size of the obstruction and the vehicle's driving speed and direction. This ensures that even if the boundary target is obstructed, the system can cover enough driving space to provide effective monitoring and warning.

[0080] Optionally, after detecting that the monitoring target blocks the boundary target based on the 360-degree image data, the method further includes:

[0081] The occluded images of the monitored target's occluded boundary objects are captured from the 360-degree image data, and a corrected training sample is constructed based on the occluded image. The boundary detection model is trained based on the corrected training sample.

[0082] After detecting the boundary objects blocked by the monitored target based on the 360-degree image data, an automated model optimization process is introduced to further improve the robustness and accuracy of the system.

[0083] Among them, when the system detects that the monitoring target (such as pedestrians, vehicles, etc.) blocks the boundary target (such as the edge of the road, curbstone, etc.) through the 360-degree surround system, it will automatically capture the image containing the blocked scene from the real-time image data. For the intercepted blocked image, the system needs to perform further processing to construct a corrected training sample that can be used for model training. Including but not limited to labeling the image (marking the original position of the blocked boundary target, the position and shape of the blocking object, etc.), cropping, scaling, enhancement and other operations to ensure the diversity and representativeness of the sample. Using the constructed corrected training samples, the system can train or optimize the existing boundary detection model. Including adjusting the parameters of the model, updating the structure of the model or introducing new feature representations, so that the model can better cope with the boundary detection task under occlusion. After the model training or optimization is completed, it is evaluated to verify its performance. Based on the evaluation results, the system can decide whether it is necessary to further iterate the training or optimize the model.

[0084] Optionally, in actual applications, the size and shape of the monitoring prompt area can be adjusted in real time according to dynamic factors such as the bus's speed, turning intention, etc. For example, when turning, the monitoring area may need to be expanded in the direction of the turn.

[0085] S130, marking the monitored target on the 360-degree image screen of the bus, collecting speed information and distance information of the monitored target, marking the speed information and distance information on the 360-degree image screen, and issuing a hidden danger warning for the monitored target when the speed information and / or distance information reaches the set warning index.

[0086] Finally, the identification, speed and distance information of the monitored target can be collected on the 360-degree image of the bus, and hidden danger warnings can be issued when necessary.

[0087] Specifically, based on the above-determined monitoring targets, a border or mark of the monitoring target is drawn on the 360-degree image screen so that it can be clearly distinguished on the 360-degree image screen. For vehicle targets, their speed can be estimated by analyzing the displacement of the target in consecutive frames. For example, the target tracking algorithm tracks the position change of the monitoring target and then determines its speed. For non-vehicle targets such as pedestrians, speed estimation can be comprehensively judged by combining the target's motion trajectory and posture information.

[0088] On the other hand, the distance between the target and the bus is measured using monocular vision or binocular vision technology. Monocular vision relies on known camera parameters and the size information of the target to estimate the distance, while binocular vision calculates the distance by comparing the parallax in the images captured by the two cameras.

[0089] The collected speed and distance information is marked on the 360-degree image in digital or graphical form, such as next to or above the monitored target.

[0090] Furthermore, reasonable speed warning indicators and distance warning indicators are set according to factors such as the bus's speed, road conditions, and traffic rules. The speed information and distance information of the monitoring target are monitored in real time and compared with the set warning indicators. When the speed information or distance information reaches or exceeds the set warning indicator, the system should immediately issue a warning signal. The warning signal can be in various forms such as sound, image flashing, and seat vibration to attract the driver's attention. At the same time, the system can highlight or specially mark the monitoring target that has reached the warning indicator on the 360-degree image screen so that the driver can quickly identify and respond.

[0091] Through the above steps, the identification of monitoring targets, collection of speed and distance information, and hidden danger warning functions can be realized on the 360-degree image screen of the bus, thereby improving the driving safety of the bus and the riding experience of passengers.

[0092] In the above, during the driving process of the bus, the 360-degree image data of the bus is obtained in real time, the 360-degree image data is input into the pre-built boundary detection model, the boundary target in the 360-degree image data is determined based on the boundary detection model, the monitoring prompt area of ​​the current driving process is delineated based on the boundary target, the boundary detection model pre-constructs training samples based on the boundary target image of the bus driving section, and the boundary detection model is trained based on the training samples; the initial target within the first set distance range is determined based on the 360-degree image data, and it is determined whether the initial target is in the monitoring prompt area, and the initial target in the monitoring prompt area is determined as the monitoring target; the monitoring target is marked on the 360-degree image screen of the bus, the speed information and distance information of the monitoring target are collected, the speed information and distance information are marked on the 360-degree image screen, and when the speed information and / or distance information reaches the set warning index, the monitoring target is warned of hidden dangers. By adopting the above technical means, by detecting boundary targets, the influence of targets outside the boundary targets on driving is accurately filtered out, the accuracy of bus driving monitoring prompts is improved, the safety and reliability of bus driving are improved, unnecessary driving accidents are avoided, and the operation effect of buses is optimized.

[0093] Embodiment 2:

[0094] Based on the above embodiments, Figure 4 This is a schematic diagram of the structure of a bus 360-degree monitoring and warning device provided in Example 2 of this application. Figure 4 The bus 360-degree monitoring prompt warning device provided in this embodiment specifically includes:

[0095] The boundary delineation module 21 is used to obtain 360-degree image data of the bus in real time during the driving process of the bus, input the 360-degree image data into a pre-built boundary detection model, determine the boundary target objects in the 360-degree image data based on the boundary detection model, and delineate the monitoring prompt area of ​​the current driving process based on the boundary target objects. The boundary detection model constructs training samples based on the boundary target images of the bus driving section in advance, and trains the boundary detection model based on the training samples;

[0096] A monitoring module 22, configured to determine an initial target within a first set distance range based on the 360-degree image data, determine whether the initial target is within a monitoring prompt area, and determine that the initial target within the monitoring prompt area is a monitoring target;

[0097] The early warning module 23 is used to mark the monitored target on the 360-degree image of the bus, collect the speed information and distance information of the monitored target, mark the speed information and distance information on the 360-degree image, and issue a hidden danger early warning to the monitored target when the speed information and / or distance information reaches the set early warning index.

[0098] Specifically, after the monitoring prompt area of ​​the current driving process is delineated based on the boundary target object, it also includes:

[0099] A second set distance range outside the monitoring prompt area is defined as an auxiliary analysis area;

[0100] Detect the designated target in the auxiliary analysis area and mark the designated target at the corresponding position in the auxiliary analysis area.

[0101] Specifically, after determining the boundary target in the 360-degree image data based on the boundary detection model, the method further includes:

[0102] Target image data containing boundary objects is captured from the 360-degree image data, and the boundary detection model is optimized and trained based on the target image data.

[0103] Specifically, after the monitoring prompt area of ​​the current driving process is delineated based on the boundary target object, it also includes:

[0104] Obtain historical accident records of bus routes, determine accident-prone sections based on historical accident records, and expand the monitoring and prompting area outward based on boundary targets in accident-prone sections;

[0105] When the number of surveillance targets detected in the surveillance prompt area reaches a set threshold based on the 360-degree image data, the surveillance prompt area is expanded outward based on the boundary targets;

[0106] When it is detected based on the 360-degree image data that the monitoring target blocks the boundary target, the monitoring prompt area is expanded outward based on the boundary target;

[0107] The occluded images of the monitored target's occluded boundary objects are captured from the 360-degree image data, and a corrected training sample is constructed based on the occluded image. The boundary detection model is trained based on the corrected training sample.

[0108] In the above, during the driving process of the bus, the 360-degree image data of the bus is obtained in real time, the 360-degree image data is input into the pre-built boundary detection model, the boundary target in the 360-degree image data is determined based on the boundary detection model, the monitoring prompt area of ​​the current driving process is delineated based on the boundary target, the boundary detection model pre-constructs training samples based on the boundary target image of the bus driving section, and the boundary detection model is trained based on the training samples; the initial target within the first set distance range is determined based on the 360-degree image data, and it is determined whether the initial target is in the monitoring prompt area, and the initial target in the monitoring prompt area is determined as the monitoring target; the monitoring target is marked on the 360-degree image screen of the bus, the speed information and distance information of the monitoring target are collected, the speed information and distance information are marked on the 360-degree image screen, and when the speed information and / or distance information reaches the set warning index, the monitoring target is warned of hidden dangers. By adopting the above technical means, by detecting boundary targets, the influence of targets outside the boundary targets on driving is accurately filtered out, the accuracy of bus driving monitoring prompts is improved, the safety and reliability of bus driving are improved, unnecessary driving accidents are avoided, and the operation effect of buses is optimized.

[0109] The bus 360-degree monitoring, prompting and warning device provided in the second embodiment of the present application can be used to execute the bus 360-degree monitoring, prompting and warning method provided in the above-mentioned first embodiment, and has corresponding functions and beneficial effects.

[0110] Embodiment three:

[0111] Embodiment 3 of the present application provides an electronic device, referring to Figure 5 The electronic device includes: a processor 31, a memory 32, a communication module 33, an input device 34 and an output device 35. The number of processors in the electronic device can be one or more, and the number of memories in the electronic device can be one or more. The processor, memory, communication module, input device and output device of the electronic device can be connected via a bus or other means.

[0112] As a computer-readable storage medium, the memory can be used to store software programs, computer executable programs and modules, such as the program instructions / modules corresponding to the bus 360-degree monitoring, prompting and warning method described in any embodiment of the present application (for example, the various modules in the bus 360-degree monitoring, prompting and warning device). The memory may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the device, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0113] The communication module is used for data transmission.

[0114] The processor executes various functional applications and data processing of the device by running the software programs, instructions and modules stored in the memory, thereby realizing the above-mentioned 360-degree monitoring, prompting and warning method for buses.

[0115] The input device can be used to receive input digital or character information and generate key signal input related to user settings and function control of the device. The output device can include display devices such as display screens.

[0116] The electronic device provided above can be used to execute the bus 360-degree monitoring, prompting and warning method provided in the above embodiment 1, and has corresponding functions and beneficial effects.

[0117] Embodiment 4:

[0118] The embodiment of the present application also provides a storage medium containing computer executable instructions, which are used to execute a 360-degree monitoring, prompting and warning method for a bus when executed by a computer processor. The 360-degree monitoring, prompting and warning method for a bus includes: acquiring 360-degree image data of the bus in real time during the driving process of the bus, inputting the 360-degree image data into a pre-constructed boundary detection model, determining boundary targets in the 360-degree image data based on the boundary detection model, and delineating a monitoring prompt area of ​​the current driving process based on the boundary targets, the boundary detection model pre-constructing training samples based on boundary target images of the bus driving section, and training the boundary detection model based on the training samples; determining an initial target within a first set distance range based on the 360-degree image data, judging whether the initial target is within the monitoring prompt area, and determining that the initial target within the monitoring prompt area is a monitoring target; marking the monitoring target on the 360-degree image screen of the bus, collecting speed information and distance information of the monitoring target, marking the speed information and distance information on the 360-degree image screen, and issuing a hidden danger warning for the monitoring target when the speed information and / or distance information reaches the set warning index.

[0119] Storage medium - any of various types of memory devices or storage devices. The term "storage medium" is intended to include: installation media, such as CD-ROM, floppy disk or tape device; computer system memory or random access memory, such as DRAM, DDR RAM, SRAM, EDO RAM, Rambus RAM, etc.; non-volatile memory, such as flash memory, magnetic media (such as hard disk or optical storage); registers or other similar types of memory elements, etc. Storage media may also include other types of memory or combinations thereof. In addition, the storage medium may be located in the first computer system in which the program is executed, or may be located in a different second computer system, which is connected to the first computer system via a network (such as the Internet). The second computer system can provide program instructions to the first computer for execution. The term "storage medium" may include two or more storage media residing in different locations (for example, in different computer systems connected by a network). The storage medium may store program instructions (for example, embodied as a computer program) that can be executed by one or more processors.

[0120] Of course, the storage medium containing computer executable instructions provided in the embodiment of the present application is not limited to the 360-degree monitoring, prompt and warning method for buses as described above, and can also execute related operations in the 360-degree monitoring, prompt and warning method for buses provided in any embodiment of the present application.

[0121] The 360-degree monitoring, prompting and warning device, storage medium and electronic device for buses provided in the above embodiments can execute the 360-degree monitoring, prompting and warning method for buses provided in any embodiment of the present application. For technical details not described in detail in the above embodiments, please refer to the 360-degree monitoring, prompting and warning method for buses provided in any embodiment of the present application.

[0122] The above are only preferred embodiments of the present application and the technical principles used. The present application is not limited to the specific embodiments described herein, and various obvious changes, readjustments and substitutions that can be made by those skilled in the art will not deviate from the scope of protection of the present application. Therefore, although the present application is described in more detail through the above embodiments, the present application is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present application, and the scope of the present application is determined by the scope of the claims.

Claims

1. A 360-degree monitoring and warning method for buses, characterized in that: include: During the driving process of the bus, 360-degree image data of the bus is acquired in real time, the 360-degree image data is input into a pre-constructed boundary detection model, boundary targets in the 360-degree image data are determined based on the boundary detection model, and a monitoring prompt area of ​​the current driving process is delineated based on the boundary targets. The boundary detection model constructs training samples based on boundary target images of the bus driving section in advance, and the boundary detection model is trained based on the training samples; Determine an initial target within a first set distance range based on the 360-degree image data, judge whether the initial target is in the monitoring prompt area, and determine that the initial target in the monitoring prompt area is a monitoring target; The monitoring target is marked on the 360-degree image of the bus, the speed information and the distance information of the monitoring target are collected, the speed information and the distance information are marked on the 360-degree image, and when the speed information and / or the distance information reaches the set warning index, a hidden danger warning is issued to the monitoring target.

2. The bus 360-degree monitoring prompt and early warning method according to claim 1 is characterized in that: After the monitoring prompt area of ​​the current driving process is demarcated based on the boundary target object, the method further includes: A second set distance range outside the monitoring prompt area is defined as an auxiliary analysis area, a designated target in the auxiliary analysis area is detected, and the designated target is marked at a corresponding position in the auxiliary analysis area.

3. The bus 360-degree monitoring prompt and warning method according to claim 1 is characterized in that: After the monitoring prompt area of ​​the current driving process is demarcated based on the boundary target object, the method further includes: The historical accident records of the bus route are obtained, and the accident-frequently occurring sections are determined based on the historical accident records. The monitoring prompt area is expanded outwardly based on the boundary target object in the accident-frequently occurring sections.

4. The bus 360-degree monitoring prompt and warning method according to claim 1 is characterized in that: After the monitoring prompt area of ​​the current driving process is demarcated based on the boundary target object, the method further includes: When it is detected based on the 360-degree image data that the monitoring targets in the monitoring prompt area reach a set quantity threshold, the monitoring prompt area is expanded outward based on the boundary target object.

5. The bus 360-degree monitoring prompt and warning method according to claim 1 is characterized in that: After the monitoring prompt area of ​​the current driving process is demarcated based on the boundary target object, the method further includes: When it is detected based on the 360-degree image data that the monitoring target obstructs the boundary target, the monitoring prompt area is expanded outward based on the boundary target.

6. The bus 360-degree monitoring prompt and warning method according to claim 5 is characterized in that: After detecting that the monitoring target blocks the boundary target based on the 360-degree image data, the method further includes: An occlusion image in which the monitored target occludes the boundary target is intercepted from the 360-degree image data, a corrected training sample is constructed based on the occlusion image, and the boundary detection model is trained based on the corrected training sample.

7. The bus 360-degree monitoring prompt and warning method according to claim 1 is characterized in that: After determining the boundary target in the 360-degree image data based on the boundary detection model, the method further includes: Target image data including the boundary target object is intercepted from the 360-degree image data, and the boundary detection model is optimized and trained based on the target image data.

8. A 360-degree monitoring, prompting and warning device for buses, characterized in that: include: A boundary delineation module is used to obtain 360-degree image data of the bus in real time during the driving process of the bus, input the 360-degree image data into a pre-constructed boundary detection model, determine boundary targets in the 360-degree image data based on the boundary detection model, and delineate a monitoring prompt area of ​​the current driving process based on the boundary targets. The boundary detection model constructs training samples based on boundary target images of the bus driving section in advance, and trains the boundary detection model based on the training samples; A monitoring module, configured to determine an initial target within a first set distance range based on the 360-degree image data, determine whether the initial target is within the monitoring prompt area, and determine that the initial target within the monitoring prompt area is a monitoring target; The early warning module is used to mark the monitored target on the 360-degree image screen of the bus, collect the speed information and distance information of the monitored target, mark the speed information and the distance information on the 360-degree image screen, and issue a hidden danger early warning to the monitored target when the speed information and / or the distance information reaches the set early warning index.

9. An electronic device, characterized in that: include: memory and one or more processors; The memory is used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the bus 360-degree monitoring, prompting and warning method as described in any one of claims 1-7.

10. A storage medium containing computer executable instructions, characterized in that: The computer executable instructions, when executed by a computer processor, are used to execute the bus 360-degree monitoring, prompting and warning method as described in any one of claims 1-7.

Images