Method, device and equipment for determining intersection information based on unmanned aerial vehicle

Abstract

The embodiment of the invention provides an intersection information determination method, device and equipment based on an unmanned aerial vehicle, and is used for solving the problem of inaccurate determination of channelization information in related technologies. According to the embodiment of the invention, the high-definition video frame of the intersection is obtained through high-altitude shooting of the unmanned aerial vehicle, and the complete channelized structure and the dynamic traffic flow are covered. Moreover, the zebra crossing is identified firstly, and then the stop line is detected based on the zebra crossing position cutting area, so that mutual interference of the two types of key points is avoided, and the key point identification accuracy can be improved.

Description

Technical Field

[0001] This application relates to the technical fields of computer vision, UAV target recognition and tracking, and intelligent transportation, and in particular to a method, apparatus and equipment for determining intersection information based on UAVs. Background Technology

[0002] With the continuous development of Intelligent Transportation Systems (ITS), urban road traffic management has placed higher demands on high-precision and high-efficiency information collection methods. Especially in applications such as traffic signal control optimization, intersection reconstruction design, traffic simulation modeling, traffic problem diagnosis, and channelization problem analysis, the demand for collecting channelization information at intersections is increasing. This channelization information includes key points at intersections, such as key points of zebra crossings and stop lines.

[0003] Traditional information gathering methods mainly rely on the following types of approaches:

[0004] Manual on-site surveys and video recording: This involves manually measuring and photographing intersections or setting up fixed cameras to record channelization information. This method is labor-intensive, time-consuming, and the data is highly subjective, posing safety hazards, especially during peak hours or at complex intersections. Summary of the Invention

[0005] This application provides a method for determining intersection information based on unmanned aerial vehicles (UAVs) to solve the problem of inaccurate determination of channelization information in related technologies.

[0006] In a first aspect, embodiments of this application provide a method for determining intersection information based on unmanned aerial vehicles (UAVs), the method comprising:

[0007] Acquire video streams captured by drones;

[0008] Extract video frames from the video stream, input the video frames into a pre-trained zebra crossing recognition model, and obtain the first position information of the zebra crossing key points in the video frames output by the zebra crossing recognition model;

[0009] Based on the first location information, the video frame is cropped to obtain a cropped image containing zebra crossings. The cropped image is then input into a pre-trained stop line recognition model to obtain the second location information of the stop line key points in the cropped image output by the stop line recognition model.

[0010] The above-mentioned technical solution has the following advantages or beneficial effects: In this embodiment, high-definition video frames of the intersection are obtained by drone photography, covering the complete channelization structure and dynamic traffic flow. Furthermore, this application first identifies zebra crossings, and then detects stop lines based on the zebra crossing location by cropping the area, avoiding interference between the two types of key points and improving the accuracy of key point identification.

[0011] In one possible implementation, cropping the video frame based on the first location information to obtain a cropped image containing zebra crossings includes:

[0012] Based on the first target location information of the zebra crossing key points, determine the rectangle containing the zebra crossing;

[0013] The video frame is binarized, and the binarized video frame is subjected to a preset number of erosion and dilation morphological processing. Based on the morphologically processed image, the region with the largest connected component is determined. The first boundary of the region is determined to be closest to the center point of the rectangle, and the second boundary of the line segment of the rectangle is determined to be farthest from the first boundary. Based on the first boundary and the second boundary, a stop line detection region is determined. Based on the stop line detection region, the video frame is cropped.

[0014] The above-mentioned technical solution has the following advantages or beneficial effects: In this embodiment, the inherent spatial constraints of zebra crossings and stop lines are utilized to narrow the detection range; the influence of illumination changes and partial occlusion is effectively overcome through connected component analysis; and the established dynamic boundary detection mechanism can adapt to different road design specifications. This, in turn, can improve the accuracy of lane line key point detection.

[0015] In one possible implementation, the method further includes:

[0016] Based on the stop line corresponding to the determined stop line key point, the image is sliced ​​at a preset distance away from the zebra crossing. The sliced ​​image is then subjected to erosion and dilation morphological processing. Based on the number of connected components contained in the processed image, the number of lanes is determined, and the center point of each connected component is determined as the center point of the lane line.

[0017] The above-mentioned technical solution has the following advantages or beneficial effects: It reduces computational load and avoids perspective distortion problems in long-distance lane line detection by tiling the image at a preset distance away from the zebra crossing. Morphological processing enhances robustness, effectively handling complex scenarios such as uneven lighting, partial occlusion, or road surface wear, ensuring lane line continuity. By preset extension distances and connected component filtering thresholds, it can adapt to different road designs.

[0018] In one possible implementation, extracting video frames from the video stream includes:

[0019] Extract multiple video frames from the video stream;

[0020] The method further includes:

[0021] Based on the K-means clustering algorithm, the first location information of zebra crossing key points in multiple video frames is clustered to obtain the first target location information of zebra crossing key points.

[0022] Based on the K-means clustering algorithm, the second location information of the stop line key points in multiple determined video frames is clustered to obtain the second target location information of the stop line key points.

[0023] The above technical solution has the following advantages or beneficial effects: the clustering algorithm is naturally robust to outliers and can automatically filter out false detections that deviate significantly from the main group; moreover, by adjusting the number of clusters, it can flexibly adapt to different road scenarios, such as double zebra crossings in school areas or diagonal stop lines at special intersections.

[0024] In one possible implementation, the method further includes:

[0025] Target video frames are extracted from the video stream. Based on a rotating bounding box detection algorithm, rotating rectangular boxes that fit the target contour are extracted from the target video frames. A rectangular box detection method is then used to obtain the smallest bounding horizontal rectangular box that completely contains the target in the target video frame. The rotating rectangular boxes and the horizontal rectangular boxes are matched to obtain each successfully matched target rotating rectangular box. The targets include vehicles and pedestrians.

[0026] The movement direction is determined based on the long side of the target rotation bounding box in each target video frame, and the target corresponding to the successfully matched target rotation bounding box is tracked based on the determined movement direction.

[0027] The above-mentioned technical solution has the following advantages or beneficial effects: In the embodiments of this application, the introduction of the rotating frame reduces the target coverage area, significantly reduces background interference, and reduces direction estimation error. The matching verification mechanism reduces the false negative rate, especially improving the robustness of detection for tilted vehicles (such as trucks turning).

[0028] In one possible implementation, the matching based on the rotated rectangle and the horizontal rectangle to obtain each successfully matched target rotated rectangle includes:

[0029] For each horizontal rectangle in the video frame, determine the overlapping area between the horizontal rectangle and each rotating rectangle, and determine the ratio of each overlapping area to the area of ​​the horizontal rectangle; the rotating rectangles whose corresponding ratio is greater than a threshold are determined as rotating rectangles that contain the same target as the horizontal rectangle, and the rotating rectangles are determined as successfully matched target rotating rectangles.

[0030] The above-mentioned technical solution has the following advantages or beneficial effects: by adopting the method provided in the embodiments of this application, the horizontal rectangular box is calculated efficiently, the rotated rectangular box can closely fit the target contour, and can be matched accurately and effectively, thereby improving the accuracy of target detection.

[0031] In one possible implementation, after matching the rotated rectangle and the horizontal rectangle to obtain each successfully matched target rotated rectangle, and before tracking the target corresponding to the successfully matched target rotated rectangle based on the determined movement direction, the method further includes:

[0032] For each target rotating rectangle, based on the position information of the first center point of the target rotating rectangle and the second center point of the horizontal rectangle that successfully matches the target rotating rectangle in the target video frame, the distance deviation between the first center point and the second center point in the target video frame is determined. The target video frame and its adjacent video frames are then input into the recognition model to obtain the predicted direction corresponding to the target rotating rectangle output by the recognition model. The direction deviation between the predicted direction and the movement direction determined based on the long side of the target rotating rectangle is determined. Based on the direction deviation and the distance deviation, the matching degree between the target rotating rectangle and the matching horizontal rectangle is determined. If the matching degree is greater than a preset matching threshold, the subsequent step of tracking the target corresponding to the successfully matched target rotating rectangle based on the determined movement direction is executed.

[0033] The above technical solution has the following advantages or beneficial effects: by using the method provided in the embodiments of this application, and further calculating the matching degree based on the angle deviation and position deviation, the accuracy of matching can be improved, thereby improving the effect of target tracking.

[0034] In one possible implementation, the method further includes:

[0035] Based on the location information of each key point in the corresponding video frame and the global positioning system (GPS) information of the UAV, the actual location information of each key point is determined.

[0036] The above-mentioned technical solution has the following advantages or beneficial effects: In this embodiment of the application, mapping each key point from the location information of the video frame to the actual location information of each key point helps to locate each key point on the actual urban road. Whether it is an urban road during peak hours or a complex intersection, it can ensure the accuracy and efficiency of road traffic management in various application scenarios.

[0037] Secondly, embodiments of this application provide a UAV-based intersection information determination device, the device comprising:

[0038] The acquisition module is used to acquire the video stream captured by the drone;

[0039] The processing module is used to extract video frames from the video stream, input the video frames into a pre-trained zebra crossing recognition model, obtain the first position information of the zebra crossing key points in the video frames output by the zebra crossing recognition model; crop the video frames based on the first position information to obtain a cropped image containing zebra crossings, input the cropped image into a pre-trained stop line recognition model, and obtain the second position information of the stop line key points in the cropped image output by the stop line recognition model.

[0040] In one possible implementation, the processing module is specifically used for

[0041] Based on the first target location information of the zebra crossing key points, determine the rectangle containing the zebra crossing;

[0042] The video frame is binarized, and the binarized video frame is subjected to a preset number of erosion and dilation morphological processing. Based on the morphologically processed image, the region with the largest connected component is determined. The first boundary of the region is determined to be closest to the center point of the rectangle, and the second boundary of the line segment of the rectangle is determined to be farthest from the first boundary. Based on the first boundary and the second boundary, a stop line detection region is determined. Based on the stop line detection region, the video frame is cropped.

[0043] In one possible implementation, the processing module is further configured to slice the image at a preset distance away from the zebra crossing based on the stop line corresponding to the determined stop line key point, perform erosion and dilation morphological processing on the sliced ​​image, determine the number of lanes based on the number of connected components contained in the processed image, and determine the center point of each connected component as the center point of the lane line.

[0044] In one possible implementation, the processing module is further configured to extract multiple video frames from the video stream;

[0045] The processing module is specifically used to cluster the first location information of zebra crossing key points in multiple video frames based on the K-means clustering algorithm to obtain the first target location information of the zebra crossing key points; and to cluster the second location information of stop line key points in multiple video frames based on the K-means clustering algorithm to obtain the second target location information of the stop line key points.

[0046] In one possible implementation, the processing module is further configured to extract target video frames from the video stream, extract rotating rectangular boxes that fit the target contour in the target video frames based on a rotating bounding box detection algorithm, and obtain the smallest bounding horizontal rectangular box that completely contains the target in the target video frames using a rectangular box detection method; perform matching based on the rotating rectangular boxes and the horizontal rectangular boxes to obtain each successfully matched target rotating rectangular box; wherein the target includes vehicles and pedestrians; determine the movement direction based on the long side of each target rotating bounding box in the target video frames, and track the target corresponding to the successfully matched target rotating rectangular box based on the determined movement direction.

[0047] In one possible implementation, the processing module is specifically configured to, for each horizontal rectangle in the video frame, determine the overlapping area between the horizontal rectangle and each rotating rectangle, determine the ratio of each overlapping area to the area of ​​the horizontal rectangle; determine rotating rectangles whose corresponding ratio is greater than a threshold as rotating rectangles that contain the same target as the horizontal rectangle, and determine the rotating rectangles as successfully matched target rotating rectangles.

[0048] In one possible implementation, the processing module is further configured to, for each target rotating rectangle, determine the distance deviation between the first center point and the second center point in the target video frame based on the position information of the first center point of the target rotating rectangle and the second center point of the horizontal rectangle that successfully matches the target rotating rectangle in the target video frame, and input the target video frame and its adjacent video frames into the recognition model to obtain the predicted direction corresponding to the target rotating rectangle output by the recognition model, and determine the direction deviation between the predicted direction and the movement direction determined based on the long side of the target rotating rectangle; determine the matching degree between the target rotating rectangle and the matching horizontal rectangle based on the direction deviation and the distance deviation; if the matching degree is greater than a preset matching threshold, then execute the subsequent step of tracking the target corresponding to the successfully matched target rotating rectangle based on the determined movement direction.

[0049] In one possible implementation, the processing module is further configured to determine the actual location information of each key point based on the location information of each key point in the corresponding video frame and the GPS information of the drone.

[0050] Thirdly, this application provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0051] Memory, used to store computer programs;

[0052] A processor, used to execute a program stored in memory, implements the method described.

[0053] In this embodiment, high-definition video frames of the intersection are obtained through aerial photography using a drone, covering the complete channelization structure and dynamic traffic flow. Furthermore, this application first identifies the zebra crossing and then detects the stop line based on the zebra crossing location by cropping the area, avoiding interference between the two types of key points and improving the accuracy of key point identification. Attached Figure Description

[0054] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0055] Figure 1 A schematic diagram illustrating the process of a method for determining intersection information based on an unmanned aerial vehicle (UAV) provided in an embodiment of this application;

[0056] Figure 2 A schematic diagram of a video frame provided in an embodiment of this application;

[0057] Figure 3 A schematic diagram illustrating a defined key point of a zebra crossing, provided as an embodiment of this application;

[0058] Figure 4 A schematic diagram illustrating a defined key point of a zebra crossing, provided as an embodiment of this application;

[0059] Figure 5 A schematic diagram of a morphologically processed image provided in an embodiment of this application;

[0060] Figure 6 A schematic diagram illustrating a cropped image provided in an embodiment of this application;

[0061] Figure 7 This is a schematic diagram of an image obtained through morphological processing after slicing, as provided in an embodiment of this application.

[0062] Figure 8 A schematic diagram of a video frame including a rotating rectangle provided in an embodiment of this application;

[0063] Figure 9 A schematic diagram of a tracking method provided for an embodiment of this application;

[0064] Figure 10 A detailed schematic diagram illustrating the process of determining a trajectory map, provided in an embodiment of this application;

[0065] Figure 11 This application provides a schematic diagram illustrating the process of determining intersection information in an embodiment of the present application.

[0066] Figure 12 A detailed schematic diagram illustrating the process of determining intersection information provided in this application embodiment;

[0067] Figure 13 A schematic diagram of a UAV-based intersection information determination device provided in an embodiment of this application;

[0068] Figure 14 This is a schematic diagram of an electronic device structure provided in an embodiment of this application. Detailed Implementation

[0069] To make the objectives and implementation methods of this application clearer, the exemplary implementation methods of this application will be clearly and completely described below with reference to the accompanying drawings of the exemplary embodiments of this application. Obviously, the exemplary embodiments described are only some embodiments of this application, and not all embodiments.

[0070] It should be noted that the brief descriptions of terms in this application are only for the convenience of understanding the embodiments described below, and are not intended to limit the embodiments of this application. Unless otherwise stated, these terms should be understood in their ordinary and common meaning.

[0071] The terms "first," "second," "third," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar or related objects or entities, and do not necessarily imply a specific order or sequence, unless otherwise specified. It should be understood that such terms are interchangeable where appropriate.

[0072] The terms “comprising” and “having”, and any variations thereof, are intended to cover but not exclude inclusion, for example, a product or device that includes a range of components is not necessarily limited to all of the components that are clearly listed, but may include other components that are not clearly listed or that are inherent to such product or device.

[0073] The term "module" refers to any known or subsequently developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code that is capable of performing the functions associated with that element.

[0074] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

[0075] For ease of explanation, the above description has been provided in conjunction with specific embodiments. However, the above exemplary discussion is not intended to be exhaustive or to limit the embodiments to the specific forms disclosed above. Various modifications and variations can be obtained based on the above teachings. The selection and description of the above embodiments are for the purpose of better explaining the principles and practical applications, thereby enabling those skilled in the art to better utilize the described embodiments and various different variations of embodiments suitable for specific use considerations.

[0076] Figure 1 A schematic diagram illustrating a method for determining intersection information based on unmanned aerial vehicles (UAVs) provided in this application embodiment, the process including the following steps:

[0077] S101: Acquire the video stream captured by the drone.

[0078] The UAV-based intersection information determination method provided in this application is applied to an electronic device, which can be a smart device such as a PC or server.

[0079] To accurately and effectively determine information, electronic devices can acquire video streams collected by drones. Drones offer advantages such as flexible deployment, comprehensive viewing angles, and wide coverage. Compared to traditional ground-based camera deployment methods, this approach is not limited by fixed viewing angles, thus significantly reducing the manual workload of intersection documentation and providing efficient data support for signal optimization and intersection problem diagnosis and analysis.

[0080] In this embodiment, a drone is used to take aerial photos of the target intersection, acquiring a panoramic video stream with sufficient detail, ensuring the video duration is greater than one signal cycle, and the drone's camera is parallel to the ground. This video stream can be a high-altitude video stream, for example, a video stream from a height of 100 meters.

[0081] S102: Extract video frames from the video stream, input the video frames into a pre-trained zebra crossing recognition model, and obtain the first position information of the zebra crossing key points in the video frames output by the zebra crossing recognition model.

[0082] After acquiring the video stream, the electronic device can extract video frames from the acquired video stream. In one example, the electronic device can perform a video frame extraction operation on the video stream to extract individual video frames from the continuous video stream as the basic unit for subsequent analysis.

[0083] In this embodiment of the application, it can be ensured that the acquired video stream contains sufficient traffic flow and road structure information. In order to improve the accuracy of intersection information determination, the electronic device can preprocess the video stream after acquiring it, including video format conversion, image stabilization processing and resolution enhancement, to ensure that the image data is clear.

[0084] Figure 2 This is a schematic diagram of a video frame provided in an embodiment of this application.

[0085] Depend on Figure 2 It can be seen that the video frame contains sufficient traffic flow and road structure information, and that the video frame is a video frame from an intersection. Among them, Figure 2 The simplified zebra crossings, stop lines, and lane lines marked in the Chinese text are the intersection information described in the embodiments of this application.

[0086] In order to accurately and effectively determine intersection information, the electronic device locally stores a pre-trained zebra crossing recognition model. This zebra crossing recognition model can be a YOLOv11 key point detection model. The electronic device can input the video frame into the pre-trained zebra crossing recognition model to obtain the first position information of the zebra crossing key points in the video frame output by the zebra crossing recognition model.

[0087] Figure 3 This is a schematic diagram of a defined zebra crossing key point provided in an embodiment of this application.

[0088] Figure 3 This is a schematic diagram of the zebra crossing key points determined based on the first target key point, and... Figure 3 This refers to a portion of a video frame, not the entire video frame. Figure 3 The circle in the image represents the key point of the zebra crossing. Figure 3 The straight line in the diagram represents the boundary of the zebra crossing area, determined by connecting key points of the zebra crossing. Figure 3 It can be seen that multiple zebra crossing key points can be determined, and the positions of the determined multiple zebra crossing key points almost coincide with the actual key points of the zebra crossing. In other words, the method provided in the embodiments of this application can accurately and effectively determine the key points of the zebra crossing.

[0089] S103: Based on the first position information, the video frame is cropped to obtain a cropped image containing zebra crossings. The cropped image is then input into a pre-trained stop line recognition model to obtain the second position information of the stop line key points in the cropped image output by the stop line recognition model.

[0090] Since the acquired video frames contain a lot of information, directly determining the stop line based on the video frames may mistakenly identify other information as the stop line. Therefore, electronic devices can crop the video frames and determine the key points of the stop line based on the cropped video frames.

[0091] Specifically, after obtaining the first location information of the zebra crossing key points, the electronic device can crop the video frame based on the first location information of the zebra crossing key points to obtain a cropped image containing the zebra crossing. In one example, the electronic device can expand the minimum bounding rectangle of the zebra crossing key points in each direction by a certain proportion to obtain the cropped image.

[0092] In order to accurately and effectively determine intersection information, the electronic device locally stores a pre-trained stop line recognition model. This stop line recognition model can be a YOLOv11 key point detection model. The electronic device can input the video frame into the pre-trained stop line recognition model to obtain the second position information of the stop line key points in the video frame output by the stop line recognition model.

[0093] Figure 4 This is a schematic diagram of a defined zebra crossing key point provided in an embodiment of this application.

[0094] Figure 4 This is a schematic diagram of the stop line key points determined based on the second target key points, and... Figure 4 This refers to a portion of a video frame, not the entire video frame. Figure 3 The circle in the diagram represents the key point of the stop line. Figure 4 The straight line in the diagram is the stop line determined by connecting the key points of the stop line. Figure 4 It can be seen that multiple stop line key points can be determined, and the positions of the determined multiple stop line key points almost coincide with the actual key points of the stop line. In other words, the method provided by the embodiments of this application can accurately and effectively determine the stop line key points.

[0095] This application utilizes deep learning target detection and target tracking technologies to process drone videos of intersections, thereby identifying the channelization features of the current intersection and quickly obtaining high-resolution intersection channelization information and signal timing logic.

[0096] The development of the low-altitude economy and the application of drones and video artificial intelligence (AI) technologies have led to the application of automated recognition and channeling technologies based on drones or other high-altitude video. However, existing technologies still have significant problems. To improve coverage, drones typically need to fly at higher altitudes (e.g., above 30 meters), but at these altitudes, image resolution is limited, making it difficult to clearly identify details such as lane lines, text markings, and directional arrows. While low-altitude flight can improve image clarity, it is susceptible to factors such as poor stability, high occlusion rates, and compliance restrictions, making it difficult to widely promote in real-world urban environments. Furthermore, there are significant differences in the style of road markings across different cities or intersections, such as inconsistent directional arrow shapes, fonts, and edge line types. In addition, some markings are blurred due to aging, dirt, uneven lighting, or occlusion (such as parked vehicles, tree shadows, or building shadows), resulting in insufficient robustness of existing recognition algorithms, high false recognition rates, and poor generalization ability. During peak hours, dense traffic flow can severely obscure road markings, and the dynamic movement of non-motorized vehicles and pedestrians can cause short-term occlusion, further affecting recognition stability. At the same time, the datasets currently used to train recognition models generally suffer from problems such as insufficient quantity, inconsistent labeling, and limited coverage. The data distribution varies greatly in different cities and at different collection times, resulting in poor transferability and insufficient generalization performance of the models when they are actually deployed.

[0097] To address the aforementioned issues, this application proposes a novel solution combining drone aerial photography and AI recognition algorithms. By utilizing the movement trajectory of lanes and the partial channelization position relationship, the solution automatically extracts the channelization structure parameters of intersections. This solution can be used to quickly complete tasks such as urban intersection modeling, signal timing analysis, simulation modeling data input, and traffic problem diagnosis and analysis, greatly improving the level of intelligent traffic management and investigation efficiency, and filling the gap in existing technologies in this specific field.

[0098] In this embodiment, high-definition video frames of the intersection are obtained through aerial photography using a drone, covering the complete channelization structure and dynamic traffic flow. Furthermore, this application first identifies the zebra crossing and then detects the stop line based on the zebra crossing location by cropping the area, avoiding interference between the two types of key points and improving the accuracy of key point identification.

[0099] To accurately and effectively determine the location information of key points of the stop line, based on the above embodiments, in this embodiment, the step of cropping the video frame based on the first location information to obtain a cropped image containing the zebra crossing includes:

[0100] Based on the first target location information of the zebra crossing key points, determine the rectangle containing the zebra crossing;

[0101] The video frame is binarized, and the binarized video frame is subjected to a preset number of erosion and dilation morphological processing. Based on the morphologically processed image, the region with the largest connected component is determined. The first boundary of the region is determined to be closest to the center point of the rectangle, and the second boundary of the line segment of the rectangle is determined to be farthest from the first boundary. Based on the first boundary and the second boundary, a stop line detection region is determined. Based on the stop line detection region, the video frame is cropped.

[0102] When determining intersection information, accurate detection of zebra crossings and stop lines is crucial. Since stop line recognition models perform poorly when performing key point detection on the entire stop line map, this embodiment first uses the zebra crossing location for preliminary stop line localization to determine the location of subsequent stop line key point detection. This fully utilizes the fixed spatial relationship between the two in the road scene to achieve efficient joint detection.

[0103] Specifically, the electronic device can determine a bounding box containing the zebra crossing based on the first location information of the zebra crossing key points. This bounding box can be the minimum bounding box. To improve the accuracy of subsequent processing, the electronic device can perform adaptive binarization on each video frame, eliminating noise interference and enhancing linear features through multiple rounds of erosion-dilation morphological operations. In the morphologically optimized image, the region with the largest connected component area is extracted. In one example, the electronic device can perform N_e erosion operations and N_d dilation operations with a kernel of a preset size, such as (s*s).

[0104] In one example, a video frame may contain multiple stop lines. Based on this, the electronic device can determine the subsequent stop line detection region based on each region whose area of ​​the acquired connected components is greater than a threshold.

[0105] Figure 5 This is a schematic diagram of a morphologically processed image provided in an embodiment of this application.

[0106] Figure 5 This is the lane binarized image after morphological processing.

[0107] Through spatial relationship analysis, the boundary closest to the center of the rectangular bounding box within the connected component boundary is determined as the first boundary. Simultaneously, the corresponding boundary furthest from this first boundary is determined within the original rectangular bounding box edge, and this is defined as the second boundary. The electronic device can then determine the stop line detection region based on the first and second boundaries. In one example, the first boundary can be extended a predetermined number of pixels away from the zebra crossing to form the stop line detection region. The electronic device can then crop video frames based on the stop line detection region to obtain a cropped image containing the zebra crossing.

[0108] Figure 6 This is a schematic diagram of a cropped image provided in an embodiment of this application.

[0109] Depend on Figure 6 It can be seen that the cropped image contains stop lines and zebra crossings.

[0110] The above-mentioned technical solution has the following advantages or beneficial effects: In this embodiment, the inherent spatial constraints of zebra crossings and stop lines are utilized to narrow the detection range; the influence of illumination changes and partial occlusion is effectively overcome through connected component analysis; and the established dynamic boundary detection mechanism can adapt to different road design specifications. This, in turn, can improve the accuracy of lane line key point detection.

[0111] To accurately and effectively determine lane line information, based on the above embodiments, the method in this application embodiment further includes:

[0112] Based on the stop line corresponding to the determined stop line key point, the image is sliced ​​at a preset distance away from the zebra crossing. The sliced ​​image is then subjected to erosion and dilation morphological processing. Based on the number of connected components contained in the processed image, the number of lanes is determined, and the center point of each connected component is determined as the center point of the lane line.

[0113] Accurate identification of lane lines and their quantity is crucial for determining intersection information. Traditional lane detection methods typically process the entire image directly, making them susceptible to interference from complex scenes (such as shadows, vehicle occlusion, or road wear). This application's embodiments utilize the spatial correlation between stop lines and lane lines, achieving more robust lane detection by extending the analysis outward from the stop line region.

[0114] Specifically, the electronic device can extend a predetermined distance away from the zebra crossing, for example, 5-10 meters, based on the detected stop line key points. The specific value can be converted into pixel distance through camera calibration to capture the region of interest. This step ensures that subsequent processing focuses on the key area of ​​lane line distribution and eliminates irrelevant background interference.

[0115] The captured image is converted to grayscale and adaptively binarized. Then, erosion-dilation morphological operations are used to eliminate noise and enhance lane line continuity. Erosion removes small, isolated noise points, while dilation helps connect broken lane lines.

[0116] In the morphologically processed binary image, the number of horizontally distributed white regions is counted through connected component analysis, with each connected component representing a potential lane line. Further filtering of eligible connected components yields the final count of lanes. The center point of each connected component is determined as the center position of the corresponding lane line.

[0117] Figure 7 This is a schematic diagram of an image obtained through morphological processing after slicing, as provided in an embodiment of this application.

[0118] The above-mentioned technical solution has the following advantages or beneficial effects: It reduces computational load and avoids perspective distortion problems in long-distance lane line detection by tiling the image at a preset distance away from the zebra crossing. Morphological processing enhances robustness, effectively handling complex scenarios such as uneven lighting, partial occlusion, or road surface wear, ensuring lane line continuity. By preset extension distances and connected component filtering thresholds, it can adapt to different road designs.

[0119] To accurately and effectively determine intersection information, the extraction of video frames from the video stream includes:

[0120] Extract multiple video frames from the video stream;

[0121] The method further includes:

[0122] Based on the K-means clustering algorithm, the first location information of zebra crossing key points in multiple video frames is clustered to obtain the first target location information of zebra crossing key points.

[0123] Based on the K-means clustering algorithm, the second location information of the stop line key points in multiple determined video frames is clustered to obtain the second target location information of the stop line key points.

[0124] In real-world scenarios, the zebra crossings and stop lines detected in a single frame often suffer from noise and positional jitter. To address this issue, this application employs the K-means clustering algorithm to cluster the positional information of key points across multiple video frames.

[0125] Specifically, the electronic device can extract multiple video frames from a video stream. In one example, video frames can be extracted at preset time intervals. Based on a pre-trained zebra crossing recognition model, the first location information of the zebra crossing key points in each video frame is determined. The first location information of all detected zebra crossing key points is used as the input dataset for a K-means clustering algorithm. The K-means algorithm iteratively optimizes the positions of the cluster centers based on the number of cluster centers, and the final output cluster centers are the first target location information of the zebra crossing key points. In another example, the electronic device can input the location information of each zebra crossing key point into a large language model, which then outputs the number of cluster centers.

[0126] After obtaining the first target location information, the electronic device can crop video frames based on this information to obtain each cropped image. Then, based on a pre-trained stop-line recognition model, it can obtain the second location information of the stop-line keypoints in each cropped image. The processing of the stop-line keypoints uses the same technical approach. The electronic device can use the second location information of all detected stop-line keypoints as the input dataset for the K-means clustering algorithm. The K-means algorithm iteratively optimizes the center point positions based on the number of cluster centers, and the final output cluster centers are the second target location information of the stop-line keypoints. In one example, the electronic device can input the location information of each stop-line keypoint into a large language model, which will then output the number of cluster centers.

[0127] The above technical solution has the following advantages or beneficial effects: the clustering algorithm is naturally robust to outliers and can automatically filter out false detections that deviate significantly from the main group; moreover, by adjusting the number of clusters, it can flexibly adapt to different road scenarios, such as double zebra crossings in school areas or diagonal stop lines at special intersections.

[0128] To accurately and effectively determine intersection information, based on the above embodiments, the method in this application embodiment further includes:

[0129] Target video frames are extracted from the video stream. Based on a rotating bounding box detection algorithm, rotating rectangular boxes that fit the target contour are extracted from the target video frames. A rectangular box detection method is then used to obtain the smallest bounding horizontal rectangular box that completely contains the target in the target video frame. The rotating rectangular boxes and the horizontal rectangular boxes are matched to obtain each successfully matched target rotating rectangular box. The targets include vehicles and pedestrians.

[0130] The movement direction is determined based on the long side of the target rotation bounding box in each target video frame, and the target corresponding to the successfully matched target rotation bounding box is tracked based on the determined movement direction.

[0131] In video analytics of urban traffic scenarios, accurate detection and tracking of moving targets such as vehicles and pedestrians are crucial. Traditional detection methods typically use only horizontal bounding boxes, but these boxes contain large background areas when the target rotates or tilts, severely impacting motion direction determination and tracking accuracy. Therefore, this application proposes a collaborative detection strategy that integrates rotated bounding boxes and horizontal bounding boxes. Through geometric matching and complementary verification of these two methods, more accurate target representation and motion analysis are achieved.

[0132] Specifically, the electronic device first extracts key target video frames from the video stream. For each target video frame, a target detection operation is performed, combining two detection strategies: one uses a rotated bounding box detection method, suitable for situations with large angle deviations or significant changes in viewing angle; the other uses a traditional rectangular box detection method, suitable for scenarios with stable image viewing angles and clear target outlines. The parallel application of these two methods enables accurate identification of target information such as vehicles and pedestrians within intersections under different shooting conditions, providing reliable data support for subsequent dynamic analysis. Specifically, the electronic device uses the rotated bounding box detection algorithm to extract rotated rectangular boxes that fit the target outline from the target video frame; and uses a rectangular box detection method to obtain the minimum bounding horizontal rectangle that completely contains the target in the target video frame. In one example, the electronic device can input the target video frame into the rotation recognition model and the bounding box recognition model respectively. The rotation recognition model outputs video frames marked with rotated rectangular boxes that fit the target outline, and the bounding box recognition model outputs video frames marked with the minimum bounding horizontal rectangle that completely contains the target. The video frames output by the rotation recognition model and the external recognition model can also contain the identifier of each target. The rotation recognition model can be used for bounding box target detection based on PaddlePaddle You Only Look Once Enhanced (PP-YOLOE), and the external recognition model can be used for bounding box target detection based on YOLOv11.

[0133] Electronic devices can match targets by calculating the spatial overlap rate and center point distance between two types of bounding boxes. Specifically, if the center deviation between the rotated box and the horizontal box is less than a threshold and the overlap rate exceeds a threshold, they are considered the same target, and the geometric properties of the rotated box are preserved. For targets that fail to match, a prediction box completion mechanism based on motion history can be enabled.

[0134] After target detection, the electronic device determines the movement direction based on the long side of each target's rotating bounding box, and tracks the target corresponding to the successfully matched target's rotating bounding box based on the movement direction.

[0135] Specifically, electronic devices can use target tracking technology to correlate and track targets detected in consecutive frames, thereby obtaining the spatiotemporal trajectory of the target within the intersection area. By extracting statistical features from these trajectory data, information such as the target's motion patterns, direction of travel, and dwell time can be obtained.

[0136] In this embodiment, the electronic device can further input the target video frame into a pre-trained recognition model to obtain the velocity of each target output by the recognition model, determine the corresponding movement angle based on the movement direction, and predict the position of each target based on the velocity, the movement angle, and the position information of each target in the target video frame. Specifically, the electronic device can predict the position of a target using the following formula:

[0137] x t+1 =x t +v*cos(θ)

[0138] y t+1 =y t +v*sin(θ)

[0139] Where, x t+1 x is the predicted x-coordinate of the center point of the target in the next video frame. t Let θ be the predicted ordinate of the target's center point in the next video frame, θ be the movement angle, v be the determined movement speed, and x be the distance from the center point. t Let y be the x-coordinate of the target's center point in the target video frame. t This represents the ordinate of the target's center point in the target video frame.

[0140] In one example, the electronic device can populate all tracked targets for that segment of the drone. First, it iterates through all tracked targets, then populates them sequentially based on all frames of each tracked target. The specific method is as follows: Let the target be obj. i The center target position in the t-th frame is (x t ,y t The center target position in frame t+1 is (x t+1 ,y t+1 If ), then update obj at frame t. i The filling matrix r i The length is [(x t ,y t ),(x t+1 ,y t+1 The matrix consists of line segments of width N. The entire target matrix is ​​output when all frames of all targets have been traversed.

[0141] Figure 8 This is a schematic diagram of a video frame containing a rotating rectangle, provided as an embodiment of this application.

[0142] Depend on Figure 8 It can be seen that a rotating rectangular frame that fits the target contour can be determined.

[0143] Figure 9 This is a schematic diagram of a tracking method provided in an embodiment of this application.

[0144] Figure 9 The trajectory diagram is obtained using the embodiments of this application. Figure 9 It contains multiple filling matrices.

[0145] Figure 10 This is a schematic diagram illustrating a detailed process for determining a trajectory map, as provided in an embodiment of this application.

[0146] Depend on Figure 10 It can be seen that the video stream acquired by the drone can be first frame-segmented, and the rotated bounding box is determined based on PP-YOLOE rectangular bounding box target detection, and the minimum bounding horizontal bounding box is determined based on YOLOv11 rectangular bounding box target detection. The rectangular bounding box and the rotated bounding box are fused, and the target tracking algorithm based on Byte-wise Association for Multi-Object Tracking (Bytetrack) based on the rotation box direction prediction is used for tracking, and a set of target tracking results for multiple frames is obtained. A blank image of size (W,H) is generated, all targets are traversed, the filling matrix of the target's center point is calculated, the filling matrix is ​​filled into the image, the traversal ends, and the trajectory map is generated.

[0147] The above-mentioned technical solution has the following advantages or beneficial effects: In the embodiments of this application, the introduction of the rotating frame reduces the target coverage area, significantly reduces background interference, and reduces direction estimation error. The matching verification mechanism reduces the false negative rate, especially improving the robustness of detection for tilted vehicles (such as trucks turning).

[0148] The continuous development of ITS has placed higher demands on urban road traffic management for high-precision and high-efficiency information collection methods. In particular, in applications such as traffic signal control optimization, intersection reconstruction design, traffic simulation modeling, traffic problem diagnosis, and channelization problem analysis, the demand for collecting channelization information (such as the number of lanes, lane attributes, and turning signs) and signal timing information (such as cycle, phase sequence, and green light duration) at intersections is increasing.

[0149] Traditional information collection methods mainly rely on the following approaches: Manual on-site surveys and video capture: This involves manually measuring, photographing, or recording intersection information using fixed-point cameras. This method is labor-intensive, time-consuming, and the data is highly subjective, posing safety hazards, especially during peak hours or at complex intersections. Deployment of traffic survey equipment (such as geomagnetic sensors, coils, radar, and fixed-point cameras): This is suitable for fixed monitoring points to acquire traffic operation data, but it cannot quickly extract comprehensive information on signal schemes and channelization structures. Furthermore, the equipment deployment cost is high, and its adaptability is poor. Remote sensing and map data (such as high-precision maps and satellite imagery) are used for macro-level data collection, but their resolution and timeliness are low, making it difficult to meet the data requirements at the detailed intersection level. In this application embodiment, the intersection channelization feature extraction method based on trajectory maps and morphological processing uses trajectory maps and target detection and tracking results to identify intersection channelization features. This includes identifying channelization information such as zebra crossings, stop lines, number of lanes, lane midpoints, and lane north-south angles in various directions.

[0150] To accurately and effectively perform target matching, based on the above embodiments, in this embodiment, the step of matching based on the rotated rectangle and the horizontal rectangle to obtain each successfully matched target rotated rectangle includes:

[0151] For each horizontal rectangle in the video frame, determine the overlapping area between the horizontal rectangle and each rotating rectangle, and determine the ratio of each overlapping area to the area of ​​the horizontal rectangle; the rotating rectangles whose corresponding ratio is greater than a threshold are determined as rotating rectangles that contain the same target as the horizontal rectangle, and the rotating rectangles are determined as successfully matched target rotating rectangles.

[0152] In object detection tasks in complex scenes, a single type of bounding box often struggles to balance accuracy and robustness. While horizontal bounding boxes are computationally efficient, they offer poor coverage of rotated or tilted targets; rotated bounding boxes can closely fit the target contour, but are prone to false matching in dense scenes. This application proposes a two-stage matching strategy based on geometric feature fusion, which significantly improves the accuracy and stability of object detection through the collaborative verification of horizontal and rotated bounding boxes.

[0153] Electronic devices can perform intersection over union (IOU) fusion on horizontal and rotated bounding boxes. Specifically, for each horizontal bounding box in a video frame, the electronic device can determine the overlapping area between the horizontal bounding box and each rotated bounding box, and determine the ratio of each overlapping area to the area of ​​the horizontal bounding box. Rotated bounding boxes with a ratio greater than a threshold are identified as rotating bounding boxes that contain the same target as the horizontal bounding box, and these rotating bounding boxes are identified as successfully matched target rotating bounding boxes.

[0154] The above-mentioned technical solution has the following advantages or beneficial effects: by adopting the method provided in the embodiments of this application, the horizontal rectangular box is calculated efficiently, the rotated rectangular box can closely fit the target contour, and can be matched accurately and effectively, thereby improving the accuracy of target detection.

[0155] To accurately and effectively track targets, based on the above embodiments, in this embodiment, after matching the rotating rectangle and the horizontal rectangle to obtain each successfully matched rotating rectangle, and before tracking the target corresponding to the successfully matched rotating rectangle based on the determined movement direction, the method further includes:

[0156] For each target rotating rectangle, based on the position information of the first center point of the target rotating rectangle and the second center point of the horizontal rectangle that successfully matches the target rotating rectangle in the target video frame, the distance deviation between the first center point and the second center point in the target video frame is determined. The target video frame and its adjacent video frames are then input into the recognition model to obtain the predicted direction corresponding to the target rotating rectangle output by the recognition model. The direction deviation between the predicted direction and the movement direction determined based on the long side of the target rotating rectangle is determined. Based on the direction deviation and the distance deviation, the matching degree between the target rotating rectangle and the matching horizontal rectangle is determined. If the matching degree is greater than a preset matching threshold, the subsequent step of tracking the target corresponding to the successfully matched target rotating rectangle based on the determined movement direction is executed.

[0157] Traditional single-index matching methods face several challenges in complex scenarios: appearance changes caused by target rotation, detection fluctuations due to brief occlusion, and identity confusion when multiple targets intersect. This application proposes a dual verification mechanism that integrates spatial location and motion direction, improving the reliability of target association through geometric consistency checks.

[0158] Specifically, for each target rotating rectangle, the electronic device can determine the distance deviation between the first center point and the second center point in the target video frame based on the position information of the first center point of the target rotating rectangle and the second center point of the horizontal rectangle that successfully matches the target rotating rectangle in the target video frame. The electronic device can then input the target video frame and its adjacent video frames into the recognition model to obtain the predicted direction corresponding to the target rotating rectangle output by the recognition model. The electronic device can then determine the directional deviation between the predicted direction and the movement direction determined based on the long side of the target rotating rectangle. Based on the directional deviation and the distance deviation, the electronic device can determine the matching degree between the target rotating rectangle and the matched horizontal rectangle.

[0159] Specifically, the matching degree can be determined using the following formula:

[0160] matchscore = Dis cnter *exp(-λ*|θ track -θ det |)

[0161] Where matchscore represents the determined matching degree, Dis... cnter The distance deviation is defined, λ is a hyperparameter for the penalty of the preset direction control angle, and θ is the distance deviation. track To predict the direction, θ det Indicates the direction of movement.

[0162] For each target rotating rectangle, if the matching degree is greater than the preset matching threshold, then proceed to the next step of tracking the target corresponding to the successfully matched target rotating rectangle based on the determined movement direction.

[0163] The above technical solution has the following advantages or beneficial effects: by using the method provided in the embodiments of this application, and further calculating the matching degree based on the angle deviation and position deviation, the accuracy of matching can be improved, thereby improving the effect of target tracking.

[0164] To further improve the accuracy and efficiency of road traffic management, based on the above embodiments, the method in this application embodiment further includes:

[0165] Based on the location information of each key point in the corresponding video frame and the GPS information of the drone, the actual location information of each key point is determined.

[0166] The GPS information of a drone can be collected by sensors provided by the drone's flight control system to represent the drone's attitude and position information. For example, but not limited to, the drone's GPS information includes the drone's precise GPS coordinates, flight altitude, yaw, pitch, roll, and other attitude parameters when capturing the corresponding video frame.

[0167] In one possible implementation, the location information of each key point is first transformed from the image coordinate system to the UAV coordinate system, and then transformed from the UAV coordinate system to the geographic coordinate system, thereby obtaining the actual location information of each key point. The image coordinate system in which the location information of each key point is located includes internal parameters (such as focal length, distortion coefficients, etc.) and external parameters (such as position and orientation relative to the UAV coordinate system).

[0168] In this implementation, the actual distance of each keypoint to a unit pixel in the geographic coordinate system can be obtained. For example, based on the length of the long side of all rotated rectangles, the mode N of the long side of all targets can be calculated.mode Combined with formula S pixel =4.8 / N mode The actual distance S per unit pixel can then be calculated. pixel The unit is m / pixel.

[0169] When converting between the image coordinate system and the UAV coordinate system, the image center point can be selected as the UAV's GPS location, and the image y-direction can be the UAV's true north deviation angle. Therefore, the GPS calculation method for any pixel in the image coordinate system is as follows:

[0170] Δu=uu c uv = vv c

[0171] x m =Δu·S pixel y m =Δv·S pixel

[0172] d north =y m ·cosθ-x m ·sinθ

[0173] d east =y m ·sinθ+x m ·cosθ

[0174]

[0175] lat pixel =lat c +ulat

[0176] lon pixel =lon c +Δlon

[0177] Where (u,v) are the pixel coordinates of the target in the video frame, (u c ,v c ) represents the pixel coordinates of the center point of the video frame, Δu, Δv represent the pixel offsets of the target relative to the center point, and S represents the pixel coordinates of the center point. pixel The actual physical size of a single pixel (unit: meters / pixel) is determined by the camera's focal length and resolution. θ is the angle between the camera's orientation and true north (the cardinal directions described here refer to the actual cardinal directions in the scene). m ,y m (d) represents the coordinates in the camera coordinate system. north ,d east Let ) be the coordinates of the target in the local coordinate system of the northeast sky, l atc The latitude of the reference point (such as the image center), lon For the longitude of the reference point, A lat S is the latitude conversion factor. lon Here, Δlat is the latitude offset of the target relative to the reference point, Δlon is the longitude offset of the target relative to the reference point, R is the Earth's radius, and (latpixel, lonpixel) are the latitude and longitude of the target.

[0178] The above-mentioned technical solution has the following advantages or beneficial effects: In the embodiments of this application, mapping each key point from the location information of the video frame to the actual location information of each key point helps to locate each key point on the actual urban road. Whether it is an urban road during peak hours or a complex intersection, it can ensure the accuracy and efficiency of road traffic management in various application scenarios.

[0179] In this embodiment, image coordinate system matching with GPS coordinate system is performed to map the image position of each detected target in the video frame extraction to the real geographic coordinate space. This step is typically achieved using the UAV's own positioning system or ground control point data to ensure accurate spatial positioning of the extracted data. After coordinate matching is completed, the electronic device integrates multi-target trajectory data to generate a comprehensive traffic trajectory map of the entire intersection. Based on this trajectory map, the morphological features of the intersection are further processed, such as using spatial clustering or streamline modeling methods to identify different lane distributions and vehicle driving patterns, ultimately achieving automated extraction of intersection channelization features. This includes quantifying and outputting information such as the number of lanes, lane width, directional signs, and turning distribution, providing data support for traffic planning, road design, and traffic management decisions.

[0180] Figure 11 This is a schematic diagram illustrating the process of determining intersection information as provided in an embodiment of this application.

[0181] Figure 11 The example of first tracking the target and then obtaining information about key points will be used to illustrate this approach.

[0182] Depend on Figure 11 As can be seen, the drone first collects video streams from the intersection. Electronic equipment processes the drone's video information, extracts video frames, and performs target detection and tracking based on video rotation box target detection and rectangular box target detection to generate a trajectory map. Morphological processing is then performed on the video frames to extract road channelization information, i.e., information related to key points.

[0183] Figure 12 This is a detailed schematic diagram illustrating the process of determining intersection information as provided in an embodiment of this application.

[0184] S1201: Perform zebra crossing key point detection on all frames of the full image based on YOLOv11.

[0185] S1202: Cluster key points from multiple frames based on K-means clustering to form the final zebra crossing key points.

[0186] S1203: Based on morphological processing of zebra crossing key points and road trajectory maps, stop line area extraction is achieved.

[0187] S1204: Perform stop line key point detection on all frame stop line regions based on YOLOv11.

[0188] S1205: Multi-frame keypoint clustering based on K-meams clustering to form the final stop line keypoints.

[0189] S1206: Calculate the actual distance per unit pixel based on statistics of all rotated box sizes.

[0190] S1207: Convert image coordinates to GPS coordinates based on UAV GPS center point data.

[0191] S1208: Calculate the direction of the zebra crossing and the direction of the stop line based on GPS data and the north deflection angle information of the UAV.

[0192] S1209: Traverse each stop line direction and identify the number of lanes, lane midpoint, and lane north deflection angle.

[0193] S1210: Based on the stop line, perform image tiling and morphological processing to realize the number of lanes and lane centerline recognition.

[0194] S1211: Convert the image coordinates of key points to GPS coordinates.

[0195] S1212: Lane steering decision based on lane target tracking results.

[0196] S1213: Channelization information output.

[0197] The channelization information refers to the intersection information described in the embodiments of this application.

[0198] Figure 13 The schematic diagram of the UAV-based intersection information determination structure provided in this application shows that the device includes:

[0199] The acquisition module 1301 is used to acquire the video stream captured by the drone;

[0200] The processing module 1302 is used to extract video frames from the video stream, input the video frames into a pre-trained zebra crossing recognition model, obtain the first position information of the zebra crossing key points in the video frame output by the zebra crossing recognition model; crop the video frame based on the first position information to obtain a cropped image containing the zebra crossing, input the cropped image into a pre-trained stop line recognition model, and obtain the second position information of the stop line key points in the cropped image output by the stop line recognition model.

[0201] In one possible implementation, the processing module 1302 is specifically configured to: determine a rectangular frame containing the zebra crossing based on the first target location information of the zebra crossing key points; perform binarization processing on the video frame; perform morphological processing of erosion and dilation on the binarized video frame a preset number of times; determine the region with the largest connected component based on the morphologically processed image; determine the first boundary of the boundary of the region that is closest to the center point of the rectangular frame; determine the second boundary of the line segment of the rectangular frame that is farthest from the first boundary; determine a stop line detection region based on the first boundary and the second boundary; and crop the video frame based on the stop line detection region.

[0202] In one possible implementation, the processing module 1302 is further configured to slice the image at a preset distance away from the zebra crossing based on the stop line corresponding to the determined stop line key point, perform erosion and dilation morphological processing on the sliced ​​image, determine the number of lanes based on the number of connected components contained in the processed image, and determine the center point of each connected component as the center point of the lane line.

[0203] In one possible implementation, the processing module 1302 is further configured to extract multiple video frames from the video stream;

[0204] The processing module 1302 is specifically used to cluster the first position information of zebra crossing key points in multiple video frames based on the K-means clustering algorithm to obtain the first target position information of the zebra crossing key points; and to cluster the second position information of stop line key points in multiple video frames based on the K-means clustering algorithm to obtain the second target position information of the stop line key points.

[0205] In one possible implementation, the processing module 1302 is further configured to extract target video frames from the video stream, extract rotating rectangular boxes that fit the target contour in the target video frames based on a rotating bounding box detection algorithm, and obtain the smallest bounding horizontal rectangular box that completely contains the target in the target video frames using a rectangular box detection method; match the rotating rectangular boxes and the horizontal rectangular boxes to obtain each successfully matched target rotating rectangular box; wherein the target includes vehicles and pedestrians; determine the movement direction based on the long side of each target rotating bounding box in the target video frames, and track the target corresponding to the successfully matched target rotating rectangular box based on the determined movement direction.

[0206] In one possible implementation, the processing module 1302 is specifically configured to, for each horizontal rectangle in the video frame, determine the overlapping area of ​​the horizontal rectangle with each rotating rectangle, determine the ratio of each overlapping area to the area of ​​the horizontal rectangle; determine the rotating rectangles whose corresponding ratios are greater than a threshold as rotating rectangles that contain the same target as the horizontal rectangles, and determine the rotating rectangles as successfully matched target rotating rectangles.

[0207] In one possible implementation, the processing module 1302 is further configured to, for each target rotating rectangle, determine the distance deviation between the first center point and the second center point in the target video frame based on the position information of the first center point of the target rotating rectangle and the second center point of the horizontal rectangle that successfully matches the target rotating rectangle in the target video frame, and input the target video frame and its adjacent video frames into the recognition model to obtain the predicted direction corresponding to the target rotating rectangle output by the recognition model, and determine the direction deviation between the predicted direction and the movement direction determined based on the long side of the target rotating rectangle; determine the matching degree between the target rotating rectangle and the matching horizontal rectangle based on the direction deviation and the distance deviation; if the matching degree is greater than a preset matching threshold, then execute the subsequent step of tracking the target corresponding to the successfully matched target rotating rectangle based on the determined movement direction.

[0208] In one possible implementation, the processing module 1302 is further configured to determine the actual location information of each key point based on the location information of each key point in the corresponding video frame and the GPS information of the UAV.

[0209] Figure 14 This application provides a schematic diagram of an electronic device structure based on an embodiment of the present application. In addition to the above embodiments, this application also provides an electronic device, such as... Figure 14As shown, it includes: processor 1401, communication interface 1402, memory 1403 and communication bus 1404, wherein processor 1401, communication interface 1402 and memory 1403 communicate with each other through communication bus 1404.

[0210] The memory 1403 stores a computer program, which, when executed by the processor 1401, causes the processor 1401 to perform any of the above method steps.

[0211] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0212] Communication interface 502 is used for communication between the above-mentioned electronic device and other devices.

[0213] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0214] The processors mentioned above can be general-purpose processors, including central processing units, network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits, field-programmable gate arrays or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0215] This application also provides a computer-readable storage medium storing a computer program executable by an electronic device, which, when run on the electronic device, causes the electronic device to perform any of the above method steps.

[0216] This application provides a computer program product, which includes an executable program that, when executed by a processor, implements the method described herein.

[0217] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0218] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A method for determining intersection information based on unmanned aerial vehicles (UAVs), characterized in that, The method includes: Acquire video streams captured by drones; Extract video frames from the video stream, input the video frames into a pre-trained zebra crossing recognition model, and obtain the first position information of the zebra crossing key points in the video frames output by the zebra crossing recognition model; Based on the first location information, the video frame is cropped to obtain a cropped image containing zebra crossings. The cropped image is then input into a pre-trained stop line recognition model to obtain the second location information of the stop line key points in the cropped image output by the stop line recognition model.

2. The method according to claim 1, characterized in that, The step of cropping the video frame based on the first location information to obtain a cropped image containing zebra crossings includes: Based on the first target location information of the zebra crossing key points, determine the rectangle containing the zebra crossing; The video frame is binarized, and the binarized video frame is subjected to a preset number of erosion and dilation morphological processing. Based on the morphologically processed image, the region with the largest connected component is determined. The first boundary of the region is determined to be closest to the center point of the rectangle, and the second boundary of the line segment of the rectangle is determined to be farthest from the first boundary. Based on the first boundary and the second boundary, a stop line detection region is determined. Based on the stop line detection region, the video frame is cropped.

3. The method according to claim 2, characterized in that, The method further includes: Based on the stop line corresponding to the determined stop line key point, the image is sliced ​​at a preset distance away from the zebra crossing. The sliced ​​image is then subjected to erosion and dilation morphological processing. Based on the number of connected components contained in the processed image, the number of lanes is determined, and the center point of each connected component is determined as the center point of the lane line.

4. The method according to claim 1, characterized in that, Extracting video frames from the video stream includes: Extract multiple video frames from the video stream; The method further includes: Based on the K-means clustering algorithm, the first location information of zebra crossing key points in multiple video frames is clustered to obtain the first target location information of zebra crossing key points. Based on the K-means clustering algorithm, the second location information of the stop line key points in multiple determined video frames is clustered to obtain the second target location information of the stop line key points.

5. The method according to claim 1, characterized in that, The method further includes: Target video frames are extracted from the video stream. Based on a rotating bounding box detection algorithm, rotating rectangular boxes that fit the target contour are extracted from the target video frames. A rectangular box detection method is then used to obtain the smallest bounding horizontal rectangular box that completely contains the target in the target video frame. The rotating rectangular boxes and the horizontal rectangular boxes are matched to obtain each successfully matched target rotating rectangular box. The targets include vehicles and pedestrians. The movement direction is determined based on the long side of the target rotation bounding box in each target video frame, and the target corresponding to the successfully matched target rotation bounding box is tracked based on the determined movement direction.

6. The method according to claim 5, characterized in that, The matching based on the rotated rectangle and the horizontal rectangle to obtain each successfully matched target rotated rectangle includes: For each horizontal rectangle in the video frame, determine the overlapping area between the horizontal rectangle and each rotating rectangle, and determine the ratio of each overlapping area to the area of ​​the horizontal rectangle; the rotating rectangles whose corresponding ratio is greater than a threshold are determined as rotating rectangles that contain the same target as the horizontal rectangle, and the rotating rectangles are determined as successfully matched target rotating rectangles.

7. The method according to claim 5, characterized in that, After matching the rotating rectangle and the horizontal rectangle to obtain each successfully matched target rotating rectangle, and before tracking the target corresponding to the successfully matched target rotating rectangle based on the determined movement direction, the method further includes: For each target rotating rectangle, based on the position information of the first center point of the target rotating rectangle and the second center point of the horizontal rectangle that successfully matches the target rotating rectangle in the target video frame, the distance deviation between the first center point and the second center point in the target video frame is determined. The target video frame and its adjacent video frames are then input into the recognition model to obtain the predicted direction corresponding to the target rotating rectangle output by the recognition model. The direction deviation between the predicted direction and the movement direction determined based on the long side of the target rotating rectangle is determined. Based on the direction deviation and the distance deviation, the matching degree between the target rotating rectangle and the matching horizontal rectangle is determined. If the matching degree is greater than a preset matching threshold, the subsequent step of tracking the target corresponding to the successfully matched target rotating rectangle based on the determined movement direction is executed.

8. The method according to any one of claims 1-7, characterized in that, The method further includes: Based on the location information of each key point in the corresponding video frame and the GPS information of the UAV, the actual location information of each key point is determined.

9. A state monitoring device based on edge computing, characterized in that, The device includes: The acquisition module is used to acquire the video stream captured by the drone; The processing module is configured to extract a preset number of video frames from the video stream; for each video frame, input the video frame into a pre-trained zebra crossing recognition model to obtain the location information of zebra crossing key points in the video frame output by the zebra crossing recognition model; determine the first target location information of the zebra crossing key points based on the location information of the zebra crossing key points in each video frame; crop each video frame based on the first location information to obtain each cropped image containing the zebra crossing; input each cropped image into a pre-trained stop line recognition model to obtain the location information of stop line key points in the cropped image output by the stop line recognition model; and determine the second target location information of the stop line key points based on the location information of the stop line key points in each cropped image.

10. An electronic device, characterized in that, The electronic device includes at least a processor and a memory, wherein the processor is used to execute a computer program stored in the memory to implement the steps of the UAV-based intersection information determination method as described in any one of claims 1-8.

Images