Methods, devices, equipment, media, and products for 3D human-vehicle cross-detection based on a monocular camera

Abstract

This invention discloses a 3D human-vehicle intersection detection method, device, equipment, medium, and product based on a monocular camera. The method includes: acquiring a first road video frame; determining an object segmentation mask corresponding to the first road video frame; determining image depth information when there is an intersection point between any two object mask regions in the object segmentation mask; determining target depth information corresponding to the object mask regions based on the object segmentation mask and the image depth information; determining the relative distance of the detected object in the target detection scene based on the two target depth information and the distance calibration information of the target detection scene; and generating object intersection warning information when the relative distance of the first object is less than a first distance threshold of the target detection scene. This technical solution achieves the effect of accurately determining the relative distance between two detected objects located on the road surface by combining two-dimensional image information with monocular depth information.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a method, apparatus, equipment, medium, and product for 3D human-vehicle cross-detection based on a monocular camera. Background Technology

[0002] Pedestrian-vehicle intersection detection is a core technology in intelligent transportation systems and autonomous driving, aiming to identify the interaction behavior of pedestrians and vehicles in road intersections in real time and predict potential collision risks. Pedestrian-vehicle intersection detection can improve road safety and reduce traffic accidents, especially in dangerous areas such as areas with hazardous chemicals, important government agencies, or confidential locations.

[0003] In related technologies, vehicle-pedestrian intersection detection typically employs Kalman filtering to detect and track both vehicle and pedestrian targets, thereby achieving vehicle-pedestrian intersection detection. However, this method is prone to losing targets or misjudging trajectories when targets are occluded or suddenly change lanes; furthermore, tracking algorithms relying solely on two-dimensional detection boxes may fail to update the three-dimensional position information of targets in the image in a timely manner, resulting in low detection accuracy. Summary of the Invention

[0004] This invention provides a 3D human-vehicle cross detection method, device, equipment, medium, and product based on a monocular camera, so as to achieve the effect of accurately determining the relative distance between two detection objects located on the road surface by combining the two-dimensional information of the image with monocular depth information.

[0005] According to one aspect of the present invention, a 3D human-vehicle intersection detection method based on a monocular camera is provided, the method comprising:

[0006] A first road video frame associated with the target detection scene is acquired by a monocular camera set at a preset location in the target detection scene; wherein, the first road video frame includes at least two types of detection objects located on the road in the target detection scene;

[0007] The first road video frame is segmented to obtain an object segmentation mask map corresponding to the first road video frame; wherein, the object segmentation mask map includes at least two types of object mask regions, and the object mask regions correspond to the detected objects;

[0008] In the case where there is a region intersection between any two object mask regions in the object segmentation mask image, the image depth information corresponding to the first road video frame is determined based on the pre-trained depth estimation model and the first road video frame; wherein, the depth estimation model is used to determine the depth information of each pixel in the first road video frame;

[0009] Based on the object segmentation mask and the image depth information, target depth information corresponding to each type of object mask region with region intersection is determined; wherein, the target depth information is used to characterize the regional depth features of the target object region in the object mask region, and the target object region is the object region in the object mask region that is adjacent to the region intersection;

[0010] Based on the two target depth information and the pre-determined distance calibration information corresponding to the target detection scene, the first relative distance between the detected objects corresponding to the object mask area where the region has an intersection point is determined in the target detection scene. If the first relative distance between the objects is less than the preset first distance threshold corresponding to the target detection scene, object intersection warning information is generated and displayed.

[0011] According to another aspect of the present invention, a 3D human-vehicle intersection detection device based on a monocular camera is provided, the device comprising:

[0012] A video frame acquisition module is used to acquire a first road video frame associated with the target detection scene using a monocular camera set at a preset position in the target detection scene; wherein, the first road video frame includes at least two types of detection objects located on the road in the target detection scene;

[0013] An object segmentation module is used to segment the first road video frame into objects to obtain an object segmentation mask map corresponding to the first road video frame; wherein, the object segmentation mask map includes at least two types of object mask regions, and the object mask regions correspond to the detected objects;

[0014] The depth information determination module is used to determine the image depth information corresponding to the first road video frame based on a pre-trained depth estimation model and the first road video frame when there is a region intersection point between any two object mask regions in the object segmentation mask image; wherein, the depth estimation model is used to determine the depth information of each pixel in the first road video frame;

[0015] The target depth information determination module is used to determine the target depth information corresponding to each type of object mask region with region intersection points based on the object segmentation mask image and the image depth information; wherein, the target depth information is used to characterize the regional depth features of the target object region in the object mask region, and the target object region is the object region in the object mask region that is adjacent to the region intersection point;

[0016] The object cross detection module is used to determine the first relative distance between the detected objects corresponding to the object mask area where the regions intersect in the target detection scene based on the two target depth information and the pre-determined distance calibration information corresponding to the target detection scene, and generate and display object cross warning information when the first relative distance between the objects is less than the preset first distance threshold corresponding to the target detection scene.

[0017] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0018] At least one processor; and

[0019] A memory communicatively connected to the at least one processor; wherein,

[0020] The memory stores a computer program that can be executed by the at least one processor, which enables the at least one processor to perform the 3D human-vehicle cross-detection method based on a monocular camera as described in any embodiment of the present invention.

[0021] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions, the computer instructions being configured to cause a processor to execute and implement the 3D human-vehicle cross-detection method based on a monocular camera as described in any embodiment of the present invention.

[0022] According to another aspect of the present invention, a computer program product is provided, the computer program product comprising a computer program that, when executed by a processor, implements the 3D human-vehicle cross-detection method based on a monocular camera as described in any embodiment of the present invention.

[0023] The technical solution of this invention involves acquiring road video frames associated with the target detection scene using a monocular camera positioned at a preset location within the target detection scene. Each road video frame includes at least two types of detection objects located on the road within the target detection scene. Further, object segmentation is performed on the road video frames to obtain an object segmentation mask map corresponding to a first road video frame. The object segmentation mask map includes object mask regions corresponding to each type of detection object. Further, when an intersection point is detected between any two object mask regions in the object segmentation mask map, image depth information corresponding to the road video frame is determined based on a pre-trained depth estimation model and the road video frame. Further, based on the object segmentation mask map and the image depth information, target depth information corresponding to the two object mask regions with the intersection point is determined. Further, based on the two target depth information and pre-determined distance calibration information corresponding to the target detection scene, the target depth information corresponding to the object mask regions with the intersection point is determined. This method determines the relative distance between two objects in a target detection scene. If the relative distance is less than a preset first distance threshold corresponding to the target detection scene, it generates and displays object intersection warning information. This solves the problems in related technologies where targets are easily lost or misjudged when they are occluded or suddenly change lanes, and where the 3D position information of targets in the image may not be updated in time, leading to low detection accuracy. It achieves the effect of accurately determining the relative distance between two types of objects on the road surface by combining 2D image information with monocular depth information. This improves the accuracy of object intersection detection while reducing the cost, thereby improving the accuracy of object intersection warnings, enhancing the safety of target detection scenes, and using depth information helps to develop more reasonable object intersection solutions. Furthermore, the image acquisition and processing method based on a monocular camera has high real-time performance, meeting the real-time requirements of most target detection scenarios.

[0024] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1This is a flowchart of a 3D human-vehicle intersection detection method based on a monocular camera according to an embodiment of the present invention;

[0027] Figure 2 This is a flowchart of a 3D human-vehicle intersection detection method based on a monocular camera according to an embodiment of the present invention;

[0028] Figure 3 This is a flowchart of a 3D human-vehicle intersection detection method based on a monocular camera according to an embodiment of the present invention;

[0029] Figure 4 This is a flowchart of a 3D human-vehicle intersection detection method based on a monocular camera according to an embodiment of the present invention;

[0030] Figure 5 This is a schematic diagram of the structure of a 3D human-vehicle intersection detection device based on a monocular camera according to an embodiment of the present invention;

[0031] Figure 6 This is a schematic diagram of the structure of an electronic device that implements the 3D human-vehicle cross-detection method based on a monocular camera according to an embodiment of the present invention. Detailed Implementation

[0032] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0033] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0034] Figure 1This is a flowchart of a 3D pedestrian-vehicle intersection detection method based on a monocular camera provided in an embodiment of the present invention. This embodiment is applicable to situations where a monocular camera is set in a target detection scene to detect intersections of objects on a road. The method can be executed by a 3D pedestrian-vehicle intersection detection device based on a monocular camera, which can be implemented in hardware and / or software. This device can be configured in a terminal and / or server. Figure 1 As shown, the method includes:

[0035] S110. Acquire a first road video frame associated with the target detection scene using a monocular camera set at a preset position in the target detection scene; wherein the first road video frame includes at least two types of detection objects located on the road in the target detection scene.

[0036] The target detection scenario can be a scenario requiring cross-detection of at least two types of road objects. In other words, the target detection scenario can be a scenario requiring cross-detection of at least two types of road objects to avoid dangerous situations. Optionally, the target detection scenario includes road intersections, T-junctions, hazardous chemical areas, important government areas, or key protected areas. The preset location can be a location in the target detection scenario where the road surface and road objects on the road can be captured. Optionally, the preset location includes above building entrances / exits, corners of important indoor areas, on streetlight poles, on traffic sign poles, or above the inside of the windshield of a vehicle in the target detection scenario, above the rear license plate, or other locations at the rear of the vehicle. A monocular camera refers to a camera module composed of a single camera, mainly including an image sensor, lens, processing chip, and other components. Monocular cameras can include various types, including RGB cameras, infrared cameras, wide-angle cameras, and telephoto cameras. It should be noted that the advantages of using a monocular camera for image acquisition in target detection scenarios are: monocular cameras are low in cost, have a simple hardware structure, are easy to install and use, are applicable to a wide range of scenarios, and can accurately perceive road conditions and identify vehicles, pedestrians, traffic signs and traffic lights, providing data support for cross-object detection on the road.

[0037] The first road video frame can be an image including the road surface and / or objects located on the road surface. In this embodiment, a monocular camera positioned at a preset location in the target detection scene can capture images of the road surface at a certain frame rate (e.g., 25 frames / second, 30 frames / second, etc.) to obtain multiple consecutive video frames. Furthermore, to enable object cross-detection between at least two types of objects based on the video frames, the captured video frames can be filtered to select those containing at least two types of objects located on the road in the target detection scene, and these selected video frames are used as the first road video frame. The detection objects can be objects that require object cross-detection. Optionally, the at least two detection objects include pedestrians, vehicles, or static objects on the road (e.g., buildings, etc.).

[0038] It should be noted that the first road video frame includes at least two types of detection objects to detect whether there is object overlap between different object types on the road. Furthermore, in the target detection scenario, object overlap between different object types may lead to more serious accidents. Therefore, in this embodiment, the first road video frame used for subsequent detection may be a video frame including at least two types of detection objects located on the road in the target detection scenario.

[0039] In practical implementation, in a target detection scenario, a monocular camera positioned at a preset location can capture video frames of the road at a preset frame rate. Furthermore, the captured video frames can be filtered to select those containing at least two types of objects located on the road, and these selected frames are used as the first road video frame.

[0040] S120. Perform object segmentation on the first road video frame to obtain an object segmentation mask map corresponding to the first road video frame; wherein, the object segmentation mask map includes at least two types of object mask regions; the object mask regions correspond to the detected objects.

[0041] Object segmentation involves separating the detected objects from the first road video frame to distinguish them from other regions in the resulting object segmentation mask. The object segmentation mask can be understood as an image representing the object contours of various detected objects in the first road video frame. The object segmentation mask includes object mask regions corresponding to at least two types of detected objects; that is, one detected object can correspond to one object mask region, and each object mask region can be represented using different pixel values. For example, suppose the first road video frame includes two detected objects, detected object A and detected object B. Further, the object segmentation mask obtained by performing object segmentation on the first road video frame can be an image composed of pixel values ​​0, 1, and 2. During object segmentation, the pixel values ​​in the object region containing detected object A in the first road video frame can be adjusted to 1, the pixel values ​​in the object region containing detected object B can be adjusted to 2, and the pixel values ​​in other image regions besides detected objects A and B can be adjusted to 0. The image with adjusted pixel values ​​can then be used as the object segmentation mask.

[0042] In this embodiment, there are multiple ways to segment the first road video frame into objects. Optional methods include using an edge detection segmentation algorithm to segment the first road video frame into objects; or using a machine learning-based segmentation algorithm (including clustering-based segmentation or random forest-based segmentation) to segment the first road video frame into objects; or using a deep learning-based segmentation algorithm (including fully convolutional networks, encoder-decoder convolutional neural networks, or masked region convolutional neural networks, etc.) to segment the first road video frame into objects, etc.

[0043] It should be noted that object segmentation of the first road video frame can be performed using the processing chip in the monocular camera, or it can be performed on a server connected to the monocular camera. In this embodiment, to improve object recognition and detection efficiency, various image processing operations on the first road video frame can be performed on the server.

[0044] In practical implementation, after acquiring a first road video frame associated with the target detection scene, the first road video frame can be sent to a server connected to a monocular camera. Furthermore, object recognition and segmentation can be performed on the detected objects in the first road video frame based on an object segmentation algorithm deployed on the server. Consequently, an object segmentation mask map corresponding to the first road video frame can be obtained, which includes the object mask region corresponding to each type of detected object.

[0045] S130. When there is a region intersection between any two object mask regions in the object segmentation mask map, determine the image depth information corresponding to the first road video frame based on the pre-trained depth estimation model and the first road video frame.

[0046] The existence of intersection points between any two object mask regions can be understood as overlapping areas between them. It should be noted that an object mask image can include one or more sets of object mask regions with intersection points; regardless of the number, the following processing steps can be applied. Furthermore, the existence of intersection points between at least two object mask regions indicates that the two detected objects are relatively close in the first video frame. To more accurately determine the relative distance between these two detected objects in the target detection scene, subsequent processing steps are required.

[0047] It's important to note that when at least two object mask regions are identified as having intersections based on the object segmentation mask, introducing depth information for further object cross-detection offers several advantages. Firstly, overlapping object mask regions in the object segmentation mask only indicate overlap in a two-dimensional plane, not their order in three-dimensional space. Depth information clarifies the relative positional relationship between the two detected objects, providing an accurate basis for subsequent security assessments. Secondly, relying solely on two-dimensional images makes it difficult to accurately determine the actual distance between these two objects. Depth information maps pixel positions in a two-dimensional image to three-dimensional space, and combined with camera calibration parameters, allows for the calculation of the actual distance between the two detected objects in three-dimensional space. Thirdly, when object segmentation regions overlap, the segmentation algorithm may not be able to accurately distinguish their boundaries. Fourthly, depth information serves as additional clues to further refine the segmentation results. Fifthly, when partial occlusion occurs between two detected objects, depth information helps identify the occluded portion, resulting in a more complete segmentation of the corresponding object segmentation regions. Finally, analyzing depth information allows for the identification of which pixels belong to the occluded target, thus correcting and supplementing the object segmentation mask.

[0048] The depth estimation model can be a deep learning model used to determine the depth information of each pixel in the first road video frame. The depth estimation model can be trained on the deep learning model based on sample road video frames and the true depth information of each pixel in the sample road video frames. The depth estimation model can be a deep learning model with any model structure capable of determining depth information; optionally, it can be ZoeDepth. ZoeDepth is a monocular depth estimation model that combines relative and absolute depth, achieving excellent optimization performance while maintaining the metric scale. ZoeDepth employs a two-stage framework. First, it uses a general encoder-decoder architecture for pre-training of depth estimation, learning to extract the fine-grained system of depth differences between adjacent pixels. Then, in the second stage, a lightweight head for absolute depth estimation is added and fine-tuned using an absolute depth dataset to directly predict absolute depth values. In this way, the depth information from the existing dataset is transferred to the new target dataset, achieving zero-shot depth estimation. Furthermore, ZoeDepth's core feature lies in its multimodal learning algorithm, which can simultaneously process data of different scales, types, and sources, and use deep learning algorithms to estimate the relative position and absolute depth information of the detected object in the target detection scene.

[0049] It should be noted that the depth estimation model can be deployed on the local terminal where the monocular camera is located, or it can be deployed on the server connected to the monocular camera. This embodiment does not make specific limitations on this.

[0050] In specific implementation, after obtaining the object segmentation mask, the distances between the object mask regions included in the object segmentation mask can be detected. Furthermore, if at least two object mask regions in the object segmentation mask are detected to have intersection points, the first road video frame can be input into a pre-trained depth estimation model. Further, the first road video frame can be processed based on the depth estimation model, and image depth information corresponding to the first road video frame can be output.

[0051] The image depth information may include the depth information of each pixel in the first road video frame, which refers to the distance information from the corresponding pixel to the monocular camera. Image depth information can be represented in various forms; optionally, it can be represented as a depth information matrix, or as a depth image, etc.

[0052] S140. Based on the object segmentation mask and image depth information, determine the target depth information corresponding to each object mask region with region intersections.

[0053] The target depth information is used to characterize the regional depth features of the target object region within the object mask region. The target object region is determined based on two types of object mask regions that have region intersections. The target object region can be an object region within the object mask region that is adjacent to the region intersection. The target depth information can be average depth information or weighted average depth information, etc.

[0054] In this embodiment, the existence of an intersection point between any two object mask regions only indicates that these two object mask regions overlap in a two-dimensional plane, and their relative positional relationship in three-dimensional space cannot be determined. Therefore, in order to further determine the relative positional relationship and relative distance between the two detected objects, the region depth information of the neighboring region at the intersection point of each object mask region can be determined, and the region depth information can be processed to obtain the target depth information corresponding to the object's bounding box. Furthermore, based on the two target depth information, the relative positional relationship and relative distance of the detected objects corresponding to these two object mask regions in three-dimensional space can be determined.

[0055] Optionally, based on the object segmentation mask and image depth information, the target depth information corresponding to each object mask region with region intersection is determined, including: determining the intersection region corresponding to the two object mask regions with region intersection based on the object segmentation mask; extending each edge of the intersection region outward by a preset distance with the intersection region as the center to obtain the dilated neighborhood corresponding to the intersection region; extracting the target object region corresponding to each object mask region with region intersection from the dilated neighborhood; determining the region depth information corresponding to each target object region based on the image depth information, and converting each region depth information into target depth information corresponding to each object mask region through weighted average operation or average operation.

[0056] The intersection region can be the area where two object mask regions overlap in the object segmentation mask image. The preset distance can be any pixel distance, optionally 5 pixels, 10 pixels, or 15 pixels. The dilated neighborhood can be the region obtained by dilating and expanding the intersection region. The dilated neighborhood contains the intersection region, and the distance between each edge of the dilated neighborhood and the corresponding edge in the intersection region is the preset distance. The target object region can be the image region where the detected object is located in the dilated neighborhood. The region depth information can include the depth information of each pixel in the object region.

[0057] In practical implementation, when an intersection point is detected between any two object mask regions in the object segmentation mask image, the intersection region corresponding to the two object mask regions with the intersection point can be determined. Then, with the intersection region as the center, each edge of the intersection region can be extended outwards by a preset distance, and the extended intersection region is used as the dilated neighborhood. Further, object recognition can be performed on the dilated neighborhood based on the pixel values ​​corresponding to the detected objects to extract the target object region corresponding to each detected object with the intersection point from the dilated neighborhood. Further, for the target object region corresponding to each detected object, the region depth information corresponding to the target object region can be determined based on image depth information through pixel matching. This region depth information includes the depth information of each pixel in the target object region. Further, the region depth information can be averaged, and the average value is used as the target depth information corresponding to the object mask region.

[0058] S150. Based on the depth information of the two targets and the pre-determined distance calibration information corresponding to the target detection scene, determine the first relative distance between the detected objects corresponding to the object mask areas where there are regional intersections in the target detection scene, and generate and display object intersection warning information if the first relative distance is less than the preset first distance threshold corresponding to the target detection scene.

[0059] The distance calibration information can be used to characterize the correspondence between the depth information of a pixel and the relative distance between the pixel's actual point and the monocular camera. That is, the distance calibration information includes at least one relative distance and its corresponding depth information. It should be noted that the distance calibration information corresponds to the target detection scene; different target detection scenes may correspond to different distance calibration information. Generally, distance information calibration can be performed in the target detection scene based on the monocular camera to obtain distance calibration information corresponding to the target detection scene. The first object relative distance can be the actual relative distance between two detected objects in the target detection scene. The first distance threshold is a predetermined standard used to determine whether the relative distance between two detected objects meets the warning conditions. In this embodiment, the first distance threshold is the minimum distance value at which a cross-warning is not triggered between two detected objects when there is an intersection point. It should be noted that the first distance threshold corresponds to the target detection scene; different target detection scenes may correspond to different first distance thresholds. The scale for object cross detection varies depending on the object detection scenario. Some scenarios have a stricter scale for object cross detection, resulting in a relatively smaller first distance threshold. Other scenarios have a more lenient scale for object cross detection, resulting in a relatively larger first distance threshold.

[0060] The object crossover warning information can be a system alert issued when two detected objects cross over, and this situation may indicate a risk, anomaly, or a problem requiring attention. The object crossover warning information can include multiple pieces of information associated with the object crossover, optionally including the object identifier of the object that crossed over, the location information of the object in the target detection scene, the degree of crossover, and / or the time information of the crossover.

[0061] In practical implementation, after obtaining the target depth information corresponding to the two types of object mask regions with intersection points, the relative distance of each detected object to the monocular camera in the target detection scene can be determined based on each target depth information and pre-determined distance calibration information corresponding to the target detection scene. Furthermore, a first relative distance between detected objects corresponding to the object mask regions with intersection points in the target detection scene can be determined based on the relative distances of the two objects. Further, this first relative distance can be compared with a preset first distance threshold corresponding to the target detection scene. Then, if the relative distance is less than the first distance threshold, object intersection warning information can be generated and displayed.

[0062] The display of cross-object warning information can be done by displaying it on the server-side display interface, and / or by sending the cross-object warning information to the target terminal for display on the target terminal's display interface.

[0063] The technical solution of this invention involves acquiring a first road video frame associated with the target detection scene using a monocular camera positioned at a preset location within the scene. The first road video frame includes at least two types of objects to be detected on the road within the target detection scene. Further, object segmentation is performed on the first road video frame to obtain an object segmentation mask corresponding to the first road video frame. The object segmentation mask includes at least two types of object mask regions, each corresponding to a detected object. Further, when an intersection point is detected between any two object mask regions in the object segmentation mask, image depth information corresponding to the first road video frame is determined based on a pre-trained depth estimation model and the first road video frame. Further, based on the object segmentation mask and the image depth information, target depth information corresponding to the two object mask regions with the intersection point is determined. Further, based on the two target depth information and pre-determined distance calibration information corresponding to the target detection scene, the intersection point is determined. The system identifies the relative distance between the first object in the target detection scene corresponding to the object mask area. If this relative distance is less than a preset first distance threshold corresponding to the target detection scene, an object crossover warning is generated and displayed. This solves the problems in related technologies where targets are easily lost or misjudged when they are occluded or suddenly change lanes, and where the 3D position information of the target in the image may not be updated in time, leading to low detection accuracy. It achieves the effect of accurately determining the relative distance between two types of detected objects on the road surface by combining 2D image information with monocular depth information. This improves the accuracy of object crossover detection while reducing its cost, thereby enhancing the accuracy of object crossover warnings and improving the safety of the target detection scene. Furthermore, the application of depth information helps to develop more reasonable object crossover solutions. The image acquisition and processing method based on a monocular camera has high real-time performance, meeting the real-time requirements of most target detection scenarios.

[0064] Building upon the above embodiments, there may be instances where no two object mask regions in the object segmentation mask map intersect, but the corresponding detected objects may overlap in three-dimensional space. In such cases, to more accurately determine whether object overlap exists in three-dimensional space, the relative distance between detected objects can be determined based on the object mask regions and the corresponding region depth information. Furthermore, the relative distance can be compared with a preset distance threshold to determine whether an object overlap warning is necessary.

[0065] Based on this, the method further includes: when there is no intersection between any two object mask regions in the object segmentation mask image, for any two object mask regions in the object segmentation mask image, taking one object mask region as the first object mask region and the other object mask region as the second object mask region; determining the shortest relative distance between the first object mask region and the second object mask region, and determining the third pixel point in the first object mask region corresponding to the shortest relative distance, and determining the fourth pixel point in the second object mask region corresponding to the shortest relative distance; based on the depth information of the third pixel point and a pre-determined distance relative to the target... The distance calibration information corresponding to the detection scene is used to determine the object distance of the detected object corresponding to the third pixel point relative to the monocular camera in the target detection scene; and, based on the depth information of the fourth pixel point and the pre-determined distance calibration information corresponding to the target detection scene, the object distance of the detected object corresponding to the fourth pixel point relative to the monocular camera in the target detection scene is determined; based on the object distances corresponding to the two detected objects, the second object relative distance between the two detected objects in the target detection scene is determined, and if the second object relative distance is less than the preset second distance threshold corresponding to the target detection scene, object crossover warning information is generated and displayed. The advantages of this setup are as follows: it can accurately determine the actual relative distance between two detected objects in the target detection scene based on the region depth information, even when there is no intersection between the mask regions of any two objects in the image. This allows for timely issuance of warning signals when the two detected objects are not yet in the same spatial position but are close to the danger zone. Furthermore, by focusing on the closest point between the mask regions of two objects to calculate the distance, rather than relying solely on fixed detection areas or preset motion trajectories, it can more flexibly adapt to various different scenarios. It also significantly reduces the amount of computation, improves the efficiency of object cross-detection, reduces hardware costs and energy consumption, and enables rapid response and timely issuance of warning signals.

[0066] The shortest relative distance can be the minimum relative distance between the first and second object mask regions, which can be the relative distance between the edges of the first and second object mask regions. The third pixel can be any pixel involved in the shortest relative distance within the first object mask region. The fourth pixel can be any pixel involved in the shortest relative distance within the second object mask region. In other words, the relative distance between the third and fourth pixels is the shortest relative distance. The detection object corresponding to the third pixel is the detection object corresponding to the first object mask region. The detection object corresponding to the fourth pixel is the detection object corresponding to the second object mask region. The second distance threshold can be the minimum distance value at which no cross-intersection warning is triggered between two detection objects when there is no region intersection. It should be noted that the second distance threshold corresponds to the target detection scenario; different target detection scenarios may correspond to different second distance thresholds.

[0067] In practical implementation, when there is no intersection between any two object mask regions in the object segmentation mask image, for any two object mask regions in the object segmentation mask image, one of the two object mask regions can be used as the first object mask region, and the other object mask region can be used as the second object mask region. Further, the relative distance between the region edges of the first object mask region and the region edges of the second object mask region can be determined, and the minimum value among multiple relative distances can be determined to obtain the shortest relative distance. Further, the pixel point in the first object mask region corresponding to the shortest relative distance can be determined as the third pixel point, and the pixel point in the second object mask region corresponding to the shortest relative distance can be determined as the fourth pixel point. Furthermore, the depth information corresponding to the third pixel can be determined based on the image depth information corresponding to the first road video frame. Then, based on this depth information, a traversal query is performed in the distance calibration information corresponding to the target detection scene to determine the closest first depth information from at least one first depth information included in the distance calibration information. The first relative distance corresponding to this first depth information is taken as the object distance of the detected object corresponding to the third pixel relative to the monocular camera in the target detection scene. Similarly, the depth information corresponding to the fourth pixel can be determined based on the image depth information corresponding to the first road video frame. Then, a traversal query is performed in the distance calibration information corresponding to the target detection scene to determine the closest first depth information from at least one first depth information included in the distance calibration information. The first relative distance corresponding to this first depth information is taken as the object distance of the detected object corresponding to the fourth pixel relative to the monocular camera in the target detection scene. Furthermore, after obtaining the object distances corresponding to the two detected objects, the difference between the two object distances can be determined, and the absolute value of this difference can be determined. Furthermore, the obtained absolute value can be used as the second relative distance between the two detected objects in the target detection scene. This second relative distance can then be compared with a preset second distance threshold corresponding to the target detection scene. If the second relative distance is less than this second distance threshold, object crossover warning information can be generated and displayed.

[0068] It should be noted that, in this embodiment, in order to determine whether the object segmentation mask includes any two object mask regions with a region intersection point, the degree of overlap between any two object mask regions can be determined for at least two object mask regions included in the object segmentation mask. Then, based on the result of the degree of overlap determination, it can be determined whether the object segmentation mask includes any two object mask regions with a region intersection point.

[0069] Based on this, in addition to the above technical solutions, the method further includes: determining the object mask regions corresponding to any two detection objects based on the object segmentation mask map; determining the intersection and union information of the two object mask regions, and determining the absolute value of the ratio between the intersection and union information to obtain the intersection-union ratio (IUR) corresponding to the object mask regions; determining that there is a region intersection point between the two object mask regions if the IUR is greater than a preset value; and determining that there is no region intersection point between the two object mask regions if the IUR is not greater than the preset value. The advantage of this setting is that the IUR can accurately measure the overlap and degree of overlap between two object mask regions. Furthermore, the calculation is simple and efficient. In practical applications, whether for detection systems with high real-time requirements or analysis tasks that need to process large amounts of image data, this simple and efficient calculation method can quickly provide judgment results, improving the system's operating efficiency.

[0070] The intersection information can be the area or number of pixels of the overlapping (or intersection) region of the two object mask regions. The union information can be the area or number of pixels of the covered (or union) region of the two object mask regions. The preset value can be 0 or 0.1.

[0071] In practical implementation, two object mask regions can be arbitrarily determined based on the object mask regions included in the object segmentation mask map. Further, the intersection and union regions of these two object mask regions can be determined, and intersection information and union information can be determined based on the intersection region and the union region, respectively. Further, the ratio between the intersection and union information can be determined, and the absolute value of this ratio can be determined, using the obtained absolute value as the intersection-union ratio (CIRR) corresponding to the two object mask regions. Further, the CIRR can be compared with a preset value. If the CIRR is greater than the preset value, it can be determined that there is a region intersection between the two object mask regions; if the CIRR is not greater than the preset value, it can be determined that there is no region intersection between the two object mask regions.

[0072] Figure 2 This is a flowchart of a 3D human-vehicle intersection detection method based on a monocular camera provided by an embodiment of the present invention. Building upon the aforementioned embodiments, before applying the distance calibration information corresponding to the target detection scene, distance calibration information corresponding to the target detection scene can be constructed first. Specific implementation details can be found in the technical solution of this embodiment. Technical terms that are the same as or similar to those in the above embodiments will not be repeated here.

[0073] like Figure 2 As shown, the method includes:

[0074] S210. For at least one target detection scenario, a road surface image corresponding to the target detection scenario is acquired by a monocular camera set at a preset position in the target detection scenario; wherein the road surface image includes the road surface located in the target detection scenario and containing a polygonal calibration area.

[0075] The road surface image can be an image that primarily captures or highlights the road surface. The road surface image includes road surfaces located within the target detection scene and containing polygonal calibration regions. In other words, the road surface image includes road surfaces located within the target detection scene, and these road surfaces contain polygonal calibration regions. The number of road surface images corresponding to the target detection scene can be one or more. Generally, to improve calibration accuracy and facilitate algorithm processing and calibration during the calibration process, a polygonal calibration region can be drawn on the road surface within the shooting range of the monocular camera. Furthermore, at least one road surface image containing the polygonal calibration region can be acquired using the monocular camera to perform distance calibration based on the acquired at least one road surface image.

[0076] The polygonal calibration region is a specific closed region with a polygonal shape delineated on the road surface. Optionally, the polygonal calibration region includes triangles, quadrilaterals, pentagons, etc. It should be noted that, to improve calibration accuracy, when delineating the polygonal calibration region corresponding to the target detection scene, the shape, size, and / or position of the polygonal calibration region on the road surface can be determined based on factors such as the road surface type (including cement, asphalt, dirt, and mountain roads), road width, number of lanes, and the location of the monocular camera. In other words, the polygonal calibration region drawn on the road surface can be different for different target detection scenes.

[0077] In specific implementation, for at least one target detection scenario, a monocular camera can be pre-positioned at a predetermined location within the target detection scenario. Then, a closed polygonal region can be drawn on the road surface within the image acquisition range of the monocular camera, and this closed region is used as the polygonal calibration region. Furthermore, the monocular camera positioned at the predetermined location within the target detection scenario can acquire images of the road surface containing the polygonal calibration region. This results in a road surface image corresponding to the target detection scenario, which includes the road surface located within the target detection scenario and containing the polygonal calibration region.

[0078] S220. Determine the pixel position information of each region vertex in the polygon calibration region in the road surface image to obtain the vertex position information corresponding to the road surface image.

[0079] In this context, "region vertices" refers to the corner points that constitute the polygonal calibration region, i.e., the endpoints of each edge within the polygonal calibration region. Pixel position information can be a set of data used to determine the specific location of region vertices on the road surface image. Pixel position information can be represented in various forms, including, optionally, absolute position coordinates, relative position information, and / or sub-pixel precision. Absolute position coordinates are the coordinate values ​​of the region vertex in a Cartesian coordinate system established with the upper left corner of the road surface image as the origin. Relative position information can be the offset of the region vertex relative to any point in the road surface image, calculated using that point as a reference. Sub-pixel precision refers to the pixel coordinates obtained by further refining the pixels; for example, dividing a pixel into 10 or 100 parts, the coordinates of the region vertex can be represented as (x, y), where x and y can be decimals, providing a more accurate description of the region vertex's position in the road surface image. Vertex position information includes the pixel position information of each region vertex in the road surface image.

[0080] In practical implementation, after obtaining a road map image containing polygonal calibration regions, the road surface image can be processed according to a preset image processing method to determine the pixel position information of each vertex in the polygonal calibration regions within the road surface image. Furthermore, based on the pixel position information of each vertex, the vertex position information corresponding to the road surface image can be determined. The preset image processing method includes image processing algorithms (including edge detection algorithms, Hough transform algorithms, or corner detection algorithms, etc.) and / or image processing models.

[0081] S230. Based on the vertex position information and the area size information corresponding to the polygon calibration area, determine the first relative distance between the first ground point corresponding to multiple first pixel points in the road ground image and the monocular camera in the target detection scene.

[0082] The region size information can be a series of quantified data describing the size and shape of the polygonal calibration region. Optionally, the region size information includes the region's side length and the angle between adjacent sides. The first pixel can be any pixel in the road surface image. The first ground point can refer to the actual geographical location point on the road surface corresponding to the first pixel in the road surface image. The first relative distance can be the relative distance between the first ground point and the monocular camera.

[0083] In this embodiment, after the polygonal calibration region is drawn on the road surface, the size of the drawn polygonal calibration region can be measured to obtain the region size information corresponding to the polygonal calibration region. Furthermore, having obtained the vertex position information corresponding to the polygonal calibration region, the ground position information of the first ground point corresponding to each first pixel in the road surface image in the target detection scene can be determined based on the region size information and the vertex position information. Furthermore, for each first pixel, the first relative distance of the first ground point relative to the monocular camera in the target detection scene can be determined based on the ground position information of the first ground point corresponding to the first pixel.

[0084] Optionally, based on vertex position information and region size information corresponding to the polygon calibration region, the first relative distance of the first ground point corresponding to multiple first pixels in the road ground image relative to the monocular camera in the target detection scene is determined, including: processing the vertex position information and region size information corresponding to the polygon calibration region according to a preset calibration function to determine the actual position information of the second ground point corresponding to each second pixel in the road ground image; for each second pixel, determining the second relative distance of the second ground point relative to the monocular camera in the target detection scene based on the preset position of the monocular camera and the actual position information of the second ground point corresponding to the second pixel; performing deduplication processing on multiple second relative distances, taking the deduplicated second relative distance as the first relative distance, taking the second ground point corresponding to the first relative distance as the first ground point, and taking the second pixel point corresponding to the first ground point as the first pixel point.

[0085] The preset calibration function can be a mathematical function used to establish the correspondence between image pixels and ground points in the real world. Its core is based on the monocular camera imaging model and known calibration area information, converting image coordinates into actual coordinates. The preset calibration function can include any calibration function capable of converting image coordinates to world coordinates, optionally including a camera calibration function. The second pixel can be any pixel in the road surface image. The second ground point can refer to the actual geographical location point on the road surface corresponding to the second pixel in the road surface image. The second relative distance can be the relative distance between the second ground point and the monocular camera.

[0086] In specific implementation, given the region size information, the actual position information of each region vertex in the target detection scene can be determined based on the region size information to obtain the actual vertex position information corresponding to the polygon calibration region. Further, the vertex position information and the actual vertex position information can be input into a preset calibration function. Then, the vertex position information and the actual vertex position information can be processed according to the preset calibration function to obtain a transformation matrix. This transformation matrix can be used to characterize the projection transformation relationship between the image coordinate system corresponding to the road surface image and the world coordinate system corresponding to the target detection scene. Further, for each second pixel in the road surface image, the transformation matrix and the pixel position information of the second pixel in the road surface image can be processed using the preset calibration function to obtain the actual position information of the second ground point corresponding to the second pixel in the target detection scene. Further, for each second pixel, the position information of the preset position where the monocular camera is located and the actual position information of the second ground point corresponding to the second pixel can be processed according to the distance formula to obtain the second relative distance of the second ground point relative to the monocular camera in the target detection scene. Thus, the second relative distances corresponding to multiple second ground points can be obtained. Furthermore, to ensure that the final distance calibration information includes multiple different relative distances and their corresponding depth information, the multiple second relative distances can be deduplicated. The remaining second relative distances after deduplication can be used as the first relative distance, the second ground point corresponding to the first relative distance can be used as the first ground point, and the second pixel point corresponding to the first ground point can be used as the first pixel point. Thus, the first relative distances of the first ground points corresponding to multiple first pixels in the road ground image relative to the monocular camera in the target detection scene can be obtained.

[0087] For example, the second relative distance can be determined based on the following formula:

[0088]

[0089] Where d represents the second relative distance; (x1, y1, z1) represents the actual location information of the second ground point; and (x2, y2, z2) represents the location information of the preset position of the monocular camera.

[0090] S240. Based on the pre-acquired ground image depth information corresponding to the road ground image, determine the first depth information of multiple first pixel points, and construct distance calibration information corresponding to the target detection scene based on multiple first relative distances and their corresponding first depth information.

[0091] The ground image depth information can include the depth information of each pixel in the road surface image, which refers to the distance from the corresponding pixel to the monocular camera. Ground image depth information can be represented in various forms, optionally, as a depth information matrix; or as a depth image, etc.

[0092] In practical implementation, the road surface image can be processed according to a preset depth information determination method to obtain ground image depth information corresponding to the road surface image. Furthermore, the first depth information of each first pixel can be obtained from the ground image depth information through pixel matching. Further, a mapping relationship can be established based on the first relative distances corresponding to multiple first pixels and their corresponding first depth information, and distance calibration information corresponding to the target detection scene can be generated based on the established mapping relationship. The depth information determination method can include processing the road surface image based on a depth information determination algorithm; or processing the road surface image based on a pre-trained depth estimation model, etc. The distance calibration information includes multiple different first relative distances and their corresponding first depth information.

[0093] S250, A first road video frame associated with the target detection scene is acquired by a monocular camera set at a preset position in the target detection scene; wherein, the first road video frame includes at least two types of detection objects located on the road in the target detection scene.

[0094] S260. Perform object segmentation on the first road video frame to obtain an object segmentation mask map corresponding to the first road video frame; wherein, the object segmentation mask map includes at least two types of object mask regions; the object mask regions correspond to the detected objects.

[0095] S270. When there is a region intersection between any two object mask regions in the object segmentation mask map, determine the image depth information corresponding to the first road video frame based on the pre-trained depth estimation model and the first road video frame.

[0096] S280. Based on the object segmentation mask and image depth information, determine the target depth information corresponding to each object mask region with region intersections.

[0097] S290. Based on the depth information of the two targets and the pre-determined distance calibration information corresponding to the target detection scene, determine the first relative distance between the detected objects corresponding to the object mask area where the regions intersect, and generate and display object intersection warning information if the first relative distance is less than the preset first distance threshold corresponding to the target detection scene.

[0098] The technical solution of this invention involves acquiring a road surface image corresponding to at least one target detection scenario using a monocular camera positioned at a preset location within the target detection scenario. The road surface image includes a road surface located within the target detection scenario and containing a polygonal calibration region. Further, the pixel position information of each vertex in the polygonal calibration region within the road surface image is determined to obtain vertex position information corresponding to the road surface image. Further, based on the vertex position information and the region size information corresponding to the polygonal calibration region, a first relative distance is determined between multiple first pixels in the road surface image and the monocular camera within the target detection scenario. Further, based on the pre-acquired distance information corresponding to the road surface image... The system uses surface image depth information to determine the first depth information of multiple first pixels. Based on multiple first relative distances and their corresponding first depth information, it constructs distance calibration information corresponding to the target detection scene. This achieves the effect of accurately determining the actual distance of surface points relative to the monocular camera by integrating vertex position information, region size information, and image depth information. By fusing the first relative distance and first depth information to construct distance calibration information, the overall accuracy of distance calibration is improved. Furthermore, for different target detection scenes, corresponding distance calibration information is constructed, which improves the adaptability and reliability of the distance calibration system in various scenarios. By flexibly using distance calibration information, it can meet the diverse needs of distance measurement in different application scenarios, thereby improving the accuracy of object localization in subsequent applications.

[0099] Figure 3 This is a flowchart of a 3D human-vehicle intersection detection method based on a monocular camera provided by an embodiment of the present invention. Based on the aforementioned embodiments, the method for determining the relative distance is further refined. Specific implementation details can be found in the technical solution of this embodiment. Technical terms that are the same as or similar to those in the above embodiments will not be repeated here.

[0100] like Figure 3 As shown, the method includes:

[0101] S310. Acquire a first road video frame associated with the target detection scene using a monocular camera set at a preset position in the target detection scene; wherein the first road video frame includes at least two types of detection objects located on the road in the target detection scene.

[0102] S320. Perform object segmentation on the first road video frame to obtain an object segmentation mask map corresponding to the first road video frame; wherein, the object segmentation mask map includes at least two types of object mask regions; the object mask regions correspond to the detected objects.

[0103] S330. When there is a region intersection between any two object mask regions in the object segmentation mask map, determine the image depth information corresponding to the first road video frame based on the pre-trained depth estimation model and the first road video frame.

[0104] S340. Based on the object segmentation mask and image depth information, determine the target depth information corresponding to each object mask region with region intersections.

[0105] S350. Based on the two target depth information and the pre-determined distance calibration information corresponding to the target detection scene, determine the object distance of each detected object relative to the monocular camera in the target detection scene.

[0106] Among them, the object distance can be the actual relative distance between the detected object and the monocular camera.

[0107] In practical implementation, after obtaining the target depth information corresponding to the two object mask regions, the target depth information can be used as the depth information of the detected objects included in the corresponding object mask regions. Furthermore, for the two target depth information, a traversal query can be performed on the distance calibration information corresponding to the target detection scene based on the target depth information, to determine the first depth information closest to the target depth information from at least one depth information included in the distance calibration information, and the first relative distance corresponding to the first depth information is used as the object distance of the detected object corresponding to the target depth information relative to the monocular camera in the target detection scene. The first depth information closest to the target depth information can be the first depth information with the smallest depth difference from the target depth information.

[0108] S360. Based on the object distances corresponding to the two detected objects, determine the first relative distance between the detected objects corresponding to the object mask areas where the regions intersect in the target detection scene, and generate and display object intersection warning information if the first relative distance is less than the preset first distance threshold corresponding to the target detection scene.

[0109] In this embodiment, after obtaining the object distances corresponding to two detected objects, the relative distance between the detected objects corresponding to the two object mask regions with intersecting regions in the target detection scene can be determined based on these two object distances.

[0110] Optionally, based on the object distances corresponding to the two detected objects, the relative distance between the detected objects corresponding to the object mask regions where the regions intersect is determined in the target detection scene, including: determining the absolute value of the difference between the two object distances to obtain the relative distance between the two detected objects in the target detection scene;

[0111] The relative distance is determined based on the following formula:

[0112] D 相对 =|D1-D2|

[0113] Among them, D 相对 D1 represents the relative distance; D2 represents the distance of one type of detected object with region intersection in the target detection scene; D2 represents the distance of another type of detected object with region intersection in the target detection scene.

[0114] In practical implementation, after obtaining the object distances corresponding to two detected objects, the difference between the two object distances can be determined, and the absolute value of this difference can be determined. Then, the obtained absolute value can be used as the relative distance between the two detected objects in the target detection scene. Furthermore, this relative distance can be compared with a preset first distance threshold corresponding to the target detection scene. Then, if the relative distance is less than the first distance threshold, object intersection warning information can be generated and displayed.

[0115] The technical solution of this invention determines the object distance of each detected object relative to the monocular camera in the target detection scene based on two target depth information and pre-determined distance calibration information corresponding to the target detection scene. Furthermore, based on the object distances corresponding to the two detected objects, the relative distance between the detected objects corresponding to the object mask areas with regional intersections in the target detection scene is determined. This achieves the effect of accurately determining the relative distance between two types of detected objects located on the road surface by combining monocular depth information and distance calibration information based on the two-dimensional image information, thereby improving the accuracy of object cross-detection while reducing the cost of object cross-detection.

[0116] Figure 4 This is a flowchart illustrating a 3D pedestrian-vehicle intersection detection method based on a monocular camera, provided by an embodiment of the present invention. This embodiment is a preferred embodiment of the above embodiments, and the specific implementation of this embodiment can be described using pedestrians and vehicles as examples. Figure 4 As shown, the method includes:

[0117] First, road video frames are acquired, and the video frames are segmented to detect objects, specifically pedestrians and vehicles, resulting in an object segmentation mask. If both pedestrian and vehicle mask regions exist in the object segmentation mask, the intersection-union ratio (IU) of the pedestrian and vehicle mask regions is determined. Further, if the IU is greater than 0, image depth information corresponding to the road video frame is acquired, and the average depth information of the neighboring regions of the pedestrian and vehicle mask regions at their intersection points is determined based on this depth information. Further, the actual distance between pedestrians and vehicles is determined based on these two average depth information and preset distance calibration information. Further, the actual distance is compared with a preset distance. If the actual distance is not greater than the preset distance, an object intersection warning is generated. If the actual distance is greater than the preset distance, no action is taken.

[0118] If the intersection-union ratio is not greater than 0, the actual distance between pedestrians and vehicles is determined using preset distance calibration information. The actual distance is then compared with a preset distance. If the actual distance is not greater than the preset distance, an object intersection warning is generated. If the actual distance is greater than the preset distance, no action is taken.

[0119] No operation is performed if there are no pedestrian mask regions and vehicle mask regions simultaneously in the object segmentation mask map.

[0120] Figure 5 This is a schematic diagram of the structure of a 3D human-vehicle intersection detection device based on a monocular camera, provided in an embodiment of the present invention. Figure 5 As shown, the device includes: a video frame acquisition module 510, an object segmentation module 520, a depth information determination module 530, a target depth information determination module 540, and an object cross detection module 550.

[0121] The video frame acquisition module 510 is used to acquire a first road video frame associated with the target detection scene using a monocular camera set at a preset position in the target detection scene; wherein the first road video frame includes at least two types of detection objects located on the road in the target detection scene; the object segmentation module 520 is used to perform object segmentation on the first road video frame to obtain an object segmentation mask map corresponding to the first road video frame; wherein the object segmentation mask map includes at least two types of object mask regions, and the object mask regions correspond to the detection objects; the depth information determination module 530 is used to determine the image depth information corresponding to the first road video frame based on a pre-trained depth estimation model and the first road video frame when there is a region intersection between any two object mask regions in the object segmentation mask map; wherein the depth estimation model is used to determine the depth information of the first road video frame. The depth information of each pixel in the first road video frame is determined; the target depth information determination module 540 is used to determine the target depth information corresponding to each type of object mask region with region intersection based on the object segmentation mask map and image depth information; wherein, the target depth information is used to characterize the regional depth features of the target object region in the object mask region, and the target object region is the object region in the object mask region that is adjacent to the region intersection; the object cross detection module 550 is used to determine the first relative distance between the detected objects corresponding to the object mask regions with region intersection in the target detection scene based on the two target depth information and the pre-determined distance calibration information corresponding to the target detection scene, and generate and display object cross warning information when the first relative distance is less than the preset first distance threshold corresponding to the target detection scene.

[0122] The technical solution of this invention involves acquiring road video frames associated with the target detection scene using a monocular camera positioned at a preset location within the target detection scene. Each road video frame includes at least two types of detection objects located on the road within the target detection scene. Further, object segmentation is performed on the road video frames to obtain an object segmentation mask map corresponding to a first road video frame. The object segmentation mask map includes object mask regions corresponding to each type of detection object. Further, when an intersection point is detected between any two object mask regions in the object segmentation mask map, image depth information corresponding to the road video frame is determined based on a pre-trained depth estimation model and the road video frame. Further, based on the object segmentation mask map and the image depth information, target depth information corresponding to the two object mask regions with the intersection point is determined. Further, based on the two target depth information and a pre-determined target depth information corresponding to the target detection scene... Distance calibration information determines the relative distance between the first object in the target detection scene corresponding to the object mask area where there is an intersection. When the relative distance of the first object is less than a preset first distance threshold corresponding to the target detection scene, object intersection warning information is generated and displayed. This solves the problems in related technologies where targets are easily lost or misjudged when they are occluded or suddenly change lanes, as well as the problem that the three-dimensional position information of the target in the image may not be updated in time, resulting in low detection accuracy. It achieves the effect of accurately determining the relative distance between two types of detection objects on the road surface by combining the two-dimensional information of the image with monocular depth information. It achieves the effect of improving the accuracy of object intersection detection while reducing the cost of object intersection detection, thereby improving the accuracy of object intersection warning, enhancing the safety of the target detection scene, and the application of depth information helps to develop more reasonable object intersection solutions.

[0123] Optionally, the device further includes: a road surface image acquisition module, a vertex position information determination module, a first relative distance determination module, and a distance calibration information construction module. The road surface image acquisition module is used to acquire a road surface image corresponding to at least one target detection scenario using a monocular camera positioned at a preset location in the target detection scenario; wherein the road surface image includes a road surface located within the target detection scenario and containing a polygonal calibration region; the vertex position information determination module is used to determine the pixel position information of each vertex in the polygonal calibration region in the road surface image to obtain vertex position information corresponding to the road surface image; the first relative distance determination module is used to determine, based on the vertex position information and the region size information corresponding to the polygonal calibration region, the first relative distance of a first ground point corresponding to multiple first pixels in the road surface image relative to the monocular camera in the target detection scenario; the distance calibration information construction module is used to determine the first depth information of multiple first pixels based on pre-acquired ground image depth information corresponding to the road surface image, and construct distance calibration information corresponding to the target detection scenario based on multiple first relative distances and their corresponding first depth information.

[0124] Optionally, the first relative distance determination module includes: an actual position information determination unit, a second relative distance determination unit, and a first relative distance determination unit. The actual position information determination unit is used to process vertex position information and region size information corresponding to the polygon calibration area according to a preset calibration function to determine the actual position information of the second ground point corresponding to each second pixel in the road ground image. The second relative distance determination unit is used to determine, for each second pixel, the second relative distance of the second ground point relative to the monocular camera in the target detection scene, based on the preset position of the monocular camera and the actual position information of the second ground point corresponding to the second pixel. The first relative distance determination unit is used to perform deduplication processing on multiple second relative distances, using the deduplicated second relative distance as the first relative distance, the second ground point corresponding to the first relative distance as the first ground point, and the second pixel point corresponding to the first ground point as the first pixel point.

[0125] Optionally, the target depth information determination module 540 includes: an intersection region determination unit, an expanded neighborhood determination unit, a target object region extraction unit, and a target depth information determination unit. Specifically, the intersection region determination unit determines the intersection region corresponding to two object mask regions with intersection points based on the object mask image; the expanded neighborhood determination unit extends each edge of the intersection region outwards by a preset distance, centered on the intersection region, to obtain an expanded neighborhood corresponding to the intersection region; the target object region extraction unit extracts the target object region corresponding to each type of object mask region with intersection points from the expanded neighborhood; and the target depth information determination unit determines the region depth information corresponding to each type of target object region based on image depth information, and converts each type of region depth information into target depth information corresponding to each type of object mask region through a weighted average operation or an average calculation.

[0126] Optionally, the object cross detection module 550 includes: an object distance determination unit and an object relative distance determination unit. The object distance determination unit is used to determine the object distance of each detected object relative to the monocular camera in the target detection scene based on the depth information of the two targets and pre-determined distance calibration information corresponding to the target detection scene. The object relative distance determination unit is used to determine the first object relative distance between detected objects corresponding to the object mask regions where regions intersect in the target detection scene based on the object distances corresponding to the two detected objects.

[0127] Optionally, the object relative distance determination unit is specifically used to determine the absolute value of the difference between the distances of two objects in order to obtain the first object relative distance between the two detected objects in the target detection scene;

[0128] The relative distance is determined based on the following formula:

[0129] D 相对 =|D1-D2|

[0130] Among them, D 相对 D1 represents the relative distance of the first object; D2 represents the object distance of one type of detected object with region intersection in the target detection scene; D3 represents the object distance of another type of detected object with region intersection in the target detection scene.

[0131] Optionally, the device further includes: an object mask region first determination module, a pixel point determination module, an object distance determination module, and an object relative distance determination module. The object mask region determination module is used to, when there is no region intersection between any two object mask regions in the object segmentation mask image, designate one object mask region as the first object mask region and the other object mask region as the second object mask region. The pixel point determination module is used to determine the shortest relative distance between the first object mask region and the second object mask region, and to determine a third pixel point in the first object mask region corresponding to the shortest relative distance, and a fourth pixel point in the second object mask region corresponding to the shortest relative distance. The object distance determination module is used to determine the object distance based on the depth information of the third pixel point and a pre-defined distance. The system uses the determined distance calibration information corresponding to the target detection scene to determine the object distance of the detected object corresponding to the third pixel point relative to the monocular camera in the target detection scene; and, based on the depth information of the fourth pixel point and the predetermined distance calibration information corresponding to the target detection scene, to determine the object distance of the detected object corresponding to the fourth pixel point relative to the monocular camera in the target detection scene; the object relative distance determination module is used to determine the second object relative distance between the two detected objects in the target detection scene based on the object distances corresponding to the two detected objects, and to generate and display object crossover warning information if the second object relative distance is less than a preset second distance threshold corresponding to the target detection scene.

[0132] Optionally, the device further includes: a second object mask region determination module, an intersection-union ratio (IU / U) determination module, and a region intersection point determination module. The second object mask region determination module is used to determine the object mask regions corresponding to any two detected objects based on the object segmentation mask image; the IU / U is used to determine the intersection and union information of the two object mask regions, and determine the absolute value of the ratio between the intersection and union information to obtain the IU / U is corresponding to the object mask region; the region intersection point determination module is used to determine that there is a region intersection point between the two object mask regions if the IU / U is greater than a preset value, and to determine that there is no region intersection point between the two object mask regions if the IU / U is not greater than the preset value.

[0133] The 3D human-vehicle intersection detection device based on a monocular camera provided in this embodiment of the invention can execute the 3D human-vehicle intersection detection method based on a monocular camera provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the method.

[0134] Figure 6A schematic diagram of an electronic device 10 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0135] like Figure 6 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 may also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0136] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0137] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as a 3D human-vehicle cross-detection method based on a monocular camera.

[0138] In some embodiments, the monocular camera-based 3D human-vehicle intersection detection method can be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the monocular camera-based 3D human-vehicle intersection detection method described above can be performed. Alternatively, in other embodiments, processor 11 can be configured to perform the monocular camera-based 3D human-vehicle intersection detection method by any other suitable means (e.g., by means of firmware).

[0139] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0140] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0141] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0142] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0143] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), target blockchain networks, and the Internet.

[0144] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0145] In particular, according to embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication unit 19, or installed from storage unit 18, or installed from ROM 12. When the computer program is executed by processor 11, it performs the functions defined in the methods of the embodiments of the present invention.

[0146] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0147] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A 3D human-vehicle intersection detection method based on a monocular camera, characterized in that, include: A first road video frame associated with the target detection scene is acquired by a monocular camera set at a preset location in the target detection scene; wherein, the first road video frame includes at least two types of detection objects located on the road in the target detection scene; The first road video frame is segmented to obtain an object segmentation mask map corresponding to the first road video frame; wherein, the object segmentation mask map includes at least two types of object mask regions, and the object mask regions correspond to the detected objects; In the case where there is a region intersection between any two object mask regions in the object segmentation mask image, the image depth information corresponding to the first road video frame is determined based on the pre-trained depth estimation model and the first road video frame; wherein, the depth estimation model is used to determine the depth information of each pixel in the first road video frame; Based on the object segmentation mask and the image depth information, target depth information corresponding to each type of object mask region with region intersection is determined; wherein, the target depth information is used to characterize the regional depth features of the target object region in the object mask region, and the target object region is the object region in the object mask region that is adjacent to the region intersection; Based on the two target depth information and the pre-determined distance calibration information corresponding to the target detection scene, the first relative distance between the detected objects corresponding to the object mask area with the region intersection is determined in the target detection scene. If the first relative distance between the objects is less than the preset first distance threshold corresponding to the target detection scene, object intersection warning information is generated and displayed. The method further includes: when there is no intersection between any two object mask regions in the object segmentation mask image, for any two object mask regions in the object segmentation mask image, taking one of the object mask regions as the first object mask region and the other object mask region as the second object mask region. Determine the shortest relative distance between the first object mask region and the second object mask region, and determine the third pixel point in the first object mask region corresponding to the shortest relative distance, and determine the fourth pixel point in the second object mask region corresponding to the shortest relative distance; Based on the depth information of the third pixel and the pre-determined distance calibration information corresponding to the target detection scene, the object distance of the detected object corresponding to the third pixel in the target detection scene relative to the monocular camera is determined; and, Based on the depth information of the fourth pixel and the pre-determined distance calibration information corresponding to the target detection scene, the object distance of the detection object corresponding to the fourth pixel in the target detection scene relative to the monocular camera is determined; Based on the object distance between the two detected objects, a second relative object distance between the two detected objects in the target detection scene is determined. If the second relative object distance is less than a preset second distance threshold corresponding to the target detection scene, object crossover warning information is generated and displayed.

2. The 3D human-vehicle intersection detection method based on a monocular camera according to claim 1, characterized in that, Also includes: For at least one target detection scenario, a road surface image corresponding to the target detection scenario is acquired by a monocular camera set at a preset position in the target detection scenario; wherein, the road surface image includes the road surface located in the target detection scenario and containing a polygonal calibration area; The pixel position information of each vertex in the polygon calibration region in the road surface image is determined respectively to obtain the vertex position information corresponding to the road surface image; Based on the vertex position information and the region size information corresponding to the polygon calibration region, the first relative distance of the first ground point corresponding to multiple first pixels in the road ground image relative to the monocular camera in the target detection scene is determined. Based on the pre-acquired ground image depth information corresponding to the road ground image, the first depth information of multiple first pixel points is determined, and distance calibration information corresponding to the target detection scene is constructed based on multiple first relative distances and their corresponding first depth information.

3. The 3D human-vehicle intersection detection method based on a monocular camera according to claim 2, characterized in that, The step of determining the first relative distance between the first ground points corresponding to multiple first pixels in the road ground image and the monocular camera in the target detection scene, based on the vertex position information and the region size information corresponding to the polygon calibration region, includes: The vertex position information and the area size information corresponding to the polygon calibration area are processed according to a preset calibration function to determine the actual position information of the second ground point corresponding to each second pixel in the road ground image; For each second pixel, based on the preset position of the monocular camera and the actual position information of the second ground point corresponding to the second pixel, a second relative distance of the second ground point relative to the monocular camera in the target detection scene is determined; The multiple second relative distances are deduplicated, and the deduplicated second relative distances are used as the first relative distances. The second ground points corresponding to the first relative distances are used as the first ground points, and the second pixel points corresponding to the first ground points are used as the first pixel points.

4. The 3D human-vehicle intersection detection method based on a monocular camera according to claim 1, characterized in that, The step of determining the target depth information corresponding to each type of object mask region with region intersections based on the object segmentation mask map and the image depth information includes: Based on the object segmentation mask map, the intersection regions corresponding to the two types of object mask regions with region intersection points are determined; Centered on the intersection region, each edge of the intersection region is extended outward by a preset distance to obtain an expanded neighborhood corresponding to the intersection region; Extract the target object region corresponding to each object mask region that intersects with the existing region from the expanded neighborhood; Based on the image depth information, the region depth information corresponding to each target object region is determined, and each region depth information is converted into target depth information corresponding to each object mask region through weighted average operation or average operation.

5. The 3D human-vehicle intersection detection method based on a monocular camera according to claim 1, characterized in that, The step of determining the first relative distance between the detected objects in the target detection scene corresponding to the object mask regions with region intersections, based on the two target depth information and pre-determined distance calibration information corresponding to the target detection scene, includes: Based on the two target depth information and the pre-determined distance calibration information corresponding to the target detection scene, the object distance of each detection object relative to the monocular camera in the target detection scene is determined respectively; Based on the object distances corresponding to the two detected objects, the first relative distance between the detected objects in the target detection scene corresponding to the object mask areas where the regions intersect is determined.

6. The 3D human-vehicle intersection detection method based on a monocular camera according to claim 5, characterized in that, The step of determining the first relative distance between the detected objects in the target detection scene corresponding to the object mask region where the regions intersect, based on the object distances corresponding to the two detected objects, includes: Determine the absolute value of the difference between the distances of two objects to obtain the first relative distance between the two detected objects in the target detection scene; The relative distance is determined based on the following formula: in, Indicates the relative distance to the first object; The distance between the detected object and the target detection scene is represented by a region intersection point. The distance to another detected object in the target detection scene where there is an intersection point between regions is indicated.

7. The 3D human-vehicle intersection detection method based on a monocular camera according to claim 1, characterized in that, Also includes: Based on the object segmentation mask map, the object mask regions corresponding to any two of the detected objects are determined; Determine the intersection and union information of the two object mask regions, and determine the absolute value of the ratio between the intersection information and the union information to obtain the intersection-union ratio corresponding to the object mask region; If the intersection-union ratio is greater than a preset value, it is determined that there is a region intersection point between the two object mask regions; If the intersection-union ratio is not greater than a preset value, it is determined that there is no intersection point between the two object mask regions.

8. A 3D human-vehicle intersection detection device based on a monocular camera, characterized in that, include: A video frame acquisition module is used to acquire a first road video frame associated with the target detection scene using a monocular camera set at a preset position in the target detection scene; wherein, the first road video frame includes at least two types of detection objects located on the road in the target detection scene; An object segmentation module is used to segment the first road video frame into objects to obtain an object segmentation mask map corresponding to the first road video frame; wherein, the object segmentation mask map includes at least two types of object mask regions, and the object mask regions correspond to the detected objects; The depth information determination module is used to determine the image depth information corresponding to the first road video frame based on a pre-trained depth estimation model and the first road video frame when there is a region intersection point between any two object mask regions in the object segmentation mask image; wherein, the depth estimation model is used to determine the depth information of each pixel in the first road video frame; The target depth information determination module is used to determine the target depth information corresponding to each type of object mask region with region intersection points based on the object segmentation mask image and the image depth information; wherein, the target depth information is used to characterize the regional depth features of the target object region in the object mask region, and the target object region is the object region in the object mask region that is adjacent to the region intersection point; The object cross detection module is used to determine the first relative distance between the detected objects corresponding to the object mask area where the regions intersect in the target detection scene based on the two target depth information and the pre-determined distance calibration information corresponding to the target detection scene, and generate and display object cross warning information when the first relative distance between the objects is less than the preset first distance threshold corresponding to the target detection scene. The device further includes: an object mask region first determination module, a pixel point determination module, an object distance determination module, and an object relative distance determination module; wherein, the object mask region determination module is used to, when there is no region intersection between any two object mask regions in the object segmentation mask image, select one of the object mask regions as a first object mask region and the other as a second object mask region for any two object mask regions in the object segmentation mask image; the pixel point determination module is used to determine the shortest relative distance between the first object mask region and the second object mask region, determine a third pixel point in the first object mask region corresponding to the shortest relative distance, and determine a fourth pixel point in the second object mask region corresponding to the shortest relative distance; the object distance... The distance determination module is used to determine the object distance of the detected object corresponding to the third pixel point relative to the monocular camera in the target detection scene based on the depth information of the third pixel point and the pre-determined distance calibration information corresponding to the target detection scene; and to determine the object distance of the detected object corresponding to the fourth pixel point relative to the monocular camera in the target detection scene based on the depth information of the fourth pixel point and the pre-determined distance calibration information corresponding to the target detection scene; the object relative distance determination module is used to determine the second object relative distance between the two detected objects in the target detection scene based on the object distances corresponding to the two detected objects, and to generate and display object crossover warning information if the second object relative distance is less than a preset second distance threshold corresponding to the target detection scene.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the 3D human-vehicle cross-detection method based on a monocular camera as described in any one of claims 1-7.

Images