Target detection method and device, electronic equipment and computer program product

Abstract

The invention discloses a target detection method and device, electronic equipment and a computer program product, and the method comprises the steps: obtaining a road image collected by roadside equipment, carrying out the target detection of the road image through a preset 3D target detection model, and obtaining a target detection result, which comprises 3D target frame information and 2D target frame information; determining a 3D position of the target according to the 3D target frame information and the 2D target frame information; according to the road image and the 2D target frame information, using a preset key point detection model to detect the target key point to obtain a target key point detection result; and determining course angle information of the target according to the 3D position of the target and the target key point detection result. The road end image is detected on the basis of the 3D target detection model adopted by the vehicle end, the accuracy of target 3D position detection is improved, the key point detection model is used for further recognizing the specific key point information of the target, the target course angle is calculated accordingly, and the accuracy and reliability of course angle calculation are improved.

Description

Technical Field

[0001] The present application relates to the technical field of target detection, and in particular to a target detection method, device, electronic equipment, and computer program product. Background Art

[0002] Vehicle-road-cloud collaboration refers to a complex system composed of vehicles, other traffic participants, roadside infrastructure, cloud control platform, related support platform and communication network. This system can realize information interaction and collaboration between vehicles, vehicles and roadsides, and vehicles and the cloud, thereby improving the perception, decision-making and execution capabilities of the entire transportation system. Cameras are an important part of roadside infrastructure, and roadside visual target detection is an essential capability for roadside visual perception.

[0003] Roadside visual target detection uses images obtained by roadside cameras as input to detect target information within the road section, including 3D position, 3D size, heading angle, category and other information of vehicles, pedestrians, cyclists, etc., to support decision-making of road autonomous driving vehicles and the entire transportation system.

[0004] Existing roadside visual target detection solutions mainly detect 2D target information based on the traditional visual 2D target detection model, and then realize the 3D position recognition of the target in combination with the fixed position and posture of the roadside camera, or directly use the 3D target detection model for detection. However, these methods have problems such as inaccurate detection of information such as the target's 3D position and heading angle. Summary of the invention

[0005] The embodiments of the present application provide a target detection method, device, electronic device, and computer program product to improve the accuracy of roadside 3D target detection.

[0006] The present application embodiment adopts the following technical solutions:

[0007] In a first aspect, an embodiment of the present application provides a target detection method, wherein the target detection method includes:

[0008] Acquire a road image collected by a roadside device, and use a preset 3D target detection model to perform target detection on the road image to obtain a target detection result, wherein the target detection result includes 3D target frame information and 2D target frame information;

[0009] Determine a 3D position of a target according to the 3D target frame information and the 2D target frame information;

[0010] According to the road image and the 2D target frame information, the target key points are detected using a preset key point detection model to obtain a target key point detection result;

[0011] The heading angle information of the target is determined according to the 3D position of the target and the target key point detection result.

[0012] Optionally, the preset 3D object detection model is trained in the following manner:

[0013] Obtain a first training sample image for training a 3D object detection model;

[0014] Using the first training sample image to train the 3D object detection model to obtain the preset 3D object detection model;

[0015] Among them, the first training sample image is annotated with 3D target frame information and 2D target frame information, the 3D target frame information includes the offset of the projection point position of the center point of the 3D target frame on the first training sample image, and the 2D target frame information includes the width and height of the 2D target frame determined based on the projection point position after the offset is corrected.

[0016] Optionally, the 3D target frame information includes 3D bounding box information output by a 3D bounding box regression branch, the 3D bounding box information includes an offset of a projection point position of a center point of the 3D target frame on a road image, the 2D target frame information includes a projection point position of a center point of a 3D target frame output by a key point classification branch of a preset 3D target detection model on the road image, and determining the 3D position of the target according to the 3D target frame information and the 2D target frame information includes:

[0017] Correcting the projection point position of the center point of the 3D target frame on the road image by using the offset to obtain a corrected projection point position of the center point of the 3D target frame on the road image;

[0018] According to the preset mapping relationship between pixel points and three-dimensional points, the corrected projection point position of the center point of the 3D target frame on the road image is mapped to a three-dimensional point position to obtain the 3D position of the target.

[0019] Optionally, the 2D target frame information further includes a width and a height of the 2D target frame determined based on the projection point position after the offset correction, and the target key points are detected using a preset key point detection model according to the road image and the 2D target frame information to obtain the target key point detection result, including:

[0020] Cutting out the road image according to the width and height of the 2D target frame determined based on the projection point position after the offset correction to obtain a target area image;

[0021] The preset key point detection model is used to perform key point detection on the target area image to obtain a target key point detection result.

[0022] Optionally, the target key point detection result includes the projection point position of the key point of the 3D target frame on the road image, and determining the heading angle information of the target according to the 3D position of the target and the target key point detection result includes:

[0023] According to a preset mapping relationship between pixel points and three-dimensional points, the projection point positions of the key points of the 3D target frame on the road image are mapped to three-dimensional point positions to obtain the 3D positions of the key points of the 3D target frame;

[0024] The heading angle information of the target is determined according to the 3D position of the target and the 3D position of the key point of the 3D target frame.

[0025] Optionally, the key points of the 3D target frame include the center points of the front and / or back bottom edges of the 3D target frame, and determining the heading angle information of the target according to the 3D position of the target and the 3D position of the key points of the 3D target frame includes:

[0026] If the 3D positions of the key points of the 3D target frame include the 3D positions of the center points of the front and back bottom edges of the 3D target frame, determining the heading angle information of the target according to the 3D positions of the center points of the front and back bottom edges of the 3D target frame;

[0027] If the 3D position of the key point of the 3D target frame only includes the 3D position of the center point of the front bottom edge of the 3D target frame, determining the heading angle information of the target according to the 3D position of the target and the 3D position of the center point of the front bottom edge of the 3D target frame;

[0028] If the 3D position of the key point of the 3D target frame only includes the 3D position of the center point of the back bottom edge of the 3D target frame, the heading angle information of the target is determined according to the 3D position of the target and the 3D position of the center point of the back bottom edge of the 3D target frame.

[0029] Optionally, the preset key point detection model is trained in the following manner:

[0030] Obtain a second training sample image for training a key point detection model, wherein positions of visible key points are marked in the second training sample image;

[0031] The key point detection model is trained using the training sample images to obtain the preset key point detection model.

[0032] In a second aspect, an embodiment of the present application further provides a target detection device, wherein the target detection device comprises:

[0033] A first detection unit is used to obtain a road image collected by a roadside device, and perform target detection on the road image using a preset 3D target detection model to obtain a target detection result, wherein the target detection result includes 3D target frame information and 2D target frame information;

[0034] A first determining unit, configured to determine a 3D position of a target according to the 3D target frame information and the 2D target frame information;

[0035] A second detection unit is used to detect target key points using a preset key point detection model according to the road image and the 2D target frame information to obtain a target key point detection result;

[0036] The second determination unit is used to determine the heading angle information of the target according to the 3D position of the target and the target key point detection result.

[0037] In a third aspect, an embodiment of the present application further provides an electronic device, including:

[0038] Processor; and

[0039] A memory arranged to store computer executable instructions which, when executed, cause the processor to perform any of the methods described above.

[0040] In a fourth aspect, an embodiment of the present application further provides a computer program product, comprising a computer program or instructions, which, when executed by a processor, implements any of the aforementioned target detection methods.

[0041] At least one of the above-mentioned technical solutions adopted in the embodiments of the present application can achieve the following beneficial effects: the target detection method in the embodiments of the present application first obtains a road image collected by a roadside device, and uses a preset 3D target detection model to perform target detection on the road image to obtain a target detection result, and the target detection result includes 3D target frame information and 2D target frame information; then, the 3D position of the target is determined according to the 3D target frame information and the 2D target frame information; then, according to the road image and the 2D target frame information, the target key points are detected using a preset key point detection model to obtain the target key point detection result; finally, the heading angle information of the target is determined according to the 3D position of the target and the target key point detection result. The target detection method in the embodiment of the present application performs target detection on images collected by roadside equipment based on the 3D target detection model adopted by the vehicle side. On the one hand, the accurate target 3D position can be calculated based on the detected 3D target frame information and 2D target frame information. On the other hand, the preset key point detection model is used to further identify the specific key point information of the target, and the heading angle of the target is calculated in combination with the target 3D position and the specific key point information of the target, which can avoid the problem of reverse misidentification of the heading angle, thereby improving the accuracy and reliability of the target heading angle calculation. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0043] Figure 1 It is a schematic diagram of a target detection effect in the prior art;

[0044] Figure 2 A schematic diagram of a target detection method in an embodiment of the present application;

[0045] Figure 3 A schematic diagram of the architecture of a 3D target detection model in an embodiment of the present application;

[0046] Figure 4 A schematic diagram of the prediction of the width and height of a 2D target frame in an embodiment of the present application;

[0047] Figure 5 This is a schematic diagram of the structure of a target detection device in an embodiment of the present application;

[0048] Figure 6 This is a schematic diagram of the structure of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0050] The technical solutions provided by various embodiments of the present application are described in detail below in conjunction with the accompanying drawings.

[0051] To facilitate the understanding of the embodiments of the present application, two traditional roadside visual target detection solutions are first introduced:

[0052] The first solution is to first use the visual 2D target detection model to detect the 2D box of the target in the image, and then use the midpoint of the bottom edge of the 2D box as the location point of the target. Since the position and posture of the roadside camera are fixed, the mapping relationship between the pixel point and the 3D point can be calculated in advance through the internal and external parameters of the camera. The 3D coordinates of the location point can be found using this mapping relationship, and finally the 3D position recognition of the target is achieved by translating it on the plane by half the width or length.

[0053] However, this solution has at least the following technical problems: 1) The 2D target detection model cannot identify the 3D size and heading angle of the target and can only be inferred through multi-frame motion information; 2) When the target orientation is not parallel and perpendicular to the bottom edge of the image, the position point found is inaccurate, which will lead to inaccurate 3D position recognition. Figure 1 As shown, a schematic diagram of a target detection effect in the prior art is provided. For the car in the upper left corner, its actual position point is near the white point, while the position point identified by the above scheme is near the black point.

[0054] The second solution is to apply the vehicle-side 3D target detection model to the roadside. The vehicle-side 3D target detection model usually predicts the 3D coordinates of the target's 3D box center point, the size of the 3D box, the heading angle, category, etc. However, since distance prediction in visual 3D target detection is an ill-posed problem, that is, predicting three-dimensional information from a two-dimensional image is actually a statistical prediction based on the size of the vehicle in the image, the accuracy and generalization of this solution are not high. In addition, although the heading angle prediction of this solution is relatively accurate in most cases, it may sometimes cause 180-degree reverse misidentification.

[0055] Based on this, the present application embodiment provides a target detection method, such as Figure 2 As shown, a flow chart of a target detection method in an embodiment of the present application is provided, and the target detection method at least includes the following steps S210 to S240:

[0056] Step S210, obtaining a road image collected by a roadside device, and performing target detection on the road image using a preset 3D target detection model to obtain a target detection result, wherein the target detection result includes 3D target frame information and 2D target frame information.

[0057] The target detection method of the embodiment of the present application is applied to the roadside. When performing target detection, it is necessary to first obtain the road image captured by the roadside camera, and then use the preset 3D target detection model to detect the target in the road image to obtain the target detection result including 3D target frame information and 2D target frame information.

[0058] The preset target detection model of the embodiment of the present application can be modified based on the 3D target detection model on the vehicle side. The 3D target detection model on the vehicle side can adopt the SMOKE (Single-Stage Monocular 3D Object Detection via Keypoint Estimation) model, for example. The SMOKE model is an end-to-end monocular 3D target detection model. It abandons the two-stage method of performing 2D detection and then 3D detection in the traditional method. It believes that 2D detection is redundant for monocular 3D detection tasks and may introduce noise to affect 3D detection performance. Therefore, SMOKE directly implements 3D target detection through key point prediction and 3D box regression. Of course, the specific 3D target detection model can be flexibly selected by technicians in this field in combination with the existing technology, and no specific limitation is made here.

[0059] The purpose of modifying the original 3D target detection model on the vehicle side in the embodiment of the present application is, on the one hand, to focus on the target information with higher accuracy output by the 3D target detection model, and on the other hand, to enable the model to output more other information of the target, so as to provide a basis for the subsequent calculation of accurate target heading angle.

[0060] Step S220 , determining the 3D position of the target according to the 3D target frame information and the 2D target frame information.

[0061] The preset target detection model of the embodiment of the present application can directly output 3D target frame information and 2D target frame information. Here, the 3D target frame information may include, for example, information such as the offset of the projection point position of the center point of the 3D target frame on the road image, and the 2D target frame information may include information such as the projection point position of the center point of the 3D target frame on the road image. Therefore, by combining the projection point position of the center point of the 3D target frame on the road image detected by the model, that is, the corresponding offset information, and the characteristics that the position and posture of the roadside camera are basically fixed, the accurate 3D position of the target can be calculated.

[0062] Step S230 , detecting target key points using a preset key point detection model according to the road image and the 2D target frame information to obtain target key point detection results.

[0063] Based on the above 2D target frame information, the embodiment of the present application further uses a pre-trained key point detection model to detect specific target key points in the road image, thereby obtaining target key point detection results. The preset key point detection model here can be trained based on an existing human key point detection model such as MoveNet. The specific training method can be flexibly determined by those skilled in the art in combination with existing technologies, and is not specifically limited here.

[0064] The target key points of the embodiment of the present application are mainly used for the subsequent calculation of the target's heading angle. Therefore, when defining the target key points specifically, they can be set in combination with the target's orientation characteristics and the calculation requirements of the heading angle in the roadside visual target detection scenario. You can also consider whether the set target key points are easy to mark and whether they can meet the requirements of accurate detection.

[0065] Step S240, determining the heading angle information of the target according to the 3D position of the target and the target key point detection result.

[0066] The target key point detection result obtained in the above steps is equivalent to further providing the position information of other key points of the target based on the 3D center point position of the target. Therefore, the heading angle of the target can be calculated by combining the 3D center point position of the target and the position information of other key points of the target. Since the selected key points have obvious characteristics and can be accurately detected, this method can avoid the problem of reverse misidentification of the heading angle of 180 degrees.

[0067] The target detection method in the embodiment of the present application performs target detection on images collected by roadside equipment based on the 3D target detection model adopted by the vehicle side. On the one hand, the accurate target 3D position can be calculated based on the detected 3D target frame information and 2D target frame information. On the other hand, the preset key point detection model can be used to further identify the specific key point information of the target. The heading angle of the target is calculated in combination with the target 3D position and the specific key point information of the target, which can avoid the problem of reverse misidentification of the heading angle, thereby improving the accuracy and reliability of the target heading angle calculation.

[0068] In some embodiments of the present application, the preset 3D target detection model is trained in the following manner: obtaining a first training sample image for training the 3D target detection model; using the first training sample image to train the 3D target detection model to obtain the preset 3D target detection model; wherein the first training sample image is annotated with 3D target frame information and 2D target frame information, the 3D target frame information includes the offset of the projection point position of the center point of the 3D target frame on the first training sample image, and the 2D target frame information includes the width and height of the 2D target frame determined based on the projection point position after the offset is corrected.

[0069] When training a preset 3D target detection model, the embodiment of the present application needs to first annotate the training sample images. When annotating the true value information of the image, the requirement of the embodiment of the present application for outputting specific information needs to be considered.

[0070] Taking the SMOKE 3D object detection model on the vehicle side as an example, the network structure of the SMOKE model is relatively simple, mainly consisting of a backbone network (Backbone), a key point classification branch (Keypoint Classification) and a 3D box regression branch (3D BoxRegression). Figure 3 As shown, a schematic diagram of the architecture of a 3D target detection model in an embodiment of the present application is provided.

[0071] Backbone uses the DLA-34 (Deep Layer Aggregation) structure as the feature extraction network. DLA-34 fuses feature information from different layers through iterative deep aggregation and layered deep aggregation to improve feature expression capabilities. In the SMOKE model, all layered aggregation connections are replaced with deformable convolutional networks (DCNs) to enhance the model's ability to model geometric deformations.

[0072] The key point classification branch is used to predict the key point position. Each target corresponds to a key point on the image. The key point is generally defined as the projection point of the center point of the object's 3D box on the image plane, for example, named (xc, yc). C represents the number of categories. For example, for a model that recognizes car, truck, bus, person, rider, and others, C is 6.

[0073] Based on the known key point positions, the 3D bounding box regression branch further regresses the parameters of the 3D bounding box. These parameters generally include the depth offset of the target, the offset of the key point coordinates, the scaling of the length, width and height, and the vectorized representation of the rotation angle, etc. For example, they can be expressed in the form of the following 8-tuple:

[0074]

[0075] Among them, δ z is the offset prediction of the target distance value z, Indicates x c The offset prediction of Indicates y c The offset prediction, δ h , δ w , δ l They represent the prediction of the height, width and length offset of the target respectively, and sinα and cosα are the prediction of the sine and cosine values ​​of the heading angle.

[0076] Since the 3D object detection model often has a large error in predicting z, and the heading angle prediction may have a 180-degree reverse misidentification. The δ z, sinα and cosα, and on the other hand, the prediction of the width and height of the 2D target box is added.

[0077] Therefore, in the sample annotation stage of the training model, it is possible to omit the z , sinα and cosα, and the true value information of the width and height of the 2D target box are annotated to train the model so that the model can predict the width and height information of the 2D target box.

[0078] It should also be noted that the width and height of the 2D target frame marked in the embodiment of the present application are based on the actual position of the projection point of the bottom center point of the 3D target frame on the image. Figure 4 As shown, a schematic diagram of the prediction of the width and height of a 2D target frame in an embodiment of the present application is provided. The projection point position of the center point of the 3D target frame on the image plane is (x c ,y c ), because the center point of the 3D target box is offset and Therefore, the projection point of the bottom center point of the 3D target box on the image should actually be The actual position of the projected point As a reference, mark the offsets in the four directions of up, down, left and right, namely h1, h2, w1, and w2. Then h1+h2 represents the height of the 2D target box, and w1+w2 represents the width of the 2D target box.

[0079] In some embodiments of the present application, the 3D target frame information includes 3D bounding box information output by a 3D bounding box regression branch, the 3D bounding box information includes an offset of a projection point position of a center point of the 3D target frame on a road image, and the 2D target frame information includes a projection point position of a center point of the 3D target frame on a road image output by a key point classification branch of a preset 3D target detection model, and determining the 3D position of the target based on the 3D target frame information and the 2D target frame information includes: correcting the projection point position of the center point of the 3D target frame on the road image using the offset to obtain a corrected projection point position of the center point of the 3D target frame on the road image; and mapping the corrected projection point position of the center point of the 3D target frame on the road image to a three-dimensional point position according to a preset mapping relationship between pixel points and three-dimensional points to obtain the 3D position of the target.

[0080] The 3D bounding box information used in the embodiment of the present application is mainly the offset of the projection point position of the center point of the 3D target box on the road image. and Using offset and The projection point position (x c ,y c ) to correct the projected point position

[0081] Since the posture and position of the roadside camera are basically fixed, and the road section photographed is basically unchanged, the mapping relationship between pixel points and 3D points can be determined in advance through the internal and external calibration of the camera. This mapping relationship can accurately reflect the 3D coordinates corresponding to each pixel point in the image. By searching and matching in this mapping relationship, the corrected projection point position can be obtained. The corresponding three-dimensional point coordinates are used as the 3D position of the target.

[0082] In some embodiments of the present application, the 2D target frame information also includes the width and height of the 2D target frame determined based on the projection point position after the offset correction, and the target key points are detected according to the road image and the 2D target frame information using a preset key point detection model to obtain the target key point detection result, including: cutting out the road image according to the width and height of the 2D target frame determined based on the projection point position after the offset correction to obtain a target area image; performing key point detection on the target area image using the preset key point detection model to obtain a target key point detection result.

[0083] The 2D target frame information of the embodiment of the present application mainly includes the width and height of the 2D target frame determined based on the projection point position after offset correction, which is equivalent to determining the position and size of the 2D target frame in the road image. Therefore, the embodiment of the present application can use the 2D target frame information to cut out the original road image, thereby obtaining the image area where the 2D target frame is located as the target area image. This method can filter out the interference of information in the image that is not related to each target to a certain extent, thereby improving the accuracy and detection efficiency of subsequent target key point detection.

[0084] The embodiment of the present application adds a key point detection model based on the preset 3D target detection model, which is used to predict the projection point position of the pre-set target key point in the above-mentioned target area image, thereby serving as a basis for the subsequent calculation of the heading angle.

[0085] In some embodiments of the present application, the target key point detection result includes the projection point position of the key point of the 3D target frame on the road image, and determining the heading angle information of the target based on the 3D position of the target and the target key point detection result includes: mapping the projection point position of the key point of the 3D target frame on the road image to a three-dimensional point position according to a preset mapping relationship between pixel points and three-dimensional points to obtain the 3D position of the key point of the 3D target frame; determining the heading angle information of the target based on the 3D position of the target and the 3D position of the key point of the 3D target frame.

[0086] Since the target key point detection results directly output by the key point detection model mainly refer to the projection point positions of the key points of the 3D target frame on the road image, and the three-dimensional point information of the target is required for calculation when calculating the heading angle, the embodiment of the present application can also be based on the preset mapping relationship between the pre-calibrated pixel points and the three-dimensional points, first mapping the projection point positions of the key points of the 3D target frame on the road image to three-dimensional points, thereby obtaining the 3D positions of the key points of the 3D target frame, and then combining the 3D position of the target and the 3D positions of the key points of the 3D target frame to calculate the heading angle information of the target.

[0087] In some embodiments of the present application, the key points of the 3D target frame include the center points of the front and / or back bottom edges of the 3D target frame, and determining the heading angle information of the target according to the 3D position of the target and the 3D position of the key points of the 3D target frame includes: if the 3D position of the key points of the 3D target frame includes the 3D position of the center points of the front and back bottom edges of the 3D target frame, then determining the heading angle information of the target according to the 3D position of the center points of the front and back bottom edges of the 3D target frame; if the 3D position of the key points of the 3D target frame only includes the 3D position of the center point of the front bottom edge of the 3D target frame, then determining the heading angle information of the target according to the 3D position of the target and the 3D position of the center point of the front bottom edge of the 3D target frame; if the 3D position of the key points of the 3D target frame only includes the 3D position of the center point of the back bottom edge of the 3D target frame, then determining the heading angle information of the target according to the 3D position of the target and the 3D position of the center point of the back bottom edge of the 3D target frame.

[0088] In the actual detection stage, due to the uncertainty of the direction of the target relative to the roadside camera, the model may not be able to detect these two key points at the same time, and it is also uncertain which one will be detected. Therefore, the key points of the 3D target frame annotated in the sample annotation stage can simultaneously annotate the center points of the front and back bottom edges of the 3D target frame. In actual detection, the calculation method of the heading angle is determined according to the specific detection results to ensure the accuracy and reliability of the heading angle calculation. The specific calculation methods can be divided into the following three types:

[0089] 1) If the model detects the 3D positions of the center points of the front and back bottom edges of the 3D target box at the same time, indicating that the target is currently facing the camera, the heading angle of the target can be calculated using the vector from the 3D position of the center point of the front bottom edge of the 3D target box to the 3D position of the center point of the back bottom edge;

[0090] 2) If the model only detects the 3D position of the center point of the front bottom edge of the 3D target box, indicating that the target tends to face the camera, the heading angle of the target can be calculated using the vector from the 3D position of the center point of the bottom surface of the target to the 3D position of the center point of the front bottom edge of the 3D target box;

[0091] 3) If the 3D position of the key point of the 3D target frame only includes the 3D position of the center point of the back bottom edge of the 3D target frame, indicating that the target tends to face away from the camera, the heading angle of the target can be calculated using a vector from the 3D position of the center point of the bottom surface of the target to the 3D position of the center point of the back bottom edge of the 3D target frame.

[0092] In some embodiments of the present application, the preset key point detection model is trained in the following manner: obtaining a second training sample image for training the key point detection model, the second training sample image having the positions of visible key points marked therein; and using the training sample image to train the key point detection model to obtain the preset key point detection model.

[0093] The key point detection model of the embodiment of the present application can be obtained based on deep learning training, and the specific model structure of the key point detection model can be implemented by referring to the key point detection model of the human body such as MoveNet. For a target, two key points can be defined in advance, including the center points of the front and back bottom edges of the 3D target frame. The reason for selecting these two points is that they are closely related to the calculation of the target heading angle and are convenient for annotation and image feature extraction.

[0094] Taking into account that the two key points in the actual sample image do not necessarily appear on the image at the same time, when annotating the data, only the visible key points in the image are annotated, and the invisible key points are directly ignored. The key point model trained based on this will only output the positions of the visible key points in the image. The advantage of this is that it can make the key points predicted by the model more accurate, avoid the model's guessing about invisible key points, and also improve the efficiency of sample annotation.

[0095] The present application also provides a target detection device 500, such as Figure 5 As shown, a schematic diagram of the structure of a target detection device in an embodiment of the present application is provided, wherein the target detection device 500 includes: a first detection unit 510, a first determination unit 520, a second detection unit 530 and a second determination unit 540, wherein:

[0096] The first detection unit 510 is used to obtain a road image collected by a roadside device, and perform target detection on the road image using a preset 3D target detection model to obtain a target detection result, wherein the target detection result includes 3D target frame information and 2D target frame information;

[0097] A first determining unit 520, configured to determine a 3D position of a target according to the 3D target frame information and the 2D target frame information;

[0098] A second detection unit 530 is used to detect target key points using a preset key point detection model according to the road image and the 2D target frame information to obtain a target key point detection result;

[0099] The second determination unit 540 is used to determine the heading angle information of the target according to the 3D position of the target and the target key point detection result.

[0100] In some embodiments of the present application, the preset 3D target detection model is trained in the following manner: obtaining a first training sample image for training the 3D target detection model; using the first training sample image to train the 3D target detection model to obtain the preset 3D target detection model; wherein the first training sample image is annotated with 3D target frame information and 2D target frame information, the 3D target frame information includes the offset of the projection point position of the center point of the 3D target frame on the first training sample image, and the 2D target frame information includes the width and height of the 2D target frame determined based on the projection point position after the offset is corrected.

[0101] In some embodiments of the present application, the 3D target box information includes 3D bounding box information output by a 3D bounding box regression branch, the 3D bounding box information includes an offset of a projection point position of a center point of the 3D target box on a road image, and the 2D target box information includes a projection point position of a center point of the 3D target box on a road image output by a key point classification branch of a preset 3D target detection model. The first determination unit 520 is specifically used to: correct the projection point position of the center point of the 3D target box on the road image using the offset to obtain a corrected projection point position of the center point of the 3D target box on the road image; and map the corrected projection point position of the center point of the 3D target box on the road image to a three-dimensional point position according to a preset mapping relationship between pixel points and three-dimensional points to obtain the 3D position of the target.

[0102] In some embodiments of the present application, the 2D target frame information also includes the width and height of the 2D target frame determined based on the projection point position after the offset correction, and the second detection unit 530 is specifically used to: cut out the road image according to the width and height of the 2D target frame determined based on the projection point position after the offset correction to obtain a target area image; perform key point detection on the target area image using the preset key point detection model to obtain a target key point detection result.

[0103] In some embodiments of the present application, the target key point detection result includes the projection point position of the key point of the 3D target frame on the road image, and the second determination unit 540 is specifically used to: map the projection point position of the key point of the 3D target frame on the road image to a three-dimensional point position according to a preset mapping relationship between pixel points and three-dimensional points, and obtain the 3D position of the key point of the 3D target frame; determine the heading angle information of the target according to the 3D position of the target and the 3D position of the key point of the 3D target frame.

[0104] In some embodiments of the present application, the key points of the 3D target frame include the center points of the front and / or back bottom edges of the 3D target frame, and the second determination unit 540 is specifically used to: if the 3D positions of the key points of the 3D target frame include the 3D positions of the center points of the front and back bottom edges of the 3D target frame, then determine the heading angle information of the target according to the 3D positions of the center points of the front and back bottom edges of the 3D target frame; if the 3D positions of the key points of the 3D target frame only include the 3D position of the center point of the front bottom edge of the 3D target frame, then determine the heading angle information of the target according to the 3D position of the target and the 3D position of the center point of the front bottom edge of the 3D target frame; if the 3D positions of the key points of the 3D target frame only include the 3D position of the center point of the back bottom edge of the 3D target frame, then determine the heading angle information of the target according to the 3D position of the target and the 3D position of the center point of the back bottom edge of the 3D target frame.

[0105] In some embodiments of the present application, the preset key point detection model is trained in the following manner: obtaining a second training sample image for training the key point detection model, the second training sample image having the positions of visible key points marked therein; and using the training sample image to train the key point detection model to obtain the preset key point detection model.

[0106] It can be understood that the above-mentioned target detection device can implement each step of the target detection method provided in the above-mentioned embodiment, and the relevant explanations about the target detection method are applicable to the target detection device and will not be repeated here.

[0107] Figure 6 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present application. Figure 6 At the hardware level, the electronic device includes a processor, and optionally also includes an internal bus, a network interface, and a memory. The memory may include a memory, such as a high-speed random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk storage. Of course, the electronic device may also include hardware required for other services.

[0108] The processor, network interface and memory can be interconnected through an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0109] The memory is used to store the program. Specifically, the program may include a program code, and the program code includes a computer operation instruction. The memory may include a memory and a non-volatile memory, and provides instructions and data to the processor.

[0110] The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it, forming a target detection device at the logical level. The processor executes the program stored in the memory and is specifically used to perform the following operations:

[0111] Acquire a road image collected by a roadside device, and use a preset 3D target detection model to perform target detection on the road image to obtain a target detection result, wherein the target detection result includes 3D target frame information and 2D target frame information;

[0112] Determine a 3D position of a target according to the 3D target frame information and the 2D target frame information;

[0113] According to the road image and the 2D target frame information, the target key points are detected using a preset key point detection model to obtain a target key point detection result;

[0114] The heading angle information of the target is determined according to the 3D position of the target and the target key point detection result.

[0115] The above application Figure 1The method performed by the target detection device disclosed in the illustrated embodiment can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor or an instruction in the form of software. The above processor may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in conjunction with the embodiments of the present application can be directly embodied as being executed by a hardware decoding processor, or executed by a combination of hardware and software modules in a decoding processor. The software module can be located in a storage medium mature in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0116] The present application also provides a computer-readable storage medium, which stores one or more programs, wherein the one or more programs include instructions, which, when executed by an electronic device including multiple application programs, enable the electronic device to execute Figure 1 The method performed by the target detection device in the illustrated embodiment is specifically used to perform:

[0117] Acquire a road image collected by a roadside device, and use a preset 3D target detection model to perform target detection on the road image to obtain a target detection result, wherein the target detection result includes 3D target frame information and 2D target frame information;

[0118] Determine a 3D position of a target according to the 3D target frame information and the 2D target frame information;

[0119] According to the road image and the 2D target frame information, the target key points are detected using a preset key point detection model to obtain a target key point detection result;

[0120] The heading angle information of the target is determined according to the 3D position of the target and the target key point detection result.

[0121] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0122] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0123] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0124] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0125] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0126] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0127] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0128] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0129] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0130] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A target detection method, wherein: The target detection method comprises: Acquire a road image collected by a roadside device, and use a preset 3D target detection model to perform target detection on the road image to obtain a target detection result, wherein the target detection result includes 3D target frame information and 2D target frame information; Determine a 3D position of a target according to the 3D target frame information and the 2D target frame information; According to the road image and the 2D target frame information, the target key points are detected using a preset key point detection model to obtain a target key point detection result; The heading angle information of the target is determined according to the 3D position of the target and the target key point detection result.

2. The target detection method according to claim 1, wherein: The preset 3D object detection model is trained in the following way: Obtain a first training sample image for training a 3D object detection model; Using the first training sample image to train the 3D object detection model to obtain the preset 3D object detection model; Among them, the first training sample image is annotated with 3D target frame information and 2D target frame information, the 3D target frame information includes the offset of the projection point position of the center point of the 3D target frame on the first training sample image, and the 2D target frame information includes the width and height of the 2D target frame determined based on the projection point position after the offset is corrected.

3. The target detection method according to claim 1, wherein: The 3D target frame information includes the 3D bounding box information output by the 3D bounding box regression branch, the 3D bounding box information includes the offset of the projection point position of the center point of the 3D target frame on the road image, the 2D target frame information includes the projection point position of the center point of the 3D target frame output by the key point classification branch of the preset 3D target detection model on the road image, and determining the 3D position of the target according to the 3D target frame information and the 2D target frame information includes: Correcting the projection point position of the center point of the 3D target frame on the road image by using the offset to obtain a corrected projection point position of the center point of the 3D target frame on the road image; According to the preset mapping relationship between pixel points and three-dimensional points, the corrected projection point position of the center point of the 3D target frame on the road image is mapped to a three-dimensional point position to obtain the 3D position of the target.

4. The target detection method according to claim 3, wherein: The 2D target frame information also includes a width and a height of the 2D target frame determined based on the projection point position after the offset correction. The target key points are detected using a preset key point detection model according to the road image and the 2D target frame information to obtain the target key point detection result, including: Cutting out the road image according to the width and height of the 2D target frame determined based on the projection point position after the offset correction to obtain a target area image; The preset key point detection model is used to perform key point detection on the target area image to obtain a target key point detection result.

5. The target detection method according to claim 1, wherein: The target key point detection result includes the projection point position of the key point of the 3D target frame on the road image, and the determining the heading angle information of the target according to the 3D position of the target and the target key point detection result includes: According to a preset mapping relationship between pixel points and three-dimensional points, the projection point positions of the key points of the 3D target frame on the road image are mapped to three-dimensional point positions to obtain the 3D positions of the key points of the 3D target frame; The heading angle information of the target is determined according to the 3D position of the target and the 3D position of the key point of the 3D target frame.

6. The target detection method according to claim 5, wherein: The key points of the 3D target frame include the center points of the front and / or back bottom edges of the 3D target frame, and the determining of the heading angle information of the target according to the 3D position of the target and the 3D position of the key points of the 3D target frame includes: If the 3D positions of the key points of the 3D target frame include the 3D positions of the center points of the front and back bottom edges of the 3D target frame, determining the heading angle information of the target according to the 3D positions of the center points of the front and back bottom edges of the 3D target frame; If the 3D position of the key point of the 3D target frame only includes the 3D position of the center point of the front bottom edge of the 3D target frame, determining the heading angle information of the target according to the 3D position of the target and the 3D position of the center point of the front bottom edge of the 3D target frame; If the 3D position of the key point of the 3D target frame only includes the 3D position of the center point of the back bottom edge of the 3D target frame, the heading angle information of the target is determined according to the 3D position of the target and the 3D position of the center point of the back bottom edge of the 3D target frame.

7. The target detection method according to any one of claims 1 to 6, wherein: The preset key point detection model is trained in the following way: Obtain a second training sample image for training a key point detection model, wherein positions of visible key points are marked in the second training sample image; The key point detection model is trained using the training sample images to obtain the preset key point detection model.

8. A target detection device, wherein: The target detection device comprises: A first detection unit is used to obtain a road image collected by a roadside device, and perform target detection on the road image using a preset 3D target detection model to obtain a target detection result, wherein the target detection result includes 3D target frame information and 2D target frame information; A first determining unit, configured to determine a 3D position of a target according to the 3D target frame information and the 2D target frame information; A second detection unit is used to detect target key points using a preset key point detection model according to the road image and the 2D target frame information to obtain a target key point detection result; The second determination unit is used to determine the heading angle information of the target according to the 3D position of the target and the target key point detection result.

9. An electronic device, comprising: processor; as well as A memory arranged to store computer executable instructions, wherein when the executable instructions are executed, the processor performs the target detection method according to any one of claims 1 to 7.

10. A computer program product, comprising a computer program or instructions, wherein when the computer program or instructions are executed by a processor, the object detection method according to any one of claims 1 to 7 is implemented.

Images