3D target detection method based on YOLOv7-pose

Abstract

The embodiment of the invention provides a 3D target detection method based on YOLOv7-pose. The technical problem that existing 3D target detection is poor in detection capacity can be solved. The method comprises the following steps: acquiring first image data; generating a target detection data set based on the first image data; performing three-dimensional detection frame labeling on each image of the target detection data set to obtain a label file; modifying the format of the label file according to a preset storage format to obtain a label conversion file; uploading the label conversion file to a labeling platform, and modifying the visibility of each key point in the plurality of key points to obtain a key point set; determining eight key points of the key point set based on a key point detection algorithm, and training a key point detection model in combination with the first image data and the label conversion file; inputting a to-be-detected image into the key point detection model to obtain coordinate information of eight target key points; and connecting the eight target key points based on a preset connection sequence and the coordinate information. In this way, rapid and high-precision target detection is realized.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a 3D target detection method based on YOLOv7-pose. Background Art

[0002] In the field of computer vision, target detection algorithms are widely used in video surveillance in traffic, security and other scenarios. Target detection algorithms are used to locate targets and determine target categories, and are the basis for downstream tasks such as target tracking and behavior recognition. Target detection algorithms include two-dimensional (2D) target detection algorithms and three-dimensional (3D) target detection algorithms. 2D target detection algorithms predict the 2D detection box of the target based on the RGB image, that is, output the pixel coordinates of the upper left corner or center point of the detection box as well as the width and height. When the targets to be detected are dense and the detection task has high requirements for target positioning accuracy, the results of the 2D target detection algorithm are difficult to meet expectations. Compared with the 2D target detection algorithm, the 3D target detection algorithm has higher accuracy.

[0003] At present, 3D object detection algorithms can be roughly divided into two categories according to the different sensor inputs, namely multimodal 3D object detection algorithms and monocular vision 3D object detection algorithms. Multimodal 3D object detection algorithms obtain point cloud data through LiDAR and realize 3D object detection in combination with images taken by camera sensors (for example, RGB format). Monocular vision 3D object detection algorithms realize 3D object detection on 2D images only through camera sensors, which is low-cost and easy to install.

[0004] However, multimodal 3D target detection algorithms cannot be deployed on a large scale because LiDAR is expensive and easily affected by extreme weather, which in turn affects detection accuracy. Monocular vision 3D target detection algorithms usually use a heatmap-based approach to predict the coordinates of the target's eight key points, that is, for each key point, the score at each position on the feature map is predicted, and the coordinate position of each key point on the original image is obtained through complex post-processing calculations. This type of approach takes a long time in the model reasoning and post-processing stages and cannot meet the real-time requirements of practical applications. Therefore, how to achieve fast and high-precision target detection has become an urgent problem to be solved. Summary of the invention

[0005] The embodiment of the present application provides a 3D object detection method based on YOLOv7-pose, which can solve the technical problem of poor detection capability of existing 3D object detection.

[0006] To achieve the above objectives, the embodiments of the present application adopt the following technical solutions:

[0007] In a first aspect, an embodiment of the present application provides a 3D target detection method based on YOLOv7-pose, and the 3D target detection method based on YOLOv7-pose includes: collecting first image data; the first image data includes an image captured by a camera sensor; based on the first image data, generating a target detection data set; the target detection data set includes multiple key points; annotating each image of the target detection data set with a three-dimensional detection frame to obtain a label file; each image corresponds to a label file; the label file includes a target category index and the absolute coordinates of multiple corner points of the three-dimensional detection frame; according to a preset storage format, modifying the format of the label file to obtain a label conversion file; uploading the label conversion file to a labeling platform, modifying the visibility of each key point in multiple key points, and obtaining a key point set; based on a key point detection algorithm, confirming eight key points of the key point set; based on the eight key points, the first image data and the label conversion file, training a key point detection model; inputting the image to be detected into the key point detection model to obtain coordinate information of eight target key points; connecting the eight target key points based on a preset connection order and coordinate information.

[0008] Based on the above description of the YOLOv7-pose-based 3D target detection method provided in the embodiment of the present application, it can be known that the YOLOv7-pose-based 3D target detection method includes applying the key point detection algorithm to the 3D target detection task, directly predicting the coordinate positions of the 8 corner points of the 3D detection frame based on regression, and having good accuracy and real-time performance. The key point detection model based on regression realizes 3D target detection with good accuracy and real-time performance, and only requires a camera sensor to complete the deployment, which is low-cost and convenient for large-scale application. Secondly, the key point label file in the model training data set is generated by the Cubiod cube annotation tool, which can fit the edge of the target and highlight the perspective relationship of the target. In the reasoning application stage, a standard 3D detection frame can be output, which provides accurate target positioning for downstream tasks such as target tracking.

[0009] In a feasible implementation of the first aspect, when performing the step of annotating three-dimensional detection frames for each image of the target detection data set to obtain a label file, the three-dimensional detection frame includes two front and back rectangular frames, and the 3D target detection method based on YOLOv7-pose also includes: based on the target size, adjusting the size of each of the two rectangular frames, and adjusting the relative position between the two rectangular frames to fit the edge of the target and highlight the perspective relationship of the target.

[0010] In a feasible implementation of the first aspect, the x-axis coordinates of two upper and lower key points on one side of each rectangular frame are the same.

[0011] In this way, the upper and lower bottom surfaces of the 3D detection frame can be guaranteed to be the same size and parallel. The advantage of this is that it can limit a certain degree of freedom while ensuring that the 3D detection frame can fit the target, ensuring that the labels of different targets are consistent. If key point labeling is directly used, it will bring about relative errors between key points, making data labeling difficult and not conducive to subsequent model training.

[0012] In a feasible implementation of the first aspect, when executing the step of modifying the format of the label file according to the preset storage format to obtain the label conversion file, the preset storage format is the txt format, and the 3D target detection method based on YOLOv7-pose also includes: determining the first relative coordinate and the second relative coordinate based on the absolute coordinate; the first relative coordinate is the relative coordinate of the target two-dimensional rectangular box relative to the image; the target two-dimensional rectangular box is the circumscribed rectangle of multiple corner points; the second relative coordinate is the relative coordinate of the key point relative to the image.

[0013] In a feasible implementation of the first aspect, the key point detection model includes a backbone network and a detection head; the backbone network includes a feature extraction module and a feature fusion module; the feature fusion module includes a spatial pyramid pooling sub-module and a path aggregation network sub-module; the detection head includes a two-dimensional target detection module and a key point detection module; the key point detection module is used to confirm the key points; the 3D target detection method based on YOLOv7-pose also includes: adding an object key point similarity loss function, and updating the model parameters through a back propagation mechanism.

[0014] In a feasible implementation of the first aspect, the YOLOv7-pose-based 3D target detection method further includes: for each key point, learning a confidence parameter.

[0015] This way, it is possible to show whether this key point exists in the target.

[0016] In a feasible implementation of the first aspect, the image to be detected includes multiple targets to be detected; when executing the step of inputting the image to be detected into the key point detection model to obtain the coordinate information of eight target key points; the 3D target detection method based on YOLOv7-pose also includes: confirming the category index and key point relative coordinates of each target to be detected; mapping the key point relative coordinates to the image to obtain the key point absolute coordinates.

[0017] In a feasible implementation of the first aspect, the calculation formula of the confidence parameter includes:

[0018]

[0019] in, is the key point confidence loss; is the confidence of the nth predicted key point; δ(vn ) represents the visibility of the key point; BCE is the binary cross entropy loss, which is used to calculate the binary cross entropy loss between the predicted probability and the true label.

[0020] In a second aspect, an embodiment of the present application provides a 3D target detection system based on YOLOv7-pose, and the 3D target detection system based on YOLOv7-pose includes: at least one processor; a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method provided in the first aspect.

[0021] The 3D target detection system based on YOLOv7-pose applies the key point detection algorithm to the 3D target detection task by executing the method provided in the first aspect, and directly predicts the coordinate positions of the 8 corner points of the 3D detection box based on regression, with good accuracy and real-time performance. The key point detection model based on regression realizes 3D target detection with good accuracy and real-time performance. At the same time, it only requires camera sensors to complete the deployment, which is low-cost and convenient for large-scale application. Secondly, the key point label file in the model training data set is generated by the Cubiod cube annotation tool, which can fit the edge of the target and highlight the perspective relationship of the target. In the reasoning application stage, it can output a standard 3D detection box, which provides accurate target positioning for downstream tasks such as target tracking.

[0022] In a third aspect, an embodiment of the present application provides a computer-readable medium having computer program instructions stored thereon, and the computer program instructions can be executed by a processor to implement the method provided in the first aspect.

[0023] The computer program instructions in the computer-readable medium implement the method provided in the first aspect, apply the key point detection algorithm to the 3D target detection task, and directly predict the coordinate positions of the 8 corner points of the 3D detection frame based on regression, with good accuracy and real-time performance. The regression-based key point detection model realizes 3D target detection with good accuracy and real-time performance. At the same time, it only requires camera sensors to complete the deployment, which is low-cost and convenient for large-scale application. Secondly, the key point label file in the model training data set is generated by the Cubiod cube annotation tool, which can fit the edge of the target and highlight the perspective relationship of the target. In the reasoning application stage, it can output a standard 3D detection frame, which provides accurate target positioning for downstream tasks such as target tracking. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 A schematic diagram of the structure of a 3D target detection system based on YOLOv7-pose provided in an embodiment of the present application;

[0025] Figure 2 A schematic diagram of a 3D target detection method based on YOLOv7-pose provided in an embodiment of the present application;

[0026] Figure 3 A schematic diagram of a 3D target detection method based on YOLOv7-pose provided in an embodiment of the present application;

[0027] Figure 4 The present invention provides a target detection result of a 3D target detection method based on YOLOv7-pose. DETAILED DESCRIPTION

[0028] The technical solutions in the embodiments of the present invention will be described below in conjunction with the accompanying drawings in the embodiments of the present invention. In the description of the embodiments of the present invention, unless otherwise specified, "multiple" refers to two or more than two. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0029] In addition, in order to facilitate the clear description of the technical solutions of the embodiments of the present invention, in the embodiments of the present invention, the words "first", "second" and the like are used to distinguish the same items or similar items with substantially the same functions and effects. Those skilled in the art will understand that the words "first", "second" and the like do not limit the quantity and execution order, and the words "first", "second" and the like do not necessarily limit the difference. At the same time, in the embodiments of the present invention, the words "exemplary" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or design. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a concrete way for easy understanding.

[0030] The principles and features of the present application are described below. The examples given are only used to explain the present application and are not used to limit the scope of the present application.

[0031] Key point detection algorithm is also a common detection method in the field of computer vision, usually used in human posture detection or face key point detection tasks. In addition to the above-mentioned key point detection algorithm based on heat map, the regression-based method is also widely used in key point detection algorithm, which is characterized by directly predicting the coordinate position of each key point and having good real-time performance.

[0032] At the same time, 3D object detection has also been applied to fields such as autonomous driving and robotics to achieve the positioning (such as target length, width, and depth information) and recognition of targets in the real-world 3D coordinate system, and is a hot research direction in the field of computer vision.

[0033] YOLOv7 pose estimation (YOLOv7-pose) is a target detection and pose estimation model based on the YOLOv7 architecture. It inherits the efficient target detection capability of YOLOv7 and integrates the key point detection function, which can quickly and accurately predict the pose of objects, such as key points of the human body or feature points of other specific objects. A major feature of YOLOv7-Pose is its strong real-time performance. It uses the optimized structure of YOLOv7 to minimize computational overhead and maintain a high reasoning speed even in complex scenes. By adding a key point detection branch to the original detection network, YOLOv7-Pose achieves multi-task fusion, which can predict the position of key points while completing the target bounding box detection.

[0034] This model has the characteristics of end-to-end training. Through multi-task loss functions, it optimizes target detection and key point prediction at the same time, which significantly simplifies the training process. In addition, it is widely used in sports motion analysis, motion monitoring and rehabilitation training in health care, behavior recognition in smart security, gesture recognition in human-computer interaction, and motion capture in animation production. The workflow of YOLOv7-Pose starts with the input image, extracts features through the backbone network, generates the target bounding box through the detection head, and predicts the pose in the target area through the key point branch, and finally outputs the category, bounding box and key point coordinates of each target.

[0035] This model is usually implemented based on PyTorch and trained and verified with datasets such as COCO-Keypoints and MPII-Human-Pose. It also supports integration with OpenCV to facilitate actual deployment and testing. It is a powerful tool that is particularly suitable for application scenarios with high requirements for real-time performance and accuracy.

[0036] The embodiment of the present application provides a 3D target detection method based on YOLOv7-pose, which is applicable to various fields that require target detection. The key point detection algorithm is applied to the 3D target detection task, and the coordinate positions of the 8 corner points of the 3D detection box are directly predicted based on regression, with good accuracy and real-time performance. The key point detection model based on regression realizes 3D target detection with good accuracy and real-time performance. At the same time, it only requires a camera sensor to complete the deployment, which is low-cost and convenient for large-scale application. Secondly, the key point label file in the model training data set is generated by the Cubiod cube annotation tool, which can fit the edge of the target and highlight the perspective relationship of the target. In the reasoning application stage, a standard 3D detection box can be output, which provides accurate target positioning for downstream tasks such as target tracking.

[0037] The embodiment of the present application provides a YOLOv7-pose-based 3D target detection system, which can execute the YOLOv7-pose-based 3D target detection method provided in the embodiment of the present application. Figure 1 A schematic diagram of the structure of a 3D target detection system based on YOLOv7-pose provided in an embodiment of the present application.

[0038] like Figure 1 As shown, the YOLOv7-pose-based 3D object detection system 001 includes at least one processor 011 and a memory 012 communicatively connected to the at least one processor; wherein the memory 012 stores instructions executable by the at least one processor 011, and the instructions are executed by the at least one processor 011 so that the at least one processor 011 can execute the YOLOv7-pose-based 3D object detection method provided in an embodiment of the present application.

[0039] Figure 2 A flowchart of a 3D target detection method based on YOLOv7-pose provided in an embodiment of the present application. Figure 2 As shown, in some embodiments, the YOLOv7-pose-based 3D target detection method includes the following steps:

[0040] S1, collecting first image data.

[0041] The first image data includes an image captured by a camera sensor. Exemplarily, image data in RGB format is collected by a vehicle-mounted camera or a roadside camera as the first image data.

[0042] In some embodiments, the multiple images of the first image data may have different scenes, perspectives, etc. In this way, during the training phase of the key point detection model, the detection capability and generalization capability of the model can be improved.

[0043] S2: Generate a target detection data set based on the first image data.

[0044] The object detection dataset includes multiple key points. The object detection dataset is a 3D object detection dataset.

[0045] S3, annotate each image of the target detection dataset with a three-dimensional detection box to obtain a label file.

[0046] Each image corresponds to a label file. For example, in a 3D object detection dataset, each RGB image corresponds to a label file.

[0047] The label file is a text file with the suffix .txt. In the label file, the category index of the object in the corresponding image and the absolute coordinates of the eight corner points of the 3D detection box in the image are recorded line by line.

[0048] Exemplarily, the eight corner points of the 3D detection box are composed of four corner points of the two rectangular boxes in front and behind. The storage format of the label file is cls_id, x1, y1, ..., x8, y8. Among them, cls_id is the target category index, and xi and yi represent the absolute coordinates of the i-th key point in the image.

[0049] In some embodiments, a Cuboid cube annotation tool provided by a data annotation platform is used to annotate a three-dimensional (3D) detection frame for the target to be detected on the image. The target to be detected may be a pre-selected target, such as Figure 4 Vehicle shown.

[0050] The label file includes the target category index and the absolute coordinates of multiple corner points of the 3D detection box.

[0051] In some embodiments, the three-dimensional detection frame includes two front and rear rectangular frames. When executing step S3, the 3D object detection method based on YOLOv7-pose further includes:

[0052] S311, based on the size of the target, adjusting the size of each of the two rectangular frames, and adjusting the relative position between the two rectangular frames to fit the edge of the target and highlight the perspective relationship of the target.

[0053] In one implementation, the x-axis coordinates of two key points at the top and bottom of one side of each rectangular frame are the same.

[0054] The 3D target detection algorithm can predict the coordinate positions of the eight corner points of the target and then obtain a 3D detection frame, which can highlight the target itself and the perspective relationship between different targets on the 2D image, thereby achieving accurate target positioning.

[0055] S4, modifying the format of the label file according to the preset storage format to obtain a label conversion file.

[0056] In some embodiments, the format of the label file is modified to the key point label storage format required by the YOLOv7-pose key point detection algorithm.

[0057] In some embodiments, the three-dimensional detection frame includes two front and rear rectangular frames. When executing step S4, the preset storage format is txt format. The 3D target detection method based on YOLOv7-pose also includes:

[0058] Based on the absolute coordinates, the first relative coordinates and the second relative coordinates are determined. The first relative coordinates are the relative coordinates of the target two-dimensional rectangular frame relative to the image. The target two-dimensional rectangular frame is the circumscribed rectangle of multiple corner points. The second relative coordinates are the relative coordinates of the key point relative to the image.

[0059] For example, the boundingRect function provided by OpenCV is used to calculate the bounding rectangle of the eight corner points of the 3D detection box as the 2D detection box. The key point detection label file is stored in the format of cls_id,x,y,w,h, ......, Where cls_id is the target category index, x, y, w, h are the relative coordinates of the upper left corner of the 2D detection box and the relative width and height, Represents the relative coordinates and visibility of the i-th key point. The key point coordinates are obtained from the coordinates of the eight corner points of the 3D detection box marked in step 2.

[0060] S5, uploading the label conversion file to the annotation platform, modifying the visibility of each key point among the multiple key points, and obtaining a key point set.

[0061] In some embodiments, the number "0" indicates that the visibility of the key point is "invisible". The number "1" indicates that the visibility of the key point is "occluded". The number "2" indicates that the visibility of the key point is "visible". Exemplarily, the visibility of each key point in the target is modified according to the actual situation. For example, when a key point in the target to be labeled is occluded by itself or other targets, the visibility of the key point is marked as 1, that is, "occluded". When the target to be labeled is at the edge of the image and part of the position is truncated, the corresponding key point at this position is marked as 0, that is, "invisible". In other cases, the visibility of the key point is marked as 2, that is, "visible".

[0062] For example, according to the target key points being invisible, occluded, and visible, Scaled to 0, 1, or 2.

[0063] The annotation platform can use the Computer Vision Annotation Tool (CVAT) to annotate images. CVAT is an open source computer vision task data annotation tool that provides users with a simple and intuitive web interface for various types of annotation of images and videos, including target detection, image segmentation, pose estimation, etc. The embodiment of the present application uses the Cubiod and Skeleton annotation tools provided by CVAT to achieve three-dimensional annotation and key point annotation of targets in images.

[0064] S6, based on the key point detection algorithm, confirm the eight key points of the key point set.

[0065] In some embodiments, the YOLOv7-pose key point detection algorithm is used to modify the number of predicted key points to eight, so as to fit the 3D object detection frame.

[0066] S7, training a key point detection model based on the eight key points, the first image data and the label conversion file.

[0067] like Figure 3 As shown, in some embodiments, the key point detection model includes a backbone network and a detection head. The backbone network includes a feature extraction module (Cross Stage Partial Network, CSP) and a feature fusion module. The feature fusion module includes a spatial pyramid pooling submodule (Spatial Pyramid Pooling, SPP) and a path aggregation network submodule (Path Aggregation Network, PAN). The detection head includes a two-dimensional target detection module and a key point detection module. The key point detection module is used to confirm the key points.

[0068] In some embodiments, when executing step S7, the YOLOv7-pose-based 3D object detection method further includes:

[0069] S711, add the object key point similarity loss function and update the model parameters through the back propagation mechanism.

[0070] During the training process, the Object Keypoint Similarity Loss (OKSLoss) is added as a loss function for the key point detection part, and the model parameters are updated through the back propagation mechanism. The calculation formula of OKSLoss is as follows:

[0071]

[0072] in, represents the key point loss; N kpts Indicates the number of key points; d n represents the Euclidean distance between the nth predicted key point and the true label key point; k n represents the weight coefficient of the nth key point; s represents the scale of the target; δ(v n ) indicates the visibility of the key point.

[0073] In the key point OKSLoss, δ(v n ) Only key points with visibility of 1 or 2 are calculated, i.e., key points marked as occluded and visible in step S5.

[0074] In some embodiments, N kpts The value of is 8.

[0075] In some embodiments, when executing step S7, the YOLOv7-pose-based 3D object detection method further includes:

[0076] S721, for each key point, learn a confidence parameter.

[0077] In some embodiments, the calculation formula of the confidence parameter includes:

[0078]

[0079] in, is the key point confidence loss; is the confidence of the nth predicted key point; δ(v n ) represents the visibility of the key point; BCE is the binary cross entropy loss, which is used to calculate the binary cross entropy loss between the predicted probability and the true label.

[0080] S8, input the image to be detected into the key point detection model to obtain the coordinate information of eight target key points.

[0081] In this way, the image to be detected is input into the key point detection model to realize reasoning and obtain coordinate information.

[0082] In some embodiments, when executing step S7, the YOLOv7-pose-based 3D object detection method further includes:

[0083] S811, confirm the category index and key point relative coordinates of each target to be detected.

[0084] S812, mapping the relative coordinates of the key points to the image to obtain the absolute coordinates of the key points.

[0085] S9, connecting eight target key points based on a preset connection sequence and coordinate information.

[0086] In this way, the trained key point detection model can predict the category and eight key point coordinates for each target to be detected in the input image. The key point coordinates are relative coordinates in the image. The relative coordinates are obtained by post-processing methods, such as mapping the relative coordinates to the original image, to obtain the absolute coordinates of the key points in the original input image.

[0087] Exemplarily, the predicted key points are connected according to the connection order of the corner points of the 3D detection box in step 3, and the 3D detection box is drawn in the original image to achieve 3D object detection.

[0088] The embodiment of the present application applies the key point detection algorithm to the 3D object detection task, and directly predicts the coordinate positions of the 8 corner points of the 3D detection frame based on regression, with good accuracy and real-time performance. Through the monocular vision 3D object detection algorithm, 3D object detection can be achieved only through the camera sensor, which can be widely deployed in vehicle-mounted cameras and roadside cameras, easy to install and low cost.

[0089] Based on the same application concept, a 3D target detection system based on YOLOv7-pose is also provided in an embodiment of the present application. The method corresponding to the 3D target detection system based on YOLOv7-pose may be the 3D target detection method based on YOLOv7-pose in the aforementioned embodiment, and its principle of solving the problem is similar to that of the method. The 3D target detection system based on YOLOv7-pose provided in an embodiment of the present application includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the methods and / or technical solutions of the aforementioned multiple embodiments of the present application.

[0090] Another embodiment of the present application further provides a computer-readable storage medium having computer program instructions stored thereon, wherein the computer program instructions can be executed by a processor to implement the methods and / or technical solutions of any one or more embodiments of the present application described above.

[0091] Specifically, the present embodiment may adopt any combination of one or more computer-readable media. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination thereof. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device, or device.

[0092] Computer readable signal media may include a data signal propagated in baseband or as part of a carrier wave, which carries a computer readable program code. Such propagated data signals may take a variety of forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination of the above. Computer readable signal media may also be any computer readable medium other than a computer readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0093] Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0094] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0095] The flow chart or block diagram in the accompanying drawings shows the possible architecture, function and operation of the equipment, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated system for hardware that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0096] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0097] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or page components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0098] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0099] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0100] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform some steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store program codes.

[0101] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

[0102] In addition, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in a device claim can also be implemented by one unit or device through software or hardware. The words first, second, etc. are used to indicate names, and do not indicate any particular order.

Claims

1. A 3D target detection method based on YOLOv7-pose, characterized in that: include: collecting first image data; The first image data includes an image captured by a camera sensor; Based on the first image data, generating a target detection data set; the target detection data set includes a plurality of key points; Annotate each image of the target detection data set with a three-dimensional detection frame to obtain a label file; each image corresponds to one label file; the label file includes a target category index and absolute coordinates of multiple corner points of the three-dimensional detection frame; Modify the format of the label file according to a preset storage format to obtain a label conversion file; Uploading the label conversion file to the annotation platform, modifying the visibility of each of the multiple key points, and obtaining a key point set; Based on a key point detection algorithm, confirm eight key points of the key point set; Based on the eight key points, the first image data and the label conversion file, training a key point detection model; Input the image to be detected into the key point detection model to obtain the coordinate information of eight target key points; Based on a preset connection order and the coordinate information, the eight target key points are connected.

2. The 3D target detection method based on YOLOv7-pose according to claim 1, characterized in that: When performing the step of annotating the three-dimensional detection frame of each image of the target detection data set to obtain a label file, the three-dimensional detection frame includes two front and rear rectangular frames, and the 3D target detection method based on YOLOv7-pose further includes: Based on the size of the target, the size of each of the two rectangular frames is adjusted, and the relative position between the two rectangular frames is adjusted to fit the edge of the target and highlight the perspective relationship of the target.

3. The 3D target detection method based on YOLOv7-pose according to claim 2, characterized in that: The x-axis coordinates of the two key points on one side of each rectangular frame are the same.

4. The 3D target detection method based on YOLOv7-pose according to any one of claims 1 to 3, characterized in that: When performing the step of modifying the format of the label file according to the preset storage format to obtain the label conversion file, the preset storage format is a txt format, and the 3D target detection method based on YOLOv7-pose further includes: Based on the absolute coordinates, determine the first relative coordinates and the second relative coordinates; the first relative coordinates are the relative coordinates of the target two-dimensional rectangular frame relative to the image; the target two-dimensional rectangular frame is the circumscribed rectangle of the multiple corner points; the second relative coordinates are the relative coordinates of the key point relative to the image.

5. The 3D target detection method based on YOLOv7-pose according to any one of claims 1 to 3, characterized in that: The key point detection model includes a backbone network and a detection head; the backbone network includes a feature extraction module and a feature fusion module; The feature fusion module includes a spatial pyramid pooling submodule and a path aggregation network submodule; the detection head includes a two-dimensional target detection module and a key point detection module; The key point detection module is used to confirm the key points; The YOLOv7-pose-based 3D target detection method further includes: The object key point similarity loss function is added, and the model parameters are updated through the back-propagation mechanism.

6. The 3D target detection method based on YOLOv7-pose according to claim 5, characterized in that: The YOLOv7-pose-based 3D target detection method further includes: For each of the keypoints, a confidence parameter is learned.

7. The 3D target detection method based on YOLOv7-pose according to claim 5, characterized in that: The image to be detected includes a plurality of targets to be detected; when executing the step of inputting the image to be detected into the key point detection model to obtain coordinate information of eight target key points; the 3D target detection method based on YOLOv7-pose also includes: Confirming the category index and relative coordinates of key points of each target to be detected; The relative coordinates of the key points are mapped to the image to obtain the absolute coordinates of the key points.

8. The 3D target detection method based on YOLOv7-pose according to claim 6, characterized in that: The calculation formula of the confidence parameter includes: in, is the key point confidence loss; is the confidence of the nth predicted key point; δ(v n ) represents the visibility of the key point; BCE is the binary cross entropy loss, which is used to calculate the binary cross entropy loss between the predicted probability and the true label.

9. A 3D object detection system based on YOLOv7-pose, characterized in that: include: at least one processor; a memory communicatively coupled to the at least one processor; The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method according to any one of claims 1 to 8.

10. A computer readable medium having computer program instructions stored thereon, wherein the computer program instructions can be executed by a processor to implement the method according to any one of claims 1 to 8.

Images