Target size identification method and device based on monocular camera
Through the monocular camera combining depth estimation model and object detection algorithm, the three-dimensional coordinates of the target are directly obtained from the monocular camera, solving the problem of target size recognition efficiency and accuracy in the existing technology, and achieving efficient and low-cost target size recognition .
Patent Information
- Application Number
- CN202411160689.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-22
- Publication Date
- 2025-01-10
AI Technical Summary
The prior art lacks a highly efficient, low-cost and highly accurate target size recognition method in a wide range of applications, especially in the absence of reference objects or costly binocular cameras and complex radar sensors.
Using a monocular camera-based method, the target bounding box is obtained through the object detection algorithm, and the absolute depth information is obtained by combining the depth estimation model, and converted into three-dimensional point cloud coordinates, the actual size of the target is calculated, and the accuracy is improved through scaling and specification comparison.
It achieves efficient and low-cost identification of the target size, and has high accuracy in a wide range of application scenarios, avoiding the limitations of reference objects and high-cost binocular cameras.
Smart Images
Figure CN119295718A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of object recognition, and in particular to a method and device for recognizing target size based on a monocular camera. Background Art
[0002] In order to manage or process objects, it is often necessary to know the actual size of the target in advance. For example, traffic signs are usually set up beside traffic roads. They are traffic assets. The management department needs to know the size of the traffic signs set up beside the road in order to update and manage them. The methods for estimating the size of traffic signs in the existing technical solutions mainly include the following methods:
[0003] (1) Method based on reference object annotation
[0004] This method uses a reference object of known size in the image and uses object detection algorithms such as YOLO and Faster R-CNN to detect the object in the image and obtain its bounding box. The actual size of the object is calculated by combining the pixel size ratio of the object and the reference object in the image with the actual size of the reference object.
[0005] However, the method based on reference object annotation relies on the presence of reference objects of known size in the image. It cannot be applied when there are no reference objects or the size of the reference objects is unknown, and the reference objects are difficult to adapt to aerial targets such as highway signs.
[0006] (2) Method based on binocular camera
[0007] The binocular camera captures the same scene through two cameras at a certain distance, and uses the parallax principle to calculate the absolute depth information to estimate the actual size of the target.
[0008] However, binocular cameras are expensive, and in scenes with varying lighting and less texture, the stereo matching effect is not ideal, resulting in large errors in the results.
[0009] (3) Radar-based methods
[0010] Radar (such as LiDAR) generates 3D point cloud data of the scene by emitting laser light and measuring its reflection time, thereby calculating the actual size of the object.
[0011] However, LiDAR sensors are expensive, data processing is complex, and real-time performance is poor.
[0012] That is, there is a lack of an efficient, low-cost target size recognition method with high accuracy in a wide range of application scenarios. Summary of the invention
[0013] In order to solve the above problems in the prior art, the present invention provides a method and device for object size recognition based on a monocular camera, which can achieve high efficiency and low cost in object size recognition and have high accuracy in a wide range of application scenarios.
[0014] In order to achieve the above object, the technical solution adopted by the present invention is:
[0015] In a first aspect, the present invention provides a method for identifying target size based on a monocular camera, comprising the steps of:
[0016] S1. Using a monocular camera to obtain a current target scene image, and detecting and locating a target bounding box of the current target scene image;
[0017] S2. Obtaining absolute depth information of each pixel in the current target scene image through a depth estimation model;
[0018] S3, obtaining three-dimensional point cloud coordinates according to the focal length of the monocular camera, the horizontal and vertical distances from each pixel point in the current target scene image to the image center, and the absolute depth information of each pixel;
[0019] S4. Calculate the actual target size according to the three-dimensional point cloud coordinates corresponding to the vertex information of the target bounding box.
[0020] The beneficial effects of the present invention are as follows: the target in the target scene picture is identified and located through the existing target detection algorithm, the absolute depth information of the target scene picture acquired by the monocular camera is identified through the depth estimation model, and the actual target size is obtained after converting it into three-dimensional point cloud coordinates, thereby eliminating the need for a binocular camera and being unaffected by reference objects, thereby achieving efficient and low-cost target size recognition with high accuracy in a wide range of application scenarios.
[0021] Optionally, step S3 includes the following steps:
[0022] The focal length f of the monocular camera and the horizontal and vertical distances (l x ,l y ) is substituted into the first formula to obtain the X coordinate and Y coordinate of the pixel p in the world coordinate system, and the absolute depth information H of the pixel p is used as the Z coordinate of the pixel p in the world coordinate system, thereby obtaining the three-dimensional point cloud coordinates (X, Y, Z) of each pixel. The first formula is:
[0023]
[0024] Optionally, step S4 includes the following steps:
[0025] The scaled target size is obtained by calculating the three-dimensional point cloud coordinates corresponding to the vertex information after the target bounding box is scaled, and the actual target size is obtained by enlarging the scaled target size in the same proportion.
[0026] Optionally, the S4 specifically includes the following steps:
[0027] Get the position (x1, y1, w, h) of the target bounding box in the current target scene image, where x1 and y1 are the coordinates of the upper left corner vertex of the target bounding box, w is the length of the target bounding box, and h is the width of the target bounding box;
[0028] Reduce the target bounding box by a times to obtain the scaled position of the target bounding box (x1+w / 4, y1+h / 4, w / 2, h / 2);
[0029] According to the scaled position (x1+w / 4, y1+h / 4, w / 2, h / 2) of the target bounding box, the vertex information of the target bounding box is obtained, thereby obtaining the corresponding three-dimensional point cloud coordinates;
[0030] The Euclidean distance between each vertex is calculated according to the three-dimensional point cloud coordinates of each vertex of the target bounding box to obtain the scaled target size, and the actual target size is obtained after the scaled target size is magnified a times.
[0031] According to the above description, by scaling the target bounding box, the problem of inaccurate point cloud at the edge of the target is reduced, thereby further improving the accuracy of the actual target size.
[0032] Optionally, a is from 1.5 to 3.
[0033] Optionally, the method further comprises the following steps:
[0034] S5. Acquire various sizes of pre-collected targets, compare the actual target size with the various sizes, and take the size closest to the actual target size as the final target size.
[0035] According to the above description, the accuracy of the actual target size can be further improved by comparing the actual specification size with the value.
[0036] Optionally, step S2 includes the following steps:
[0037] The general depth estimation model is trained using the data set corresponding to the target scene to obtain a fine-tuned depth estimation model;
[0038] The absolute depth information of each pixel in the current target scene image is obtained through the fine-tuned depth estimation model.
[0039] Optionally, the depth estimation model is a Depth Anything model, and the algorithm for detecting and locating the target bounding box of the current target scene image is a Yolo target detection algorithm.
[0040] Optionally, the target is a traffic sign, and the data set corresponding to the target scene is a KITTI data set.
[0041] In a second aspect, the present invention provides an object size recognition device based on a monocular camera, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, an object size recognition method based on a monocular camera according to the first aspect is implemented.
[0042] Among them, the technical effect corresponding to the object size recognition device based on a monocular camera provided in the second aspect refers to the relevant description of the object size recognition method based on a monocular camera provided in the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS
[0043] Figure 1 A schematic diagram of the main flow of a method for identifying target size based on a monocular camera according to an embodiment of the present invention;
[0044] Figure 2 is an original schematic diagram of the current target scene image involved in the embodiment of the present invention;
[0045] Figure 3 This is a schematic diagram of a current target scene image after depth estimation according to an embodiment of the present invention;
[0046] Figure 4 A schematic diagram of a current target scene image converted into a three-dimensional point cloud according to an embodiment of the present invention;
[0047] Figure 5 The present invention is a schematic diagram of a framework of a monocular camera-based object size recognition device according to an embodiment of the present invention.
[0048] Description of reference numerals:
[0049] 1. A target size recognition device based on a monocular camera;
[0050] 2. Processor;
[0051] 3. Memory. DETAILED DESCRIPTION
[0052] In order to better understand the above technical solution, exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to enable a clearer and more thorough understanding of the present invention and to fully convey the scope of the present invention to those skilled in the art.
[0053] Embodiment 1
[0054] This embodiment is suitable for application scenarios where it is necessary to identify the size of targets on the road. In particular, it is necessary to identify the size of traffic signs on the road. Existing target size recognition algorithms are not compatible with cost, application scenarios and efficiency. This embodiment uses a depth estimation model and coordinate system conversion to achieve target size recognition only under the target scene image obtained by a monocular camera, achieving efficient and low-cost target size recognition with high accuracy in a wide range of application scenarios. See the following description for details.
[0055] Please refer to Figures 1 to 5 , a target size recognition method based on a monocular camera, comprising the steps of:
[0056] S1. Use a monocular camera to obtain a current target scene image, and detect and locate a target bounding box of the current target scene image.
[0057] In this embodiment, the target is a traffic sign.
[0058] In this embodiment, the monocular camera has the advantages of low cost and easy deployment, and is suitable for large-scale application scenarios.
[0059] in, Figure 2 is the current target scene image. In this embodiment, the Yolo target detection algorithm is used to detect and locate the target bounding box of the current target scene image, that is, the boundary of the traffic sign is selected in the current target scene image. The Yolo target detection algorithm can be obtained after training with the labeled data set.
[0060] S2. Obtain the absolute depth information of each pixel in the current target scene image through the depth estimation model.
[0061] In this embodiment, step S2 includes the following steps:
[0062] S21, training a general depth estimation model using a data set corresponding to the target scene to obtain a fine-tuned depth estimation model;
[0063] The depth estimation model is the Depth Anything model, and the dataset corresponding to the target scene is the KITTI dataset. The KITTI dataset is a large dataset used for autonomous driving research, which contains a large number of annotated images and lidar data, mainly traffic scene pictures. Therefore, the KITTI dataset is used to fine-tune the Depth Anything model to make it more suitable for the depth estimation of traffic signs, thereby improving the accuracy of target recognition.
[0064] S22. Obtain the absolute depth information of each pixel in the current target scene image through the fine-tuned depth estimation model.
[0065] Therefore, the current target scene image Figure 2 The transformation shown Figure 3 shown.
[0066] S3, obtaining the three-dimensional point cloud coordinates according to the focal length of the monocular camera, the horizontal and vertical distances from each pixel point in the current target scene image to the center of the image, and the absolute depth information of each pixel;
[0067] In this embodiment, step S3 includes the following steps:
[0068] The focal length f of the monocular camera and the horizontal and vertical distances (l x ,l y ) is substituted into the first formula to obtain the X and Y coordinates of pixel p in the world coordinate system, and the absolute depth information H of pixel p is used as the Z coordinate of pixel p in the world coordinate system, thereby obtaining the three-dimensional point cloud coordinates (X, Y, Z) of each pixel. The first formula is:
[0069]
[0070] Therefore, each pixel in the current target scene image has a corresponding 3D point cloud coordinate in the world coordinate system. Figure 3 The transformation shown Figure 4 shown.
[0071] S4. Calculate the actual target size according to the three-dimensional point cloud coordinates corresponding to the vertex information of the target bounding box.
[0072] Therefore, by using the vertex information of the traffic sign, the length and width of the traffic sign are obtained by calculating the distance of the corresponding three-dimensional point cloud coordinates (X, Y, Z), that is, the actual target size.
[0073] Embodiment 2
[0074] Please refer to Figures 1 to 5, a method for identifying target size based on a monocular camera, based on the above embodiment 1, step S4 includes the following steps:
[0075] The scaled target size is obtained by calculating the three-dimensional point cloud coordinates corresponding to the vertex information after the target bounding box is scaled, and the actual target size is obtained by enlarging the scaled target size by the same proportion.
[0076] In this embodiment, S4 specifically includes the following steps:
[0077] S41. Obtain the position (x1, y1, w, h) of the target bounding box in the current target scene image, where x1 and y1 are the coordinates of the upper left corner vertex of the target bounding box, w is the length of the target bounding box, and h is the width of the target bounding box.
[0078] S42. Reduce the target bounding box by a times to obtain the scaled position of the target bounding box (x1+w / 4, y1+h / 4, w / 2, h / 2).
[0079] In this embodiment, a is 2, so this embodiment is to reduce the target bounding box by 2 times, so the length and width are half of the original, and the vertices on both sides are indented by half in total, that is, the vertices on one side are indented by one quarter inward, so the position of the target bounding box after scaling is (x1+w / 4, y1+h / 4, w / 2, h / 2). In other embodiments, the optional range of a is 1.5 to 3.
[0080] Therefore, after scaling, the vertex information of the target bounding box is closer to the inside of the target, thereby avoiding the problem that the absolute depth information at each vertex of the target bounding box is at a position where it changes dramatically and the vertices of the rectangular box are often located outside the target, thereby improving the accuracy of target recognition size.
[0081] S43. According to the scaled position (x1+w / 4, y1+h / 4, w / 2, h / 2) of the target bounding box, the vertex information of the target bounding box is obtained, thereby obtaining the corresponding three-dimensional point cloud coordinates.
[0082] Among them, the corresponding three-dimensional point cloud coordinates can be obtained according to the vertex information of the target bounding box. At this time, the absolute depth information of the scaled target bounding box is relatively stable, that is, the corresponding three-dimensional point cloud coordinates are more stable and reliable.
[0083] S44. Calculate the Euclidean distance between each vertex according to the three-dimensional point cloud coordinates of each vertex of the target bounding box to obtain the scaled target size, and then magnify the scaled target size by a times to obtain the actual target size.
[0084] In this embodiment, the length and width of the traffic sign are obtained by magnifying the scaled target size by two times, which reduces the problem of inaccurate point cloud at the edge of the target, thereby further improving the accuracy of the actual target size.
[0085] Embodiment 3
[0086] Please refer to Figures 1 to 5 , a method for identifying target size based on a monocular camera, based on the above-mentioned embodiment 1 or 2, further comprising the following steps:
[0087] S5. Obtain various sizes of the pre-collected targets, compare the actual target size with the various sizes, and take the size closest to the actual target size as the final target size.
[0088] Among them, the existing traffic signs all have corresponding size specifications. This embodiment collects the size specifications of various traffic signs in advance, and takes the size specification to which the calculated actual target size is closer as the final target size, thereby further improving the accuracy of the actual target size.
[0089] For example, there are traffic signs with sizes of 600*600mm and 800*800mm. At this time, the actual target size is 593mm*602mm, so 600*600mm is taken as the final target size.
[0090] In summary, this embodiment, based on the use of a monocular camera, obtains a highly accurate actual target size through target detection algorithm, depth estimation model, coordinate system conversion, scaling processing and specification comparison value taking, thereby eliminating the need for a binocular camera and being unaffected by reference objects, thereby achieving efficient and low-cost target size recognition with high accuracy in a wide range of application scenarios.
[0091] Embodiment 4
[0092] Please refer to Figure 5 A target size recognition device 1 based on a monocular camera includes a memory 3, a processor 2, and a computer program stored in the memory 3 and executable on the processor 2. When the processor 2 executes the computer program, the steps in the above-mentioned embodiments one, two or three are implemented.
[0093] Since the system / device described in the above embodiments of the present invention is a system / device used to implement the method of the above embodiments of the present invention, a person skilled in the art can understand the specific structure and deformation of the system / device based on the method described in the above embodiments of the present invention, and thus will not be described in detail here. All systems / devices used in the method of the above embodiments of the present invention belong to the scope of protection of the present invention.
[0094] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, devices or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0095] The present invention is described with reference to the flowcharts and / or block diagrams of the methods, devices (apparatus) and computer program products according to the embodiments of the present invention. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions.
[0096] It should be noted that in the claims, any reference numerals placed between brackets shall not be construed as limiting the claims. The word "comprising" does not exclude the presence of components or steps not listed in the claims. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. The invention may be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. In the claims enumerating several means, several of these means may be embodied by the same hardware. The use of the words first, second, third, etc., is for convenience of expression only and does not indicate any order. These words may be understood as part of the component name.
[0097] In addition, it should be noted that, in the description of this specification, the description of the terms "one embodiment", "some embodiments", "embodiment", "example", "specific example" or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, unless they are contradictory.
[0098] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments after knowing the basic creative concept. Therefore, the claims should be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present invention.
[0099] Obviously, those skilled in the art can make various modifications and variations to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention should also include these modifications and variations.
Claims
1. A method for identifying target size based on a monocular camera, characterized in that: Includes steps: S1. Using a monocular camera to obtain a current target scene image, and detecting and locating a target bounding box of the current target scene image; S2. Obtaining absolute depth information of each pixel in the current target scene image through a depth estimation model; S3, obtaining three-dimensional point cloud coordinates according to the focal length of the monocular camera, the horizontal and vertical distances from each pixel point in the current target scene image to the image center, and the absolute depth information of each pixel; S4. Calculate the actual target size according to the three-dimensional point cloud coordinates corresponding to the vertex information of the target bounding box.
2. The method for object size recognition based on a monocular camera according to claim 1, characterized in that: The step S3 comprises the following steps: The focal length f of the monocular camera and the horizontal and vertical distances (l x ,l y ) is substituted into the first formula to obtain the X coordinate and Y coordinate of the pixel p in the world coordinate system, and the absolute depth information H of the pixel p is used as the Z coordinate of the pixel p in the world coordinate system, thereby obtaining the three-dimensional point cloud coordinates (X, Y, Z) of each pixel. The first formula is:
3. The method for object size recognition based on a monocular camera according to claim 1, characterized in that: The step S4 comprises the following steps: The scaled target size is obtained by calculating the three-dimensional point cloud coordinates corresponding to the vertex information after the target bounding box is scaled, and the scaled target size is enlarged by the same proportion to obtain the actual target size.
4. The method for object size recognition based on a monocular camera according to claim 3, characterized in that: The S4 specifically comprises the following steps: Get the position (x1, y1, w, h) of the target bounding box in the current target scene image, where x1 and y1 are the coordinates of the upper left corner vertex of the target bounding box, w is the length of the target bounding box, and h is the width of the target bounding box; Reduce the target bounding box by a times to obtain the scaled position of the target bounding box (x1+w / 4, y1+h / 4, w / 2, h / 2); According to the scaled position (x1+w / 4, y1+h / 4, w / 2, h / 2) of the target bounding box, the vertex information of the target bounding box is obtained, thereby obtaining the corresponding three-dimensional point cloud coordinates; The Euclidean distance between each vertex is calculated according to the three-dimensional point cloud coordinates of each vertex of the target bounding box to obtain the scaled target size, and the actual target size is obtained after the scaled target size is magnified a times.
5. The method for object size recognition based on a monocular camera according to claim 3, characterized in that: The a is 1.5 to 3.
6. A method for identifying target size based on a monocular camera according to any one of claims 1 to 5, characterized in that: The following steps are also included: S5. Acquire various sizes of the pre-collected targets, compare the actual target size with the various sizes, and take the size closest to the actual target size as the final target size.
7. A method for identifying target size based on a monocular camera according to any one of claims 1 to 5, characterized in that: The step S2 comprises the following steps: The general depth estimation model is trained using the data set corresponding to the target scene to obtain a fine-tuned depth estimation model; The absolute depth information of each pixel in the current target scene image is obtained through the fine-tuned depth estimation model.
8. The method for object size recognition based on a monocular camera according to claim 7, characterized in that: The depth estimation model is a Depth Anything model, and the algorithm for detecting and locating the target bounding box of the current target scene image is a Yolo target detection algorithm.
9. The method for object size recognition based on a monocular camera according to claim 7, characterized in that: The target is a traffic sign, and the data set corresponding to the target scene is the KITTI data set.
10. A target size recognition device based on a monocular camera, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method for identifying the size of an object based on a monocular camera as described in any one of claims 1 to 9 is implemented.