Long-distance monocular distance measurement method, system, medium, equipment and application
Through the combination of the depth calculation module, the object detection module and the information fusion module, combined with the attention mechanism to deal with missing parameters, the efficient and accurate positioning of long-distance targets in high-altitude environments is achieved, and the large long-distance distance measurement error in the existing technology is solved. Problem, suitable for high-altitude atmospheric monitoring.
Patent Information
- Application Number
- CN202411125897.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-16
- Publication Date
- 2024-11-26
AI Technical Summary
The existing monocular distance measurement technology is difficult to accurately measure long-distance targets in high-altitude monitoring, especially in environmental pollution monitoring. The existing methods have large calculation errors and lack research on long-distance scenarios. Most of the existing research is aimed at indoor and short-distance. Outdoor and mobile vehicles lack effective solutions for long-distance scenarios.
A long-distance monocular distance measurement method is adopted to collect images through a monocular camera, and the depth calculation module is used to obtain the depth map. The target detection module recognizes the target object. The information fusion module integrates the depth map and camera parameters. The distance prediction module is end-to-end Distance prediction uses encoder-decoder network and attention mechanism to deal with missing parameters to achieve efficient and accurate positioning.
Efficient and accurate positioning of long-distance targets under large-scale space improves the accuracy of distance measurement, adapts to the unavailable camera parameters, and has outstanding advantages in high-altitude atmospheric monitoring.
Smart Images

Figure CN119027492A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of distance measurement technology, and in particular to a long-distance monocular distance measurement method, system, medium, equipment and application. Background Art
[0002] Monocular ranging technology refers to the technology of using a single camera to estimate the depth or distance of objects in the scene. At present, there are mainly the following methods for existing monocular ranging technology. The first is to obtain a large number of internal and external parameters of the camera, and deduce the distance of the target through optical and geometric principles. However, it is difficult for the high-altitude monitoring system to obtain accurate external parameters, which leads to large errors in the calculated distance. At the same time, due to the large number of cameras, the workload of obtaining external parameters is large. The second is a method based on deep learning, including supervised learning and self-supervised learning. The scenes include indoor depth estimation and outdoor depth estimation. The result of depth estimation is usually the relative depth from the scene point to the camera, rather than the actual straight-line distance between the object and the camera. In the actual application process, the actual distance is required. In order to obtain the actual distance, it is also necessary to combine the camera's internal parameters and possible external parameters, scale information, etc. to transform the results of monocular depth estimation to obtain the actual distance. However, since this information still needs to be obtained through the calibration process, it is difficult and laborious to calibrate the installed high-altitude camera system. In addition, most existing studies are mainly based on indoor, short-distance outdoor and mobile vehicle outdoor data sets, and lack research on long-distance scenes.
[0003] As the country attaches great importance to environmental protection, various regions have also strengthened the management and supervision of environmental protection. One of the important means is to use high-altitude cameras to monitor the atmospheric environment. When environmental pollution occurs, the staff needs to determine the location of the environmental pollution points based on on-site experience and intervene in time. While the high-altitude monitoring system can ensure wide-area coverage and high resolution, it also faces some challenges, including how to efficiently and accurately locate remote pollution points to assist staff in rapid intervention. Summary of the invention
[0004] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a long-distance monocular ranging method, system, readable storage medium and device, which can efficiently and accurately locate distant targets under wide area coverage.
[0005] The purpose of the present invention is achieved through the following technical solutions:
[0006] The present invention provides a long-distance monocular ranging method, comprising the following steps:
[0007] S1, collect the image to be tested through a monocular camera;
[0008] S2, obtaining a depth map of the image to be tested through a depth calculation module;
[0009] S3, identifying the target in the image to be tested through the target detection module, and obtaining the category and location information of the target;
[0010] S4, using the target center point coordinates, width and height data obtained in S3 to crop the depth map obtained in S2 to obtain the depth information of the target object;
[0011] S5, inputting the above information and the monocular camera parameters into the information fusion module, performing information fusion, and obtaining fused feature information;
[0012] S6. Input the fused feature information into the distance prediction module to predict the distance between the target object and the camera, thereby realizing end-to-end long-distance monocular ranging.
[0013] In step S2 of the present invention, the depth calculation module is an encoder-decoder network; when obtaining the depth map of the image to be tested, the encoder part in the depth calculation module uses multiple stacked MobileNetv2block and MobileViTblock to extract and encode the image to be tested; then the multiple stacked decoder blocks Decoderblock used by the decoder part use multi-scale jump connections to restore image details from the feature map of the encoder to reduce feature loss; finally, transposed convolution is used to expand the spatial resolution, and then the depth map DepthMap of the image to be tested is output:
[0014] DepthMap=DepthExtraction(I)
[0015] I represents the image to be tested input into the depth calculation module, and DepthExtraction represents the encoder-decoder network in the depth calculation module.
[0016] Optionally or preferably, when outputting the depth map, the calculation process X of each decoder block Decoderblock i As shown below:
[0017]
[0018] Among them, U(.) represents the upsampling block, which consists of a 3×3 convolutional layer with ELU activation and bilinear interpolation; M(.) represents the maximum pooling layer for downsampling shallow features; D(.) represents the feature fusion block composed of a 3×3 convolutional layer with ELU activation; [.] represents the concatenation operation.
[0019] Optionally or preferably, when identifying the target in the image to be detected in step S3, yolov5, yolov8 target detection network or other target detection network is used to obtain the category and position information of the target in the image.
[0020] Optionally or preferably, when the depth map is cropped in step S4, the depth map output in step S2 is cropped with the center point of the target detection frame as the center point and the width and height of the target detection frame as the size.
[0021] Optionally or preferably, when information fusion is performed in step S5, the following steps are included:
[0022] S501, using a normalization layer to normalize the data;
[0023] The normalized data includes: target object depth information d, width W and height H of the image captured by the monocular camera, height CH of the monocular camera, width CMOSw and height CMOSh of the monocular camera sensor, PTZ parameters of the monocular camera, center point coordinates (x, y) of the target detection frame, and width and height (w, h) of the target detection frame;
[0024] S502, using the mask mechanism, marking the unavailable information position as 0 and marking the available information position as 1, so as to create a corresponding mask tensor mask, and determine the data to be input for the information fusion module;
[0025] S503, using the attention mechanism, selectively processing the weight of the input data according to the mask tensor mask. By using the mask and the attention mechanism, the model can dynamically select available parameters, where the calculation process of the attention mechanism AttentionWeights is as follows:
[0026] AttentionWeights=softmax(FC(ReLU(DWConv(FC(LN(X)*mask)))))
[0027] Wherein, X represents the input information, which is concatenated from the information in step S501 by using concat; LN represents layer normalization; FC represents a fully connected layer; ReLU represents a ReLU activation function; DWConv represents a deep convolution;
[0028] S504, using the fully connected layer to fuse the information, to obtain the fused features FusedFeatures, where the calculation process is:
[0029] FusedFeatures=FC(LN(X)*AttentionWeights).
[0030] Optionally or preferably, the distance prediction module applies a fully connected layer to predict the distance using the fused features to obtain a predicted distance predicted_distance, wherein the calculation process is:
[0031] predicted_distance=FC(ReLU(FC(FusedFeatures))).
[0032] The present invention provides a long-distance monocular ranging system, comprising: a monocular camera, a depth calculation module, a target detection module, an information fusion module and a distance prediction module, wherein:
[0033] The monocular camera is used to collect video information of the detected area and obtain the image to be tested;
[0034] The depth calculation module crops the image to be tested and obtains a depth map of the image to be tested through average pooling calculation;
[0035] The target detection module uses the target detection network to obtain the category and location information of the target object in the image to be detected;
[0036] The information fusion module uses a mask and attention mechanism to fuse the depth map, the monocular camera parameter information, and the location information of the target object;
[0037] The distance prediction module inputs the fused information into the fully connected layer for distance prediction.
[0038] Optionally or preferably, the depth calculation module is an encoder-decoder network; the encoder is composed of stacked MobileNetv2 blocks and MobileViTblocks, and performs feature extraction and encoding on the image to be tested; the decoder is composed of a plurality of stacked decoder blocks.
[0039] The present invention also provides a readable storage medium. Based on any of the above-mentioned ranging methods and / or systems, the readable storage medium stores a computer program, and when the computer program is executed by a processor, any of the above-mentioned ranging methods is implemented.
[0040] The present invention also provides a long-distance monocular distance measurement device, based on any of the above distance measurement methods and / or systems, the distance measurement device includes a computer-readable storage medium and a processor;
[0041] The computer-readable storage medium is used to store executable instructions; the executable instructions are any of the above-mentioned ranging methods; and the processor is used to read the executable instructions stored in the computer-readable storage medium.
[0042] The present invention also provides a long-distance monocular ranging application, which adopts any of the above methods, systems, readable storage media or devices for the monocular ranging link in atmospheric environment detection.
[0043] Based on the above technical solution, the following technical effects can be produced:
[0044] The present invention provides a long-distance monocular ranging method, which can measure and locate long-distance targets in a large-scale space, has high accuracy, low requirements on camera hardware, can adapt to situations where camera parameters are unavailable, and has outstanding advantages in the field of high-altitude atmospheric monitoring. BRIEF DESCRIPTION OF THE DRAWINGS
[0045] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying creative work.
[0046] Figure 1 is a flow chart of the method of the present invention;
[0047] Figure 2 is a schematic diagram of a depth calculation module in the present invention;
[0048] Figure 3 is a schematic diagram of a decoder block in the present invention;
[0049] Figure 4 Schematic diagram of the structure of the information fusion module and the distance prediction module. DETAILED DESCRIPTION
[0050] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.
[0051] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0052] Embodiment 1:
[0053] This embodiment provides a long-distance monocular ranging method, such as Figure 1 As shown, the following steps are included:
[0054] S1, collect the image to be tested through a monocular camera;
[0055] S2, obtaining a depth map of the image to be tested through a depth calculation module;
[0056] In this embodiment, if Figure 2 As shown, the depth calculation module is a lightweight encoder-decoder network to facilitate inference deployment on portable devices;
[0057] Specifically, the encoder uses stacked MobileNetv2 block (MV2) and MobileViTblock to extract and encode features of the image to be tested; the decoder is composed of a plurality of decoder blocks stacked; each decoder block uses multi-scale jump connections to recover image details from the feature map of the encoder to reduce feature loss; Figure 3 As shown, the calculation process of the decoder block Decoder block X i for:
[0058]
[0059] Among them, U(.) represents the upsampling block, which consists of a 3×3 convolutional layer with ELU activation and bilinear interpolation; M(.) represents the maximum pooling layer for downsampling shallow features; D(.) represents the feature fusion block composed of a 3×3 convolutional layer with ELU activation; [.] represents the concatenation operation.
[0060] Then, transposed convolution is used to expand the spatial resolution, and finally the depth map DepthMap of the image to be tested is output, where:
[0061] DepthMap=DepthExtraction(I)
[0062] I represents the image to be tested input into the depth calculation module, and DepthExtraction represents the encoder-decoder network in the depth calculation module.
[0063] S3, identifying the target in the image to be tested by the target detection module, and obtaining the category and position information of the target; specifically, the category of the target includes objects and scenes, and the position information of the target includes the center point coordinates (x, y) of the target detection frame and the width and height (w, h) of the target detection frame; the target detection frame is the target detection frame when the target detection module identifies the target;
[0064] Specifically, when identifying the target object, the yolov8 target detection network is used to obtain the category and location information of the target object in the image. The calculation process is as follows:
[0065] [cls,x,y,w,h]=TargetDetectionModule(I)
[0066] Among them, I represents the input image; TargetDetectionModule represents the target detection network; cls represents the category of the detected target; x and y represent the horizontal and vertical coordinates of the center point of the target detection box respectively; w and h represent the width and height of the target detection box respectively.
[0067] S4, using the target center point coordinates and width and height data obtained in S3 to crop the depth map obtained in S2 to obtain the depth information of the target object, specifically including the following steps;
[0068] S401, taking the center point of the target detection frame as the center and the width and height of the target detection frame as the size, cropping the depth map DepthMap output in step S2;
[0069] S402, calculating the target object depth information d, where:
[0070]
[0071] Where A represents the area of the cropped depth map.
[0072] S5, input the information, camera parameters, and camera parameter indicator information obtained through steps S1-S4 into the information fusion module for information fusion; the information fusion module uses an information normalization layer, an attention mechanism layer, and an information fusion layer to perform information fusion, and the specific steps are as follows:
[0073] S501, using a normalization layer to normalize the data; specifically, the normalized data includes: target depth information d, width W and height H of the image captured by the monocular camera, height CH of the monocular camera, width CMOSw and height CMOSh of the camera sensor, PTZ parameters of the camera, center point coordinates (x, y) of the target detection frame, and width and height (w, h) of the target detection frame;
[0074] S502, using the mask mechanism, marking the unavailable information position as 0 and marking the available information position as 1, so as to create a corresponding mask tensor mask, and determine the data to be input for the information fusion module;
[0075] S503, using the attention mechanism to selectively process the weight of the input data according to the mask tensor mask. The calculation process of the attention mechanism AttentionWeights is as follows:
[0076] AttentionWeights=softmax(FC(ReLU(DWCone(FC(KN(X)*mask)))))
[0077] Wherein, X represents the input information, which is concatenated from the information in step S501 by using concat; LN represents layer normalization; FC represents a fully connected layer; ReLU represents a ReLU activation function; DWConv represents a deep convolution;
[0078] S504, using the fully connected layer to fuse the information, to obtain the fused features FusedFeatures, where the calculation process is:
[0079] FusedFeatures=FC(LN(X)*AttentionWeights).
[0080] S6, inputting the fused feature information into the distance prediction module to predict the distance between the target object and the camera, thereby realizing end-to-end long-distance monocular ranging;
[0081] Specifically, the distance prediction module uses the fully connected layer to predict the distance using the fused features to obtain the predicted distance predicted_distance, where the calculation process is:
[0082] predicted_distance=FC(ReLU(FC(FusedFeatures))).
[0083] The present invention provides a long-distance monocular ranging system, comprising a monocular camera, a depth calculation module, a target detection module, an information fusion module and a distance prediction module, wherein:
[0084] The monocular camera is used to collect video information of the detected area and obtain the image to be tested;
[0085] The depth calculation module crops the image to be tested and obtains a depth map of the image to be tested through average pooling calculation;
[0086] The target detection module uses the target detection network to obtain the category and location information of the target object in the image to be detected;
[0087] The information fusion module uses a mask and attention mechanism to fuse the depth map, camera parameter information, and target location information;
[0088] The distance prediction module inputs the fused information into the fully connected layer for distance prediction.
[0089] The present invention also provides a long-distance monocular distance measurement device, based on the above-mentioned distance measurement method and system, comprising: a computer-readable storage medium and a processor;
[0090] The computer-readable storage medium is used to store executable instructions; the executable instructions are any of the above-mentioned distance measurement methods;
[0091] The processor is configured to read executable instructions stored in the computer-readable storage medium.
[0092] The present invention also provides a readable storage medium. Based on the above-mentioned ranging method and system, the readable storage medium stores a computer program, and when the computer program is executed by a processor, any of the above-mentioned ranging methods is implemented.
[0093] This embodiment has the following advantages:
[0094] 1. In the present invention, the depth calculation module, the target detection module and the camera parameters are combined, the information fusion module is used to perform feature fusion, and the distance prediction module is used to estimate the actual distance, thereby realizing end-to-end long-distance monocular ranging and improving the accuracy of long-distance monocular ranging;
[0095] 2. In the present invention, the stacked MobileNetv2 network block and MobileViT network block are used to extract features of the image, and multi-scale skip links are used to restore image details from the feature map of the encoder, thereby improving the accuracy of depth estimation;
[0096] 3. In the present invention, considering that some parameters of the camera are unavailable, a mask mechanism is adopted, special marks are used to process the missing parameters, and an attention mechanism is used to enable the model to dynamically select available parameters. In this way, the model can better adapt to the situation where parameters are unavailable and further improve the accuracy of distance calculation;
[0097] 4. In the present invention, combining a lightweight depth calculation module with a quantized target detection module not only enables the overall structure of the model to be deployed in a portable device or instrument, but also expands the application scope of the long-distance monocular ranging method.
[0098] The above is only a preferred embodiment of the present invention. It should be understood that the present invention is not limited to the form disclosed herein, and should not be regarded as excluding other embodiments, but can be used in various other combinations, modifications and environments, and can be modified within the scope of the concept described herein through the above teachings or the technology or knowledge of the relevant field. The changes and modifications made by those skilled in the art shall not deviate from the spirit and scope of the present invention, and shall be within the scope of protection of the claims attached to the present invention.
Claims
1. A long-distance monocular ranging method, characterized in that: The following steps are involved: S1, collect the image to be tested through a monocular camera; S2, obtaining a depth map of the image to be tested through a depth calculation module; S3, identifying the target in the image to be tested through the target detection module, and obtaining the category and position information of the target; the position information of the target includes the coordinates of the target center point and the width and height data; S4, using the target center point coordinates, width and height data obtained in S3 to crop the depth map obtained in S2 to obtain the depth information of the target object; S5, inputting the above information and the monocular camera parameters into the information fusion module, performing information fusion, and obtaining fused feature information; S6. Input the fused feature information into the distance prediction module to predict the distance between the target object and the camera, thereby realizing end-to-end long-distance monocular ranging.
2. A long-distance monocular ranging method according to claim 1, characterized in that: In step S2, the depth calculation module is an encoder-decoder network; when obtaining the depth map of the image to be tested, the encoder part in the depth calculation module uses multiple stacked MobileNetv2blocks and MobileViTblocks to extract and encode the image to be tested; and then the multiple stacked decoder blocks Decoderblock used by the decoder part use multi-scale jump connections to recover image details from the feature map of the encoder to reduce feature loss; Finally, transposed convolution is used to expand the spatial resolution, and then the depth map DepthMap of the image to be tested is output: DepthMap=DepthExtraction(I), where I represents the image to be tested input into the depth calculation module, and DepthExtraction represents the encoder-decoder network in the depth calculation module.
3. A long-distance monocular ranging method according to claim 2, characterized in that: When outputting the depth map, the calculation process of each decoder block Decoderblock is X i As shown below: Among them, U(.) represents the upsampling block, which consists of a 3×3 convolutional layer with ELU activation and bilinear interpolation; M(.) represents the maximum pooling layer for downsampling shallow features; D(.) represents the feature fusion block composed of a 3×3 convolutional layer with ELU activation; [.] represents the concatenation operation; f i represents the features from the encoder; X i Represents the output of the decoder.
4. A long-distance monocular ranging method according to claim 1, characterized in that: When identifying the target in the image to be tested in step S3, yolov5, yolov8 target detection network or other target detection network is used to obtain the category and position information of the target in the image; when cropping the depth map in step S4, the depth map output in step S2 is cropped with the center point of the target detection frame as the center point and the width and height of the target detection frame as the size.
5. A long-distance monocular ranging method according to claim 1, characterized in that: When information fusion is performed in step S5, the following steps are specifically included: S501, using a normalization layer to normalize the data; The normalized data includes: target object depth information d, width W and height H of the image captured by the monocular camera, height CH of the monocular camera, width CMOSw and height CMOSh of the monocular camera sensor, PTZ parameters of the monocular camera, center point coordinates (x, y) of the target detection frame, and width and height (w, h) of the target detection frame; S502, using a mask mechanism to mark unavailable information positions as 0 and available information positions as 1, so as to create a corresponding mask tensor mask to determine the data to be input for the information fusion module; S503, using the attention mechanism, selectively processing the weight of the input data according to the mask tensor mask. By using the mask and the attention mechanism, the model can dynamically select available parameters, where the calculation process of the attention mechanism AttentionWeights is as follows: AttentionWeights=softmax(FC(ReLU(DWConv(FC(LN(X)*mask)))) Wherein, X represents the input information, which is concatenated from the information in step S501 by using concat; LN represents layer normalization; FC represents a fully connected layer; ReLU represents a ReLU activation function; DWConv represents a deep convolution; S504, using the fully connected layer to fuse the information, to obtain the fused features FusedFeatures, where the calculation process is: FusedFeatures=FC(LN(X)*AttentionmWeights).
6. A long-distance monocular ranging method according to claim 1, characterized in that: The distance prediction module uses the fully connected layer to predict the distance using the fused features to obtain the predicted distance predicted_distance, where the calculation process is: predicted_distance=FC(ReLU(FC(FusedFeatures))) Among them, FC represents the fully connected layer; ReLU represents the ReLU activation function; FusedFeatures represents the fused features.
7. A long-distance monocular ranging system, characterized in that: It includes a monocular camera, a depth calculation module, a target detection module, an information fusion module and a distance prediction module, among which: The monocular camera is used to collect video information of the detected area and obtain the image to be tested; The depth calculation module crops the image to be tested and obtains a depth map of the image to be tested through average pooling calculation; The target detection module uses the target detection network to obtain the category and location information of the target object in the image to be detected; The information fusion module uses a mask and attention mechanism to fuse the depth map, the monocular camera parameter information, and the location information of the target object; The distance prediction module inputs the fused information into the fully connected layer for distance prediction; the depth calculation module is an encoder-decoder network; the encoder is composed of stacked MobileNetv2 blocks and MobileViT blocks, which extract features and encode the image to be tested; the decoder is composed of multiple decoder blocks stacked.
8. A readable storage medium, characterized in that: The readable storage medium stores a computer program, and when the computer program is executed by a processor, any one of the long-distance monocular ranging methods in claims 1-6 is implemented.
9. A long-distance monocular distance measurement device, characterized in that: The ranging device includes a computer-readable storage medium and a processor; The computer-readable storage medium is used to store executable instructions; the executable instructions are any of the above-mentioned distance measurement methods; The processor is configured to read executable instructions stored in the computer-readable storage medium.
10. A long-distance monocular ranging application, characterized by: The long-distance monocular ranging method as described in any one of claims 1 to 6 is used in the monocular ranging link in atmospheric environment detection.
