Distance measurement method and device, electronic equipment and storage medium
By analyzing image feature correlation using attention mechanism in the target model in the automotive factory scenario, the robustness and efficiency problems of traditional convolutional neural networks and pure visual Transformer algorithms are solved during the deployment of the end-side side, and more efficient Distance measurement and detection accuracy.
Patent Information
- Application Number
- CN202411203058.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-29
- Publication Date
- 2024-12-13
AI Technical Summary
In the automotive factory scenario, the deep learning-based object detection method and monocular depth estimation method are easily affected by light, angle and background interference when deployed on the end side, and the calculation complexity is high, resulting in insufficient robustness and efficiency, which is difficult to adapt. Practical application requirements.
By adopting a distance measurement method, the distance information between the target moving object and the distance measuring device is determined by acquiring initial image data and inputting it into the target model, using the feature correlation between multiple image features. The target model includes the target detection. The first sub-model of depth estimation and the second sub-model of depth estimation use the preset attention mechanism to perform feature correlation analysis, including convolution module, feature correlation module and fusion module, reducing detection errors and improving feature extraction efficiency.
Capture the feature associations of target running objects in a global scope, reduce detection errors in strong light, dark light and complex backgrounds, avoid loss of data information compression, improve detection accuracy and efficiency, and adapt to the end-side deployment requirements.
Smart Images
Figure CN119131107A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image data processing, and in particular to a distance measurement method and device, electronic equipment, and a storage medium. Background Art
[0002] In the automobile factory scenario, the distance measurement between the delivery robot and the moving object is carried out using the deep learning-based target detection method and the monocular depth estimation method. This not only improves the delivery accuracy and efficiency and ensures the safety of operation, but also can perceive and adapt to the dynamic environment in real time, enhance the flexibility and responsiveness of the production line, optimize the production layout, and reduce equipment costs, thereby promoting the development of automobile factories towards intelligence and automation.
[0003] In automobile production plants, commonly used methods for factory moving object detection and depth estimation include methods based on traditional convolutional neural networks such as ResNet and EfficientNet and algorithms based on pure vision Transformer. Although these methods perform well to a certain extent, they still face challenges when deployed on the edge. Traditional convolutional neural network methods are easily affected by factors such as lighting, angle, and background interference, resulting in performance degradation. Although algorithms based on pure vision Transformer have strong modeling capabilities, they have high computational complexity and high requirements for edge hardware and power consumption, making them less suitable for practical applications. Therefore, it is urgent to improve the robustness and efficiency of the detection algorithm to meet the needs of edge deployment. Summary of the invention
[0004] In view of the above problems, a distance measurement method and device, electronic device, and storage medium are proposed to overcome the above problems or at least partially solve the above problems, including:
[0005] A distance measurement method, applied to a distance measurement device, comprising:
[0006] Acquire initial image data collected for the moving object;
[0007] The initial image data is input into a target model, and the distance information between the target moving object in the initial image data and the distance measuring device is determined in the target model based on the feature correlation between multiple image features in the initial image data, and the distance information is output, wherein the target model is used to determine the feature correlation between image features based on a preset attention mechanism.
[0008] Optionally, the target model includes a first sub-model for detecting a target moving object and a second sub-model for performing depth estimation, the image feature includes a first image feature and a second image feature, the feature correlation includes a first feature correlation and a second feature correlation, and determining the distance information between the target moving object in the initial image data and the distance measuring device based on the feature correlation between multiple image features in the initial image data in the target model includes:
[0009] In the first sub-model, based on the first feature correlation between the plurality of first image features in the initial image data, target image data corresponding to the target running object is determined from the initial image data;
[0010] The target image data is input into the second sub-model, and in the second sub-model, the distance information between the target running object and the target distance measuring device is determined based on the second feature correlation between multiple second image features in the target image data.
[0011] Optionally, the first sub-model includes a first convolution module, a first feature association module and a fusion module, and the determining of target image data corresponding to the target running object from the initial image data based on the first feature association between multiple first image features in the initial image data includes:
[0012] In the first sub-model, the first convolution module is used to determine first image data corresponding to the initial image data and second image data obtained based on the first image data, and the dimension of the first image data is greater than the dimension of the second image data;
[0013] Determine a plurality of first image features in the second image data using the feature association module, and determine first feature associations of the first image features;
[0014] generating third image data based on the first feature association and the second image data;
[0015] Using the fusion module to fuse the first image data, the second image data and the third image data to obtain fourth image data;
[0016] A preset target detection algorithm is used to extract target image data corresponding to the target running object from the fourth image data.
[0017] Optionally, the second sub-model includes a second convolution module, a second feature association module and a decoding layer, and the target image data is input into the second sub-model, and in the second sub-model, based on the second feature association between multiple second image features in the target image data, the distance information between the target running object and the target distance measurement device is determined, including:
[0018] In the second sub-model, the second convolution module is used to determine fifth image data corresponding to the target image data and sixth image data obtained based on the fifth image data, and the dimension of the fifth image data is greater than the dimension of the sixth image data.
[0019] Determine a plurality of second image features in the sixth image data using the second feature association module, and generate eighth image data based on second feature associations of the second image features and the seventh image data;
[0020] using the second feature association module to determine a plurality of first image features in the eighth image data, and determine second feature associations of the first image features, and generating ninth image data based on the second feature associations of the second image features and the eighth image data;
[0021] Receiving the fifth image data, the sixth image data, the seventh image data and the ninth image data transmitted through the jump connection by using the decoding layer, and obtaining tenth image data having the same dimension as the target image data based on the fifth image data, the sixth image data, the seventh image data and the ninth image data;
[0022] Distance information between the target running object and the target distance measuring device is determined based on the tenth image data.
[0023] Optionally, the using the first convolution module to determine first image data corresponding to the initial image data and second image data obtained based on the first image data includes:
[0024] Obtaining a first weight matrix corresponding to the first convolution module;
[0025] Performing convolution processing on the initial image data based on the first weight matrix to obtain first image data;
[0026] The first image data is convolved based on the first weight matrix to obtain second image data.
[0027] Optionally, it is characterized in that, in the second sub-model, the second convolution module is used to determine fifth image data corresponding to the target image data and sixth image data obtained based on the fifth image data, including:
[0028] Obtaining a second weight matrix corresponding to the second convolution module;
[0029] Performing convolution processing on the target image data based on the second weight matrix to obtain fifth image data;
[0030] The fifth image data is convolved based on the second weight matrix to obtain sixth image data.
[0031] Optionally, it also includes:
[0032] Preprocessing the initial image data, wherein the preprocessing includes any one or more of the following:
[0033] Add random noise processing, data enhancement processing, and normalization processing.
[0034] A distance measuring device, comprising:
[0035] An initial image data acquisition module, used to acquire initial image data collected for a moving object;
[0036] A distance information determination module is used to input the initial image data into a target model, determine the distance information between the target moving object in the initial image data and the distance measuring device based on the feature correlation between multiple image features in the initial image data in the target model, and output the distance information, wherein the target model is used to determine the feature correlation between image features based on a preset attention mechanism.
[0037] An electronic device comprises a processor, a memory and a computer program stored in the memory and capable of running on the processor, wherein the computer program implements the distance measurement method as described above when executed by the processor.
[0038] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the distance measurement method described above is implemented.
[0039] The embodiments of the present invention have the following advantages:
[0040] In an embodiment of the present invention, initial image data collected for a moving object can be obtained; then the initial image data is input into a target model, and in the target model, the distance information between the target moving object in the initial image data and the distance measuring device is determined based on the feature correlation between multiple image features in the initial image data, and the distance information is output, so that the correlation between the features of the target moving object can be captured globally, which helps the model to better understand the position, shape and posture information of the detection target in the overall image. At the same time, it can reduce detection errors in strong light, dark light and complex backgrounds and avoid data information compression loss. In addition, in an embodiment of the present invention, an attention mechanism is used to extract associated features to improve feature extraction efficiency. BRIEF DESCRIPTION OF THE DRAWINGS
[0041] In order to more clearly illustrate the technical solution of the present invention, the accompanying drawings required for use in the description of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.
[0042] Figure 1 is a flow chart of steps of a distance measurement method provided by one embodiment of the present invention;
[0043] Figure 2 is a flowchart of another distance measurement method provided by an embodiment of the present invention;
[0044] Figure 3a It is a schematic diagram of an overall architecture of a detection network provided by an embodiment of the present invention;
[0045] Figure 3b It is a schematic diagram of the overall architecture of a depth estimation network provided by an embodiment of the present invention;
[0046] Figure 4 It is a schematic diagram of the structure of a distance measurement device provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0047] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0048] Reference Figure 1 , shows a flow chart of the steps of a distance measurement method provided by an embodiment of the present invention, which is applied to a distance measurement device and may specifically include the following steps:
[0049] Step S101, acquiring initial image data collected for a moving object;
[0050] Among them, the embodiments of the present invention can be applied to distance measuring equipment, and the distance measuring equipment can be a robot in an automobile factory. The robot can be provided with an image device module for collecting image data, and then analyzing the collected image data to determine the real-time distance between the robot and the surrounding running objects, thereby ensuring the safety of the working environment.
[0051] In one embodiment of the present invention, the initial image data is preprocessed; the preprocessing may include any one or more of adding random noise processing, data enhancement processing, and normalization processing.
[0052] Random noise processing can improve the robustness and generalization ability of the model, data enhancement processing can help the model learn a wider range of features, and normalization processing can make the data have the same scale, thus facilitating calculation.
[0053] Among them, data enhancement processing can include rotation, scaling, cropping, flipping, color change and the like.
[0054] In the embodiments of the present invention, various data processing operations (such as image scaling, noise addition, data enhancement, etc.) can be used to improve the accuracy of data and the generalization ability of the model, thereby enhancing the adaptability of the model to different scenarios.
[0055] Step S102, input the initial image data into the target model, determine the distance information between the target moving object in the initial image data and the distance measuring device based on the feature correlation between multiple image features in the initial image data in the target model, and output the distance information, wherein the target model can be used to determine the feature correlation between image features based on a preset attention mechanism.
[0056] After the collected initial image data is processed, it can be input into the target model for model processing. The target model is a distance model used to detect the distance between the target running object and the distance measuring device in the initial image data. The input data of the target model is the initial image data or data preprocessed based on the initial image data. The output data of the target model is the distance information between the distance measuring device and the target running object.
[0057] A self-attention mechanism can be pre-trained in the target model. The self-attention mechanism can learn the correlation between various features in each image data, and then perform feature extraction based on the correlation to obtain the distance information between the target running object and the distance measurement device.
[0058] In an embodiment of the present invention, initial image data collected for a moving object can be obtained; then the initial image data is input into a target model, and in the target model, the distance information between the target moving object in the initial image data and the distance measuring device is determined based on the feature correlation between multiple image features in the initial image data, and the distance information is output, so that the correlation between the features of the target moving object can be captured globally, which helps the model to better understand the position, shape and posture information of the detection target in the overall image. At the same time, it can reduce detection errors in strong light, dark light and complex backgrounds and avoid data information compression loss. In addition, in an embodiment of the present invention, an attention mechanism is used to extract associated features to improve feature extraction efficiency.
[0059] Reference Figure 2 , shows a flowchart of another distance measurement method provided by an embodiment of the present invention, which may specifically include the following steps:
[0060] Step S201, acquiring initial image data collected for a moving object;
[0061] Step S202, inputting the initial image data into a target model, where the target model includes a first sub-model for performing target running object detection and a second sub-model for performing depth estimation;
[0062] In the embodiment of the present invention, the first sub-model is used to determine the running object detection, that is, the image feature area corresponding to the target running object can be determined by the initial image data. The second sub-model determines the distance relationship between each target running object and the distance measurement device based on the image feature area determined by the first sub-model.
[0063] Step S203, in the first sub-model, based on the first feature correlation between the plurality of first image features in the initial image data, determining the target image data corresponding to the target running object from the initial image data;
[0064] After the initial image data is input into the target model, the target sub-model may be applied first to obtain the target image data, and then the target sub-model may be applied to process the target image data to obtain the distance information.
[0065] In the first sub-model, the self-attention mechanism can be applied to perform global analysis of image features to determine the first feature correlation, and then feature extraction can be performed based on the first feature correlation. In this way, the correlation between target object features can be captured globally, which helps the model better understand the position, shape and posture information of the detection target in the overall image.
[0066] In one embodiment of the present invention, the first sub-model includes a first convolution module, a first feature association module and a fusion module, and based on the first feature association between multiple first image features in the initial image data, determining the target image data corresponding to the target running object from the initial image data includes the following sub-steps:
[0067] Sub-step 11, in the first sub-model, a first convolution module is used to determine first image data corresponding to the initial image data and second image data obtained based on the first image data, and the dimension of the first image data is greater than the dimension of the second image data.
[0068] Specifically, a first weight matrix corresponding to the first convolution module is obtained; the initial image data is convolved based on the first weight matrix to obtain first image data; the first image data is convolved based on the first weight matrix to obtain second image data.
[0069] The first weight matrix may be set according to actual needs, and is not limited in the embodiment of the present invention.
[0070] The first convolution module contains one or more convolution kernels, each of which is a small, fixed-size weight matrix (e.g. 3x3, 5x5, etc.). Each convolution kernel slides over the input feature map, performing element-wise multiplication and addition operations to generate one channel of the output feature map.
[0071] Specifically, the convolution kernel starts from the upper left corner of the input feature map and slides on the feature map according to a certain stride. At each position, the convolution kernel performs an element-by-element multiplication operation with the corresponding area of the input feature map, and then adds the results to form a pixel value of the output feature map. This process is repeated on the entire input feature map until the entire feature map is covered.
[0072] Sub-step 12, using a first feature association module to determine a plurality of first image features in the second image data, and determine first feature associations of the first image features;
[0073] In one example, the first feature association module can be composed of UIB (universal inverted neck structure) and MMQA (mobile accelerator optimized lightweight multi-head attention). During the convolution process of the convolution layer, the first feature association module is combined to analyze the global image features to determine the correlation of image features.
[0074] Sub-step 13, generating third image data based on the first feature correlation and the second image data;
[0075] When performing the convolution operation, the convolution module can process the second image data in combination with the first feature association to generate third image data.
[0076] Sub-step 14, using a fusion module to fuse the first image data, the second image data and the third image data to obtain fourth image data;
[0077] In the convolution module, the image size can be reduced by convolution layer by layer, and the image features can be extracted in a concentrated manner. The image data of the second image data dimension can be decoded based on the third image data, and the image data can be fused with the second image data. Similarly, the fused image data can be decoded and then fused with the first image data to obtain the fourth image data. In the aforementioned fusion process, image data of different scales are finally combined, thereby avoiding detection errors and improving data accuracy.
[0078] Sub-step 15: extracting target image data corresponding to the target running object from the fourth image data using a preset target detection algorithm.
[0079] In one embodiment of the present invention, the second sub-model includes a second convolution module, a second feature association module and a decoding layer, and the target image data is input into the second sub-model. In the second sub-model, based on the second feature association between multiple second image features in the target image data, the distance information between the target running object and the target distance measurement device is determined, including the following steps:
[0080] Sub-step S21, in the second sub-model, a second convolution module is used to determine fifth image data corresponding to the target image data and sixth image data obtained based on the fifth image data, and the dimension of the fifth image data is greater than the dimension of the sixth image data.
[0081] Specifically, a second weight matrix corresponding to the second convolution module is obtained; convolution processing is performed on the target image data based on the second weight matrix to obtain fifth image data; and convolution processing is performed on the fifth image data based on the second weight matrix to obtain sixth image data.
[0082] The second weight matrix may be set according to actual needs, and is not limited in the embodiment of the present invention.
[0083] The second convolution module contains one or more convolution kernels, each of which is a small, fixed-size weight matrix (e.g., 3x3, 5x5, etc.). Each convolution kernel slides over the input feature map, performing element-wise multiplication and addition operations to generate one channel of the output feature map.
[0084] Specifically, the convolution kernel starts from the upper left corner of the input feature map and slides on the feature map according to a certain stride. At each position, the convolution kernel performs an element-by-element multiplication operation with the corresponding area of the input feature map, and then adds the results to form a pixel value of the output feature map. This process is repeated on the entire input feature map until the entire feature map is covered.
[0085] Sub-step S22, using a second feature association module to determine a plurality of second image features in the sixth image data, and generating eighth image data based on the second feature associations of the second image features and the seventh image data;
[0086] Sub-step S23, using a second feature association module to determine a plurality of first image features in the eighth image data, and determine a second feature association of the first image features, and generate ninth image data based on the second feature association of the second image features and the eighth image data;
[0087] Sub-step S24, using a decoding layer to receive fifth image data, sixth image data, seventh image data and ninth image data transmitted through the jump connection, and obtaining tenth image data having the same dimension as the target image data based on the fifth image data, sixth image data, seventh image data and ninth image data;
[0088] Sub-step S25 , determining the distance information between the target running object and the target distance measuring device based on the tenth image data.
[0089] Step S204: input the target image data into the second sub-model, and in the second sub-model, determine the distance information between the target running object and the target distance measuring device based on the second feature correlation between multiple second image features in the target image data.
[0090] In the embodiment of the present invention, it can be applied to a robot based on an automobile production scenario. The robot can pre-train a target model and then apply the target model on the vehicle side. Specifically, it can include: data collection, data labeling, data labeling, data processing, model training, and distance measurement. The specific process of each stage is as follows:
[0091] (i) Data collection: The data involved in this method is the moving object data collected from the perspective of the robot in the automobile production scene, such as pedestrians, vehicles and other moving targets, which are specifically represented by their corresponding RGB images and depth maps;
[0092] (ii) Data labeling: This step is only performed on data related to motion object detection. First, a small amount of the collected data is manually labeled, and then fine-tuned based on the pre-trained weights of the large detection model grounding dino with general zero-shot generalization capability. Then, the fine-tuned model weights are used to pre-label the remaining large batches of data.
[0093] (III) Data labeling: Manual verification of the data that has been pre-labeled by the aforementioned large model.
[0094] (IV) Data processing: For the data processing of the detection task, the image is scaled and adjusted proportionally, and random noise, Mosiac data enhancement, ColorJitter operation and normalization are added. A batch of negative sample data is collected, for example, the batch of negative sample data is sampled according to a preset probability.
[0095] The data augmentation operations used on the sampled data may include image flipping, rotation, scaling, translation, color conversion, cropping and padding, adding noise, Gaussian blur, occlusion, affine transformation, geometric perturbation, and lighting enhancement, etc., to improve the generalization ability and robustness of the corresponding model.
[0096] (V) Model training: The embodiment of the present invention can apply the Mobilenetv4 small network for training. The output of the target model can include two output task heads, one corresponding to the Gaussian heat map of the center point of the target box, and the other corresponding to the upper left and lower right of the target box relative to the center of the robot.
[0097] (VI) Distance measurement: The trained model is quantified and deployed on the robot side. The detection model is used to extract the target frame of the moving object, and the depth estimation model is used to estimate the depth of the frame image. The average of the depth values of the center point of the moving object's target frame and the eight nearest points around it is calculated as the distance from the moving object to the robot.
[0098] The following combination Figure 3a and Figure 3b The distance measurement process of the above embodiment in the embodiment of the present invention is exemplarily described as follows:
[0099] The first sub-model may include a convolutional layer, a FPN (Feature Pyramid Network), a feature association module, and a SSH (Single Stage Headless, detector), wherein the feature association module is composed of a UIB and MMQA structure, UIB (universal inverted neck structure) and MMQA (mobile accelerator optimized lightweight multi-head attention) complement each other and jointly realize the determination of feature relevance, wherein,
[0100]
[0101] Among them, "SR" means space reduction. In practical applications, a depth-wise separable convolution (DW) with a preset stride (the preset stride is set according to actual needs and is not restricted too much in the embodiments of the present invention) can be used, or when space reduction is not used, SR means the identity function.
[0102] In the embodiment of the present invention, the UIB and MMQA structures can realize global detection of image data, determine the correlation of features in the image, and thus reduce information loss during each convolution operation.
[0103] Two optional depthwise convolutions (DWs) are introduced in the inverted bottleneck block, one before the expansion layer and the other between the expansion layer and the projection layer. The presence or absence of these two depthwise convolutions is part of the neural architecture search (NAS) optimization process, which ultimately generates novel network architectures.
[0104] The inverted neck structure is flexibly constructed during the neural network architecture search process, eliminating the need to manually formulate scaling rules. In addition, combined with the neural network architecture search algorithm, this method enables parameter sharing across different instantiations, making neural architecture search extremely efficient.
[0105] The first sub-model includes 5 convolutional layers, which can extract features from input data (usually image data) through convolution operations.
[0106] Taking a 480*288*3 RGB image as an example, when extracting moving object features based on the MobileNetV4 detection method, the overall process of feature map conversion is as follows: Figure 3a shown.
[0107] After the convolution operation of the first convolution layer, the image data of 480*288*3 is converted from 480*288*3 to 240*144, reducing the dimension of the image data. After the convolution processing of the second convolution layer, 240*144 is converted to 120*72. After the convolution processing of the third convolution layer, 120*72 is converted to 60*36. At this time, the UIB and MMQA of the feature association module can be applied to determine the global feature association information within the image 60*36. According to the association information, the convolution operation is performed again to obtain 30*18 image data. Then, the UIB and MMQA of the feature association module are applied to determine the global feature association information within the image data 30*18. According to the association information, the convolution operation is performed again to obtain 15*9 image data.
[0108] In FPN, multiple image data of different scales transmitted by the convolutional layer can be received, and feature fusion can be realized through the decoder. For example, 30*18 image data can be obtained by decoding 15*9 image data, and the decoded 30*18 image data is fused with the previous 30*18 image data, and then the 30*18 image data is decoded to obtain 60*36 image data, and the decoded 60*36 image data is fused with the previous 60*36 image data, and the fused 60*36 is decoded to obtain 120*72 image data, and then the 120*72 image data is fused and the 120*72 image data is input into SSH. In SSH, different sizes of objects are processed by applying detection modules at different levels of the network, which helps to detect small and large objects.
[0109] Taking a 224*224*3 RGB image as an example, when extracting moving object features based on the depth estimation method of MobileNetV4, the overall process of feature map conversion is as follows: Figure 3b shown.
[0110] After the convolution operation of the first convolution layer, the image data of 224*224*3 is converted from 224*224*3 to 112*112, reducing the dimension of the image data. After the convolution processing of the second convolution layer, 112*112 is converted to 56*56. After the convolution processing of the third convolution layer, 56*56 is converted to 28*28. At this time, the UIB and MMQA of the feature association module can be applied to determine the global feature association information within the image 28*28, and the convolution operation is performed again according to the association information to obtain 14*14 image data. Then, the UIB and MMQA of the feature association module are applied to determine the global feature association information within the image data 14*14, and then the convolution operation is performed again according to the association information to obtain 7*7 image data.
[0111] In the second sub-model, skip connections can be set between the convolutional layer and the decoding layer, similar to the shortcut connections in networks such as U-Net or DeeperLab, to restore image details. These skip connections add the feature maps in the encoder to the corresponding layers in the decoder instead of concatenating them, which can reduce the subsequent calculation pressure.
[0112] "Upsample" in the field of deep learning and image processing refers to the process of increasing the spatial resolution of data or images. In neural networks, upsampling is used in the decoder part, especially in tasks such as generative models, image segmentation, super-resolution reconstruction, and depth estimation. The purpose of upsampling is to convert low-resolution feature maps into higher-resolution feature maps so that more detailed output can be generated. Specifically, upsampling can be achieved through interpolation, transposed convolution, up-pooling, sub-pixel convolution, etc.
[0113] In the decoding layer, the 7*7 image data is converted to 14*14 image data by upsampling, and then upsampled based on the 14*14 image data to obtain 28*28 image data. The decoding layer fuses the 28*28 image data received from the convolution layer through the skip connection with the 28*28 image data obtained by upsampling, and upsamples the fused 28*28 image data to obtain 56*56 image data. The 56*56 image data is fused with the 56*56 image data received from the convolution layer by the decoding layer through the skip connection, and the fused 56*56 image data is obtained. After the image data is upsampled, 112*112 image data is obtained, and then the 112*112 image data is fused with the 112*112 image data received by the decoding layer from the convolution layer through the jump connection, and then upsampled to obtain 224*224 image data. A convolution operation is performed using a 1×1 matrix weight to output a depth map. The depth map contains depth information between the moving object and the target moving object. The distance from the moving object to the robot (i.e., the distance measurement device in the embodiment of the present invention) is calculated by calculating the average of the depth values of the center point of the target frame of the moving object and the eight nearest points around it.
[0114] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.
[0115] Reference Figure 4 , shows a schematic diagram of the structure of a distance measurement device provided by an embodiment of the present invention, which may specifically include the following modules:
[0116] An initial image data acquisition module 401 is used to acquire initial image data collected for a moving object;
[0117] The distance information determination module 402 is used to input the initial image data into a target model, determine the distance information between the target moving object in the initial image data and the distance measuring device based on the feature correlation between multiple image features in the initial image data in the target model, and output the distance information, wherein the target model is used to determine the feature correlation between image features based on a preset attention mechanism.
[0118] In one embodiment of the present invention, the target model includes a first sub-model for performing target running object detection and a second sub-model for performing depth estimation, the image feature includes a first image feature and a second image feature, the feature correlation includes a first feature correlation and a second feature correlation, and the distance information determination module 402 may include:
[0119] A first sub-model processing sub-module, configured to determine, in the first sub-model, target image data corresponding to the target running object from the initial image data based on first feature correlations between a plurality of first image features in the initial image data;
[0120] The second sub-model processing sub-module is used to input the target image data into the second sub-model, and in the second sub-model, determine the distance information between the target running object and the target distance measuring device based on the second feature correlation between multiple second image features in the target image data.
[0121] In one embodiment of the present invention, the first sub-model includes a first convolution module, a first feature association module and a fusion module, and the first sub-model processing sub-module may include:
[0122] The first convolution processing unit is used to determine, in the first sub-model, first image data corresponding to the initial image data and second image data obtained based on the first image data by using the first convolution module, wherein the dimension of the first image data is greater than the dimension of the second image data.
[0123] a first feature association unit, configured to determine a plurality of first image features in the second image data using the first feature association module, and determine first feature associations of the first image features;
[0124] A third image generating unit, configured to generate third image data based on the first feature correlation and the second image data;
[0125] a fusion unit, configured to fuse the first image data, the second image data and the third image data using the fusion module to obtain fourth image data;
[0126] The target image data determining unit is used to extract the target image data corresponding to the target running object from the fourth image data by using a preset target detection algorithm.
[0127] In one embodiment of the present invention, the second sub-model includes a second convolution module, a second feature association module and a decoding layer, and the second sub-model processing sub-module may include:
[0128] A second convolution processing unit is used to determine, in the second sub-model, fifth image data corresponding to the target image data and sixth image data obtained based on the fifth image data by using the second convolution module, wherein the dimension of the fifth image data is greater than the dimension of the sixth image data.
[0129] a second feature association unit, configured to determine a plurality of second image features in the sixth image data by using the second feature association module, and generate eighth image data based on second feature associations of the second image features and the seventh image data;
[0130] a ninth image data generating unit, configured to determine a plurality of first image features in the eighth image data by using the second feature association module, determine second feature associations of the first image features, and generate ninth image data based on the second feature associations of the second image features and the eighth image data;
[0131] a tenth image data generating module, configured to receive, by using the decoding layer, the fifth image data, the sixth image data, the seventh image data, and the ninth image data transmitted through the jump connection, and obtain, based on the fifth image data, the sixth image data, the seventh image data, and the ninth image data, tenth image data having the same dimension as the target image data;
[0132] A distance information determining unit is used to determine the distance information between the target running object and the target distance measuring device based on the tenth image data.
[0133] In one embodiment of the present invention, the first convolution processing unit may include:
[0134] A first weight matrix determining subunit, used to obtain a first weight matrix corresponding to the first convolution module;
[0135] A first image data determination subunit, configured to perform convolution processing on the initial image data based on the first weight matrix to obtain first image data;
[0136] The second image data determination subunit is used to perform convolution processing on the first image data based on the first weight matrix to obtain second image data.
[0137] In one embodiment of the present invention, in the second sub-model, the second convolution processing unit may include:
[0138] A second weight matrix determination subunit, used to obtain a second weight matrix corresponding to the second convolution module;
[0139] a fifth image data determination subunit, configured to perform convolution processing on the target image data based on the second weight matrix to obtain fifth image data;
[0140] The sixth image data determination subunit is used to perform convolution processing on the fifth image data based on the second weight matrix to obtain sixth image data.
[0141] In one embodiment of the present invention, the device may further include:
[0142] A preprocessing module is used to preprocess the initial image data, wherein the preprocessing includes any one or more of the following:
[0143] Add random noise processing, data enhancement processing, and normalization processing.
[0144] In an embodiment of the present invention, initial image data collected for a moving object can be obtained; then the initial image data is input into a target model, and in the target model, the distance information between the target moving object in the initial image data and the distance measuring device is determined based on the feature correlation between multiple image features in the initial image data, and the distance information is output, so that the correlation between the features of the target moving object can be captured globally, which helps the model to better understand the position, shape and posture information of the detection target in the overall image. At the same time, it can reduce detection errors in strong light, dark light and complex backgrounds and avoid data information compression loss. In addition, in an embodiment of the present invention, an attention mechanism is used to extract associated features to improve feature extraction efficiency.
[0145] An embodiment of the present invention further provides an electronic device, which may include a processor, a memory, and a computer program stored in the memory and capable of running on the processor. When the computer program is executed by the processor, the above distance measurement method is implemented.
[0146] An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the distance measurement method described above is implemented.
[0147] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0148] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.
[0149] Those skilled in the art will appreciate that the embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the embodiments of the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.
[0150] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0151] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0152] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0153] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.
[0154] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprises" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.
[0155] The above provides a detailed introduction to a distance measurement method and device, an electronic device, and a storage medium. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for a person skilled in the art, according to the idea of the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.
Claims
1. A distance measurement method, characterized in that: Applied to a distance measuring device, the method comprises: Acquire initial image data collected for the moving object; The initial image data is input into a target model, and the distance information between the target moving object in the initial image data and the distance measuring device is determined in the target model based on the feature correlation between multiple image features in the initial image data, and the distance information is output, wherein the target model is used to determine the feature correlation between image features based on a preset attention mechanism.
2. The method according to claim 1, characterized in that The target model includes a first sub-model for detecting a target moving object and a second sub-model for depth estimation, the image feature includes a first image feature and a second image feature, the feature correlation includes a first feature correlation and a second feature correlation, and determining the distance information between the target moving object in the initial image data and the distance measuring device based on the feature correlation between multiple image features in the initial image data in the target model includes: In the first sub-model, based on the first feature correlation between the plurality of first image features in the initial image data, target image data corresponding to the target running object is determined from the initial image data; The target image data is input into the second sub-model, and in the second sub-model, the distance information between the target running object and the target distance measuring device is determined based on the second feature correlation between multiple second image features in the target image data.
3. The method according to claim 2, characterized in that The first sub-model includes a first convolution module, a first feature association module and a fusion module. The method of determining target image data corresponding to the target running object from the initial image data based on the first feature association between multiple first image features in the initial image data includes: In the first sub-model, the first convolution module is used to determine first image data corresponding to the initial image data and second image data obtained based on the first image data, and the dimension of the first image data is greater than the dimension of the second image data; Determine a plurality of first image features in the second image data using the first feature association module, and determine first feature associations of the first image features; generating third image data based on the first feature association and the second image data; Using the fusion module to fuse the first image data, the second image data and the third image data to obtain fourth image data; A preset target detection algorithm is used to extract target image data corresponding to the target running object from the fourth image data.
4. The method according to claim 2, characterized in that The second sub-model includes a second convolution module, a second feature association module and a decoding layer. The target image data is input into the second sub-model. In the second sub-model, based on the second feature association between multiple second image features in the target image data, the distance information between the target running object and the target distance measurement device is determined, including: In the second sub-model, the second convolution module is used to determine fifth image data corresponding to the target image data and sixth image data obtained based on the fifth image data, and the dimension of the fifth image data is greater than the dimension of the sixth image data; Determine a plurality of second image features in the sixth image data using the second feature association module, and generate eighth image data based on second feature associations of the second image features and the seventh image data; using the second feature association module to determine a plurality of first image features in the eighth image data, and determine second feature associations of the first image features, and generating ninth image data based on the second feature associations of the second image features and the eighth image data; Receiving the fifth image data, the sixth image data, the seventh image data and the ninth image data transmitted through the jump connection by using the decoding layer, and obtaining tenth image data having the same dimension as the target image data based on the fifth image data, the sixth image data, the seventh image data and the ninth image data; Distance information between the target running object and the target distance measuring device is determined based on the tenth image data.
5. The method according to claim 3, characterized in that: The using the first convolution module to determine the first image data corresponding to the initial image data and the second image data obtained based on the first image data includes: Obtaining a first weight matrix corresponding to the first convolution module; Performing convolution processing on the initial image data based on the first weight matrix to obtain first image data; The first image data is convolved based on the first weight matrix to obtain second image data.
6. The method according to any one of claims 4, characterized in that The step of using the second convolution module in the second sub-model to determine fifth image data corresponding to the target image data and sixth image data obtained based on the fifth image data includes: Obtaining a second weight matrix corresponding to the second convolution module; Performing convolution processing on the target image data based on the second weight matrix to obtain fifth image data; The fifth image data is convolved based on the second weight matrix to obtain sixth image data.
7. The method according to any one of claims 1 to 6, characterized in that Also includes: Preprocessing the initial image data; The pre-processing includes any one or more of the following: Add random noise processing, data enhancement processing, and normalization processing.
8. A distance measuring device, characterized in that: The device comprises: An initial image data acquisition module, used to acquire initial image data collected for a moving object; A distance information determination module is used to input the initial image data into a target model, determine the distance information between the target moving object in the initial image data and the distance measuring device based on the feature correlation between multiple image features in the initial image data in the target model, and output the distance information, wherein the target model is used to determine the feature correlation between image features based on a preset attention mechanism.
9. An electronic device, characterized in that: The method comprises a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein when the computer program is executed by the processor, the distance measurement method according to any one of claims 1 to 7 is implemented.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the distance measurement method according to any one of claims 1 to 7 is implemented.