Oil tank truck barrel opening positioning method, device and equipment and storage medium
By combining the object detection model and the depth estimation model, using the EfficientNet network and the structure perception module SPM, combined with the multi-scale feature fusion module and the CA attention mechanism, the problem of lightweight depth estimation algorithm prediction fuzzy at the accuracy and edge prediction is solved, achieving higher depth estimation accuracy and clearer edges.
Patent Information
- Application Number
- CN202411776078.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-05
- Publication Date
- 2025-04-11
AI Technical Summary
Existing lightweight depth estimation algorithms have problems with accuracy and edge prediction fuzzy.
By combining the object detection model and the depth estimation model, the EfficientNet network is used as the depth encoder, and the structure perception module SPM is applied on the features with the minimum resolution, and the depth estimation is performed by combining the multi-scale feature fusion module and the CA attention mechanism.
It improves the accuracy of depth estimation, solves the problem of blurring edges of lightweight models, and enhances the understanding of scene structure.
Smart Images
Figure CN119810191A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of filling technology, and in particular to a method, device, equipment and storage medium for locating the barrel opening of a tanker truck. Background Art
[0002] The depth information of an object is generally obtained through commercial sensors such as lasers and radars. This method is not only costly, but also has the disadvantage of short perception distance, which limits the widespread application of depth estimation. With the rapid development of deep learning, depth estimation has become an important research topic in the field of computer vision. It aims to generate pixel-by-pixel depth maps from images of a specific perspective. The obtained depth information can be used to better understand 3D scenes, perform indoor positioning, and estimate height. Vision-based depth estimation is divided into two categories: supervised and self-supervised. Supervised algorithms require RGB images to have corresponding depth labels, and the collection of depth labels usually requires depth cameras or lidars. At the same time, the collected depth labels are usually sparse points that cannot be well matched with the original image. Therefore, self-supervised methods are a research trend in recent years.
[0003] With the gradual application of self-supervised depth estimation in the fields of robots and automated equipment, the efficiency of the model has become increasingly important. Existing technologies achieve model lightweighting by reducing the number of layers in the feature extraction network or changing the convolution in the network. For example, DNA-Depth (Deep neighbor layer aggregation for lightweight selfsupervised monocular depth estimation) uses EfficientNet as the backbone network and redesigns the feature fusion module to achieve prediction results comparable to those of large models with fewer training parameters. However, after the depth estimation model is lightweighted, although the number of model parameters is reduced, the network will be unable to capture complex features in the image, thereby reducing the accuracy of the model's depth estimation. Summary of the invention
[0004] In view of this, the present application aims to provide a method, device, equipment and storage medium for locating the barrel mouth of a tank truck to solve the technical problems of decreased accuracy and blurred edge prediction of existing lightweight depth estimation algorithms.
[0005] On the one hand, the present application provides a method for locating the barrel opening of a tank truck, the method comprising the following steps:
[0006] S10, acquiring a target frame image of the tanker truck;
[0007] S20, inputting the acquired tanker target frame image into the trained target detection model and depth estimation model;
[0008] S30, detecting the barrel mouth position in the input tanker target frame image through the target detection model, and outputting the barrel mouth position detection result; performing pixel-by-pixel depth estimation on the input tanker target frame image through the depth estimation model, specifically comprising: S31, first inputting the input tanker target frame image into the depth encoder of the depth estimation model to obtain feature maps of different resolutions; S32, then inputting the obtained feature maps of different resolutions into the multi-scale feature fusion module of the depth estimation model, using the visual attention method to fuse different features, and obtaining a fused feature map; S33, finally sending the obtained fused feature map into the depth decoder of the depth estimation model to obtain a corresponding depth map, and outputting the depth value of each pixel in the image;
[0009] S40, combining the depth value result of the pixel point in the image with the barrel mouth position detection result to obtain the depth value of the barrel mouth of the tank truck, thereby completing the positioning of the barrel mouth of the tank truck.
[0010] Furthermore, the deep encoder uses the EfficientNet network as a feature extraction network and applies a structure perception module SPM on the minimum resolution features to simulate the interdependence between channels.
[0011] Furthermore, the structure perception module SPM first extracts the feature map X∈R C×H×W Reshape to X′∈R C×N , where N = H × W, C represents the number of image channels, and then matrix multiplication is performed between X′ and the transpose of X′ to calculate the feature similarity matrix S∈R C×C , S is calculated as follows: Where i and j represent different channels, then take the maximum value of S row by row to obtain a new matrix, and perform element-by-element subtraction from S to convert the similarity matrix S into the discriminant matrix D∈R C×C , calculated as formula D ij =max i (S)-S i,j , where max represents the maximum value operation row by row; next, the softmax layer is applied to obtain the attention mapping matrix A∈R C×C , calculated as follows Where exp represents the application of an exponential function to D, and ∑ represents a summation operation; finally, a matrix multiplication is performed between the transpose of A and the reshaped feature X', and the result is reshaped, and then an element-by-element summation operation is performed between the result and the reshaped feature X' to obtain the final output E∈R C×H×W , calculated as follows
[0012] Furthermore, the multi-scale feature fusion module adopts a detail fusion module DFM, which learns the importance of different resolution features and different channel features of the same resolution features in a learnable way, and assigns different weights to different channel features according to their importance, so that the network focuses on specific resolution features and channel features, emphasizes important details, and fuses the effects of different scale features to generate clear edges; the multi-scale feature fusion module downsamples the high-resolution features and upsamples the low-resolution features. When the sizes of the fused features are consistent, the feature maps of different resolutions are sent to the DFM for fusion to obtain the fused features.
[0013] Furthermore, the multi-scale feature fusion module also performs a residual connection on the initial features and the fused features.
[0014] Furthermore, the detail fusion module DFM first concatenates the low-resolution feature L and the high-resolution feature H to obtain the feature M, and then sends the feature M to a convolution layer consisting of a 3×3 convolution, a BN normalization and a GELU activation function to obtain the feature U, and the calculation formula is: U=GELU(BN(Conv3×3(M))), where Conv3×3 represents a 3×3 convolution operation, BN represents a normalization operation, and GELU represents a GELU activation function; then, U is compressed into a vector through global average pooling to obtain global context information, and then two 1×1 convolutions, a GELU activation function and a Sigmoid function are used to calculate the weight vector v to adjust the importance of different channel features. The calculation formula is: Among them, Conv1×1 represents a 1×1 convolution operation, and Sigmoid represents a Sigmoid activation function operation; then v and U are multiplied element by element to generate a weighted feature Z; finally, feature U and weighted feature Z are added to obtain the final output O, and the calculation formula is: o=v⊙U+U.
[0015] Furthermore, the depth decoder first passes the fused feature map through CA attention, and then passes through a 3×3 convolution and a Sigmoid function to obtain a depth map; the CA attention first performs global average pooling on the input features along the horizontal direction x and the vertical direction y respectively to obtain feature maps in two directions, and the calculation formulas are: Where X represents the input feature, and Z represents the features in two directions after global average pooling. Then the acquired feature maps are concatenated and passed into the shared 1×1 convolution for channel reduction. After that, the features are subjected to BN normalization and nonlinear activation function operations, and they are divided according to the ratio of length to width. The calculation formula is: f = δ(BN(Conv1×1([Z H ,Z W]))), where [·,·] represents the cascade operation of the spatial dimension, and δ represents the nonlinear activation function; then the segmented features are convolved with 1×1 to restore the number of feature channels to the number before channel reduction; finally, the feature maps in the two directions are respectively used with the Sigmoid function to obtain the attention weights, and then multiplied with the original feature maps. The calculation formulas are: g H =Sigmoid(Conv1×1(f H )), g W =Sigmoid(Conv1×1(f W )), the final feature weighted multiplication process can be expressed as: Where Z represents the output feature, X represents the input feature, and g represents the attention weight calculated above.
[0016] On the other hand, the present application also provides a tanker barrel opening positioning device, the device comprising:
[0017] An image acquisition module is used to acquire a target frame image of the tanker truck;
[0018] A target detection module is used to detect the barrel mouth position in the target frame image of the tank truck acquired by the image acquisition module using the target detection model, and output the barrel mouth position detection result in the image;
[0019] A depth estimation module is used to use the above-mentioned depth estimation model to perform pixel-by-pixel depth estimation on the target frame image of the tanker truck acquired by the image acquisition module, and output the depth value of each pixel in the image;
[0020] The depth positioning module is used to combine the depth value results of the pixel points in the image with the barrel mouth position detection results to obtain the depth value of the barrel mouth of the tank truck and complete the positioning of the barrel mouth of the tank truck.
[0021] On the other hand, the present application also provides a device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the above-mentioned method for locating the barrel mouth of a tank truck is implemented.
[0022] On the other hand, the present application also provides a computer-readable storage medium having computer program instructions stored thereon, and the computer program instructions, when executed by a processor, implement the above-mentioned method for locating the barrel mouth of a tank truck.
[0023] One or more technical solutions proposed in this application have at least the following beneficial effects compared with the prior art:
[0024] In this application, the target frame image of the tanker truck is acquired and the plane position coordinates of the mouth of the tanker truck barrel are located through the target detection model. At the same time, the depth value of each pixel in the image is estimated through the depth estimation model. The detection results of the target detection model and the depth estimation model are combined to obtain the depth value corresponding to the mouth of the tanker truck barrel. When the depth of the pixel points in the image is estimated: the depth encoder of the depth estimation model adds a structure emphasis module SPM to the minimum resolution feature, uses the self-attention mechanism to capture long-distance dependencies, aggregates the discriminant features on the channel dimension, explicitly enhances the understanding of the scene structure, obtains rich scene structure context, and effectively improves the accuracy of the lightweight depth estimation algorithm; secondly, the multi-scale feature fusion module of the depth estimation model adopts The detail fusion module DFM is used to replace the simple splicing operation, and the local features are further processed to obtain more details. A learnable method is used to make the network focus on specific channels and detail features, and effectively fuse features of different resolutions to make the edges of the depth map clearer. In addition, in order to supplement the pixel information lost in the convolution process, the original features output by the depth encoder are connected with the fused features through a residual connection to obtain the lost pixel information of the edge of the target object and increase the clarity of the edge of the target object. Then, in the depth decoder of the depth estimation model, CA attention is used to integrate position information, obtain richer features of different dimensions, and generate a spatial selective attention map, which can further improve the accuracy of the algorithm's depth estimation.
[0025] The depth estimation model provided in this application integrates visual attention and multi-scale features, which improves the accuracy of depth estimation while maintaining a small number of parameters, and solves the problem of blurred edges in lightweight models. BRIEF DESCRIPTION OF THE DRAWINGS
[0026] In order to more clearly illustrate the technical solutions of the exemplary embodiments of the present invention, the following briefly introduces the drawings required to describe the embodiments. Obviously, the drawings introduced are only drawings of a part of the embodiments to be described in the present invention, rather than all the drawings. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0027] Figure 1 The present invention is a flow chart of a method for locating a barrel opening of a tanker truck according to an exemplary embodiment.
[0028] Figure 2 It is a schematic diagram of the structure of a depth estimation model according to an exemplary embodiment.
[0029] Figure 3 The present invention is a specific flow chart of performing pixel-by-pixel depth estimation on an input image using a depth estimation model according to an exemplary embodiment.
[0030] Figure 4 FIG. 4 is a schematic diagram of the structure of a structure perception module SPM according to an exemplary embodiment.
[0031] Figure 5 The figure is a schematic diagram of the structure of a multi-scale feature fusion module according to an exemplary embodiment.
[0032] Figure 6 FIG. 4 is a schematic structural diagram of a detail feature fusion module DFM according to an exemplary embodiment.
[0033] Figure 7 The figure is a schematic diagram of the structure of a deep decoder according to an exemplary embodiment.
[0034] Figure 8 The present invention is a schematic framework diagram of a barrel opening positioning device for a tanker truck according to an exemplary embodiment. DETAILED DESCRIPTION
[0035] In order to better explain the technical solutions in the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in combination with the drawings of the specification and the specific implementation methods. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work should belong to the scope of protection of this application. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail to highlight the purpose of the present application. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.
[0036] Embodiment 1
[0037] See also Figures 1 to 7 This embodiment provides a method for locating the barrel mouth of a tanker truck, which is applied to the liquid filling industry and can realize the barrel mouth positioning of the tanker truck. Of course, it can also be applied to the barrel mouth positioning of other packaging containers in the liquid filling industry and other industrial industries. The positioning method includes the following steps:
[0038] S10, obtaining a target frame image of the tanker truck.
[0039] S20, inputting the acquired tanker truck target frame image into the trained target detection model and depth estimation model.
[0040] S30, detecting the barrel mouth position in the input tanker target frame image through the target detection model, and outputting the barrel mouth position detection result; at the same time, performing pixel-by-pixel depth estimation on the input tanker target frame image through the depth estimation model.
[0041] S40, combining the depth value result of the pixel point in the image with the barrel mouth position detection result to obtain the depth value of the barrel mouth of the tank truck, thereby completing the positioning of the barrel mouth of the tank truck.
[0042] The target detection model can adopt the current mainstream machine vision target detection model, such as YOLOv5, YOLO-G, etc. The specific structure of the target detection model is not limited, as long as the two-dimensional plane position coordinates of the target can be located from the input target image. The process of the target detection model for barrel mouth position detection refers to the existing technology and is not described here.
[0043] The structure of the depth estimation model is as follows Figure 2 As shown in the figure, it mainly includes two parts: a depth estimation network and a pose network; the depth estimation network includes a depth encoder, a multi-scale feature fusion module and a depth decoder; the pose network is the same as DNA-Depth (Deep neighbor layer aggregation for lightweight self supervised monocular depth estimation), which is built on the ResNet-18 architecture, inputs the previous frame image and the current frame image, obtains some rotation and translation information, can represent the relative position between the two frames, and obtains the pose change of the image; finally, the output results of the depth estimation network and the pose network are used to reconstruct the image, calculate the reconstruction loss and edge smoothing loss between the reconstructed image and the original image, backpropagate the loss back to the network, optimize the parameters in the network, and train the depth estimation model by minimizing the total loss, thereby realizing depth estimation.
[0044] The depth estimation model performs pixel-by-pixel depth estimation on the input tanker target frame image, which specifically includes:
[0045] S31, first input the input tanker truck target frame image into the depth encoder of the depth estimation model to obtain feature maps of different resolutions.
[0046] In this embodiment, the deep encoder uses the EfficientNet network as the feature extraction network. First, the original image of the tanker target frame image is input into the EfficientNet network for feature extraction, and 5 feature maps with different resolutions are obtained. The structure perception module SPM is applied to the feature with the smallest resolution to simulate the interdependence between channels and obtain better scene understanding.
[0047] The structure of the structure perception module SPM is as follows Figure 3 As shown, the structure perception module SPM first extracts the feature map X∈RC×H×W Reshape to X′∈R C×N , where N = H × W, C represents the number of image channels, and then matrix multiplication is performed between X′ and the transpose of X′ to calculate the feature similarity matrix S∈R C×C , S is calculated as follows: Where i and j represent different channels, then take the maximum value of S row by row to obtain a new matrix, and perform element-by-element subtraction from S to convert the similarity matrix S into the discriminant matrix D∈R C×C , calculated as formula D ij =max i (S)-S i,j , where max represents the maximum value operation row by row; next, the softmax layer is applied to obtain the attention mapping matrix A∈R C×C , calculated as follows: Where exp represents the application of an exponential function to D, and ∑ represents a summation operation; finally, a matrix multiplication is performed between the transpose of A and the reshaped feature X', and the result is reshaped, and then an element-by-element summation operation is performed between the result and the reshaped feature X' to obtain the final output E∈R C×H×W , calculated as follows:
[0048] S32, then input the acquired feature maps of different resolutions into the multi-scale feature fusion module of the depth estimation model, and fuse different features using a visual attention method to obtain a fused feature map.
[0049] The context feature fusion module of the existing depth estimation model is implemented by performing a simple splicing operation on the feature map, which ignores the semantic gap between features at different levels and lacks processing of local details, resulting in artifacts in the depth map, thereby affecting the accuracy of depth estimation. Therefore, in this embodiment, in order to effectively fuse features of different resolutions and improve the accuracy of depth estimation of important details, when fusing feature maps of different resolutions, the detail fusion module DFM is used to replace the simple splicing operation; in addition, in order to solve the problem of loss of feature information of some pixels due to convolution operation, the initial features are residually connected with the fused features.
[0050] The structure of the multi-scale feature fusion module is as follows Figure 4As shown in the figure, Fi_0 (i = 1, 2, 3, 4, 5) represents the feature maps of different resolutions output by the deep encoder, and Fi_j (i = 0, 1, 2, 3, 4; j = 1, 2, 3, 4) represents the intermediate features of feature fusion. The multi-scale feature fusion module performs downsampling operations on high-resolution features; the multi-scale feature fusion module performs upsampling operations on low-resolution features; the initial features of each layer are connected to the fused features once with a residual connection to supplement the feature information of the lost pixels; when the size of the fused features is consistent, the feature maps of different resolutions are sent to the detail fusion module DFM for fusion to obtain the fused features.
[0051] The multi-scale feature fusion module adopts the detail fusion module DFM, which learns the importance of different resolution features and different channel features of the same resolution feature in a learnable way, and assigns different weights to different channel features according to their importance, so that the network focuses on specific resolution features and channel features, emphasizes important details, and fuses the effects of different scale features to generate clear edges and structures such as Figure 5 shown.
[0052] The detail fusion module DFM first concatenates the low-resolution feature L and the high-resolution feature H to obtain the feature M, and then sends the feature M to a convolution layer consisting of a 3×3 convolution, BN normalization and GELU activation function to obtain the feature U. The calculation formula is:
[0053] U = GELU (BN (Conv3 × 3 (M))), where Conv3 × 3 represents a 3 × 3 convolution operation, BN represents a normalization operation, and GELU represents a GELU activation function. Then, U is compressed into a vector through global average pooling to obtain global context information. Then, two 1 × 1 convolutions, a GELU activation function, and a Sigmoid function are used to calculate the weight vector v to adjust the importance of different channel features. The calculation formula is: Among them, Conv1×1 represents a 1×1 convolution operation, and Sigmoid represents a Sigmoid activation function operation; then v and U are multiplied element by element to generate a weighted feature Z; finally, feature U and weighted feature Z are added to obtain the final output O, and the calculation formula is: O=v⊙U+U.
[0054] Since the weight vector represents the importance of different channels, channels containing key information will receive higher scores. Therefore, this recalibration operation can adaptively emphasize key details of multiple scales of sharp edges, making the model's estimation of these key details more accurate and improving the overall accuracy of the algorithm's depth estimation.
[0055] S33, finally sending the obtained fused feature map to the decoder of the depth estimation model to obtain the corresponding depth map, and outputting the depth value of each pixel in the image.
[0056] In order to process the spatial information of the image well, CA attention not only considers the information between channels, but also incorporates the position information, and can generate a spatially selective attention map. Therefore, in this embodiment, the depth decoder uses CA attention to replace channel attention to improve the accuracy of the algorithm's depth estimation. The structure of the depth decoder is as follows: Figure 6 shown.
[0057] The depth decoder first passes the fused feature map through CA attention, and then passes through a 3×3 convolution and a Sigmoid function to obtain a depth map; the CA attention first performs global average pooling on the input features along the horizontal direction x and the vertical direction y to obtain feature maps in two directions. The calculation formulas are: Where X represents the input feature, and Z represents the features in two directions after global average pooling. Then the acquired feature maps are concatenated and passed into the shared 1×1 convolution for channel reduction. After that, the features are subjected to BN normalization and nonlinear activation function operations, and they are divided according to the ratio of length to width. The calculation formula is: f = δ(BN(Conv1×1([Z H ,Z W ]))), where [·,·] represents the cascade operation of the spatial dimension, and δ represents the nonlinear activation function; then the segmented features are convolved with 1×1 to restore the number of feature channels to the number before channel reduction; finally, the feature maps in the two directions are respectively used with the Sigmoid function to obtain the attention weights, and then multiplied with the original feature maps. The calculation formulas are: g H =Sigmoid(Conv1×1(f H )), g W =Sigmoid(Conv1×1(f W )), the final feature weighted multiplication process can be expressed as: Where Z represents the output feature, X represents the input feature, and g represents the attention weight calculated above.
[0058] The specific application process of the above-mentioned tank truck barrel mouth positioning method on the tank truck filling machine is introduced in detail below with an example.
[0059] The above-mentioned tank truck barrel mouth positioning method is encapsulated into image positioning software, and the image positioning software is installed in a suitable position of the industrial computer of the tank truck filling machine, and is configured to communicate with the host computer of the tank truck filling machine.
[0060] The image positioning software receives the barrel mouth positioning instruction issued by the host computer, obtains the target frame image of the tank truck through the camera, and inputs the obtained target frame image of the tank truck into the trained target detection model and depth estimation model.
[0061] The target detection model detects the barrel mouth position in the input tank truck target frame image and outputs the barrel mouth position detection result in the image.
[0062] The depth estimation model performs pixel-by-pixel depth estimation on the input target frame image of the tanker truck: first, the target frame image is input into the depth encoder based on the EfficientNet network to obtain feature maps of different resolutions; 2) the feature maps of different resolutions are then input into the multi-scale feature fusion module, and a visual attention method is used to efficiently fuse different features to obtain a fused feature map; 3) finally, the feature map is sent to the depth decoder to obtain the corresponding depth map, and the depth value of each pixel in the output image is output.
[0063] The image positioning software combines the depth estimation results of the pixel points with the target detection results, outputs the distance between the barrel mouth of the tanker and the camera, and then completes the three-dimensional spatial positioning of the barrel mouth of the tanker, and returns the value to the host computer.
[0064] The upper computer controls the mechanical arm gripper of the tank truck filling machine to grab the material gun, and according to the barrel mouth depth positioning result, puts the material gun into the barrel mouth of the tank truck to complete the filling operation.
[0065] Embodiment 2
[0066] See also Figure 8 This embodiment also provides a tanker barrel opening positioning device, the device comprising:
[0067] An image acquisition module is used to acquire a target frame image of the tanker truck;
[0068] A target detection module is used to detect the barrel mouth position in the target frame image of the tank truck acquired by the image acquisition module using the target detection model, and output the barrel mouth position detection result in the image;
[0069] A depth estimation module is used to use the above-mentioned depth estimation model to perform pixel-by-pixel depth estimation on the target frame image of the tanker truck acquired by the image acquisition module, and output the depth value of each pixel in the image;
[0070] The depth positioning module is used to combine the depth value results of the pixel points in the image with the barrel mouth position detection results to obtain the depth value of the barrel mouth of the tank truck and complete the positioning of the barrel mouth of the tank truck.
[0071] Among them, the specific working processes of the above-mentioned modules can refer to the corresponding contents disclosed in the aforementioned embodiments, which will not be repeated here.
[0072] In some specific embodiments, the depth estimation module specifically includes:
[0073] A deep encoder unit, wherein the deep encoder unit extracts features from the original image of the target frame image of the tanker truck using an EfficientNet network, obtains feature maps of five different resolutions, and applies a structure perception module (SPM) to the features of the smallest resolution, simulates the interdependence between channels, obtains better scene understanding, uses a self-attention mechanism to capture long-range dependencies, aggregates discriminative features on the channel dimension, explicitly enhances the understanding of the scene structure, obtains rich scene structure context, and effectively improves the accuracy of the depth estimation of the depth estimation module;
[0074] A multi-scale feature fusion unit. In order to effectively fuse features of different resolutions and improve the accuracy of depth estimation of important details, the multi-scale feature fusion unit uses a detail fusion module DFM to replace a simple splicing operation when fusing feature maps of different resolutions. In addition, in order to solve the problem of loss of feature information of some pixel points due to convolution operation, a residual connection is performed between the initial feature and the fused feature. The fusion module DFM learns the importance of features of different resolutions and features of different channels of the same resolution in a learnable manner, and assigns different weights to different channel features according to their importance, so that the multi-scale feature fusion unit focuses on specific resolution features and channel features, emphasizes important details, fuses the effects of features of different scales, and generates clear edges.
[0075] A depth decoder unit, in order to process the spatial information of the image well, the depth decoder unit uses CA attention to replace channel attention, incorporates position information, obtains richer features in different dimensions, and generates a spatial selective attention map, which can further improve the accuracy of depth estimation of the depth estimation module.
[0076] Embodiment 3
[0077] This embodiment also provides a device, which may include: at least one processor, and at least one memory that can communicate with the processor, such as a read-only memory (ROM), a random access memory (RAM), etc., and the memory stores a computer program; the device also includes a power supply, a communication interface, an input / output interface, and a communication bus, etc. The power supply is used to provide an operating voltage for each hardware device on the device, and the communication interface can create a data transmission channel between the device and an external device. The communication protocol it follows is any communication protocol that can be applied to the technical solution of the present application, and is not specifically limited here. The input / output interface is used to obtain external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs and is not specifically limited here.
[0078] The processor can call the computer program in the memory to execute the above-mentioned tank truck barrel mouth positioning method, which includes: S10, obtaining the target frame image of the tank truck; S20, inputting the obtained target frame image of the tank truck into the trained target detection model and depth estimation model; S30, detecting the barrel mouth position in the input target frame image of the tank truck through the target detection model, and outputting the barrel mouth position detection result; at the same time, performing pixel-by-pixel depth estimation on the input target frame image of the tank truck through the depth estimation model, specifically including: S31, first inputting the input target frame image of the tank truck into the trained target detection model and depth estimation model; In the depth encoder of the depth estimation model, feature maps of different resolutions are obtained; S32, the feature maps of different resolutions are then input into the multi-scale feature fusion module of the depth estimation model, and different features are fused using the visual attention method to obtain a fused feature map; S33, the fused feature map is finally sent to the depth decoder of the depth estimation model to obtain a corresponding depth map, and the depth value of each pixel in the image is output; S40, the depth value result of the pixel in the image is combined with the barrel mouth position detection result to obtain the depth value of the barrel mouth of the tank truck, thereby completing the positioning of the barrel mouth of the tank truck.
[0079] In addition, the computer program in the memory can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art, and the computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.
[0080] When the above software is applied to a tank truck filling machine, the software is first copied to the specified location of the image industrial computer of the filling machine. The software is configured to communicate with the PLC of the filling machine, and the target frame image of the tank truck is obtained in real time through the camera. The software receives the barrel mouth positioning instruction sent by the PLC, starts to locate the barrel mouth of the tank truck, and then returns the positioning result to the PLC.
[0081] Embodiment 4
[0082] This embodiment also provides a computer-readable storage medium on which computer program instructions are stored.
[0083] When the computer program instructions are executed by the processor, the above-mentioned method for locating the barrel mouth of the tank truck is implemented. The method includes: S10, obtaining a target frame image of the tank truck; S20, inputting the obtained target frame image of the tank truck into a trained target detection model and a depth estimation model; S30, detecting the barrel mouth position in the input target frame image of the tank truck through the target detection model, and outputting the barrel mouth position detection result; at the same time, performing pixel-by-pixel depth estimation on the input target frame image of the tank truck through the depth estimation model, specifically including: S31, first inputting the input target frame image of the tank truck into the depth estimation model; The depth encoder of the depth estimation model is used to obtain feature maps of different resolutions; S32, the feature maps of different resolutions are then input into the multi-scale feature fusion module of the depth estimation model, and different features are fused using the visual attention method to obtain a fused feature map; S33, the fused feature map is finally sent to the depth decoder of the depth estimation model to obtain a corresponding depth map, and the depth value of each pixel in the image is output; S40, the depth value result of the pixel in the image is combined with the barrel mouth position detection result to obtain the depth value of the barrel mouth of the tank truck, thereby completing the positioning of the barrel mouth of the tank truck.
[0084] The above is a detailed introduction to a method, device, equipment and storage medium for locating the barrel mouth of a tank truck provided by the present application. This article uses specific examples to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the claims of the present application.
Claims
1. A method for locating the barrel opening of a tank truck, characterized in that: The following steps are involved: S10, acquiring a target frame image of the tanker truck; S20, inputting the acquired tanker target frame image into the trained target detection model and depth estimation model; S30, detecting the barrel mouth position in the input tanker target frame image through the target detection model, and outputting the barrel mouth position detection result; performing pixel-by-pixel depth estimation on the input tanker target frame image through the depth estimation model, specifically comprising: S31, first inputting the input tanker target frame image into the depth encoder of the depth estimation model to obtain feature maps of different resolutions; S32, then inputting the obtained feature maps of different resolutions into the multi-scale feature fusion module of the depth estimation model, using the visual attention method to fuse different features, and obtaining a fused feature map; S33, finally sending the obtained fused feature map into the depth decoder of the depth estimation model to obtain a corresponding depth map, and outputting the depth value of each pixel in the image; S40, combining the depth value result of the pixel point in the image with the barrel mouth position detection result to obtain the depth value of the barrel mouth of the oil tanker, thereby completing the positioning of the barrel mouth of the oil tanker.
2. The method for locating the barrel opening of a tank truck according to claim 1, characterized in that: The deep encoder uses the EfficientNet network as the feature extraction network and applies the structure perception module SPM on the minimum resolution features to simulate the interdependence between channels.
3. The method for locating the barrel opening of a tanker truck according to claim 2, characterized in that: The structure perception module SPM first extracts the feature map X∈R C×H×W Reshape to X′∈R C×N , where N = H × W, C represents the number of image channels, and then matrix multiplication is performed between X′ and the transpose of X′ to calculate the feature similarity matrix S∈R C×C , S is calculated as follows: Where i and j represent different channels, then take the maximum value of S row by row to obtain a new matrix, and perform element-by-element subtraction from S to convert the similarity matrix S into the discriminant matrix D∈R C×C , calculated as formula D ij =max i (S)-S i，j , where max represents the maximum value operation row by row; next, the softmax layer is applied to obtain the attention mapping matrix A∈R C×C , calculated as follows Where exp represents the application of an exponential function to D, and ∑ represents a summation operation; finally, a matrix multiplication is performed between the transpose of A and the reshaped feature X′, and the result is reshaped, and then an element-by-element summation operation is performed between the result and the reshaped feature X′ to obtain the final output E∈R C×H×W , calculated as follows 4. The method for locating the barrel opening of a tank truck according to claim 1, characterized in that: The multi-scale feature fusion module adopts a detail fusion module DFM, which learns the importance of different resolution features and different channel features of the same resolution features in a learnable way, and assigns different weights to different channel features according to their importance, so that the network focuses on specific resolution features and channel features, emphasizes important details, and fuses the effects of different scale features to generate clear edges; the multi-scale feature fusion module downsamples the high-resolution features and upsamples the low-resolution features. When the sizes of the fused features are consistent, the feature maps of different resolutions are sent to the DFM for fusion to obtain the fused features.
5. The method for locating the barrel opening of a tanker truck according to claim 4, characterized in that: The multi-scale feature fusion module also performs a residual connection on the initial features and the fused features.
6. The method for locating the barrel opening of a tank truck according to claim 4, characterized in that: The detail fusion module DFM first concatenates the low-resolution feature L and the high-resolution feature H to obtain the feature M, and then sends the feature M to a convolution layer composed of a 3×3 convolution, a BN normalization and a GELU activation function to obtain the feature U, and the calculation formula is U=GELU(BN(Conv3×3(M))), where Conv3×3 represents a 3×3 convolution operation, BN represents a normalization operation, and GELU represents a GELU activation function; then the feature U is compressed into a vector through global average pooling to obtain global context information, and then two 1×1 convolutions, a GELU activation function and a Sigmoid function are used to calculate the weight vector v to adjust the importance of different channel features. The calculation formula is Among them, Conv1×1 represents a 1×1 convolution operation, and Sigmoid represents a Sigmoid activation function operation; then the weight vector v and the feature U are multiplied element by element to generate a weighted feature Z; finally, the feature U and the weighted feature Z are added to obtain the final output O, and the calculation formula is O=v☉U+U.
7. The method for locating the barrel opening of a tank truck according to claim 1, characterized in that: The depth decoder first passes the fused feature map through CA attention, and then passes through a 3×3 convolution and a Sigmoid function to obtain a depth map; the CA attention first performs global average pooling on the input features along the horizontal direction x and the vertical direction y to obtain feature maps in two directions. The calculation formulas are: Where X represents the input feature, and Z represents the features in two directions after global average pooling. Then the acquired feature maps are concatenated and passed into the shared 1×1 convolution for channel reduction. After that, the features are subjected to BN normalization and nonlinear activation function operations, and they are divided according to the ratio of length to width. The calculation formula is: f = δ(BN(Conv1×1([Z H ,Z W ]))), where [·,·] represents the cascade operation of the spatial dimension, and δ represents the nonlinear activation function; then the segmented features are convolved with 1×1 to restore the number of feature channels to the number before channel reduction; finally, the feature maps in the two directions are respectively used with the Sigmoid function to obtain the attention weights, and then multiplied with the original feature maps. The calculation formulas are: g H =Sigmoid(Conv1×1(f H )), g W =Sigmoid(Conv1×1(f W )), the final feature weighted multiplication process can be expressed as: Where Z represents the output feature, X represents the input feature, and g represents the attention weight calculated above.
8. A tanker barrel opening positioning device, characterized in that: The device comprises: An image acquisition module is used to acquire a target frame image of the tanker truck; A target detection module, configured to detect the barrel mouth position in the target frame image of the tank truck acquired by the image acquisition module using the target detection model described in any one of claims 1 to 7, and output the barrel mouth position detection result in the image; A depth estimation module, configured to use the depth estimation model described in any one of claims 1 to 7 to perform pixel-by-pixel depth estimation on the target frame image of the tanker truck acquired by the image acquisition module, and output a depth value of each pixel in the image; The depth positioning module is used to combine the depth value results of the pixel points in the image with the barrel mouth position detection results to obtain the depth value of the barrel mouth of the tank truck and complete the positioning of the barrel mouth of the tank truck.
9. A device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, it implements the method for locating the barrel mouth of a tank truck as described in any one of claims 1 to 7.
10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method for locating the barrel mouth of a tank truck as described in any one of claims 1 to 7 is implemented.